Xbox party is always says its connecting me and usually 3 other people and wont stop.

Are you using WiFi? This can lead to really unstable connections. Try using ethernet and if that doesn't work then check your internet speed, ISP, and also your server. The fact you are able to connect on mobile and PC says it could be your internet connection.
Also check the number of devices you're able to have on your internet. If it's two or less that's why you can't connect.

when i join a party i just have the connecting box for me and usually 2 other people. It never loads I have let it sit for hours. This problem is only on my Xbox. I can connect on Xbox mobile and on my PC but not my Xbox. I have tried power cycling
and clearing my mac IP address casche and unluging the console and nothing works please this has been happening for a long time now.

try using a internet cable and if that dont work then contact microsoft

Try contacting Microsoft Support, or using an ethernet cable

Iev noticed the guy above me has Mediacom. I am also having the problem and 3 other people I've seen also say they have Mediacom.

Mine works fine directly connected to the modem, but then through the router it wont work. ITs not the router or ports or anything because I havent changed anything. It just stopped working on its own.

Hi there. Is this a party of 2 or are there other people in it?

The reason I ask is because I usually connect with the same friends daily but occasionally when we have some new people in often with a bad connection it tends to boot me out or the party becomes unstable. It might not be an issue with your connection but
if they have a bad connection or someone in the party has a bad connection ..that could be the reason.

You might want to make people in the party aware of the issue and have them reset their routers & turn off any downloads on their PC's before they join. This usually works for me so its a suggestion.

Good Luck !