When ever I Join a party my game sound goes silent until I leave that party

Hi

Please try adjusting the Chat mixer. You will find details on how to do this in the link below.

http://support.xbox.com/xbox-one/console/configure-audio-settings

I get the same problem. Playing BF4 get a party invite join in. Game volume goes to 60% causing me to have to turn it up beyond normal levels resulting in distortion. I am using a HTIB connected via optical and my headphones plugged into my HTIB headphone
jack. Everything is fine until I join the party it gets terrible for sound after joining. I'm in the same boat, if this doesn't get fixed I can't use party chat ever. I am using Kinect for chat during these instances.

and i can also not join parties until i try 10 times and sometimes i need to leave the party because i cant hear anyone then

Hello I see you're having a problem, but the Xbox Ambassadors are here to help.

I have problems with my party audio quite often, and generally it can be traced to game specific setups. Take fortnite for instance, if you go to the settings menu under communication and voice chat you will see that it may be switched to in game or party.
Many other games offer this same scenario. If you join a party it will automatically switch your settings to party, but it will remain the same when you leave the party. Add to this the fact if you are in an Xbox party and everyone has left and you are the
only one in the party, you cannot hear people in other games until you leave the party.

Hope this helps.

Same problem here on my xbox one x super annoying issue as skate 3 is one if my favorite games. Everytime i start the fame up the party just goes silent and stays silent.