Xbox one streaming audio issues

I am streaming from xbox one directly to twitch. Playing fortnite for example. In my headset, I hear clearly all of my teammates in game chat. My stream cannot hear some of my teammates most of the time so it sounds like I am talking to myself, or they can hear some teammates and not all teammates, or they can just barely hear teammates. My audio levels are fine and the fact that sometimes the stream can hear them vs sometimes means its an issue to do with the stream. Please advise.

:)