Xbox One Controller Stops Responding Mid Game

This problem has been extremely frustrating to me and even more so that I cannot speak to a human on the customer support line.

When I’m playing with my wireless controllers on any game on my Xbox One, my controller will stop responding mid-game. The X at the top of my controller will light up, but none of the buttons will respond. Even should I shut the controller off and turn it
back on, the controller with reconnect with the solid X light, but the buttons will not respond.

I thought it was just my controller, but this happens as well with my girlfriend’s brand new controller. Both of our controllers are up to date with the firmware, the console itself is also up to date. Happens with all of our games, Madden, Call of Duty,
etcetera. Even if I unplug the console for a couple minutes and plug it back in, the problem will STILL persist after a couple minutes of gameplay. I even went the route of buying an ethernet cable to connect my console straight to the internet as opposed
to WiFi to reduce any wireless interference and that did not work.

If anybody at all has any suggestions or even a full on solution I would greatly appreciate it!

:)