Mouse on keyboard disconnecting while playing games through the xbox app

On any game that I play through the xbox app on windows 11 (i.e., Halo Infinite and Sea of Thieves) my mouse and keyboard will intermittently disconnect. The only way to fix it is by unplugging and plugging back in my mouse and keyboard. This only happens for Microsoft games. I tested out Red Dead Redemption 2 through the Epic store and that worked fine, no mouse and keyboard issues. Mouse and keyboard functionality is perfect when not gaming as well. There seems to be no pattern to the disconnecting. It will just randomly do it. I've looked for solutions everywhere with no luck.

Thanks.

*** Moved from windows ***

:)