Xbox kicked off internet when Chromecast connects

I have an Xbox One on my living room tv and a Chromecast in my bedroom. If the Chromcast starts casting in the bedroom while the Xbox is on, the Xbox disconnects from the internet. I then have to restart the entire console. It says it's connected but then
you can't actually do anything. After restarting, both the Chromecast and the Xbox work fine. It's a problem if my my boyfriend and I go to bed at different times. Has anyone else had this problem? Anyway to fix?

:)