Is there anyway to stop people from casting videos to the youtube app on xbox one?

Basically someone i know casted a video to the youtube app on my xbox but i didn't even have it opened, or anything, i also had the xbox app connectivity settings all turned off, and i'm saying that cause i know someone will tell me to disable those settings
and i already did, and they're still able to cast random videos to the youtube app on my xbox one, is there anyway of stopping this? i also tested it out on my pc and i was able to cast random music to groove music on my xbox one without even having it opened
it just opens by it's self, does anyone know how to stop this, or if Microsoft will eventually do a software update fixing these issues? cause it's kind of annoying and who knows what people can do to your xbox if you leave your room for one second and leave
it on. But for now i'm keeping all the apps on my xbox deleted besides netflix cause as far as i know there is no casting options available on phones or pc's, and if there is i'm also gonna delete that and stick to using my pc, blocking all remote connections
and use a firewall, i don't like my privacy being invaded, by some creeps.

:)