How come people cannot join me unless I add them as friend? (Xbox Series S)

This is a problem I've been having for a long time. Basically, anytime I try to invite someone to a game and they are not on my friends' list, they always get an error message upon trying to join me. This happens even if they and I restart our consoles or do other things to try to fix it.

What is weird, though, is that I'm able to join them if they invite me.

I've checked my privacy settings and even restarted my Xbox multiple times, but these did not work. What am I supposed to do? This is getting frustrating because I don't want to add people just to let them join my game, or always having them invite me to their game.

:)