Problems playing with a specific friend

the problem is because of the NAT that should be set to open on both consoles, unless either of them has some restriction on the multiplayer side of some game, which they should or should have notified the offender, but I would bet that it is NAT problem

He is playing on his hotspot and were having no issues he connected with no problems

Check yours and your friends NAT. Follow this NAT error solution guide to assist you to resolve this issue.

Both of our NATs are open this still isn't fixing it

Hey Run

Thank you for taking the time to post this on the Xbox Forums. We understand that you are currently unable to connect to one of your buddies when you play games online.

It appears that this issue is reproduced whenever you folks are not playing on the same network, which would indicate that a configuration on your network might have changed, as to producing the difficulties with connecting to your
friend.

Considering that you had to visit your friends home, and have both of your consoles on the same network to play, I would like to see if its possible to attempt to test a network that is not yours or his to attempt to connect to an online
session momentarily, purely for testing purposes [this can be done using a mobile hotspot or even an alternative band of network if possible].

Additionally you can try to use a different MAC address on your console via your

network settings to see if using the physical address [or Wi-Fi address depending on the device you are using] of a device that does not experiencing connectivity issue and see if this yield different results.

Joining a friends game can be fairly simple and might even reduce latency issues if your on the same network, so if you can test an alternate network, we should be able to identify what the source of this connectivity issue is.