Xbox connection and lag

Xbox connection and lag
So, to start, there is nothing wrong with my router or internet service provider. I have done tests on different devices and I get really good results. About a month or two ago, I was in an Xbox party with 2 friends. Friend 1 and 2. A friend of friend
1 then joined. Friend 2 was the party leader and got a message from friend 1 to kick the guy that joined. (I was afk for all this.) Friend 2 kicked him and we started playing a game once I got back. About 15 minutes in, Friend 2 gets a MAJOR lag spike and
proceeds to disconnect from the party and game. He calls friend 1 on his phone and says he is being booted offline and his router shut off. Once he gets it fixed, he joins back and we keep playing. Then I get booted, but ever since then, I have been lagging
HORRIBLY. My latency on Xbox was around 30 up until this happened. Now, I get anywhere from 200 to 3000 constantly. I have looked up every trick in the book and nothing has worked, we got a new router with better specs than the previous one but nothing changed.
I have no idea why it's happening. Any help would be appreciated. I play on the basic Xbox one. No new edition.

:)