Xbox One Lag COD MW4. I tried everything I can think of

Xbox One Lag COD MW4. I tried everything I can think of
Hi guys, someone please please help me. This is getting me so frustrated. I only play COD and after about 3-5 games i would start lagging. I would be smashing on a game and then im running into walls. When i start to lag i would check my in-game information
by going to menu (three lines) then Options, and then right bumper until i go to Account and bottom right of the screen shows "Only Functional in Public Games" My Packet Loss stays at 0% but my Latency would fluctuate all over the place sometimes getting
up to 999 ms

all other times when i am not lagging my latency would jump up and down 20-40 ms

I tried everything. Can someone please help. My ISP is Xfinity which is the fastest around here. Im pretty good at networking and i dont believe it is my network. Can it possibly be my actual console? Im telling myself to wait for the new xbox and then
if i have the same problem then it has to be my network but i really cant wait until Nov/Dec for that. Please help. I tried everything i can think of and everything i can find online with no luck.

Some background information.

-Always hardwired

-Lag started after I moved to a new house

-Old house 200 down 4 up no lag issue at all

-New house 600 down 20 up and lags

-XBOX Network info:

-Wired

-IPv4 and IPv6

-NAT Open

-All services are available

-All XBOX test says good. Test network, test multiplayer connection, and test network speed

-Network speed latest numbers:

-Download speed: 385.71 Mbps

-Upload speed: 12.53 Mbps

-Packet loss: 0%

-Latency: 15ms

Things I tried: I know i did more but this is the ones i can think of at the moment.

-reset modem

-soft reset xbox

-factory reset xbox and re downloaded game to console

-cleared cache

-fan directly blowing on console. (heard it could be overheating)

-NAT Type always open

-changed ports through advanced menu

-swapped out all Ethernet cables

-Modem is new

-test speeds with computer and speeds are over 500 dl

-tested speed on xbox and shows over 300 dl each time

-lags at all times of day with other things on network and off network

-cleared MAC address

-Disabled cross play

-ran Ping test

:)