Slow download speed on Xbox Series X

Sorry if this question has been asked a lot in the past but is there a reason why my download speeds are dreadfully slow? For a couple of mins, it could be 20MBs per second and then it drops down to the speeds of KBPS. I don't have any games or apps running in the background while it's downloading. On my PS5, or other devices, the speed is consistent. Is this a known thing or is it something on my end?

:)