When plugging my XBox Series X into a wired ethernet connection, it causes my internet to...

I got a new XBox Series X for Christmas, plugged it into my wired ethernet connection (the same one I was using for my PS4 console with no issues), and shortly after I did this I noticed my internet started to drop out across all of my devices. I tried disconnecting from my switch and running a cable directly to my modem, I also tried other cables, but it's the same result. I first thought there was an issue with my internet, but contacted my service provider, and there was no issue.

So I finally decided to try disconnecting the ethernet cable and using Wifi. This solved my internet connectivity issues, but now I can only use the Xbox on wifi, or it causes the issues again.

Has anyone else experienced this? would it be a problem with the ethernet port?

Thank you.

:)