Should I turn my xbox of at the switch/outlet once its been powered down (energy saver mode)

Hello, first it's a pleasure to be able to help you today.

My name is welcome to the community.

Hello williamoxley, yes can do this, but the ideal if going to turn off completely switch /outlet off, is to choose the full shutdown mode of xbox,

Xbox defaults to Energy saver mode. You will have to switch it to standby for downloads to continue while powered down

My console is on energy saver mode, and it is plugged into an outlet not power strip or anything like that.

That the mode of energy saving, is activated or not, it changes nothing.

Actually, I believe it does. Energy saver mode fully shuts down the console, internet connection included, almost the same as unplugging it. Easy way to prove this is to put your console in Instant On mode, power it down, then log into your router and check
your "Connected devices". Your console (in Instant On mode) will still show up as connected. Now put the console in Energy Saver mode, power it down and check for connected devices in your router (you may need to refresh if you've stayed logged into your router).
I bet you'll find that your Xbox no longer shows as a connected device because it has been fully powered down.

I've never tried that so I could be wrong, but I can see no reason for the console to send power to wifi in energy saver mode since it doesn't check for updates.

I had similar issues and 'fixed' it by switching the power setting from energy saver to the insta on mode.