Xbox One playing a 4K bluray rip over gigabit network

I'm streaming it from my NAS on the same network with everything connected using CAT5e cable.

Erasing cache happens when you do a full shutdown you can do this by holding the power button until it shuts off or go into the power options section and select full shutdown

Sounds like you have tried the main stuff. Perhaps the apps on Xbox (Kodi/VLC) don't work well with 4K streaming.

I assume the laptop streams it over the wired connection as well, or wirelessly?

If wirelessly, try connecting it to Ethernet to test it that way, and you could try the xbox on wireless to see what it does.

Are you trying to stream it directly from the Xbox, or from something else?

If it is from say a phone or tablet, try using Allcast app, its free from the Play Store, that casts to an Xbox.

If it is from your Xbox, test your network speed, to stream a 4k movie ( download speed ) you need 25mbps min, to stream it from your xbox to a TV ( upload speed ) at least 6mbps is the bare minimum, more is highly recommended.

Its a wired connection. I switched it to wireless to see if that made a difference but was the same. I also copied the file onto a USB drive and tried to play it from there direct and it didnt work (using VLC, Kodi and the built in Windows player). So its
not a network problem. I mostly just get a white or black screen and then the program crashes about 30 seconds later.

The file is 10 bit (so is the HDMI port). I dont know if that has anything to do with it.