Does not seem to be outputting the correct color space

Does it typically make any difference if the setting is enabled when the output contains little to no over- or undershoot of legal BT.709 values? On my TV forcing xvYCC on with a regular BT.709 source makes no difference in color handling (which is correct).

Is your color depth option set higher than your TV can support? Go to
Settings > Video output.

Under "Video fidelity", you will see the "Color depth" and "Color space" options.

Color depth:

• If you have a Dolby Vision TV, set it to 36 bits per pixel (12-bit).
• If you have an HDR10-compatible TV, set it to 30 bits per pixel (10-bit).
• If your TV does not support HDR, set it to 24 bits per pixel (8-bit).

Color space:

• If you are on a TV, use Standard (recommended).
• If you are on a monitor, use PC RGB.

The color space setting should usually be set to either Auto or Source.

RGB carries the picture across in the primary colors red, green, and blue. This is what games use.

YCbCr carries the picture across recalculated as luminance and color difference. This is used for video material as it allows the color information to be stored at a lower resolution to help save space. The BT.601 flavor is used for standard
definition, and the BT.709 flavor is used for high definition. Being able to use either flavor regardless of the output resolution is something that is only useful in extremely rare cases. Using BT.601 together with a HD resolution will give inaccurate color
reproduction.

Unlike RGB, YCbCr always uses video range levels and is unaffected by the Reference Levels setting. Your display should however not expect full-range (expanded) video levels when fed YCbCr, unless it has been forced to through a setting (some displays allow
that for whatever reason).

The best starting point is setting the color space to Auto and the reference levels to Standard. This will switch the Xbox 360 between RGB and YCbCr based on the content, and automatically pick the appropriate YCbCr flavor based on output resolution.

Is there a way to see resolution played back at? I press info when watching the martian and it says HEVC as codec and in the video output settings on the xbox it says: Tv Resolution: 4K UHD. Color depth: 30 bits per pixel (10 bits) color space: standard.
Under advanced settings 24Hz is checked. Allow 4K and 3D ate also checked.

The reason was probably that they didn't care to block that setting, to keep things simple. Remember that when HDMI output was introduced with the Elite, the Xbox 360 only output full range RGB. It wasn't until one of the system updates that they added the
color space settings.

Before they migrated their framework to Silverlight and botched the behavior, YCbCr would pass video material through unconverted as far as I know (aside from the inevitable conversion between BT.601 and BT.709 when needed). Source would switch between RGB
and YCbCr, and Auto would go by the E-EDID data to decide on the color space, usually resulting in behaving like the Source setting.

The problem was that with the Color Space set to Source, the Reference Levels setting became a problem. Some displays expected RGB to be full range and some expected it to be limited range. The Xbox 360 didn't always (or maybe never) indicate the quantization
range correctly in the metadata, and forcing the range on a display to Full or Limited might affect YCbCr input as well - even though YCbCr is implicitly limited range.

As a result, video could become washed out while games looked right, or video look correct but games become murky. Letting everything be output as YCbCr was probably a convenient workaround to make sure everything looked okay, and allow for those without
insight into color space to be given simpler advice as to which settings to pick.

Many displays convert to YCbCr internally for processing though - often with 4:2:2 chroma subsampling. However, they may also perform the RGB to YCbCr conversion at a greater bit depth than the Xbox 360, giving lesser rounding errors.

Now that the Xbox 360 can no longer pass through YCbCr as-is, but converts to RGB and then back to YCbCr, the setting is of course bad even for video output.

Sony took a more tech-savvy approach, but they probably also took greater care to form a properly flagged signal.