Audio issues with headset and tv sound

Yeah I have the same set up, hdmi connected to the tv which provides the TV with both picture and sound and then the turtle beach connected to the xbox audio adapter to supply the headset with sound.... So yes with this set up, both the tv and the headset
receive audio

the audio only works with my headset plugged in, but still no tv sound

I'm talking about plugging in any stereo headset (like earplugs) to listen to all game audio instead of outputting it through your TV. The Wii U does this with their GamePad (you can play in off TV mode and listen to the audio). The sound quality is awesome.
I've heard the PS4 will let you plug in a stereo headset and output all game audio (sounds, music) to the headset in 2.1 sound. Will the Xbox One allow the same thing?

PalmettoBling may have the key to your issue. Check your console settings, you may need to use the Display & sound - Audio output - Headset audio - Using HDMI or optical audio headset checked and/or set your console party chat setting set to both headset
and TV so that it can be recorded/streamed by the Elgato external capture card.

Good luck!

Sounds coming from headphones and TV at the same time is normal for the console...the other day I left my stereo on (my TV is wired through it) and couldn't work out why the audio from my headphones sounded different!

As to the audio through a headset being lower quality, that would depend really....if the audio signal for the headset is coming directly from the Xbox (wireless headset or wired), then I'd argue it's better quality than that carried to the TV via the HDMI cable. For me, my wireless headset is plugged into my controller (using an adapter) and the sound quality is miles better than the TV is capable of (is even better than my very good stereo too)