Xbox game pass ultimate on fireTV Stick

What I mean by it tries to head into Xbox, is it sends me to the HDMI that the Xbox is plugged into... when I hold the microphone button on the Firestick to start a voice command. ....

The FireTV Stick does
CEC via HDMI. So whenever the FireTV Stick gets activated, it'll try to control the TV and tell it to switch to that HDMI port it's currently connected to. That way, you can also control the FireTV Stick with your TV's remote instead of the FireTV remote
control.

So if the stick is still connected to some HDMI port along with the Xbox, activating the FireTV stick will cause the TV to switch to that input.

But since you say, you do NOT have the stick on the Xbox anymore, it'd be interesting to know what you have on your TV connected how.

It depends, if you also want to have game pass for pc I would cancel gold and xbox game pass and subscribe to ultimate. But, if you don't want to play on pc, I'd stick with yearly gold and monthly xbox game pass.

I can confirm that a 4k signal won't passthrough my Xbox One S either. I've tried my DirecTV C61k (4k capable), TiVo Bolt and Amazon 4k FireTV. The One S won't pass through 4k from any of them. This sucks and is the MAIN reason I went ahead and got an
Xbox One S. If I wanted to stick with separate inputs for my DirecTV and Xbox One I would have just stuck with my original Xbox One. Not happy...

Here's the thing: there are options if you don't want Game Pass Ultimate for whatever reason. If you are not interested in playing Xbox games on PC, then simply get Game Pass on Console which is $10 per month alongside a separate Xbox Live Gold yearly subscription.
If not interested in that, then stick with Xbox Live Gold without Game Pass.

...Now when I try to use voice command on my Firestick, it tries to head into Xbox. ...

What do you mean "it tries to head into Xbox"?

Do you have Kinect plugged into your Xbox console, and Kinect is picking up the voice command, doing Xbox stuff, or is the FireTV Stick controlling your Xbox somehow while you have the TV app running?