Xbox Party Chat is echoing

So, this is a really weird situation, but for some reason when I increase the headset volume in Xbox party chat, I have an echo. I've tested it with the talk on Discord option, and it's not there. It seems to only happen in Xbox party chat. I've tested two sets of earbuds, although it could be a problem with the brand I'm using. I don't think it's my controller, but I don't have a spare to test it. I'm on Xbox One X, with a standard Xbox One controller. How do I troubleshoot this?

:)