Could my ISP be the reason for my high packet loss?

My issue is that my xbox has a consistently 50+% packet loss on my ISP (Hawaiian Telcom).

-I tried the box on another ISP (Spectrum) at a friend's house, 0% packet loss.

-I connected the Xbox wirelessly tethered to my phone (AT&T), 0% packet loss.

-I tried connecting the box directly to my fiber connection (bypassing all networking gear and using the ISP public IP), same high loss numbers.

-No other devices using my ISP and network have packet loss.

I've tried everything I can think of, and can't think of any other reason it's not my ISP. Could they be blocking or filtering or something that causes the high packet loss?

Of course, when I call them, the customer service people don't have any idea of what I'm talking about. They say it's Microsoft's or my particular Xbox's fault. But it simply can't be, since I've tried the same box on different ISPs with no loss. And with my limited knowledge, I can't point to anything particular and say that is causing the high packet loss. Help me obi wan...

:)