can't remotely remove account from xbox

You would have to go to his console and remove it. There is no way to remove it remotely. If he's actually using the account, you need to change your password. No one else should have access to your account.

You would have to go to that console and remove your account from it. You can't do it remotely.

If he no longer has access to his consoles then he can't remotely remove them from his account. Here is the quote from Xbox Support website,

If you no longer have access to the Xbox One console where you saved your Microsoft account, you won't be able to remove the account remotely. However, you can change your password, which will disable
access to your Microsoft account on the console to which you no longer have access.

You cannot remove your account from an Xbox remotely so you need to change the password on the account. That will render the account unusable on the remote machine.

The best way to stop access, if your account is on other Xbox's, is to change password. You can not remotely remove your account from a console.