My brother got banned for getting beaten in a 3v3

To clarify, you wouldn't be able to talk to the enforcement team. Your brother would need to go to the enforcement site for information on why he was suspended and to see if he can submit a case review. He would not have been suspended for no reason or
for being too good.

Since this is an enforcement issue, you'll have to speak with the Xbox Enforcement Team. You can reach them at https://enforcement.xbox.com.

The context of any communications he may have had, i.e. did he join the party invites or message those people back, may be the deciding factor in why he received an enforcement action. Had he replied with offensive content back to the trolls messages,
that could be the reason why he received that action. In instances like this it is best to report and block the offending individuals before things escalate. For future reference, things shared or talked about on Xbox Live should try and fall in line with
the Community Standards here: https://www.xbox.com/en-US/legal/community-standards

That is the thing with bans and submitting case reviews to get un-banned is that if there was any retaliation with foul language on your part. Your ban will stand unfortunately.

As above, that is the only way to appeal. However, your brother should also report them. However, if he retaliated using bad language then his ban will stand.