I have been banned for simply retaliating to insulting behaviour over messages, but the...

As long as you reported them, the enforcement team will investigate and take the appropriate action. Occasionally they will send a thank you message, but not always, so there is no way to know 100% when/if the other person has received an enforcement action. In situations like that, it's best to report and block instead of responding.

I understand how you feel but you should never insult people or retaliate when they insult you. If it's your first ban, you'll probably get a communication ban but not a permanent one.

it works like that on ps4. Just because a game is rated M it doesn't mean everyone is free to insult anybody. If he insulted you, you should have blocked and reported him. if you retaliate, you can get banned.

I think the system is working. Just behave,respect the rules, respect other players and never retaliate or insult others and you'll never be banned. Bans are always motivated and as you said wrong bans can be reverted.

Retaliating is not against the CoC.

It's when you retaliate with offensive language or threatening behaviour that it becomes a violation of the ToU/CoC,doesn't matter who sent the first message.

Best thing to do is not respond,just block & report the sender.