How do I add my son to my xbox one when he already has an account on his mother's family group.

I am divorced.

My sons have xbox accounts on a family group created by their mother. I am trying to set them up on my xbox one and created a family group of my own but it appears that they can't join that as well.

Their mother has suggested that I join their family group but how would that affect what I've already done on my XBox?

Or is it possible for me to add their accounts to my xbox without making them part of my family group?

:)