Why do 3 of the 4 family members sharing Gamepass Ultimate get kicked about 20 minutes into...

My family and I play Xbox PC games sharing my Gamepass Ultimate account. We are encountering a consistent and frustrating problem where 3 out of 4 of us drop to desktop with no error. This happens on multiple games but we have been playing Gears 4 Horde Mode lately so I will reference it a lot in the following details (Back 4 Blood is another one). I am a savvy technologist who has been building PCs and networking games for decades (ipx/spx Doom2 over cheaper-net).

1. We are all playing on different PCs.
   
2. We are playing through the Xbox App.
   
3. We are all logged into our own account.
   
4. Each PC is logged into my store account.
   
5. We are setup with Microsoft Family and they are all members (we use this functionality for safety features and more)
   
6. The Gamepass Ultimate badge appears under each of our individual tags in the top left.
   
7. All Firewall protection is setup correctly and/or turned off entirely.
   
8. 3rd Party Anti-Virus has been removed from the equation.
   
9. When 4 of us play Gears 4, we are all kick in the order that we logged in down to the last player that logged into the game. <-
   
10. The kicking starts about 20 minutes after we begin to play.
    
11. The kicking has no messaging at all, we drop to desktop as if the process was killed.
    
12. We can rejoin the game, only to have this happen again about 20 minutes later.
    
13. When 2 of us play Gears4 in the exact same way, no one gets kicked.
    
14. When 3 of us play Gears4 in the exact same way, no one gets kicked.
    
15. A few weeks ago we played Aragami 2 with 3 of us with no issue.
    
16. We tried a 4 player game in Back 4 Blood and the exact same thing happens.
    
17. While their Xbox accounts do not match their store account, this does not matter as I get kicked as well.
    

What I believe is happening is that the Xbox service is checking the account and seeing that 4 people are using it and the family sharing is not properly stepping in to say that we are using it legitimately (I am under the impression that a Family can be 5 people; this works for Office 365). I ran into a problem where the software for an ASUS mb on board audio card broke the store validation, so I know its fragile. I also know this is a fairly new feature they are supporting and I may have it setup wrong or misunderstand the constraints.

Any help would be appreciated.

Thank you.

:)