Is it possible to remove all my friends at once?

I also tried to do that but there is no way to do this you are going to have to do it one by one

At this time that isn't an option but if you have access to a pc it may be easier to remove friends from the Xbox site as opposed to sitting in front of the Xbox and doing it especially if you're already in front of a pc for work or something.

unfortunately, you'll have to remove them one by one.

It would be more efficient to be able to highlight or click on multiple friends to remove at once ...however the option to do so hasn't been added yet. It would also be a help if friends go inactive for a certain amount of time to auto drop off the list.
Xbox hasn't gotten that far yet. I hoping in the near future they do. In the meantime take a few minutes each time before a play session and remove a few inactives

There's isn't an option to remove multiple friends at once. You have to go to each profile and remove them, and if you want them removed as a Follower, you can Block and then Unblock them, as well. And unfortunately it's hard to tell who hasn't been active
in the last 6 months as a lot of players will put their status to appearing Offline. You'll just have to spend some time one day going through your friends list one by one, sorry about that. Good luck though and hopefully it fixes your freezing issues.