If I factory reset my Xbox but keep my games will it remove my payment and account details?

Hi, I hope you're doing great out there. I am an Independent advisor and I am here to help you in regard to your concern.

Don't worry because as long as you're connected to Xbox Live your saved progress, games, will not be deleted. It will remove all of your accounts but you can add all of them later on.

With your payment methods, they’re linked to your account which means that once you added your account back, the payment methods are still on your account and will be visible on the Xbox.

If i remove account from xbox will all my payment be removed

Payment details are all linked to your account, not the console - however I would always recommend doing a full factory reset on the console if selling or giving it away

Hi, are the games stored on the internal hard drive or an external hard drive ?

There is an option to factory restore the Xbox and keep your games & apps, you will though be required to add any accounts back to the Xbox.

Reset and keep my games & apps. You should use this option as your primary choice when troubleshooting. It will reset the OS and delete all potentially corrupted data without deleting your games or apps. This will save you from downloading or reinstalling
the large game files. In some cases, if a corrupt game file is causing the issue you're troubleshooting, you may have to choose Reset and remove everything, but you should always start with Reset and keep my games & apps and save Reset and remove everything
as a last resort.

support.xbox.com/.../reset-console-to-factory-defaults

support.xbox.com/.../add-remove-account-xbox-one

Thank you so much for the detailed information and I'm sorry because a factory reset will be the last resort. Don't worry because as long as you're connected to Xbox Live your saved progress, games, will not be deleted. It will remove all of your accounts but you can add all of them later on.

Press the Xbox button to open the guide.
Select Profile & system > Settings > System > Console info.
Select Reset console.
Reset and keep my games & apps.

Should both my and my friend factory reset or just one of us?

In this case, please try to factory reset your console:

Press the Xbox button > Settings > System > Console info > Reset this xbox > Reset and keep my games and apps. The "Reset and keep my games and apps" option will not make you lose your games, but you will need to sign in to your account again.

If the option to "Reset and keep my games and apps" does not work, you would need to try to "Reset and remove everything". This will remove all your games and apps from the console and completely restore the console operational system.