HI Adam. I'm Greg, an installation specialist, 10 years awarded Windows MVP, and Volunteer Moderator, here to help you.
See fixes for Xbox One Wireless controller "Remove failed" error:
https://www.reddit.com/r/Windows10/comments/bhy...
https://superuser.com/questions/907337/windows-...
I hope this helps. Feel free to ask back any questions and keep me posted. If you'll wait to rate whether my post resolved your problem, I will keep working with you until it's resolved.
______________________________________________
Standard Disclaimer: There are links to non-Microsoft websites. The pages appear to be providing accurate, safe information. Watch out for ads on the sites that may advertise products frequently classified as a PUP (Potentially Unwanted Products). Thoroughly research any product advertised on the sites before you decide to download and install it.