Hi Christoffer,
Regarding the “failed to accept invite” error, I have done some research and found that a community member mentioned that it can take 12-24 hours until the external user is removed from the SharePoint site, so I would suggest you wait for some time and re-invite the user to see if that makes a difference. For your reference:
https://community.office365.com/en-us/f/154/t/258100
If the issue persists after waiting for some time, I need your account information for further investigation. To protect your privacy, I have sent you a private message to collect it. You can check the PM via the link below:
https://community.office365.com/user/conversations
Regards,
Allan