I've experienced simular issue and reason is because the external user appears twice in Azure AD.
Deleting the john.doe@keyman .com mostly solves the problem.
If it persist, remove both and reinvite.
I'm searching for the root cause and I think it's in the order people click links.
First they get an invite to join a team, after that another email once we added them to a role in sharepoint for granting specific access to files.
My guess is that the external user first clicks on the second mail and triggers this behavior.
Feel free to drop me an email with your experience.