Managing external identities to enable secure access for partners, customers, and other non-employees
You could be getting this error due to network or transient issues. I am able to invite the user with or without group members assignment from Azure and Azure Admin portal. Can you please check the documentation if you have any organization policies which is preventing you from inviting any user.