@StephanG I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.
Issue: we are using SSO with Fortinet VPN to authenticate our users and leverage Conditional Access. It is working for 99% - but there are 1% that won't work. They just do not deliver the group IDs back. Today i learned about the 150 group limit - but they are below that 3 users are affected: 88,99 and 100 groups The Fortinet Log states: [364:root:75f]fsv_saml_login_response:510 No group info in SAML response. [364:root:75f]fsv_saml_login_resp_cb:173 SAML group mismatch. And there is a claim named: [http://schemas.microsoft.com/claims/groups.link] - i cannot find anything about that. But it kinda matches the information about what happens if you are above 150 groups.
Solution: (Answered by (@StephanG ) Although not hitting the limit of 150 - a link was sent. So this was the solution https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-SSL-VPN-SAML-Authentication-not-working-for/ta-p/216142 In order to resolve this issue, on the Azure portal, in the group claims settings, 'Groups Assigned to the Application' should be selected instead of 'All groups'. Once, this setting is selected, only the groups which are assigned to the application will be sent in the SAML response instead of all the groups.