@Alon
I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!
Since the Microsoft Q&A community has a policy that "[The question author cannot accept their own answer. They can only accept answers by others] (https://docs.microsoft.com/en-us/answers/support/accepted-answers#why-only-one-accepted-answer)",
I'll repost your solution in case you'd like to "[Accept] (https://docs.microsoft.com/en-us/answers/support/accepted-answers#accepted-answer-in-a-question-thread)" the answer.
As per your query you have fixed the issue by following below article,
https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-add-app-roles-in-azure-ad-apps
Basically, you can do the exact thing that you needed by creating custom roles for my app. Then, you can assign those custom roles to individual users, or to entire groups.
Just like the article states, you can map a custom claim to the user's 'assignedRoles' attribute, and all values for those custom roles are sent over the SAML response.
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.