Thank you for posting this question in Microsoft Q&A.
If you do not want to assign any users to SAML application configured in Azure AD, then you just have to disable the setting "Assignment required". You can follow below steps to disable this setting.
- Login to Azure portal with Global admin credentials.
- Access the "Azure active directory" blade and then access "Enterprise applications".
- Now select the SAML application that you have configured.
- Click on properties blade and set "assignment required" option to "No".
- Once you set this setting to "No", Azure AD will not block the user access to application irrespective of what group user is part of.
Let me know if you have any further questions on this.
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.