There are two levels of filtering/scoping - assignment to the application, and scoping filters. You can use scoping filters to narrow down the initial set of users that are brought in scope from membership in assigned groups.
Another approach that I'm 90% confident will work is to assign one or more groups to the application that contain users as direct members of the group(s), with those groups being used to control what users are granted access. Separately, create a group that then only has groups as direct members. That parent group will bring the groups that it has as members into scope without granting any entitlement to the users within.
Rough example:
Group A and Group B are assigned directly to the app. No other groups are assigned directly.
Group A's direct members: All users that should have access to the app
Group B's direct members: Groups Q, R, S, T
Groups A, B, Q, R, S, and T will be provisioned out to the other app as group objects. The members of groups Q, R, S and T in Entra ID will not be added to the SCIM app as users OR as members of groups Q, R, S or T unless they are also members of Group A.
Scoping filters can be used to restrict groups A and B from being provisioned into the SCIM app. This may be helpful as A and B are functionally "scoping" groups and may not have direct value in the SCIM app.