It is not possible at this time to configure "All Users" and "Assigned Groups" - both object types have to be set as either All or Assigned jointly.
You can accomplish this goal however by using the Sync Assigned setting, creating a dynamic "all users" group (AAD Premium P1 feature) and giving it a criteria that pulls in all members, such as users where accountEnabled equals True. You can assign that group and then filter the "all users" group object itself out via the Scoping Filters feature - creating a filter akin to "filter groups where displayName eq 'all users'" or "filter groups where objectId eq XYZ" to block the group object itself while still allowing all users inside of it to be in scope.