@skip hofmann , It would be hard to tell whether, its broken somewhere or not working because of any steps being missed while deploying it. It would be great if you can recheck the steps I am sharing below just to make sure we are configuring it correctly.
Step 1: Make sure you have create a group with users in it and the users are not a part of any nested groups.
Step 2: Nested Groups and Dynamic groups are not supported.
Step 3: Users will experience the new signin experience only if there existing tokens are invalidated and their earlier sessions have expired. You can try testing it in the incognito mode/ inprivate browsing mode present in the browsers. You can also try revoking the tokens using Revoke-AzureADUserAllRefreshToken PowerShell cmdlet.
Step 4: Make sure that the synced users with which you are testing are not Global Admins in the tenants. We recommend using non-privileged synced users.
Step 5: You can only use maximum of 10 groups per feature i.e each for Password hash Sync and Pass Through Authentication
Do let me know if these points are present in your test so that we can plan the next steps accordingly.
If this helps and if there are any more queries around this, please do let us know so that we can help you further. Also, please do not forget to accept the response as Answer if the above response helped in answering your query.