@Mike Gilbert Discussed your issue with our engineering team.
As per the discussion I had about your issue changing the groups that are sync’d, this is likely the root cause of the problem (the users being sync’d are dependent on the groups that are in/out of scope). We’ll need to look at the metadata to figure that out.
If you have a Azure Support plans with your subscription, open a support ticket, they can further engage the engineering team to investigate your issue further or if you dont have a valid support plan, let me know we can assist creating a one-time support ticket for this issue.
Let me know if you have any further questions.
Please remember to "Accept Answer" if answer helped, so that others in the community facing similar issues can easily find the solution.