What you're describing is working by design from the Azure AD side. The behavior of the generic SCIM connector offered with custom non-gallery applications is to send group memberships one per request. If the attempt to PATCH add/remove a group membership fails, it will be saved and retried later. Are you able to either temporarily or permanently increase the number of requests per second that your SCIM endpoint can accept so that the initial population of the group memberships can be done without the SCIM server sending 429s? There aren't many options on the client(Azure AD) side of this that will help you out here, unfortunately - the group memberships need to be sent over, and the 429s are going to cause that to take an incredibly long time as the generic SCIM connector doesn't have any request rate limiting available on it.
SCIM PATCH group membership throttling
I have an enterprise application with 10k users under 5 groups. The groups and users have been successfully provisioned. However, for the group membership, the requests get throttled. It looks like each time the group attempts to get their assignments, the entire membership appears to get re-assigned, but this causes requests to get throttled. Is there a reason why the memberships need to be entirely replayed each time Azure AD attempts to re-synchronize the users?