Maybe this time answer will not be deleted :D
One of the options is to set the Service Principal as the Owner of required groups that SP should be able to manage. The fact that SP will be the owner of the group gives him the ability to add and remove users and owners from them. With this setting, the application does not need to have any API permissions to manage groups Group.ReadWrite.All or Group.Read.All. Being the owner of restricted number of groups, allow SP only to read/change those groups and for others where SP is not an owner we will have error 403 Forbidden access.
Additional option is to create Administrative Unit and add required groups to it. Then create new Custom Role with permission to read or update members as required for example microsoft.directory/groups.security.assignedMembership/members/update
Then in Administrative Unit we need to navigate to 'Roles and administrators' select our new Custom role and as member add Service Principal which should have the rights to manage this AU (so also manage groups assigned to it, as AU will be selected in Scope during assignment). (side not as for now we need to assign new Custom role from AU, not from Role view as selection of AU is not available in Scope during role assignment).