Sorry to say that, but "this is not how it works" ;-)
That user "Azure Active Directory Admin" is "only" used to connect to your AAD to read/check your AAD; without that user the SQL Server won't be able to connect to your AAD, so it won't be able to link any AAD-User or group to your SQL Server.
for example, you create a user for those tasks like yourdomain\svc.sqlserveraad that user has read access to your AAD, and you configure your SQL Server to use this user as "Azure Active Directory Admin." You create/link that customer-admin-group in your database and grant them their permissions.