Hello,
Did you try to configure the access via Active Directory groups ?
Below you have an article regarding how you can use this to restrict access :
https://learn.microsoft.com/en-us/windows-server/manage/windows-admin-center/configure/user-access-control#active-directory-or-local-machine-groups
Best Regards,