Technically speaking, Graph permissions are assigned directory-wide. In the delegate permissions model though, the effective permissions are the subset of those assigned to the app and the permissions assigned to the user, so you should be able to scope it by scoping the user's role assignment. Take a look at the administrative units functionality and leverage them to scope the user's admin role(s): https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/administrative-units
Do note that no method exists to limit read permissions, as every user ("member") within the directory can read other user's information.