You can limit access via Application access policies, or the newly introduced RBAC for apps feature: https://techcommunity.microsoft.com/t5/exchange-team-blog/announcing-public-preview-of-role-based-access-control-for/ba-p/3688228
It allows you to scope access based on management scopes in Exchange, or Entra ID administrative units. Both can be scoped to include only a subset of the rooms, as needed. If you need a more detailed writeup on the method, I have one here: https://www.michev.info/blog/post/4282/exo-rbac-improvements-1-limiting-application-access