You can use application access policies as detailed here: https://learn.microsoft.com/en-us/graph/auth-limit-mailbox-access
Going forward, these will be replaced by native integration with Exchange's RBAC model, just announced in preview: https://techcommunity.microsoft.com/t5/exchange-team-blog/announcing-public-preview-of-role-based-access-control-for/ba-p/3688228