Try the new role based application access controls
https://techcommunity.microsoft.com/t5/exchange-team-blog/announcing-public-preview-of-role-based-access-control-for/ba-p/3688228
or the older application access policies and scope the Azure app to just the allowed mailbox
https://learn.microsoft.com/en-us/graph/auth-limit-mailbox-access