Application access policy will help you in this scenario. Please refer to this documentation for more details https://learn.microsoft.com/en-us/graph/auth-limit-mailbox-access .
Hope this helps.
If the answer is helpful, please click Accept Answer and kindly upvote. If you have any further questions about this answer, please click Comment.