Hi, dear @Emma
According to my test, it is currently not possible to use Sites.Selected permission to send sharing invitations. But my idea is the same as you. If only sharing invitations for specific sites, Sites.Selected permissions should be sufficient, because this is more in line with fine-grained control and access. It is definitely not a good idea to grant access to all sites just for sharing invitations for one site.
So I suggest you submit a user voice to add support for Sites.Selected permissions, and I will vote for you.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.