I would like to allow a specific set of users to have access to add and remove Shared Mailbox members.
Do you mean manage the Full Access and Send As permission for shared mailbox?
Both are controlled by "Add-MailboxPermission" command, from the information below, we can know this command contained in the "Mail Recipients" admin role:
Here is my testing, you could have a compare with yours:
I created a new admin group which contained this admin role, then added an Exchange online mailbox into it:
This mailbox could manage the delegation permission for this migrated shared mailbox successfully:
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.