You need to deploy it first then you can define the access. https://learn.microsoft.com/en-us/microsoft-365/admin/manage/manage-addins-in-the-admin-center?view=o365-worldwide#edit-add-in-access
So deploy it then exclude the group of users and then it should be fine.
I hope this information helps you resolves the questions.
Please accept as answer and upvote if the above information is helpful for the benefit of the community.