This is expected behaviour. If you activate any of the PIM Role Using Group ID, it will activate the role for all users who are part of that group.
If you want to activate a specific role only to specific user, then you will have to make the role eligible for that user only.
Other thing is you can permanently assign the role to one particular group and make the group membership activated in PIM. Whoever is eligible for the group membership they will have to activate it in PIM blade in Azure portal. And users who will be added as group members will by default get the role assigned which is assigned to the group in Azure portal.
Let me know if you have any further questions.
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.