Hello @Aman Jha Thank you for reaching out. To my knowledge, there isn't currently a way to restrict the permission to certain groups once the group.read.all permission is granted, would you consider filing a feature request on the M365 Developer platform so we can look into this?
Let me know whether this helps and if you have further questions,
Diana.