Thank you for your post and I apologize for the delayed response!
When it comes to controlling who from your AzureAD tenant can connect to your VMs, you can definitely try leveraging Azure role-based access control (Azure RBAC). RBAC helps you manage who has access to Azure resources, what they can do with those resources, and what areas they have access to. For more info.
If you have any other questions, please let me know.
Thank you for your time and patience throughout this issue.
Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.