@Nigel Morse
Thank you for your post and I apologize for the delayed response!
When it comes to controlling who from your AzureAD tenant can connect to your VMs, you can definitely try leveraging Azure role-based access control (Azure RBAC). RBAC helps you manage who has access to Azure resources, what they can do with those resources, and what areas they have access to. For more info.
Compute specific built-in roles:
Additional link - Sign in to Windows virtual machine in Azure using Azure Active Directory authentication (Preview)
If you have any other questions, please let me know.
Thank you for your time and patience throughout this issue.
----------
Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.