Hi Patrick,
For your requirement it may be simpler to use one of the third-party solutions available in Azure Marketplace. One example, with OpenVPN Access Server you can control which subnets/IP addresses each user can access. I grabbed below diagram off of their site. It shows different access for various IPs, but could easily be for different customer subnets:
In this way you could have each user connect to the Access Server in Azure from the main office using their Entra ID credentials, and then they would be able to access whichever customer IP ranges or individual customer IPs you granted them access to. Please note there are other third-party options that provide similar functionality. If you have a preferred network security virtual appliance vendor that you use it is likely they have offering in Azure Marketplace too.
Please click Accept Answer and upvote if the above was helpful.
Thanks.
-TP