Sorry for the delay in response to the above questions.
Great to hear that adding a firewall rule with appropriate ranges fixed the issue.
I want to figure out how to make it work with a virtual network or a private endpoint.
Connecting via a VNET rule or a private endpoint both require connecting into an Azure VNET. For this you would most likely want to set up a P2S or S2S VPN that your remote users can also use so that their traffic is propagated through an approved virtual network.
Configure a point-to-site VPN connection to a VNet using multiple authentication types: Azure portal
https://learn.microsoft.com/en-us/azure/vpn-gateway/tutorial-site-to-site-portal
https://learn.microsoft.com/en-us/azure/vpn-gateway/point-to-site-about
Regards,
Oury