Hi,
Can you check iperf and verify the connection + transfer speed using the iperf, this should provide some details around the speed and latency. Also configure the diagnostics on the VPN GW for investigating the root cause might throw up some pointers troubleshoot-vpn-with-azure-diagnostics
Iperf - vpn-gateway-validate-throughput-to-vnet
Upload the logs here and also raise a support case with Microsoft to investigate the backend routing.
Hope this helps.
JS
==
Please Accept the answer if the information helped you. This will help us and others in the community as well.