Thank you for getting back and sharing the required details.
Based on my understanding from your architecture above, the public IP address assigned to your VPN Gateway cannot be used to route traffic to the internet, it is used to establish the VPN connection only.
Based on your statement above
We would like to remove the reliancy on Site 1 and go out to the internet through Azure.
In order to achieve such connectivity, you can deploy an Azure Firewall on your Virtual Network in Azure so that the internet bound traffic from on-prem can routed the VPN connection to private IP of the Azure Firewall and then using SNAT the Firewall will direct the traffic to the Internet, something similar to this tutorial here.
Azure Firewall provides SNAT capability for all outbound traffic to public IP addresses. Using Azure Firewall, you can enable access to internet.
Additional reference:
Hope this helps! Please let me know if you have any additional questions. Thank you!
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.