Hello @azureuser01
Welcome to the Microsoft Q&A forum.
If I have understood the question correctly you have successfully established a VPN connection but are unable to do RDP (port 3389) into your VM.
In such scenarios most likely a NSG present in resource group is blocking this connectivity. You can perform an IP flow verify check and see any NSG is blocking this connectivity. You can follow this documentation here.
Regarding your question above.
When I turn the firewall ON on either machine, I lose the ping connection. Do you have any clue what firewall rules I need to add to keep the ping connection there?
On Windows OS, you can run the command netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol="icmpv4:8,any" dir=in action=allow
to enable V4 ICMP connectivity in the OS firewall.
Hope this helps! Please let me know if you have any additional questions. Thank you!