Thank you for reaching out.
Based on your questions above
We have a Kubernetes backend behind a public load balancer. We are building an internal LB so were can pass the traffic through a PaloAlto firewall pair hosted at our data center over a S2S VPN tunnel. I would like to nat from the DC Palo Alto firewall back to the front-end IP of the internal Load balancer. Is this possible?
Yes, I think this communication should be possible you will need to add a UDR on the NVA with next hop as internal load balancer's frontend IP.
Your architecture in a way is similar to the architecture discussed here (ILB is deployed here to load balance highly Available NVAs )
Before I move any backend pools over, I want to make sure the connectivity is solid. Should I be able to ping the front-end IP? I already have the VPN gateway up and all the other subnets are accessible in the VNET. I suspect that the frontend IP is not responding to ICMP but may still be reachable.
If HA ports load-balancing rules are configured on the Internal Load Balancer then you should be able to ping the private front end IP. You can check if correct routing is present and if there is no NSG blocking ICMP traffic. Alternate way to test this scenario will be to deploy a demo VM as a backend pool and test the end-end connectivity.
Hope this helps! Please let me know if you have any additional questions and if you can share a network diagram of your set-up here will help. Thank you!
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.