Thank you for reaching out.
Your observation above actually coincides with how Azure Load balancer is designed to work. For Azure Load Balancer a response to an inbound flow is always a response from a virtual machine. When the flow arrives on the virtual machine, the original source IP address is also preserved. Every endpoint is answered by a VM. For example, a TCP handshake occurs between the client and the selected back-end VM. A response to a request to a front end is a response generated by a back-end VM. When you successfully validate connectivity to a front end, you're validating the connectivity throughout to at least one back-end virtual machine. This is currently documented here.
As client IP is preserved the NSG associated with the VM's nic should allow the communication for any Public IP addresses.
If this is not a desired outcome, you can consider using Azure Application Gateway in your scenario.
Hope this helps! Please let me know if you have any additional questions. Thank you!
---Please "Accept the answer" if the information helped you. This will help us and others in the community as well.