Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.
Currently, "When a load balancer backend pool is configured by IP address, it will use default outbound access due to an ongoing known issue."
The product group is still working on resolutions for this issue.
See : Azure Load Balancer Known issues
Should the predicament change, I shall update this thread.
Kindly let us know if this helps or you need further assistance on this issue.
Thanks,
Kapil
Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.