Hey Gitarani,
The end solution was odd but I figured it out thanks to a reddit post.
To allow on prem to get to azure resources you need to do the following:
On the pfsense virtual appliance, I removed the lan interface (make sure you have some rules for management on the wan). Then add in static routes to your ON PREM networks via the azure wan gateway (10.0.1.1 or similar).
I had setup a separate subnet for the pfsense vpn gateway vs vms in azure.
For Azure traffic -> on prem make sure your routing tables in azure have the route to on prem pointing at your pfsense wan interface.
It honestly still doesn't make sense to me as it's completely counter intuitive to normal networking/Cisco based stuff.