I found a solution to this issue, and I hope it helps someone else.
So in order to access on premises resources while connected to Azure VPN, you need to add a route on your client computer, the one thats connecting to the Azure VPN. The best way, I've found, to do this, is by modifying the routes.txt file found here:
C:\Users\USERNAME\AppData\Roaming\Microsoft\Network\Connections\Cm#####-#######\routes.txt
Add this line, modifying as needed to specify your LAN subnet.
ADD 192.168.X.X MASK 255.255.25X.0 default METRIC default IF default
Save, close, reconnect to the Azure VPN.
You also need to make sure you have a route on your on prem VPN device in the Site to Site configuration that allows traffic for your Azure Client VPN subnet.