@Adrian Could you try to enable "Use policy based traffic selector" option in the Azure Portal. You will find this option under the connection configuration page. Attached screenshot.
Setting "Use policy based traffic selector" to true on a connection will configure the Azure VPN gateway to connect to policy-based VPN firewall on premises. If you enable this field, you need to ensure your VPN device has the matching traffic selectors defined with all combinations of your on-premises network (local network gateway) prefixes to/from the Azure virtual network prefixes, instead of any-to-any. For example, if your on-premises network prefixes are 10.1.0.0/16 and 10.2.0.0/16, and your virtual network prefixes are 192.168.0.0/16 and 172.16.0.0/16, you need to specify the following traffic selectors:
10. 1.0.0/16 <====> 192.168.0.0/16
10. 1.0.0/16 <====> 172.16.0.0/16
10. 2.0.0/16 <====> 192.168.0.0/16
10. 2.0.0/16 <====> 172.16.0.0/16
The reason I ask for this is because you have multiple traffic selectors. Please let me know if enabling this option(also renegotiating VPN after doing the same) makes any difference. Thank you!