@Adrian Could you try to enable "Use policy based traffic selector" option in the Azure Portal. You will find this option under the connection configuration page. Attached screenshot.
Just FYI-
Setting "Use policy based traffic selector" to true on a connection will configure the Azure VPN gateway to connect to policy-based VPN firewall on premises. If you enable this field, you need to ensure your VPN device has the matching traffic selectors defined with all combinations of your on-premises network (local network gateway) prefixes to/from the Azure virtual network prefixes, instead of any-to-any. For example, if your on-premises network prefixes are 10.1.0.0/16 and 10.2.0.0/16, and your virtual network prefixes are 192.168.0.0/16 and 172.16.0.0/16, you need to specify the following traffic selectors:
10. 1.0.0/16 <====> 192.168.0.0/16
10. 1.0.0/16 <====> 172.16.0.0/16
10. 2.0.0/16 <====> 192.168.0.0/16
10. 2.0.0/16 <====> 172.16.0.0/16
The reason I ask for this is because you have multiple traffic selectors. Please let me know if enabling this option(also renegotiating VPN after doing the same) makes any difference. Thank you!