RDS behind Azure Web App Proxy works through browser but not RDP file downloaded

Question

We followed this Microsoft guide to protect the RDwebclient we setup on windows 2022 with azure application proxy: https://learn.microsoft.com/en-us/entra/identity/app-proxy/application-proxy-integrate-with-remote-desktop-services

This is working perfectly when using the RDwebclient and starting the connection in the browser. But when downloading the RDP file we get an error upon connecting: "Your computer can't connect to the remote computer because authentication to the firewall failed due to missing credentials"

Due to the above issue, unable to use multiple monitors.

Done an extensive search, not able to find a solution for this issue. Unless the pre-authentications is set to passthrough, which is a huge security hole.