RDS HTML5 WebClient via AppProxy to OnPremise world slow as during initial load of credntials from Azure - fine OnPrem!

Question

My Title explains my issue I guess but a few more things...

We are mid sized org with an OnPrem foothold and Azure AD Sync to access Cloud Public services from the likes of Microsoft. Trying to rid ourselves of web facing On Prem RDSWeb - grand!

Stick it behind Azure App Proxy
Wrestled eventually through the lack of good documentation around how to do that when you have an Internal Domain Name different from Public and the Internal domain cant support Certs (dont ask!!) by buiding it out in HA style so we could effect using an internal DNZ zone that matched the external domain name and with simple loadbalancing to two front ends . So all good - secure - no public facing access to RDS unless you authenitcate at Azure and launch the app or fire it in via Edge. no hacks on the hosts to allow mismatched domain cert trust. Good I say...

what our small band of users report is that it takes an AGE to load through the process when they launch the url there is a back and forth progress bar whilst it checks presumably the token on the Brwoser Edge is up to date before it then goes on to the OnPrem AppProxy server to talk to our internal front end who must then squirt that back to Microsoft to pass through the proxy to the users web browser...but its painfully slow during that starting process.
Then maybe the next day I try its reasonable - cached?

Brand new 2022 Servers for RDS setup with a seperate SQL db - ultra quick onprem experience slow via AppProxy.

I have another AD site with a different AppProxy server sharing out different internal assets like a "Intranet" page. What I dont get is that its so much quicker an experience for the same users opening that URL compared with the RDS WebClient AppProxy URL.

Has anyone please found any options or ways of making RDWEB/WebClient via AppProxy quick enough to serve up as a solution for a 500+ sized business?

many thanks!