@Martin Garrix I have summarized our conversation in the comments above, in this answer for clarity and for others to reference.
Q: Guidance regarding using 3rd Party VPN with Azure Virtual Desktop.
A: In general, using 3rd party VPNs can get a bit complex.
You need to configure a split tunnel to maintain connection of the VM to the fabric as well as the AVD Service endpoints. Please don’t disable the windows firewall as it can break many things.
Q: Is there any documentation regarding the split tunnel path suggested above?
A: You can use this as a starting point, it also applies for AVD: Optimizing RDP Connectivity for Windows 365 - Microsoft Community Hub
Also, it is advised to have these URLs excluded from the VPN: https://learn.microsoft.com/en-us/azure/virtual-desktop/safe-url-list?tabs=azure#session-host-virtual-machines
If you have any questions at all, let me know in the "comments". Comment is the fastest way of notifying the experts.
If this answer helped or steered you in the right direction, don't forget to 'Accept answer' and hit 'Yes' to 'was this ans helpful', wherever the information provided helps you, as this can be beneficial to other community members for remediation of similar issues.