Hello Joaquin,
Firstly, I would check all of the pre-reqs, specifically the Network section: https://learn.microsoft.com/en-us/azure/virtual-desktop/prerequisites?tabs=portal
You will want to ensure any firewalls or proxies are whitelisted to connect to the public endpoint for AVD.
I also recommend the Azure Virtual Desktop Experience Estimator: https://azure.microsoft.com/en-gb/products/virtual-desktop/assessment/
The above page will show the end-user the lowest route trip time from their location, to the AVD service.
Ideally you want the AVD session hosts as close to the end-users as possible, but you can still get a good experience across the globe due to Front Door routing.
I'd also recommend reading up a bit about the RDP shortpath feature for AVD, to establish a direct UDP connection from the client to the session host. Try to use the latest Remote Desktop client for the end-users.
Any questions, please let me know
Thanks