Is there a known explanation?
On average, network speed between Azure regions is pretty fast (roughly 65ms for US West2 and the East US, for example). When making a local connection using Remote Desktop, however, you'll typically encounter increased latency due to load during peak hours depending on your service provider and location. It sounds like the evening are peak hours for your area with heavy traffic.
How can I diagnose the problem?
A quick resource for checking latency is http://azurespeedtest.azurewebsites.net/, which measures speed from your browser to blob storage services in different data centers (remember that this includes local network latency).
If you suspect that some of the latency is due to heavy Azure datacenter traffic, there are tools you can use to test your VM's network latency:
https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-test-latency
What can I do to improve latency in Labs while using Remote Desktop?
I would create a lab in a local region as long as there aren't hardware limitations. Shared image galleries make it easier to globally replicate images in different regions:
https://learn.microsoft.com/en-us/azure/devtest-labs/configure-shared-image-gallery