Hello there,
How is the networking performance on the host?
We might need more information to move further with the troubleshooting steps.
Can you share the exact error message or Event ID you are getting?
If you cannot find anything in Event Viewer I would suggest you use tools like Procmon to find the root cause for this behaviour.
Any checkpoints on them? These will not help latency
Your virtualized desktop, you have a VDA licence for that right?
You have also checked the switch they connect to?
Hope this resolves your Query !!
--If the reply is helpful, please Upvote and Accept it as an answer–