Based on the symptoms described, it is possible that the RDP Listener is misconfigured, which can cause intermittent connectivity issues. This issue is more likely to occur on a VM that uses a custom image. To troubleshoot the issue, you can follow the steps outlined in the "Remote Desktop disconnects frequently in Azure VM" article provided in the context. Additionally, you can review the event logs on your session host virtual machines (VMs) by going to Event Viewer > Windows Logs > Application. Look for events that have one of the following sources to identify your issue: WVD-Agent, WVD-Agent-Updater, RDAgentBootLoader, MsiInstaller. If you need further assistance, you can create a support request or ask Azure community support.
References: