Hi @John Champ ,
It sounds like you might be hitting by-design feature of the software load balancer in front of your VMs. By default it will close any idle connections after 4 minutes, but you can configure the timeout to be anything between 4 and 100 minutes:
Configurable Idle Timeout for Azure Load Balancer
You need to have a Standard load balancer from Azure to end the connection with a TCP RST packet, letting the app know that the connection has ended. The free load balancer that comes with the server does not have this feature.
Let me know if this helps and if you run into any issues.
If the information helped you, please Accept the answer. This will help us as well as others in the community who may be researching similar questions.