Program on VM automatically crash after long idle

Question

Hi,

I am training machine learning model on Azure VM with NC6 promo GPU. Everything was fine at the beginning, but after a while I went back to check and realized my training program was stopped. Also, I got this message "client_loop: send disconnect: Broken pipe". Is there any solution for this problem since it cost me a lot of time and money.

Answer 1

@Le Nguyen Minh Huy Thanks for the question. We have forwarded to the product team to check on this issue. You can try the following.

The sshd/server settings in /etc/ssh/sshd_config :
TCPKeepAlive yes
ClientAliveInterval 60
ClientAliveCountMax 40000

AND

the ssh/client setting in ~/.ssh/config :
ServerAliveInterval 60

We would recommend to raise a Azure support desk ticket from Help+Support blade from Azure portal for your service resource. This will help you to share the details securely and work with an engineer who can provide more insights about the issue that if it can be replicated.