If that connection is working (at least for some short time), then I would suggest that everything on SQL server side was set up correctly!
As it seems to be an intermittent problem I would start to investigate deeper into the network part of your environment…
How are your servers connected to each other?
Are there any firewalls between those servers?
Is the local firewall active?
Are there any other network device that might drop the connection after a specific time?
We once had an issue with a firewall between two servers that thought the connection was idle for too long and kicked it.