ADF Copy Data Intermittently throwing an error

Question

I have a ADF pipeline. Starting activity is "Copy Data". Sometimes this will process more than 20 files successfully and sometimes it will not be able to process 10 or 20 files.

when it will not be able to process it, that time Copy Data will be going in Error.

Error is -

"ErrorCode=SftpNetworkIssue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Meet network issue when connect to Sftp server 'bn-r12devapp04.nuance.com', SocketErrorCode: 'ConnectionReset'.,Source=Microsoft.DataTransfer.ClientLibrary.SftpConnector,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=Renci.SshNet,'"

If I reduce the total file count to 5 or 7, Copy Data works fine.

Earlier we have processed 40 files in a single run as well. I dont know why Copy Data behaving like this.

Can someone please check on this

OR

have faced similar issue. Please help me.

Thanks,

Nitin Kolte

Answer 1

I think this type of error, where the connection is forcibly closed by the remote host, is often related to network instability, connection limits on the server, or resource constraints.

Try reducing the number of concurrent connections in your ADF pipeline. You can configure this in the "Copy Data" activity under the "Data Integration Units" (DIUs) or set a limit on parallelism.

Contact the administrator of the SFTP server to see if there are any logs or configurations that might indicate a resource or connection limit issue.

Check the network connectivity between your ADF and the SFTP server. Tools like ping or traceroute might help diagnose any latency or packet loss issues.