SFTP Timeout while copying multiple files form SFTP Server to a Merged file to Blob storage

Question

I tried by loading the multiple files from SFTP and coping the files as a single file to Blob account.

Able to read all the files but unable to merge the files. It was throwing timeout issue.

I want to figure out how many connections that the SFTP is taking and how the connection with SFTP will work..?

Error Message:

ErrorCode=SftpNetworkIssue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Meet network issue when connect to Sftp server 'direct.dev.mckesson.com', SocketErrorCode: 'TimedOut'.,Source=Microsoft.DataTransfer.ClientLibrary.SftpConnector,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond,Source=Renci.SshNet,'

Answer 1

Hello @HarshithaVaka-6516,

Welcome to QnA forum.

The number of connections that the SFTP server takes depends on the configuration of the server and the number of concurrent connections allowed.

To troubleshoot the timeout issue, you can try the following steps:

• Check the network connectivity between the SFTP server and the Azure Data Factory.
• Check the firewall settings on the SFTP server to ensure that it allows connections from the Azure Data Factory.
• Increase the timeout value in the SFTP linked service configuration in Azure Data Factory
• Reduce the number of concurrent connections in the SFTP dataset configuration in Azure Data Factory

I have tried the scenario in SFTP-->Storage copy activity.

Source Settings:

Read all Files from the SFTP location.

Sink Settings:

I used the 'Merge Files'

and

Block Size (MB)---> 100 max

Output

All the files are written to a single destination file as merged content.

Please let us know if you need further help in this matter.