Errors while copying files from SFTP with ADF Copy data Activity

Question

Errors while copying files from SFTP with ADF Copy data Activity

Chukka, Rani 76

ADF pipeline failed loading the file form SFTP server to Storage account with the following error.

ErrorCode=UserErrorFailedToReadStream,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException

,Message=Failed to read data from Sftp server '', file path ''

, offset '16777216', length: '16777216'.

,Source=Microsoft.DataTransfer.ClientLibrary.SftpConnector,''Type=Renci.SshNet.Common.SshException,Message=Input/output error,Source=Renci.SshNet,'

The pipeline is working fine from last couple of months. No code changes made, and no sftp connection issues found.

Appreciate your help.

Accepted answer

2 additional answers

Your answer

Answer 1

SSingh-MSFT 16,371 Moderator

Hi @Chukka, Rani ,

Thanks for posting this question in Microsoft Q&A forum.

AS I understand from the question, ADF pipeline failed loading the file form SFTP server to Storage account with Error: Message=Failed to read data from Sftp server.

Could you please check below points for troubleshooting:

1). Could you please check if the Data which is getting transferred has increased considerably in size?

2). Does Linked Service and Dataset configuration in ADF looks fine.

3). Could you please check out what is the maximum concurrent connection limit in your sftp server?

4). If the files are large, please take steps suggested here: SFTP Connector Problem with large files

5). Utilize 'Degree of Parallelism.' in Copy Activity settings that data loading will use.

Please try options mentioned above and share your inputs to proceed further.

Thank you.

Chukka, Rani 76



I tried your recommendations, please find the details below. No change in error message. 

1). Could you please check if the Data which is getting transferred has increased considerably in size?
    No Change in data size. File sizes are - 37.2MB,41.1MB and 24MB

2). Does Linked Service and Dataset configuration in ADF looks fine.
    Yes, Linked Service and Dataset configuration is good, no issues with copying small files of size 836KB.

3). Could you please check out what is the maximum concurrent connection limit in your sftp server?
    Max concurrent connections = null; 
    I tried with Max concurrent connections = 1, No change in result, got same error.

4). If the files are large, please take steps suggested here: SFTP Connector Problem with large files
    I don't see any recommendation in the provided link

5). Utilize 'Degree of Parallelism.' in Copy Activity settings that data loading will use.
    Tried with "Degree of Parallelism = 1" and "Degree of Parallelism = 3" as well, no luck.

Appreciate your help.

Chukka, Rani 76

Hi @SSingh-MSFT ,

I tried your recommendations, please find the details below. No change in the error message.

1). Could you please check if the Data which is getting transferred has increased considerably in size?
    No Change in data size. File sizes are - 37.2MB,41.1MB and 24MB

2). Does Linked Service and Dataset configuration in ADF looks fine.
    Yes, Linked Service and Dataset configuration is good, no issues with copying small files of size 836KB.

3). Could you please check out what is the maximum concurrent connection limit in your sftp server?
    Max concurrent connections = null; 
    I tried with Max concurrent connections = 1, No change in result, got same error.

4). If the files are large, please take steps suggested here: SFTP Connector Problem with large files
    I don't see any recommendation in the provided link

5). Utilize 'Degree of Parallelism.' in Copy Activity settings that data loading will use.
    Tried with "Degree of Parallelism = 1" and "Degree of Parallelism = 3" as well, no luck.

Appreciate your help. Thanks

Answer 2

Rose Mary Job 5

I unchecked "Disable chunking" option in source in copy activity and the issue is resolved.

SSingh-MSFT 16,371 Reputation points Moderator

2023-05-05T08:39:28.6133333+00:00

Hi Rose Mary Job •,

Thank you for sharing the solution adopted by you.

Appreciate your efforts in sharing it with the community. Thanks

Answer 3

SSingh-MSFT 16,371 Moderator

Hi @Chukka, Rani ,

Thanks for detailed reply.

As mentioned by you, for smaller files there is no issue. For larger files Pipelines is failing.

I would recommend you to check at SFTP side, it might be because of the insufficient storage on SFTP server.

Please try storage optimization at SFTP level and this should help.

Do let us know if further queries.

SSingh-MSFT 16,371 Reputation points Moderator

2023-02-21T07:45:35.4633333+00:00

Hi @Chukka, Rani ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Mark Helpful for the same. And, if you have any further query do let us know.
Chukka, Rani 76 Reputation points

2023-04-25T15:08:36.0533333+00:00

The issue was resolved by itself after couple of days. Not sure about the root cause. Thank you for your support.

Share via

Errors while copying files from SFTP with ADF Copy data Activity

2 additional answers

Your answer