SFTP Sink VERY Slow in ADF Self hosted IR

Calin M 1

We'd like to use the sftp sink in a copy task to upload some very large files to the cloud. We find that the upload is so slow, it is unusable. The upload using SFTP Sink/ADF/SelfHostedIR is orders of magnitude slower than uploading the file using a regular sftp client (winscp, filezilla, cygwin)

We are running the jobs on a self hosted IR. Our files are upwards of 30g, so we cannot use the sftp sink, it would just take forever.

Has anyone successfully uploaded large files, using the SFTP sink and run the job in the self hosted IR?

Thanks!

HimanshuSinha-msft 19,476 Reputation points Microsoft Employee

2020-12-21T14:56:46.637+00:00

Hello @Calin M ,
Thanks for the ask and using the forum .
Let me start with saying thet I have not uploaded a file that big . But I have aquestion and a suggestion .
Question :Can you please elaborate what do you mean when you say "We are running the jobs on a self hosted IR" ?
To test if this is an issue with the SHIR or not , i could have tried to copy the file from the SHIR to the sink ( I think you are already doing ) . Also I could try to copy any file that large from the azure storage ( may be blob) to the sink . In the second case there is NO SHIR involved . If you are still facing while testing with NO SHIR scenario then we will have to dig more .

Let me know how it goes.
Thanks
Himanshu .
Calin M 1 Reputation point

2020-12-22T03:13:57.867+00:00

The issue is definitely with the SHIR.

Both the file system source and the SFTP Sink are associated with the SHIR. The SFTP Sink points to a sftp server in the cloud (not azure storage, just some linux server). When I use the Copy Task in a pipeline, it is many times slower than when I use winscp or some other scp client to transfer the file from the server hosting the SHIR to the cloud sftp server.

I saw this at a client site, and I could also reproduce in my environment (similar setup)

You don't have to run a big file to test this, just run any file that is big enough so you can measure the time, and compare the time it takes to upload the file to SFTP using ADF, with the time it takes to do the same thing using winscp or filezilla. Start with the file on the same server as the SHIR.
Saurabh Sharma 23,816 Reputation points Microsoft Employee

2020-12-29T19:20:14.683+00:00

@Calin M I suggest you to please open an Azure Support ticket so that a data engineer can investigate this issue. In case you have any limitations creating a support ticket please send an email to azcommunity@microsoft.com with your subscription id with this Q&A thread link and subject as "Attn: Saurabh' and I will help providing a free support ticket for you.

Thanks
Saurabh
Saurabh Sharma 23,816 Reputation points Microsoft Employee

2020-12-30T19:33:55.773+00:00

@Calin M Please let me know if you need help on creating a support ticket.
Calin M 1 Reputation point

2021-01-04T06:43:38.21+00:00

Hi Saurabh,

I emailed azcommunity@microsoft.com with my subscription ID. Please help open a ticket for me.

Thank you,

Calin
Saurabh Sharma 23,816 Reputation points Microsoft Employee

2021-01-04T23:33:23.24+00:00

@Calin M Thanks. I have received your email. I will update you with the details on email.

Share via

SFTP Sink VERY Slow in ADF Self hosted IR

Your answer