Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,656 questions
Azure datafactory Marketing cloud sFTP connection error
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 11%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Chand, Anupam SBOBNG-ITA/RX
466
Reputation points
We are trying to connect to the sFTP server hosted on Salesforce Marketing cloud. We are using the sFTP connector as a source on an Azure data factory pipeline. Even though we are able to test the connection and browse/preview the source successfully, when trying to execute the copy activity, we are getting the below error:
{ "errorCode": "2200", "message": "ErrorCode=SftpNetworkIssue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Meet network issue
when connect to Sftp server 'mc871qqp8bp5k4ljq9f4hpppn-g1.ftp.marketingcloudops.com', SocketErrorCode: 'ConnectionReset'.,Source=Microsoft.DataTransfer.ClientLibrary.SftpConnector,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=System,'", "failureType": "UserError", "target": "Copy data1", "details": [] }
When we approached the Salesforce support team, and they suggested to whitelist the Salesforce IPs on Azure but I'm aware that ADF does not have any option whitelist IPs. The only thing I can think of is whitelisting Azure IR IPs on the sFTP server. To add, we tried connecting to the sFTP server using Filezilla and we were able to connect successfully and download the files from there which tells me that this is not an IP whitelisting problem. Just wanted to check if there is something else we can try.
We have set the max concurrency to 1 and we still got the same error.
Sample pipeline run id : 4adfae63-73fd-453b-9d8d-e3cafbe5d162
Azure Data Factory
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT 26,236 Reputation points2020-10-15T00:03:24.863+00:00 Hello anonymous user and welcome to Microsoft Q&A.
I concur with your assessment that this is not an IP whitelisting problem. There are many possible reasons the Sftp server might sever the connection. From http/https to TLS versions, to an error on their server, to resource limitations and many more. We can eliminate half of these depending upon when the connection was terminated. If it was terminated after the data started flowing, we can eliminate the former. The duration of the copy activity suggests that data had started flowing.
Thank you for sharing the max concurrency. That also eliminates some causes.
I notice that the pipeline run you provided was a debug run. Do you have a triggered run you can share, and if so, does it also fail the same way?
-
Chand, Anupam SBOBNG-ITA/RX 466 Reputation points
2020-10-15T01:39:02.177+00:00 Thanks @MartinJaffer-MSFT ,
Here is the pipeline run id after it was published. 914b37e6-2193-4d82-9ef7-4b1145fb0f17. This also failed in the same way. If you look at the outputs, there is no data copied at all. I'm also convinced that it is a server problem but what can i ask those guys to try to establish successful connectivity. And how come the test connection works and we are also able to preview the files in ADF UI.
Regards,
A -
MartinJaffer-MSFT 26,236 Reputation points
2020-10-15T23:43:29.07+00:00 anonymous user I have some more information for you.
I found a similar case, where this issue was caused by running the copy activity inside a loop. The loop cause many
connections to be made, and became throttled by the SFTP server. Is your copy activity inside a loop? If so, is it sequential or parallel? -
MartinJaffer-MSFT 26,236 Reputation points
2020-10-23T17:58:57.13+00:00 anonymous user I have not heard back from you. Are you still experiencing the problem?
-
MartinJaffer-MSFT 26,236 Reputation points
2020-10-26T18:16:11.697+00:00 If you found your own solution, could you please share it with the community?
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 72%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EXT%3C/text%3E%3C/svg%3E)
Xia, T. (Tiancong) 1 Reputation point2022-10-10T08:42:34.11+00:00 @MartinJaffer-MSFT Hi Martin,
My use case is copy activities inside a loop. And I am experiencing a lot of sFTP connection error, do you know the best way to solve this? -
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 51%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
arif refai 1 Reputation point2022-11-17T18:01:31.22+00:00 Hi @MartinJaffer-MSFT , We are using the SFTP copy inside a loop and facing the same error. Do you have any resolution for this ?
Sign in to comment
Sign in to answer