question

WesselVanErp-8373 avatar image
0 Votes"
WesselVanErp-8373 asked KranthiPakala-MSFT commented

Integration runtime whitelisting IP for SFTP connection

Hi,

For a customer I'm currently working on a project where we periodically want toload data from an SFTP server with the use of Data Factory. I can't, however, get it to work. Everytime I'm trying to make a connection using the AutoResolveIntegrationRuntime, I get the message:
SocketErrorCode: 'TimedOut'. A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

I think the problem is on our side. I'm able to connect to the SFTP server using Filezilla and using a locally installed IR I was able to make to connection in Data Factory. Hence, it seems that the SFTP server is blocking Azure somehow.

We have whitelisted the IP adresses stated on: https://docs.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses. Since our Data Factory is hosted in Western Europe, I assume that I have to whitelist the IP adresses for that part of the world:

40.74.26.0/23,
40.74.24.192/26,
13.69.67.192/28


Below a link to a screenshot of the configuration of the SFTP server (I can't upload it here somehow):
https://ibb.co/7rw6q6Z

Can anybody tell what I'm doing wrong?

Thank you in advance for helping me out.

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @WesselVanErp-8373 ,

Welcome to Microsoft Q&A platform and thanks for reaching out.

Could you please tell us if you seeing this error consistently or if you are noticing it intermittently with Azure IR? Also could you please share the failed run ID of the pipeline and the activity for further analysis?

  • failed pipeline run ID

  • failed activity run ID


0 Votes 0 ·

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @WesselVanErp-8373 ,

If you choose to use the auto-resolve Azure IR for both Source and Sink , which is the default,

For copy activity, ADF will make a best effort to automatically detect your sink data store's location, then use the IR in either the same region if available or the closest one in the same geography; if the sink data store's region is not detectable, IR in the data factory region as alternative is used.

For example, you have your factory created in East US,

When copy data to Azure Blob in West US, if ADF successfully detected that the Blob is in West US, copy activity is executed on IR in West US; if the region detection fails, copy activity is executed on IR in East US.
When copy data to Salesforce of which the region is not detectable, copy activity is executed on IR in East US.

For more clarification please refer to this doc : Integration runtime location

In case if your sink is also using Azure IR, I would recommend you to please try white-listing Sink data store region IP range, else try white-listing all the Azure IR Ip range
mentioned in the document. (Azure Integration Runtime IP addresses)


Hope this helps.



Thank you

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.




· 6
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @WesselVanErp-8373 ,

Just checking to see if the above info was helpful. If this answers your query, please do consider to click “Accept Answer” and "Up-Vote" as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

0 Votes 0 ·

Hi Kranthi Pakala,

Thank you for taking time trying to help me out.

It is not the pipeline that gives these errors. I didn’t even come as far as using it in a pipeline.

When I add a new data source you have the possibility to test the connection. With using the auto IR I this test is not a success. However, when I test this with a local IR and the same settings as with the auto IR, the test gives a positive result.

I just need the auto IR to get to work. I think it has something to do with whitelisting on the sftp server’s side, but I’m not sure what is actually causing this issue.

0 Votes 0 ·

Hi @WesselVanErp-8373 ,

Thanks for your response. Yes, you are correct, in case if you are using Azure IR for your sink, then you will have to whitelist Azure IR IP range of your sink region. (For example if your sink is Azure Blob storage from East US2 then you will have to try whitelisting East US2 IP addresses - 20.41.2.0/23, 20.41.4.0/26, 20.44.17.80/28)

Here is how it works:
- If IR of source or sink are SHIR, SHIR will be choosed.
- If both source and sink are Azure IR and one of them are manually created IR, the region of manual created azure IR is choosed
- If both source and sink are azure IR and all of them are auto resolved, sink region is choosed.

Hope this info helps. In case if you still encounter the issue, could you please tell us what is your sink, its region and what is the IR (manually create or Auto resolve IR) used? And also please try whitelisting full Azure IR IP range and see if that helps to resolve the issue?

Please let us know how it goes.
Thanks

0 Votes 0 ·

Hi @KranthiPakala-MSFT,


Again, thank you for taking time trying to help me out.


This, however, still doesn't help me out. Maybe I didn't give the right information to you and lead you in the wrong direction. The thing that isn't working is the connection to a linked service. At this point it doesn't have anything to do with source and sink (I assume).


In the screenshots below you see where I get stuck. In the first one you see the linked service with the auto resolve and in the second one the linked service with the self hosted IR. All other configurations are equal, it's just the IR I manually chose.


9988-asir.png 9998-shir.png


Hopefully this gives you a better context, so you are able to help me out.


0 Votes 0 ·
asir.png (49.1 KiB)
shir.png (47.4 KiB)

Hi @WesselVanErp-8373 ,


Thanks for your response and clarification. Have you got a chance to whitelist all the Azure IP Range and see if that helps to resolve the issue? If you see the same issue even after whitelisting Azure IR IP range, please email below details to AzCommunity@microsoft.com to work closely on this issue.


Email subject: <Attn - Kranthi : Microsoft Q&A Thread title>


Thread URL: <Microsoft Q&A Thread link>


Subscription ID: <your subscription id>


Please let me know here once you have done the same.




Thank you


0 Votes 0 ·
Show more comments