An Azure service for ingesting, preparing, and transforming data at scale.
Pipeline using a new integration runtime never end or fail while attempting the copy activity
I created a new integration runtime with two nodes following the instructions of the document:
The network team added the necessary URLS to the firewall per the same document.
When testing the IR using a simple pipeline to transfer data from an SQL data base to a json file it fails or never ends.
Telnet tests to URLs and ports added in the FW work fine.
The access to the DB using SSMS works as well.
The IR diagnostic tool only fails when attempting to connect to download.microsoft.com, but this feature to download and upgrade the IR SW was disabled.
I'm able to create a linked service to the SQL server required using the IR with a succesfull conection.
Created a data set with the same linked service which works fine.
Using the preview option to show partially the data works without issue, just like if I use a lookup activity.
The problem comes when trying to use the copy data activity as when the activity starts in remains in the time to first byte step and it nevers reach a single byte of data.
As the network team is very restrictive and previously had some issues witht the registration of the nodes, I followed the trouble shooting guide
https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-troubleshoot-guide?tabs=data-factory
and used wireshark to found out if there were some URLs getting reset.
Found that the storage account URL that is used to store the json file with the data, is apparently getting blocked when it tries the node to reach it, as it can be seen in the trace that the IP address used is retransmitted until is reset. Even tough the telnet to this IP address and the port 443 works fine.
From the network team indicates that this might be caused by masking of the URL or somehitng else, as the Firewall is not showing this particular IP or URL getting block.
Do you have any ideas on how to proceed with this issue and be able to make the nodes work properly?
Thanks
Ruben