Intermittent connectivity issue between Azure Data Factory and Blob Storage — 'remote name could not be resolved'

Question

Hi everyone, For about a week now, I’ve been experiencing an intermittent issue when running a pipeline in Azure Data Factory that writes data to an Azure Blob Storage container. The pipeline sometimes fails with the following error:

Operation on target SOURCE failed: Operation on target INGEST failed: Operation on target ForEach Table failed: Activity failed because an inner activity failed; Inner activity name: Export table, Error: ErrorCode=AzureBlobFailedToCreateContainer,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Unable to create Azure Blob container. Endpoint: 'https://xxxxxxxxx.blob.core.windows.net/', Container Name: 'container01'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.WindowsAzure.Storage.StorageException,Message=The remote name could not be resolved: 'xxxxxxxxx.blob.core.windows.net',Source=Microsoft.WindowsAzure.Storage,''Type=System.Net.WebException,Message=The remote name could not be resolved: 'xxxxxxxxx.blob.core.windows.net',Source=System,'.

The issue is intermittent, sometimes the pipeline runs successfully, and other times it fails, with no configuration changes in between.

We’ve noticed that this problem only happens in pipelines that use data sources located on-premises, which require a Self-hosted Integration Runtime (SHIR) to access the data. Pipelines that do not use a SHIR (i.e., those pulling from cloud-based sources) are able to complete the copy activity without errors.

In addition, we are also experiencing similar errors when running Python code (Azure function) that attempts to write to the same Blob Storage account, where it also fails intermittently with a “remote name could not be resolved” message.

Prior to this issue, everything was working fine — both the ADF pipelines and the Python-based processes were functioning normally.

No changes have been made to our infrastructure:

• The Storage Account has not been recreated
• The managed private endpoint between Data Factory and the Storage Account remains unchanged

Has anyone else encountered this issue recently? If you’ve experienced something similar and found a solution or workaround, your help would be greatly appreciated. Thanks in advance!

Answer 1

Hi everyone,

I’m closing the thread with the solution that worked for us, in case it helps someone else. After a deep dive we confirmed the problem wasn’t the Storage firewall or our ADF settings, but our internal DNS resolution used by the Self‑Hosted Integration Runtime (SHIR).

1. Using nslookup we saw the SHIR queries three on‑prem DNS servers. The first one ( 10.27.x.x ) replied but returned an empty answer for <storage>.blob.core.windows.net, the other two servers ( 10.25.x.x and 10.26.x.x ) returned the correct private IP for the Private Endpoint. Windows treats the empty answer as “valid” and caches it, which explains the intermittent failures.

Immediate workaround: in the VNet’s DNS server list we placed the servers that resolve correctly first and left the faulty one last. From that moment the pipelines stopped failing.

Permanent fix: on the malfunctioning DNS server we added a conditional forwarder for blob.core.windows.net and privatelink.blob.core.windows.net, pointing to 168.63.129.16 (Azure DNS). That server now also returns the correct IP, so the order no longer matters.

(Optional) To speed things up you can clear the cache on the SHIR VM with ipconfig /release, ipconfig /renew and ipconfig /flushdns, but it will work as soon as the negative TTL expires anyway.

We’ve been running for more than 48 hours without a single “remote name could not be resolved” error.

Thanks to @Ganesh Gurram for the suggestions—hope this helps anyone facing the same issue. Best regards!