The remote name could not be resolve error

Question

Hi,

I have created my resources in azure as follows in azure.

1. Created storage account(ADLS2) in private network and enabled private end points.
2. Created azure data factory in private network.
3. Created managed private end point in data factory for the storage account.
4. Created managed virtual network integration runtime in azure data factory.
5. Created linked service to that storage account by using managed virtual network integration runtime and successfully tested the connectivity.
6. Also able to write files to storage account by using above created linked service.
7. created Azure windows VM in private network and installed Selfhosted integration runtime on that windows VM. SHIR is up and running successfully.
8. We have made necessary network changes to connect on-premises data stores from azure data factory.
9. Created linked service to on-premises database by using SHIR. Connection was successfully.
10. Created dataset for on-premises data base and able to preview the table data.
11. Used copy activity to read data from on-premises data store and sink data to storage account. This is the step is failing with below error.
    Error Details:
    ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: An error occurred while sending the request.. Account: 'etlcoestoragedev'. FileSystem: 'input'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=An error occurred while sending the request.,Source=mscorlib,''Type=System.Net.WebException,Message=The remote name could not be resolved: 'etlcoestoragedev.dfs.core.windows.net',Source=System,'
12. Note: My storage account, Data factory and Windows VM are in same virtual network and same subnet in azure. Regards, Mani

Answer 1

You are having a problem with the DNS resolution issue, where the domain name cannot be resolved to an IP address. In your specific case, the error occurs when trying to access the ADLS Gen2 storage account from your Azure Data Factory pipeline.

You may look to this old thread : https://learn.microsoft.com/en-us/answers/questions/1362264/adf-linked-service-unable-to-create-connection-wit

1. Private Endpoint DNS Configuration:

• Private DNS Zone: When you create a private endpoint, Azure automatically creates a DNS record for it within a private DNS zone. Ensure that this private DNS zone is correctly associated with the virtual network where your resources are located.
  • DNS Resolution: Verify that the DNS resolution within your virtual network is correctly configured. The VM should be able to resolve the storage account's private endpoint to its private IP address. You can test this by running a DNS query (e.g., using nslookup) from your VM.

  
     nslookup etlcoestoragedev.dfs.core.windows.net
  
  

  The result should show the private IP of the storage account.

2. Network Security Group (NSG) Rules:

• Check the NSG rules associated with the subnet where your resources reside. Ensure that there are no rules blocking outbound traffic from your Data Factory, VM, or other relevant resources to the storage account.

3. Routing and Peering:

• Routing: Ensure that the routing within the virtual network is correctly set up and that there's no conflict or misconfiguration that might prevent traffic from reaching the storage account's private endpoint.
  • Virtual Network Peering: If you're using VNet peering, make sure the necessary DNS and routing configurations are shared across peered networks.

4. Integration Runtime Configuration:

• Managed Virtual Network: Verify that the integration runtime is correctly configured to use the managed virtual network in your Azure Data Factory.
  • SHIR Configuration: Double-check that the Self-hosted Integration Runtime (SHIR) on your Azure VM has access to the storage account. If the SHIR is configured to route traffic through an on-premises network, ensure that the DNS settings there can resolve the storage account's private endpoint.