Miscellaneous error encountered while downloading one of the specified Azure Blob(s) during Azure Batch job in ADF

Question

I am encountering a Miscellaneous error encountered while downloading one of the specified Azure Blob(s) error when I run a Python script in Azure Data Factory in a Subscription A while everything reproduced from scratch works on a Subscription B.

Steps:

1. Setting up Azure Batch Service and Pool with Ubuntu Server 22.04, User Identity: Admin Pool, AND Virtual Network VN1 + Subnet Subnet1 selected during the Pool Configuration
2. Storage Account with container example and hello.py script. Firewall settings: Enable from selected virtual networks: VN1 and Subnet1 +
3. Azure Data Factory, where I create a Managed Virtual Network Integration Runtime and Add Batch Account with Pool details and Blob Storage from 2. as linked service

The Issue: when I set it up from scratch in Subscription B, when I create a Managed Private Endpoint in ADF from step 3 and approve it in SA from 2 - everything works just fine.

In case of Subscription A, it works if the Storage Account is public. When I change the setup to "Enable from selected virtual networks" and select VN associated with my Pool, generate Managed Privte Endpoint in ADF and approve it in the SA - I get the error:

Miscellaneous error encountered while downloading one of the specified Azure Blob(s)

FilePath - /mnt/batch/tasks/workitems/mypoolname/job-1/myjobid/wd/activity.json

Message - Failure in name resolution for https://mysaname.blob.core.windows.net/adfjobs/myjobid/runtime/activity.json?sv=2021-08-06&se=2025-03-21T12%3A36%3A07Z&sr=b&sp=r&sig=mysig

Unfortunately Subscription A is a prod environment and I don't want to experiment too much.

Answer 1

@11-4688

Thank you for posting your query!

please check the below steps and confirm us

1. DNS Resolution: The error message indicates a failure in name resolution. Ensure that the DNS settings in your virtual network are correctly configured. You might need to set up a custom DNS server that can resolve the Azure Blob Storage endpoints.
2. Network Security Group (NSG) Rules: Verify that the NSG rules associated with your virtual network and subnet allow outbound traffic to Azure Blob Storage. Sometimes, restrictive NSG rules can block necessary traffic.
3. Private Endpoint DNS Configuration: When using private endpoints, ensure that the DNS configuration is correctly set up to resolve the private endpoint's IP address. You might need to configure a DNS forwarder or use Azure Private DNS Zones.
4. Firewall Settings: Double-check the firewall settings of your storage account. Ensure that the virtual network and subnet are correctly added to the allowed networks list.
5. Managed Private Endpoint Approval: Ensure that the managed private endpoint in ADF is correctly approved in the storage account. Sometimes, re-approving the endpoint can help resolve connectivity issues.
6. Logs and Diagnostics: Enable diagnostic logs for your storage account and Azure Batch account. These logs can provide more detailed information about the error and help pinpoint the issue.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.