Access denied when I'm trying to connect to private blob storage from Azure Data Factory

Question

Hi,

My goal is to run an exe file stored in a private Azure Blob container.

The exe is simple : it creates a text file, write the current datetime in it, and then push it to the private Azure Blob container.
This has to be sent from Azure Data Factory. To do this, here is my environment :

• Azure Data Factory running with the simple pipeline :
  
  • Private storage account with the following configuration :
    
  • A linked service connected to the storage account :
    
  • A private managed virtual network approved :
    
  • A linked service connected to an Azure Batch :
    
  • A batch account linked to the right storage account
  • A pool running on this batch account

Two things that I need to add in context :

• When I set the storage account to public, it works and I find the text file in my blob storage. So the process works well, but there is a security issue somewhere I can't find.
• All the resources (ADF, Blob storage, Batch account) used have a role has contributor/owner of the blob with a managed identity.

Here is the error I get when I set the storage account to private :

   {  
      "errorCategory":0,  
      "code":"BlobAccessDenied",  
      "message":"Access for one of the specified Azure Blob(s) is denied",  
      "details":[  
         {  
            "Name":"BlobSource",  
            "Value":"https://XXXXXXXXXXXXXXXXX/testv2.exe?sv=2018-03-28&sr=b&sig=XXXXXXXXXXXXXXXXXX&sp=r"  
         },  
         {  
            "Name":"FilePath",  
            "Value":"D:\\batch\\tasks\\workitems\\XXXXXXXXXXX\\job-1\\XXXXXXXXXXXXXXXXXXXXXXXX\\testv2.exe"  
         }  
      ]  
   }  

Thank you for your help!

Answer 1

Hello @tmsagora ,

Welcome to the MS Q&A platform.

The issue has been documented here.

Resolution:

To resolve the issue, manage the Batch pool and the storage account configurations based on your scenarios.

Scenario 1: Batch pool and storage account are in the same region, and Batch pool has a virtual network

Check Subnet information under Network Configuration from the Azure portal > Batch Account > Pool > Properties. Take note and write the information down.

Navigate to the storage account, and select Networking. In the Firewalls and virtual networks setting, select Enable from selected virtual networks and IP addresses for Public network access. Add the Batch pool's subnet in the firewall allowlist.

If the subnet doesn't enable the service endpoint, when you select it, a notification will be displayed as follows:

The following networks don't have service endpoints enabled for 'Microsoft.Storage'. Enabling access will take up to 15 minutes to complete. After starting this operation, it is safe to leave and return later if you don't wish to wait.

Therefore, before you add the subnet, check it in the Batch virtual network to see if the service endpoint for the storage account is enabled.

After you complete the configurations above, the Batch nodes in the pool can access the storage account successfully.

Scenario 2: Batch pool and storage account are in different regions

Create a new Batch pool in a virtual network with a static public IP address. For more information, see Create a Batch pool with specified public IP addresses.

Because the Batch pool and storage account are in different regions, the outbound traffic will go through the public internet via the public IP address.

Write down the public IP address.

Assign the public IP address to the Batch pool public Load Balancer's IP.

Add the public IP address to the storage firewall allowlist.

Run the Batch jobs with the newly created Batch pool.

Please go over this and let us know if you have any further questions.

------------------------------

• Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators