Can I parameterize the dataset at storage account level?

Question

I have a pipeline that takes an input file from a specific storage account, with existing parameterization down to the container level. I want to extend this parameterization to include the storage account itself. Depending on whether I input storage account 1 or 2, the pipeline should fetch files from the corresponding location.

Answer 1

Yes, it is possible to parameterize the dataset at the storage account level in ADF by using parameters in your linked service and datasets.

1. Create Parameters in the Linked Service:
   • Go to the Linked Services section in ADF.
   • Edit the linked service that connects to your storage account.
   • Add a parameter for the storage account name. For example, you can add a parameter named StorageAccountName.
2. Modify the Linked Service Connection String:
   • In the linked service connection string, replace the storage account name with the parameter you just created. For example:

     
           {
     
               "type": "LinkedService",
     
               "typeProperties": {
     
                   "connectionString": {
     
                       "value": "DefaultEndpointsProtocol=https;AccountName=@{linkedService().StorageAccountName};AccountKey=<your_account_key>;EndpointSuffix=core.windows.net"
     
                   }
     
               }
     
           }
     
     

3. Create Parameters in the Dataset:
   • Go to the Datasets section in ADF.
   • Edit the dataset that you want to parameterize.
   • Add parameters for the storage account name, container name, and possibly the file path.
4. Modify the Dataset to Use Parameters:
   • In the dataset, use the parameters you created to dynamically construct the path. For example:

     
           {
     
               "type": "Dataset",
     
               "typeProperties": {
     
                   "location": {
     
                       "type": "AzureBlobStorageLocation",
     
                       "container": {
     
                           "value": "@{dataset().ContainerName}"
     
                       },
     
                       "filePath": {
     
                           "value": "@{dataset().FilePath}"
     
                       },
     
                       "storageAccount": {
     
                           "value": "@{dataset().StorageAccountName}"
     
                       }
     
                   }
     
               }
     
           }
     
     

5. Set Parameters in the Pipeline:
   • In the pipeline that uses the dataset, set the parameters for the storage account name, container name, and file path.
   • When you trigger the pipeline, you can provide different values for these parameters to fetch files from different storage accounts and locations.
6. Example Pipeline Parameters:
   • Define pipeline parameters for StorageAccountName, ContainerName, and FilePath.
   • Pass these parameters to the dataset in the activities within your pipeline.

More links :

https://learn.microsoft.com/en-us/azure/data-factory/parameterize-linked-services?tabs=data-factory

https://www.c-sharpcorner.com/article/parameterization-in-azure-data-factory-linked-services/

https://dataengineerazure.medium.com/how-to-parameterize-dataset-in-azure-data-factory-parameterization-in-azure-data-factory-4cae57d886aa

https://davidalzamendi.com/azure-data-factory-linked-services/

https://www.linkedin.com/pulse/parameterizing-linked-service-azure-data-factory-nithyanandam/