Can I parameterize the dataset at storage account level?

Hrithik Purwar 25 Reputation points
2024-08-01T09:57:58.49+00:00

I have a pipeline that takes an input file from a specific storage account, with existing parameterization down to the container level. I want to extend this parameterization to include the storage account itself. Depending on whether I input storage account 1 or 2, the pipeline should fetch files from the corresponding location.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,657 questions
0 comments No comments
{count} votes

Accepted answer
  1. Amira Bedhiafi 34,491 Reputation points Volunteer Moderator
    2024-08-01T10:57:40.3033333+00:00

    Yes, it is possible to parameterize the dataset at the storage account level in ADF by using parameters in your linked service and datasets.

    1. Create Parameters in the Linked Service:
      • Go to the Linked Services section in ADF.
      • Edit the linked service that connects to your storage account.
      • Add a parameter for the storage account name. For example, you can add a parameter named StorageAccountName.
    2. Modify the Linked Service Connection String:
      • In the linked service connection string, replace the storage account name with the parameter you just created. For example:
        
              {
        
                  "type": "LinkedService",
        
                  "typeProperties": {
        
                      "connectionString": {
        
                          "value": "DefaultEndpointsProtocol=https;AccountName=@{linkedService().StorageAccountName};AccountKey=<your_account_key>;EndpointSuffix=core.windows.net"
        
                      }
        
                  }
        
              }
        
        
    3. Create Parameters in the Dataset:
      • Go to the Datasets section in ADF.
      • Edit the dataset that you want to parameterize.
      • Add parameters for the storage account name, container name, and possibly the file path.
    4. Modify the Dataset to Use Parameters:
      • In the dataset, use the parameters you created to dynamically construct the path. For example:
        
              {
        
                  "type": "Dataset",
        
                  "typeProperties": {
        
                      "location": {
        
                          "type": "AzureBlobStorageLocation",
        
                          "container": {
        
                              "value": "@{dataset().ContainerName}"
        
                          },
        
                          "filePath": {
        
                              "value": "@{dataset().FilePath}"
        
                          },
        
                          "storageAccount": {
        
                              "value": "@{dataset().StorageAccountName}"
        
                          }
        
                      }
        
                  }
        
              }
        
        
    5. Set Parameters in the Pipeline:
      • In the pipeline that uses the dataset, set the parameters for the storage account name, container name, and file path.
      • When you trigger the pipeline, you can provide different values for these parameters to fetch files from different storage accounts and locations.
    6. Example Pipeline Parameters:
      • Define pipeline parameters for StorageAccountName, ContainerName, and FilePath.
      • Pass these parameters to the dataset in the activities within your pipeline.

    More links :

    https://learn.microsoft.com/en-us/azure/data-factory/parameterize-linked-services?tabs=data-factory

    https://www.c-sharpcorner.com/article/parameterization-in-azure-data-factory-linked-services/

    https://dataengineerazure.medium.com/how-to-parameterize-dataset-in-azure-data-factory-parameterization-in-azure-data-factory-4cae57d886aa

    https://davidalzamendi.com/azure-data-factory-linked-services/

    https://www.linkedin.com/pulse/parameterizing-linked-service-azure-data-factory-nithyanandam/

    1 person found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.