Yes, it is possible to parameterize the dataset at the storage account level in ADF by using parameters in your linked service and datasets.
- Create Parameters in the Linked Service:
- Go to the Linked Services section in ADF.
- Edit the linked service that connects to your storage account.
- Add a parameter for the storage account name. For example, you can add a parameter named
StorageAccountName
.
- Modify the Linked Service Connection String:
- In the linked service connection string, replace the storage account name with the parameter you just created. For example:
{ "type": "LinkedService", "typeProperties": { "connectionString": { "value": "DefaultEndpointsProtocol=https;AccountName=@{linkedService().StorageAccountName};AccountKey=<your_account_key>;EndpointSuffix=core.windows.net" } } }
- In the linked service connection string, replace the storage account name with the parameter you just created. For example:
- Create Parameters in the Dataset:
- Go to the Datasets section in ADF.
- Edit the dataset that you want to parameterize.
- Add parameters for the storage account name, container name, and possibly the file path.
- Modify the Dataset to Use Parameters:
- In the dataset, use the parameters you created to dynamically construct the path. For example:
{ "type": "Dataset", "typeProperties": { "location": { "type": "AzureBlobStorageLocation", "container": { "value": "@{dataset().ContainerName}" }, "filePath": { "value": "@{dataset().FilePath}" }, "storageAccount": { "value": "@{dataset().StorageAccountName}" } } } }
- In the dataset, use the parameters you created to dynamically construct the path. For example:
- Set Parameters in the Pipeline:
- In the pipeline that uses the dataset, set the parameters for the storage account name, container name, and file path.
- When you trigger the pipeline, you can provide different values for these parameters to fetch files from different storage accounts and locations.
- Example Pipeline Parameters:
- Define pipeline parameters for
StorageAccountName
,ContainerName
, andFilePath
. - Pass these parameters to the dataset in the activities within your pipeline.
- Define pipeline parameters for
More links :
https://learn.microsoft.com/en-us/azure/data-factory/parameterize-linked-services?tabs=data-factory
https://www.c-sharpcorner.com/article/parameterization-in-azure-data-factory-linked-services/
https://davidalzamendi.com/azure-data-factory-linked-services/
https://www.linkedin.com/pulse/parameterizing-linked-service-azure-data-factory-nithyanandam/