Azure Data Factory ADF - parameterize the From and To File Paths under Data Flow > Source Options > After Completion

Darpan Patel 6 Reputation points
2021-03-11T20:41:10.133+00:00

Problem: I need help with ADF to parameterize the From and To File Paths under Data Flow > Source Options > After Completion.

Environment: We are using Data Flow Activity to transform and load data from ADLS Gen 2 to a sink. I am moving a file after completion to keep files organized using Source Options in Data Flow. We use Azure DevOps and bitbucket for Source Control and CICD. We also share ADF for multiple projects so we need to have this taken care at CI-CD only.

We are using one ADLS service/Container to store both Dev and Test curated datasets in .parquet format. Hence, I need to parameterize File path for CICD. So in dev we have ContainerName & folderName/dev/subFolderName and in test we have ContainerName & folderName/tst/subFolderName along with Stage and Production get their respective paths. ContainerName are same in both Dev ADLS and Production ADLS.

So far: I added a parameters for both From and To in Data Flow > Source Options but when I export ARM, I don't see those parameters.

Thanks to all the geniuses out here.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,338 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,532 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. MartinJaffer-MSFT 26,021 Reputation points
    2021-03-15T17:04:42.303+00:00

    Hello @Darpan Patel and welcome to Microsoft Q&A.

    I haven't parameterized that part of Data Flow before. There are two routes to try.

    1. Have the Data Flow parameters be set by the pipeline (which calls the Data Flow). Then alter the pipeline parameters in CI/CD.
    2. Investigate the parameterization template options.
    3. Use Global Parameters.