How to use the Filter Activity Output as Source in Copy Activity.

Question

Issue 1: Passing Variables Between Child and Master Pipeline

 

In my setup, I have a master pipeline that triggers a child pipeline containing an Until loop activity. This loop runs until the NextpageURI is null, and while it does so, it continuously updates a variable called content.

 

The challenge I am facing is passing the final updated value of the content variable from the child pipeline back to the master pipeline after the child pipeline completes. Once I retrieve this variable, I need it to be used in a ForEach activity in the master pipeline.

 

I have tried a few approaches, but none seem to successfully pass the variable back to the master pipeline.

 

Questions and Suggestions Needed:

                •              Is there a specific way to reference or output variables from a child pipeline back to a master pipeline?

                •              Are there specific settings in the Until loop or pipeline outputs that I should modify to make this variable available after the child pipeline execution?

 

Screenshots:

1.                  Screenshot of the Until loop in the child pipeline with the expression updating the content variable.

 

2.                  Settings for the pipeline trigger in the master pipeline that calls the child pipeline.

 

3.                  The pipeline settings showing the ForEach activity in the master pipeline where the variable is expected to be used.

4.                  Expression for NextpageURI Variable

5.                  Expression for the Content Variable

6.                  Expression for the If condition

 

Issue 2: Using Filter Activity Output as Source in Copy Activity

 

In the same pipeline, I have a ForEach activity running a loop that filters JSON data based on the condition Workday: PowerApps. The filtering is working as expected, and I am able to filter the required JSON elements.

 

However, my next goal is to write the output of this filter activity into a staging Blob storage using a Copy Activity. I am struggling with how to use the filtered JSON output from the filter activity as the source for the Copy Activity.

 

Questions and Suggestions Needed:

                •              How can I take the output from the filter activity and use it as the source in the copy activity?

                •              Are there any specific expressions or variable handling mechanisms that I should be using to reference the output of the filter in the copy activity?

 

Screenshots:

1.                  ForEach Activity settings

2.                  Child Pipeline

3.                  Screenshot of the filter activity configuration, including the JSON filtering condition (Workday: PowerApps).

4.                  Configuration settings of the Copy Activity where I am attempting to use the filter’s output as the source.

 

If you need any additional information or context, please let me know.

Answer 1

Hi @Alexander Lopez ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

As per my understanding you want to pass value from child pipeline to Master pipeline. You can leverage Pipeline return value option in set variable activity to achieve this requirement.

To return a value from a child pipeline to a parent pipeline in Azure Data Factory, you can use a combination of the Set Variable and Execute Pipeline activities. The Set Variable activity is used to assign a value to a variable, and the Execute Pipeline activity is used to call the child pipeline.

To get the full details of the implementation, kindly watch out this video: Pipeline return value in Set variable in Azure Data Factory & Azure Synapse Analytics

Coming to another query, " How can I take the output from the filter activity and use it as the source in the copy activity?"

You can make use of additional column in copy activity where you can provide expression of filter activity output in additional col option, use a dummy source and in mapping tab, just keep the additional col as the schema and use the desired sink .

Hope it help. Kindly accept the answer by clicking on Accept answer button. Thankyou