Azure DataFactory Dataflow Activity read

Nirmal Chowdhary 5

Hello, I have a question regarding the use of the Data Flow activity in Azure Data Factory (ADF). I'm looking to pass data to the Data Flow activity through a foreach iteration. My scenario involves performing a join between two datasets, filtering some data, and then passing the result to another activity. However, this operation needs to run multiple times within a foreach loop.

The challenge I'm facing is that the two source datasets used in this operation are static, and I don't want to load them each time the foreach loop runs. Is there a way to pass this data to the foreach loop and reuse it multiple times within the Data Flow activity?

1 answer

ShaikMaheer-MSFT 38,501 Reputation points Microsoft Employee

2024-05-24T05:49:11.69+00:00

Hi Nirmal Chowdhary,

Thank you for posting query in Microsoft Q&A Platform.

Consider using parameterized datasets in your Dataflow and inside the loop for each iteration pass the values into parameters accordingly.

Please check below video to understand about parameterized datasets in ADF.

Parameterize dataset in ADF

Hope this helps. Please let me know if any further queries.

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.
Please sign in to rate this answer.
Nirmal Chowdhary 5 Reputation points

2024-05-24T16:04:38.98+00:00

Hello @ShaikMaheer-MSFT

Thank you for your response earlier. My dataset is already parameterized. I'm wondering if it's possible to load that data just once during each iteration of a foreach loop and then pass that same data to a Data Flow activity to execute specific steps. Currently, the Data Flow activity reloads the dataset every time it loops. I'd like to configure the Data Flow activity within the foreach loop to load the dataset only once and reuse it multiple times without refreshing the dataset.

I hope this clarifies my question. Let me know if you need any further information.

Thanks

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

ShaikMaheer-MSFT 38,501 Reputation points Microsoft Employee

2024-05-27T05:40:05.07+00:00

Hi Nirmal Chowdhary,

Not able to understand, when we say reloading dataset.

Datasets generally pointers to data. When they get use in copy activity or dataflows that time they actually pull the information and process them.

There is no such feature, where dataset is loading the data and keeping it for next step usages as per design. Datasets are only pointers to data, they cannot hold any data within.

Hope this helps. Please let me know if any further queries.

Please consider hitting Accept Answer button. Accepted answers help community as well.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure DataFactory Dataflow Activity read

1 answer

Your answer