Pass lookup activity output to dataflow

Question

I have created adf pipeline with look up activity that is reading dynamic json content key values e. g. filename and filelocation

Now these two above values (output of look up) need to pass to dataflow activity source dataset "wildcard path"

Could you please advise how it is possible to do that?

Answer 1

@Anonymous

To pass the output of a Lookup activity to a Dataflow activity in an Azure Data Factory (ADF) pipeline, you can use a WildcardPath parameter in the source dataset for the Dataflow activity.

Here's an example of how you might do this:

1. In your ADF pipeline, add a Lookup activity that retrieves the data you want to pass to the Dataflow activity.
2. In the Dataflow activity, create a source dataset that specifies the WildcardPath parameter. The WildcardPath parameter allows you to specify a file or folder path using wildcard characters (e.g., "*.csv").
3. In the Dataflow activity's source dataset, set the WildcardPath parameter to the output of the Lookup activity. You can use the @activity('lookupActivityName').output syntax to reference the output of the Lookup activity in the WildcardPath parameter.
4. Run the ADF pipeline. The Dataflow activity will use the output of the Lookup activity as the source data, with the WildcardPath parameter specifying the specific file or folder path.

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Answer 2

Hello @Anonymous ,

Thanks for the question and using MS Q&A platform.

As suggested by @Arun Siripuram you can directly use the lookup output values using pipeline expression and pass the values to your dataflow. To do so you will have to follow below steps.

First you will have to create 2 dataflow parameters as shown below for which you will pass the values from pipeline expression at pipeline level.

Then once you have your lookup activity output, your focus would be @activity('lookupActivityName').output.value which contains an array of values for your fileNames and respective fileLocations as below:

and you will have to map them to Items in ForEach activity

Inside ForEach activity you can have mapping data flow activity and pass the fileName and fileLocation for each iteration by using expressions - @item().FileName and @item().FileLocation

Now you can configure your source dataset using this dataflow parameters as needed. In case if you want to directly map the the fileNames and folder path directly in dataset instead of dataflow parameters, you can also create dataset parameters as below:

Then you can directly map the forEach item() values (nothing but fileName and folderpath for each iteration) to your dataset parameters instead of dataflow parameters below :

Hope this info helps.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how