How to pass pipeline variable into dataflow source dataset

Question

How to pass pipeline variable into dataflow source dataset

ka 20

I have a pipeline variable that contains the string "customers.txt" I want to pass it into my dataflow's source dataset's file path dataflow's source dataset

However, it is not showing up. How do I go about doing this?

ka 20 Reputation points

2023-09-25T07:45:17.5+00:00

This are my dataflow's settings, and I am passing my source into a select and then derived column activities in my data flow.

1 answer

Your answer

ka 20 Reputation points

2023-09-25T07:45:17.5+00:00

This are my dataflow's settings, and I am passing my source into a select and then derived column activities in my data flow.

Answer 1

Nandan Hegde 36,146 MVP Volunteer Moderator

Hey,

PFB the flow:

create a dataset and its parameter :

User's image

use the parameter :

User's image

at pipeline level :

User's image

when we call the dataflow via ADF pipeline, we can pass the pipeline variable to dataflow and its dataset

your approach seems to be correct, so can you share what error/issue you are facing ?

ka 20 Reputation points

2023-09-26T01:16:18.5533333+00:00

Just to add, my data flow works if i manually type in the file path (no reference to a pipeline variable)
I need this reference as I have activity in my pipeline that takes the latest file each day and i am storing the file name as a variable in my pipeline and referencing it in my dataset
ka 20 Reputation points

2023-09-26T01:28:38.71+00:00

Hi Nandan, my dataflow contains a SELECT and DERIVED COLUMN activity. The select activity has these mappings under input columns:

I used these mappings as my select activity will be used the select the columns that i want to map to my sink.

However, when i reference the pipeline variable in my filepath for my source, my source in my dataset is returning 0 columns:

and hence i am getting the error "Input column in mapping cannot be found for my select activity.
Any idea how to fix this?
Nandan Hegde 36,146 Reputation points MVP Volunteer Moderator

2023-09-27T05:42:35.06+00:00

similar thread :

https://stackoverflow.com/questions/77177880/selecting-columns-base-on-names-in-azure-data-factorys-dataflow-activity
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-09-28T22:16:56.12+00:00

Hello ka,

I am checking to see if you got a chance to look into the above response from Nandan.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Share via

How to pass pipeline variable into dataflow source dataset

1 answer

Your answer