question

PramukhTechnology-5854 avatar image
0 Votes"
PramukhTechnology-5854 asked ShaikMaheer-MSFT commented

Datacopy Destimnation dynamic file as an inputfile for data flow task

Hi,

I have created a data copy task in Azure Synapse . that basically copies 1000 records from Onprem SQL to Azure Storage - in .csv format. ( dynamic name - Filename_UTCNOW_MMHHSS.csv)

In the next phase of pipeline, I want to use the created file in datacopy as a source for the DataFlow.

How can i use the dynamically created file in the dataflow for further processing.

Thanks,
Parikshit.

azure-synapse-analytics
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @PramukhTechnology-5854,

Just checking in to see if the below answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered

Hi @PramukhTechnology-5854,

Thank you for posting query in Microsoft Q&A Platform.

We can achieve this by using a variable to store dynamic file name and then pass that variable as value to dataflow. In data flow we should have a parameter to take variable as value and then in source transformation we should use wild card path to write expression to point file.

Please check below high-level implementation steps to get better understanding.

Step1: Create a variable in pipeline to hold dynamic file name and use set variable activity to set value to it.
198668-image.png

Step2: Copy activity - to copy data from on-prem SQL to csv file. Here under sink we will be using variable from step1 as value for file to generate. Sink dataset will be parameterized on to dynamically take file name.
198752-image.png

Step3: Have parameter created in Dataflow and use expression inside source transformation to dynamically point to same created file.
198599-image.png

Hope this helps. Please let us know if any further queries.


Please consider hitting Accept Answer button. Accepted answers help community as well.


image.png (210.2 KiB)
image.png (203.1 KiB)
image.png (225.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.