Azure Data Factory simple copy file and rename

Question

Hello:

I have a simple requirement. I have a couple of files in an Azure Blob folder (receiving folder) and I need to copy them, to another fold (Azure Blob folder) and rename them. This posting is really close to what I need to do: https://learn.microsoft.com/en-us/answers/questions/569223/renaming-a-list-of-files However, of course, there are a couple of differences. This posting says: "Inside sink dataset have a parameter for file name and Pass your sink file name dynamically" . Question: the MS documentation says : "Parameters are defined at the pipeline level, and cannot be modified during a pipeline run." What gives here are parameters static? Can't be changed. Can someone kindly clarify.

For my simple case:

I have this

GetFileList & FilterforFiles work

In the Foreach the CopyFile copies the files but does not change the name.

The sink is defined as follows:

So for my test I want to remove an underbar (_) from the file names.

The final script will need to add a time stamp in the file name and

delete the file from the source.

I though the bit of code in file name above would fix the file name. But files are being copied with the original file name.

Your kind assistance is requested.

Best Regards

KD

Answer 1

Hi @KBD
I’m glad to hear that your files were successfully copied and renamed in Azure Data Factory. Let’s clarify how parameters work in ADF to address your concerns:

Pipeline Parameters vs. Dataset Parameters:

Pipeline Parameters are indeed static during a pipeline run. They are set before the pipeline starts and cannot be changed until the run is completed.

Dataset Parameters, on the other hand, can be dynamic within a pipeline run. They can take different values for each activity or iteration within that run.

For Each Activity and Dynamic Content:

The ForEach activity iterates over a collection of items. For each iteration, it can pass a different value to the dataset parameter.

Dynamic Content is used to construct the new file name for each iteration. The expression you’ve used is evaluated for each item in the loop, and the result is passed as the parameter value to the sink dataset.

Sink Dataset Filename Property:

This property specifies what the file should be named in the sink dataset. If you have set up dynamic content to generate the file name, this property will use the result of that dynamic content for each file processed.

So, when you added a parameter to the sink dataset, you were not setting a static file name. Instead, you provided an expression that ADF evaluates for each file during the pipeline run. This expression uses the current item’s name from the ForEach loop and applies the transformation (removing underscores, adding timestamps, etc.) you’ve defined.

The filename property in the sink dataset is not being ignored. It’s being used in conjunction with the dynamic content to determine the file name for each file during each iteration of the ForEach loop. The parameter you’ve added to the sink dataset is “static” in the sense that it’s a fixed part of the dataset’s configuration, but the value it takes is dynamically generated during the pipeline run.

In summary, while the parameter itself is a static part of the dataset configuration, the value it takes can be dynamically generated for each file during the pipeline run, allowing you to achieve the desired renaming functionality.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.