This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 67%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUO%3C/text%3E%3C/svg%3E)
Sorry if the question seams obvious. I am new to ADF. I have read many forum questions but am still struggling to find an efficient solution.
My storage account has two containers: daily and monthly.
Inside the containers I have the same group of directories: reportA, reportB, reportC,...
Inside each directory for the daily container I have multiple files with daily information =>
daily/reportA has many daily files called 20230101.csv, ..., 20230612.csv
daily/reportB has many daily files called 20230601.csv, ..., 20230612.csv
...
Inside each directory for the monthly container I have a file with monthly information =>
monthly/reportA has many monthly files called 202301.csv, ..., 202306.csv
monthly/reportB has many monthly files called 202301.csv, ..., 202306.csv
...
Every day I generate a new daily file for each report and would like to create monthly files, merging all daily files from a given month. On the daily activity only the current month has to be aggregated.
Sometimes (low frequency) I need to reprocess some of the reports, so new daily reports are extracted for the whole period and multiple monthly files need to be created.
Is ADF capable of performing this task on a simple way.
Thanks a lot for the support.
An Azure service for ingesting, preparing, and transforming data at scale.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
Hello @Umberto Ortolan,
Quick questions...
any screenshot would be helpful to understand the requirement!
Got it, i am looking into the scenario! Please stay tuned!
Hope the image helps.
My current solution is fixed for the current month. Date is detected and ForEach is used with fixed Directory Array.
It does not cover the scenario where a file on previous months is updated.
Hello @Umberto Ortolan,
Did you try the Copy Behavior as Merge?
Hello Satish, thanks for the message. Yes the merging part is ok. If I select a given report (eg. reportA) and a given month (eg. 202301) I can make the pipeline work.
My difficulty is on how to configurate directories to make it recursively for all reports and all prefix, considering only those with file changes on the daily files.
Regards