Capture the desired log of a pipeline in azure data factory

Question

Hi Team, I need to capture the desired log of pipeline. Like pipeline name, data written, data read, start time, edn time. My pipeline having for loop inside will have the copy activity so every loop i need to capture the log of the copy activity and need to store in a single file. Without using the SQL DB. Thanks in advance.

Answer 1

Hey,

You can follow the below approach :

1. within every iteration , copy the logs into an individual file
2. Then after completion of all iterations, merge all the individual log files into a single file and delete the individual files

Merge :

https://www.sqlservercentral.com/articles/merge-multiple-files-in-azure-data-factory

Logs :

https://www.mssqltips.com/sqlservertip/6320/logging-azure-data-factory-pipeline-audit-data/

Answer 2

@Bhajanthri Bala Sujay

In addition to @Nandan Hegde inputs, I would like to add couple more workarounds when you have multiple ForEach loops with copy activity inside them.

Another approach is to have an append variable and for each iteration append copy activity output data to append variable and then log that data all together to a file using final copy activity at the end of two For Each activities.

In order to write the append variable data to a file you can follow the similar process as explained in any of the below approaches. You can create multiple append variable for each Copy activity output fields and load that value for each iteration and at the end using additional column feature in copy activity you can copy that data to a file:

1. Transfer the output of 'Set Variable' activity into a json file [Azure Data Factory]
2. Azure data factory store variable to file in blob

Hope this info helps. Please feel free to let me know if you have any questions.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.