export CSV of complete activity of pipelines runs (status: failed/succeeded/in-progress/queued/cancelled)

Question

Hello Team,

I wanted to get the complete triggered pipelines runs activity into the CSV (all items). I can download from the particular page only and it is very difficult to go to the next page and download the items. I have more than 50 pages in my ADF and it is consuming a lot of time. I couldn't find any option in the ADF GUI to get all at once. I could see as Export Current Page and Export top 1000, but it is not helping me to get the correct output.

Can anyone help me with this?

Thanks
NM Babu

Answer 1

Hello @NM Babu ,

Thanks for the question and using MS Q&A platform.

I don't think there is an easy way to export azure data factory pipeline run history from UI, if you have so much of data. But other options I could think of is try enabling the diagnostic log settings for your data factory and send logs to a Log Analytics workspace and from there you can use Kusto queries to get the pipeline run details based on the filters like, pipeline run ID, pipeline name, run start datetime, run end datetime, etc. and then you can export the results as csv files as shown in below image.

Below is sample how it looks in log analytics:

Note: Please note that you will see logs for pipeline runs only for the ones that are being triggered after enabling the diagnostic log settings. You won't be able to get the data for the ones before enabling logging.

Here is a helpful video tutorial by one of the community volunteers: How to use Log Analytics to Capture View Azure Data Factory Logs - Azure Data Factory Tutorial 2021

Hope this info helps. Do let us know if you have further query.

Thanks