ADF logging mechanism

Question

I have a few options to get logging from Azure data factory for performing key metric analysis such as

1. Having SQL table store procedures and capture all key metrics from Azure monitor such pipeline runtimes and durations
2. Programmatically grap these through Azure functions or Powershell and CLI.
3. Enable diagnostic settings and capture all logging through log analytics

Out of these, which is a cost-effective, scalable, and enhanced strategy. Please suggest use-cases and pros and cons. Thank you

Answer 1

The ideal strategy depends on the below set of factors:

1. What are the metrics that you are trying to capture? Are those like Pipelinerunid, starttime,endtime,datarows,datasize etc? Or are those at each and every activity level metrics?

Because in case if those are the initial ones, you can have a stored procedure activity within the main pipeline itself and pass the pipeline parameters/expression to capture those metrics directly.

2. How long do you want to retain the data? Because diagnostic setting/log analytics based on what I remember has a fixed retention period post which it cleans up historical records, so in that case having a custom SQL table of your own helps you govern the retention time period.