How to store custom & default logs from Azure Data Factory to App Insights

Question

Hi Team,

I have 6 azure data factory pipelines and I want to store the default and custom logs from these pipelines into Azure App Insights. Kindly let me know about how it is done. Any kind of suggestion will be helpful.

Answer 1

You should not use App Insights for this.

You need to configure your ADF instance to send its logs to log analytics workspace. Then your ADF logs are available on AzureDiagnostics table or in ADF-specific tables.

ref: https://learn.microsoft.com/en-us/azure/data-factory/monitor-configure-diagnostics

Edit:

1. Make sure that you use secure input and secure output settings on the actions that pass/use sensitive data
2. secure output can be also used to mask output of stored proc/web/script/lookup calls. On some cases those activities generate so much log that Output field on log analytics contains only errors message and log analytics with other content will be impossible.