Azure Synapse - Send Data Factory pipeline logs to data lake/azure storage

Question

I want to send synapse Data Factory pipeline logs to data lake/azure storage. so that I don't have to login to synapse and go to individual pipeline jobs to check the logs.
the specific logs I am talking about are below:

Answer 1

Hi @Dheeraj ,

Thanks for using Microsoft Q&A forum and posting your query.

II don't see any option directly from UI to route the logs to Storage/ADLS Gen2. We are reaching out to internal team to check if there is a way and will get back to you as soon as we have an update from the team.

Also could you please confirm if you are asking regarding pipeline run logs or Apache spark application (Jobs) logs? In case if your ask is to collect data regarding Synapse pipeline runs/Activtiy runs/Trigger Runs then you can use the diagnostic settings option from Azure portal. You can diagnostic logs for the below metrics to either storage account, Event hubs, Log analytics workspace. Please see below image for more info:

Answer 2

Hi @Dheeraj ,

Apologies for the delay in response. As per my analysis you can find/monitor the information about completed/ended Apache Spark applications (log emitted by Apache Spark pools) by using the Log Analytics table: SynapseBigDataPoolApplicationsEnded

Ref doc: Azure Synapse Analytics - Apache Spark pool log

To know more about the table please refer to this doc - Azure Monitor - SynapseBigDataPoolApplicationsEnded.

In addition to above info, also noticed that only Microsoft Support and the Azure Synapse Analytics engineering team can view and download diagnostic logs that are associated with your Apache Spark pools. Ref doc: Apache Spark diagnostic logs in Azure Synapse Analytics

Hope this info helps.