Azure Synapse - Send Data Factory pipeline logs to data lake/azure storage

Dheeraj 351 Reputation points

I want to send synapse Data Factory pipeline logs to data lake/azure storage. so that I don't have to login to synapse and go to individual pipeline jobs to check the logs.
the specific logs I am talking about are below:

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,525 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. KranthiPakala-MSFT 43,397 Reputation points Microsoft Employee

    Hi @Dheeraj ,

    Thanks for using Microsoft Q&A forum and posting your query.

    II don't see any option directly from UI to route the logs to Storage/ADLS Gen2. We are reaching out to internal team to check if there is a way and will get back to you as soon as we have an update from the team.

    Also could you please confirm if you are asking regarding pipeline run logs or Apache spark application (Jobs) logs? In case if your ask is to collect data regarding Synapse pipeline runs/Activtiy runs/Trigger Runs then you can use the diagnostic settings option from Azure portal. You can diagnostic logs for the below metrics to either storage account, Event hubs, Log analytics workspace. Please see below image for more info:


    1 person found this answer helpful.

  2. KranthiPakala-MSFT 43,397 Reputation points Microsoft Employee

    Hi @Dheeraj ,

    Apologies for the delay in response. As per my analysis you can find/monitor the information about completed/ended Apache Spark applications (log emitted by Apache Spark pools) by using the Log Analytics table: SynapseBigDataPoolApplicationsEnded


    Ref doc: Azure Synapse Analytics - Apache Spark pool log

    To know more about the table please refer to this doc - Azure Monitor - SynapseBigDataPoolApplicationsEnded.

    In addition to above info, also noticed that only Microsoft Support and the Azure Synapse Analytics engineering team can view and download diagnostic logs that are associated with your Apache Spark pools. Ref doc: Apache Spark diagnostic logs in Azure Synapse Analytics

    Hope this info helps.