How to get the log from Synapse

Jia Zhang 40

Hi Team,

May I know how can I read the log of pipeline from the Synapse? When I manuelly go into UI and check pipeline spark session log, it will shows the log is too large to donwload, copy curl to donwload. After I copy it, and past !curl xcxxxxxx in a cell in synapse notebook, then it will shows download it, and then I enterd cat stderr, it will shows the result as output, but since it is too large, I can't see all then either that cell running will failed or my browser freez, may I know if anyway I can donwload it to my local to review? Thanks!

Vinodh247 18,271 Reputation points

2024-03-07T05:46:58.84+00:00

Just checking if you had a chance to look at this. if so, can you please update it?

1 answer

Vinodh247 18,271 Reputation points

2024-03-06T06:07:32.07+00:00

Hi Jia Zhang,

Thanks for reaching out to Microsoft Q&A.

Did you try this?

The Synapse Apache Spark diagnostic emitter extension is a library that enables the Apache Spark application to emit the logs, event logs, and metrics to one or more destinations, including Azure Log Analytics, Azure Storage, and Azure Event Hubs.

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/azure-synapse-diagnostic-emitters-azure-storage

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.
Please sign in to rate this answer.
Jia Zhang 40 Reputation points

2024-03-08T18:44:51.7366667+00:00

Thanks so much for the help! Just to check from the document, this will save this kind of log:

{ "timestamp": "2021-01-02T12:34:56.789Z", "category": "Log|EventLog|Metrics", "workspaceName": "<my-workspace-name>", "sparkPool": "<spark-pool-name>", "livyId": "<livy-session-id>", "applicationId": "<application-id>", "applicationName": "<application-name>", "executorId": "<driver-or-executor-id>", "properties": { // The message properties of logs, events and metrics. "timestamp": "2021-01-02T12:34:56.789Z", "message": "Registering signal handler for TERM", "logger_name": "org.apache.spark.util.SignalUtils", "level": "INFO", "thread_name": "main" // ... } }

What i want is this kind of log

I try to press copy link it will give me a curl link, and what I do is open a cell and paste the link:

After that use !cat stderr to see it. But it will shows as output under the cell, can't shows all since it is too large that it can't shows that large stuff in output. I just wondering if there is other comment other than !cat maybe can just donwload it?

Like for some log session that is small I can jsut download it by click the button in the log part, but for large log, i can only copy curl but don't know what to do next. Thanks!

Harishga 5,990 Reputation points Microsoft Vendor

2024-03-11T11:41:40.91+00:00

Jia Zhang
If the log file is still too large to view in a text editor, they can try using a tool like less or tail to view the log file in smaller chunks. Here's an example command using tail:

tail -n 1000 log.txt

This command will display the last 1000 lines of the log file.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to get the log from Synapse

1 answer

Your answer