Synapse spark logs

Question

Hello,

Is there a possibility to get access to Spark logs in Synapse other than through Spark UI? I've tried specifying "properties.defaultSparkLogFolder" when creating spark pools but it does not seem to have any effect - what is it used for?
Basically I'm looking for some mechanism that would allow to sync Spark logs to some accessible location, is it possible? (I understand there is a possibility to do that using HDInsight and Databricks)

Thanks in advance,
Iryna

Answer 1

Hello @Irene ,

Unfortunately, there is no such mechanism to copy spark logs to user accessible location.

Note: The property defaultSparkLogFolder doesn’t work in Azure Synapse Analytics.

You can download the completed application log via the following curl command:

For driver log:

curl "https://{workspace}.dev.azuresynapse.net/sparkhistory/api/v1/sparkpools/{sparkPool}/livyid/{livyId}/applications/{appId}/driverlog/stderr/?isDownload=true" -H "authorization:Bearer {AccessToken}"  

For executors log:

Fetch the metadata:

curl "https://{workspace}.dev.azuresynapse.net/sparkhistory/api/v1/sparkpools/{sparkPool}/livyid/{livyId}/applications/{appId}/yarnaggregationlog/metadata" -H "authorization:Bearer {AccessToken}"  
              

Get the container id info from the metadata:

 curl "https://{workspace}.dev.azuresynapse.net/sparkhistory/api/v1/sparkpools/{sparkPool}/livyid/{livyId}/applications/{appId}/yarnaggregationlog/containers/{containerId}/files/stderr" -H "authorization:Bearer {AccessToken}"  

Hope this helps. Do let us know if you any further queries.

------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.