azure synapse "driverLogUrl" not working

Dheeraj 351 Reputation points
2021-09-24T17:21:00.503+00:00

I am trying to access spark application driver logs from synapse using notebook's output. But it shows "Not Found" dialog for each and every "driverLogUrl". URL sections look like this:

"appInfo": {  
                    "driverLogUrl": "http://ce09f028a8174d70b3b1e9014f3297aa00016435615:8042/node/containerlogs/container_1632233005992_0497_01_000001/trusted-service-user",  
                    "sparkUiUrl": "http://ce09f028a8174d70b3b1e9014f3297aa004f2c06771:8088/proxy/application_1632233005992_0497/",  
                    "isSessionTimedOut": "false"  
                }  

135155-s1.png

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,671 questions
0 comments No comments
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 85,351 Reputation points Microsoft Employee
    2021-09-27T09:46:49.247+00:00

    Hello @Dheeraj ,

    Thanks for the question and using MS Q&A platform.

    This is excepted behaviour. We are unable to use the links provided in the output section like ( tracking URL, driverLogUrl, sparkUiUrl) because - as soon as the pipeline run completed the backend Apache Spark Livy is stopped.

    135439-image.png

    Apache Spark Application:

    135532-image.png

    How to check the logs?

    You can view full log of Livy, Prelaunch, and Driver logs via selecting the associated Apache Spark Application name and then check the logs.

    135500-image.png

    For more details, refer to Use Synapse Studio to monitor your Apache Spark applications.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful