HDinsight spark livy server stopping as soon as It starts

Gourav Sharma 1 Reputation point
2022-01-16T18:37:09.537+00:00

I am a new user of Azure HDInsight. I have created a 4 node spark cluster. When the cluster is successfully created, I saw spark history server and livy both are in a stopped state. And when I try to run them from Ambri it gets stopped by hdinsightwatchdog as soon as it gets started. Can anyone help me understand what is happening here.

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
197 questions
{count} votes

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2022-01-18T22:56:17.147+00:00

    Hello @Gourav Sharma ,
    Thanks for the ask and using Microsoft Q&A platform .

    As I understand at this time , the ask is why the Livy and history server is in stopped state . These two services does not run on all the nodes but one of the nodes in the clsuter but to maintain high avalaiblity it is installed on all the nodes . So when you want to start the services and the services is already installed & running on any other nodes , the watchdog kicks in and it stops the services on the node . ( I am sharing the screenshot below ) .

    As you mentioned you are running a 4 node cluster , this service must be running on one of them , request you please check that .

    166137-image.png

    Thanks
    Himanshu

    -------------------------------------------------------------------------------------------------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments