Synapse error - Failed to create Livy session for executing notebook

Question

Hi,

Earlier today all our spark pools started to fail on different in environments because of a Livy session error. I have try to scale the spark pool up and create a new spark pool. However, it will still not start a spark session. See the error message under:

{
"errorCode": "6002",
"message": "Exception: Failed to create Livy session for executing notebook. LivySessionId: 66, Notebook: Raw2Clean.\n--> : Livy session has failed. Session state: Error. Session yields \"Failed\" result.",
"failureType": "UserError",
"target": "raw_to_clean",
"details": []
}

Does somebody have a solution to this problem or is it a general Azure problem in West Europe a the moment?

Answer 1

Hi,

Thanks for reaching out to Microsoft Q&A.

Glad it is a temporary issue and resolved by now. But at times when you try to run a new notebook there are chances that one of your previous notebooks is still attached to the session which might not allow you to run a new notebook session. As a first step you can go through the monitor tab to see if any running sessions and kill it forcefully if it's still running. Scaling up might not work, just detaching the session from the other notebook will work.

Please Upvote and Accept as answer if the reply was helpful, this will be benefitting the other community members who go through the same issue.

Answer 2

Thanks, it was a temporary issue for a couple of hours, but it has been resolve now.

I agree with that an overview of these Microsoft issues could be great for time saving in the future

Answer 3

Hello @Stephane Massat ,
Thank you for your feedback.

Hello @Mathias A ,
As @Stephane Massat mentioned, this could be a temporary issue. Could you please confirm if this is resolved? Or if this is still happening, please let me know.

Also, I have seen this error when there are not enough resources on the cluster to launch the sessions/applications. This error could occur if the total available cores cannot accommodate the existing session/application.

Answer 4

I just had the same error today!

Answer 5

We got the same issue.
The reason was spark version 2.4 that we were using was deprecated in synapse.

To solve the issue, have to create a new one with latest version