Failed to create Livy session for executing notebook

Question

Failed to create Livy session for executing notebook

Ryan Abbey 1,136

We on occasion get the below error

Exception: Failed to create Livy session for executing notebook. LivySessionId: 17602, Notebook: nbPassthrough.\n--> Spark_User_AutoClassification_attempt_Diagnostics: Livy session has failed. Session state: Dead. Error code: Spark_User_AutoClassification_attempt_Diagnostics

A re-run goes through successfully so not a code or data issue. None of our files being processed are large so should not be a data size issue, and failing shortly after starting is just as frequent as anything longer running (although generally none of them run longer than 10-15 minutes).

There can be a number of notebooks running at once - are there shared resources among the notebooks? Or should each have it's own guaranteed set of nodes with their own resources (certainly how I've read the configuration)?

Any ideas how we can go about figuring out what is causing the failures?

PRADEEPCHEEKATLA-MSFT 59,141 Reputation points Microsoft Employee

2023-05-04T05:00:38.14+00:00
@Ryan Abbey - Thanks for the question and using MS Q&A platform.

The error message you are seeing indicates that the Livy session for executing the notebook has failed. This can happen for a variety of reasons, including resource contention, network issues, or Livy server issues.

Here are some steps you can take to troubleshoot the issue:

Check the Livy server logs: The Livy server logs can provide more information about the error that occurred. You can find the Livy server logs in the Azure Synapse Analytics workspace under the "Logs" tab. Look for any error messages or stack traces that might indicate the cause of the failure.

Check the resource usage: If there are a number of notebooks running at once, it's possible that there is resource contention that is causing the Livy session to fail. You can check the resource usage of your Spark pools to see if there are any spikes in usage that might be causing the issue. You can also try increasing the number of nodes in your Spark pool to see if that helps.

Check the network connectivity: Network issues can also cause Livy sessions to fail. Make sure that the network connectivity between your client and the Livy server is stable and that there are no firewall rules or other network restrictions that might be blocking the connection.

Contact Azure support: If you are unable to resolve the issue using the above steps, you can open a support ticket for further assistance. They can help you troubleshoot the issue and provide guidance on how to resolve it.

I hope this helps! Let me know if you have any further questions.
PRADEEPCHEEKATLA-MSFT 59,141 Reputation points Microsoft Employee

2023-05-10T08:21:21.6866667+00:00

@Ryan Abbey - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

PRADEEPCHEEKATLA-MSFT 59,141 Reputation points Microsoft Employee

2023-05-04T05:00:38.14+00:00

@Ryan Abbey - Thanks for the question and using MS Q&A platform.

The error message you are seeing indicates that the Livy session for executing the notebook has failed. This can happen for a variety of reasons, including resource contention, network issues, or Livy server issues.

Here are some steps you can take to troubleshoot the issue:

Check the Livy server logs: The Livy server logs can provide more information about the error that occurred. You can find the Livy server logs in the Azure Synapse Analytics workspace under the "Logs" tab. Look for any error messages or stack traces that might indicate the cause of the failure.

Check the resource usage: If there are a number of notebooks running at once, it's possible that there is resource contention that is causing the Livy session to fail. You can check the resource usage of your Spark pools to see if there are any spikes in usage that might be causing the issue. You can also try increasing the number of nodes in your Spark pool to see if that helps.

Check the network connectivity: Network issues can also cause Livy sessions to fail. Make sure that the network connectivity between your client and the Livy server is stable and that there are no firewall rules or other network restrictions that might be blocking the connection.

Contact Azure support: If you are unable to resolve the issue using the above steps, you can open a support ticket for further assistance. They can help you troubleshoot the issue and provide guidance on how to resolve it.

I hope this helps! Let me know if you have any further questions.
PRADEEPCHEEKATLA-MSFT 59,141 Reputation points Microsoft Employee

2023-05-10T08:21:21.6866667+00:00

@Ryan Abbey - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Failed to create Livy session for executing notebook

Activity