Failed to create Livy session for executing notebook

Ryan Abbey 1,181 Reputation points
2023-05-03T22:11:40.1866667+00:00

We on occasion get the below error

Exception: Failed to create Livy session for executing notebook. LivySessionId: 17602, Notebook: nbPassthrough.\n--> Spark_User_AutoClassification_attempt_Diagnostics: Livy session has failed. Session state: Dead. Error code: Spark_User_AutoClassification_attempt_Diagnostics

A re-run goes through successfully so not a code or data issue. None of our files being processed are large so should not be a data size issue, and failing shortly after starting is just as frequent as anything longer running (although generally none of them run longer than 10-15 minutes).

There can be a number of notebooks running at once - are there shared resources among the notebooks? Or should each have it's own guaranteed set of nodes with their own resources (certainly how I've read the configuration)?

Any ideas how we can go about figuring out what is causing the failures?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,565 questions
{count} votes