Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,884 questions
Synapse spark session LIVY_JOB_STATE_DEAD
ZUIDERVELD Sierd
1
Reputation point
When I want to use .toPandas () on a spark dataframe I receive an error that references
org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 995464. To avoid this, increase spark.kryoserializer.buffer.max value.
So, I use the Apache Spark configurations interface to change a configuration where the buffer.max is set to 512 as I've seen others do online. However, when I do this and I start my notebook (or any notebook with such a config) I receive the error
LIVY_JOB_STATE_DEAD: Livy session has failed. Session state: Dead. Error code: LIVY_JOB_STATE_DEAD. [plugins.ws-syn-pdm.pdmspark2.10 WorkspaceType:
{count} votes
-
HimanshuSinha-msft 18,536 Reputation points Microsoft Employee
2022-09-01T21:12:40.983+00:00 Hello @ZUIDERVELD Sierd ,
Thanks for the ask and using the Microsoft Q&A platform.
At this time, we are reaching out to the internal team to get some help on this . We will update you once we hear back from them.
Thanks
Himanshu
Sign in to comment
Try read the executor logs for this error and if its related to Vegas cache memory error, try below options in synapse workspace
1 > Try set the spark configurations for your pools. Set spark.synapse.vegas.useCache => false
2> Modify the pool scale settings and set the intelligent cache slider to 0%.