Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,405 questions
Synapse spark session LIVY_JOB_STATE_DEAD
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 39%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZS%3C/text%3E%3C/svg%3E)
ZUIDERVELD Sierd
1
Reputation point
When I want to use .toPandas () on a spark dataframe I receive an error that references
org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 995464. To avoid this, increase spark.kryoserializer.buffer.max value.
So, I use the Apache Spark configurations interface to change a configuration where the buffer.max is set to 512 as I've seen others do online. However, when I do this and I start my notebook (or any notebook with such a config) I receive the error
LIVY_JOB_STATE_DEAD: Livy session has failed. Session state: Dead. Error code: LIVY_JOB_STATE_DEAD. [plugins.ws-syn-pdm.pdmspark2.10 WorkspaceType:
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee2022-09-01T21:12:40.983+00:00 Hello @ZUIDERVELD Sierd ,
Thanks for the ask and using the Microsoft Q&A platform.
At this time, we are reaching out to the internal team to get some help on this . We will update you once we hear back from them.
Thanks
Himanshu -
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 23%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Ramasamy M (Americas 1 - iDEAS-AEM) 0 Reputation points2023-03-09T14:42:22.81+00:00 Try read the executor logs for this error and if its related to Vegas cache memory error, try below options in synapse workspace
1 > Try set the spark configurations for your pools. Set spark.synapse.vegas.useCache => false
2> Modify the pool scale settings and set the intelligent cache slider to 0%.
Sign in to comment