Synapse spark session LIVY_JOB_STATE_DEAD

ZUIDERVELD Sierd 1 Reputation point
2022-08-30T15:07:39.04+00:00

When I want to use .toPandas () on a spark dataframe I receive an error that references

org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 995464. To avoid this, increase spark.kryoserializer.buffer.max value.

So, I use the Apache Spark configurations interface to change a configuration where the buffer.max is set to 512 as I've seen others do online. However, when I do this and I start my notebook (or any notebook with such a config) I receive the error

LIVY_JOB_STATE_DEAD: Livy session has failed. Session state: Dead. Error code: LIVY_JOB_STATE_DEAD. [plugins.ws-syn-pdm.pdmspark2.10 WorkspaceType:

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,316 questions
{count} votes