Synapse spark session LIVY_JOB_STATE_DEAD
When I want to use .toPandas ()
on a spark dataframe I receive an error that references
org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 995464. To avoid this, increase spark.kryoserializer.buffer.max value.
So, I use the Apache Spark configurations interface to change a configuration where the buffer.max is set to 512 as I've seen others do online. However, when I do this and I start my notebook (or any notebook with such a config) I receive the error
LIVY_JOB_STATE_DEAD: Livy session has failed. Session state: Dead. Error code: LIVY_JOB_STATE_DEAD. [plugins.ws-syn-pdm.pdmspark2.10 WorkspaceType: