RuntimeConfig

User-facing configuration API, accessible through SparkSession.conf.

Supports Spark Connect

Options set here are automatically propagated to the Hadoop configuration during I/O.

Syntax

spark.conf.set(key, value)
spark.conf.get(key)

Properties

Property	Description
getAll	Returns all properties set in this configuration as a dict.

Methods

Method	Description
set(key, value)	Sets the given Spark runtime configuration property. value can be a string, integer, or boolean.
get(key, default)	Returns the value of the configuration property for the given key. Returns default if the key does not exist. Raises an exception if the key is not set and no default is provided.
unset(key)	Resets the configuration property for the given key.
isModifiable(key)	Returns True if the configuration property for the given key is modifiable in the current session.

Examples

Set and retrieve a configuration property:

spark.conf.set("key1", "value1")
spark.conf.get("key1")

'value1'

Retrieve a property with a default value:

spark.conf.get("non-existent-key", "my_default")

'my_default'

Unset a configuration property:

spark.conf.set("my_key", "my_value")
spark.conf.unset("my_key")
spark.conf.get("my_key")

pyspark...SparkNoSuchElementException: ... The SQL config "my_key" cannot be found...

Retrieve all configuration properties:

spark.conf.set("key1", "value1")
spark.conf.set("key2", "value2")
spark.conf.getAll

{'key1': 'value1', 'key2': 'value2'}