Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
User-facing configuration API, accessible through SparkSession.conf.
Supports Spark Connect
Options set here are automatically propagated to the Hadoop configuration during I/O.
Syntax
spark.conf.set(key, value)
spark.conf.get(key)
Properties
| Property | Description |
|---|---|
getAll |
Returns all properties set in this configuration as a dict. |
Methods
| Method | Description |
|---|---|
set(key, value) |
Sets the given Spark runtime configuration property. value can be a string, integer, or boolean. |
get(key, default) |
Returns the value of the configuration property for the given key. Returns default if the key does not exist. Raises an exception if the key is not set and no default is provided. |
unset(key) |
Resets the configuration property for the given key. |
isModifiable(key) |
Returns True if the configuration property for the given key is modifiable in the current session. |
Examples
Set and retrieve a configuration property:
spark.conf.set("key1", "value1")
spark.conf.get("key1")
'value1'
Retrieve a property with a default value:
spark.conf.get("non-existent-key", "my_default")
'my_default'
Unset a configuration property:
spark.conf.set("my_key", "my_value")
spark.conf.unset("my_key")
spark.conf.get("my_key")
pyspark...SparkNoSuchElementException: ... The SQL config "my_key" cannot be found...
Retrieve all configuration properties:
spark.conf.set("key1", "value1")
spark.conf.set("key2", "value2")
spark.conf.getAll
{'key1': 'value1', 'key2': 'value2'}