Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Collection function: Returns element of array at given (1-based) index or value for given key in a map. For arrays, if index is 0, Spark will throw an error. If index < 0, accesses elements from the last to the first. If 'spark.sql.ansi.enabled' is set to true, an exception will be thrown if the index is out of array boundaries instead of returning NULL. For maps, the function always returns NULL if the key is not contained in the map. Supports Spark Connect.
For the corresponding Databricks SQL function, see element_at function.
Syntax
from pyspark.databricks.sql import functions as dbf
dbf.element_at(col=<col>, extraction=<extraction>)
Parameters
| Parameter | Type | Description |
|---|---|---|
col |
pyspark.sql.Column or str |
Name of column containing array or map. |
extraction |
Any | Index to check for in array or key to check for in map. |
Returns
pyspark.sql.Column: value at given position.
Examples
Example 1: Getting the first element of an array
from pyspark.databricks.sql import functions as dbf
df = spark.createDataFrame([(["a", "b", "c"],)], ['data'])
df.select(dbf.element_at(df.data, 1)).show()
+-------------------+
|element_at(data, 1)|
+-------------------+
| a|
+-------------------+
Example 2: Getting a value from a map using a key
from pyspark.databricks.sql import functions as dbf
df = spark.createDataFrame([({"a": 1.0, "b": 2.0},)], ['data'])
df.select(dbf.element_at(df.data, dbf.lit("a"))).show()
+-------------------+
|element_at(data, a)|
+-------------------+
| 1.0|
+-------------------+