Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new values for the pairs. Supports Spark Connect.
For the corresponding Databricks SQL function, see transform_values function.
Syntax
from pyspark.databricks.sql import functions as dbf
dbf.transform_values(col=<col>, f=<f>)
Parameters
| Parameter | Type | Description |
|---|---|---|
col |
pyspark.sql.Column or str |
Name of column or expression. |
f |
function |
A binary function. |
Returns
pyspark.sql.Column: a new map of entries where new values were calculated by applying given function to each key value argument.
Examples
from pyspark.databricks.sql import functions as dbf
df = spark.createDataFrame([(1, {"IT": 10.0, "SALES": 2.0, "OPS": 24.0})], ("id", "data"))
row = df.select(dbf.transform_values(
"data", lambda k, v: dbf.when(k.isin("IT", "OPS"), v + 10.0).otherwise(v)
).alias("new_data")).head()
sorted(row["new_data"].items())
[('IT', 20.0), ('OPS', 34.0), ('SALES', 2.0)]