Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Returns the union of all given maps. For duplicate keys in input maps, the handling is governed by spark.sql.mapKeyDedupPolicy. By default, it throws an exception. If set to LAST_WIN, it uses the last map's value.
Syntax
from pyspark.sql import functions as sf
sf.map_concat(*cols)
Parameters
| Parameter | Type | Description |
|---|---|---|
cols |
pyspark.sql.Column or str |
Column names or Column |
Returns
pyspark.sql.Column: A map of merged entries from other maps.
Examples
Example 1: Basic usage of map_concat
from pyspark.sql import functions as sf
df = spark.sql("SELECT map(1, 'a', 2, 'b') as map1, map(3, 'c') as map2")
df.select(sf.map_concat("map1", "map2")).show(truncate=False)
+------------------------+
|map_concat(map1, map2) |
+------------------------+
|{1 -> a, 2 -> b, 3 -> c}|
+------------------------+
Example 2: map_concat with three maps
from pyspark.sql import functions as sf
df = spark.sql("SELECT map(1, 'a') as map1, map(2, 'b') as map2, map(3, 'c') as map3")
df.select(sf.map_concat("map1", "map2", "map3")).show(truncate=False)
+----------------------------+
|map_concat(map1, map2, map3)|
+----------------------------+
|{1 -> a, 2 -> b, 3 -> c} |
+----------------------------+
Example 3: map_concat with empty map
from pyspark.sql import functions as sf
df = spark.sql("SELECT map(1, 'a', 2, 'b') as map1, map() as map2")
df.select(sf.map_concat("map1", "map2")).show(truncate=False)
+----------------------+
|map_concat(map1, map2)|
+----------------------+
|{1 -> a, 2 -> b} |
+----------------------+
Example 4: map_concat with null values
from pyspark.sql import functions as sf
df = spark.sql("SELECT map(1, 'a', 2, 'b') as map1, map(3, null) as map2")
df.select(sf.map_concat("map1", "map2")).show(truncate=False)
+---------------------------+
|map_concat(map1, map2) |
+---------------------------+
|{1 -> a, 2 -> b, 3 -> NULL}|
+---------------------------+