to_variant_object

Converts a column containing nested inputs (array/map/struct) into a variants where maps and structs are converted to variant objects which are unordered unlike SQL structs. Input maps can only have string keys.

Syntax

from pyspark.sql import functions as sf

sf.to_variant_object(col)

Parameters

Parameter	Type	Description
col	pyspark.sql.Column or str	A column with a nested schema or column name.

Returns

pyspark.sql.Column: a new column of VariantType.

Examples

Example 1: Converting an array containing a nested struct into a variant

from pyspark.sql import functions as sf
from pyspark.sql.types import ArrayType, StructType, StructField, StringType, MapType
schema = StructType([
    StructField("i", StringType(), True),
    StructField("v", ArrayType(StructType([
        StructField("a", MapType(StringType(), StringType()), True)
    ]), True))
])
data = [("1", [{"a": {"b": 2}}])]
df = spark.createDataFrame(data, schema)
df.select(sf.to_variant_object(df.v))

DataFrame[to_variant_object(v): variant]

df.select(sf.to_variant_object(df.v)).show(truncate=False)

+--------------------+
|to_variant_object(v)|
+--------------------+
|[{"a":{"b":"2"}}]   |
+--------------------+