how to dynamically explode array type column in pyspark or scala

reddy 41 Reputation points
2020-08-19T18:34:33.73+00:00

HI,

i have a parquet file with complex column types with nested structs and arrays.

I am using the scrpit from below link to flatten my parquet file.

https://learn.microsoft.com/en-us/azure/synapse-analytics/how-to-analyze-complex-schema

I am able to flatten schema using script in STEP 1 and STEP 2 successfully. But in the above link, for STEP 3 the script uses hardcoded column names to flatten arrays. But in my case i have multiple columns of array type that need to be transformed so i cant use this method.

Is there any way to dynamically transform all the array type columns without hardcoding because in future the columns may change in my case. Something like check if a column is of array type and explode it dynamically and repeat for all columns of arrays.

Please advise
@AmanpreetSingh-MSFT @PRADEEPCHEEKATLA-MSFT @HarithaMaddi-MSFT @Saurabh Sharma

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,947 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 78,331 Reputation points Microsoft Employee
    2020-08-21T11:14:53.297+00:00

    Hello @reddy ,

    It’s hard to provide the sample code snippet which helps to dynamically transform all the array type columns without understand the underlying column types present in your dataset.

    While working with nested data types, Delta Lake on Databricks optimizes certain transformations out-of-the-box. The following notebooks contain many examples on how to convert between complex and primitive data types using functions natively supported in Apache Spark SQL.

    For more details, refer “Azure Databricks – Transform complex data types”.

    I would request you to kindly go through the below notebook which explains some data transformation examples using Spark SQL. Spark SQL supports many built-in transformation functions in the module pyspark.sql.functions therefore we will start off by importing that.

    https://learn.microsoft.com/en-us/azure/databricks/_static/notebooks/transform-complex-data-types-python.html

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.