input_file_name
Ryan Abbey
1,181
Reputation points
I am trying to read multiple parquet files and want to add the source file name to the dataframe using Synapse 2.4 cluster, however when adding the column using "input_file_name", the column is empty
spark.read.parquet(*sfile).withColumn("input_file_name", F.input_file_name())
Any known issues with this? Any alternative ways to get the filename added (short of a union loop)?
Sign in to answer