Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,951 questions
input_file_name
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 77%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Ryan Abbey
1,181
Reputation points
I am trying to read multiple parquet files and want to add the source file name to the dataframe using Synapse 2.4 cluster, however when adding the column using "input_file_name", the column is empty
spark.read.parquet(*sfile).withColumn("input_file_name", F.input_file_name())
Any known issues with this? Any alternative ways to get the filename added (short of a union loop)?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee2021-09-29T16:30:44.463+00:00 Hello @Ryan Abbey ,
Thanks for the ask and using Microsoft Q&A platform .
Can you please share more detail as to what is "F" here ? May be sharing a more of the code ( if possible ) will help .
Thanks
Himanshu -
Ryan Abbey 1,181 Reputation points
2021-09-29T19:59:32.733+00:00 pyspark sql functions
from pyspark.sql import functions as Fsfile is a list of files...
I've found if just one file is passed in, then "input_file_name" works but when more than one file, it is null
Sign in to comment
Sign in to answer