Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Does each parquet file in that folder have the same schema?
ADF dataset vs databricks dataframe
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
arkiboys
9,711
Reputation points
In databricks when I use this path to get to the delta parquet files,
delta_folder_path = "/foldername/subfoldername/LoadedData"
I see all the fields including year, month and the day fields
note: inside LoadedData folder there are folders for each of the companyNames.
Each companyName folder has delta parquet files.
However, when I use the same path in the dataflow source as delta inline dataset in ADF, I do not see the fields for year, month, day :
Any thoughts?
Thank you
Azure Data Factory
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator2022-04-22T18:33:41.327+00:00 Hello @arkiboys ,
Thanks for the question and using MS Q&A platform.As we understand the ask here is why the databricks and dataflow are behaving different pointing to the same folder , please do let us know if its not accurate.
. At this time I think we are missing some key details ideally that should not be the case . Can you please share the folder structure and also the dataflow implementation .
I am assume that all the paraquet files have all the three fields which you have called out .We will wait to hear from you . .
Thanks
Himanshu -
arkiboys 9,711 Reputation points
2022-04-24T07:23:42.977+00:00 Hi,
I have simplified the structure to explain as below...
Here you see the structure in the storage:
container1/folder1/subfolder1/subfolder2/year=2022/month=04/day=24/companyName1
container1/folder1/subfolder1/subfolder2/year=2022/month=04/day=24/companyName2
container1/folder1/subfolder1/subfolder2/year=2022/month=04/day=24/companyName3
...I then created a new dataset to read container1/folder1/subfolder1
The dataset shows the columns of the parquet files in all the above folder structure for day=24 but there are no columns in teh dataset for year, month, day
note: there is only folder for day=24 and no other days present as this is a test.Then in databricks, I point to container1/folder1/subfolder1/*. here I see all the columns in the parquet files in day=24 as well as columns for year and month and day
Question:
Why in ADF dataset I do not see the columns for year and month and day whereas in databricks I do?
each parquet file in day=24 folder have the same schema.Thanks
-
HimanshuSinha 19,637 Reputation points Microsoft Employee Moderator
2022-04-27T19:29:20.98+00:00 Hello @arkiboys ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
Sign in to comment
1 answer
Sort by: Most helpful
-
MarkKromer-MSFT 5,231 Reputation points Microsoft Employee Moderator2022-04-22T19:26:31.77+00:00