Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
Partition dimension of parquet files copied in hierarchical folders respect to year, month and date
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 41%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
pmscorca
1,052
Reputation points
Hi,
I've implemented a Synapse mapping data flow, called by a pipeline, to copy source parquet files (NYC green taxi data) into a hierarchical folders, partitioning for year, month and date keys:
At the leaf level of these hierarchical folders I've noticed that some partitions were created, but is it possible to change the dimension or the number of these partitions maintaning their related distribution in the year-->month-->date folders? Thanks
Azure Synapse Analytics
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 12%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESV%3C/text%3E%3C/svg%3E)
Subashri Vasudevan 11,306 Reputation points Volunteer Moderator2023-11-26T05:36:22.68+00:00 Hi, is your question to split the source files based on yyyy/MM/dd? Now I see it being split into YYYY MM yyyy-MM-dd. Correct if my understanding is wrong
-
pmscorca 1,052 Reputation points
2023-11-26T08:47:14.37+00:00 Hi, I've already partitioned the parquet data in an hierarchical structure as year-->month-->date, but I've noticed that in each leaf folder more files are created and I'm asking it is possible to change the dimension or the number of these partitions maintaning their related distribution in the year-->month-->date folders.
-
Subashri Vasudevan 11,306 Reputation points Volunteer Moderator
2023-11-26T09:15:35.98+00:00 Can u pls share a pic of a leaf folder.
-
pmscorca 1,052 Reputation points
2023-11-26T22:09:41.06+00:00 Hi, here is the required picture of the leaf folder:
There are 3 partitions for an unique date, with the same dimension
-
Subashri Vasudevan 11,306 Reputation points Volunteer Moderator
2023-11-27T08:30:45.3033333+00:00 Hey, looks like these files are created in different runs. They don't seem to be created at the same time in a single run (looking at the timestamp). So probably it works the way you want by creating one file at a time, but with different name everytime. Is your requirement to have one partition file with a specific name ?
If name of the file being generated is a concern here, you can go to settings tab in sink and give file name option as 'output to single file' which will prompt you to 'set single partition'. Once done, you can give a name to the partition file as you wish. This will keep over writing the same file everytime you run your pipeline.
But if your question is different from what I try to explain please let me know.
Thanks.
-
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator2023-12-01T06:52:14.0833333+00:00 Hi pmscorca ,
We haven't heard anything on the above ask. Just checking if you got a chance to look at the previous response. Kindly let us know if your issue has been resolved. Thankyou
Sign in to comment
Sign in to answer