How writing partitioned parquet files from a csv file

Question

Hi,

I need to know how I can write partitioned parquet files into an Azure Blob storage reading a single csv file.

I'd like to partition by year, creating year folders and putting the year partitioned files in the corresponding folder.

Any helps to me, please? Thanks

Answer 1

Hi

Thanks for reaching out to Microsoft Q&A.

Yes you can achieve it through ConditionalSplit transformation activity in adf data flow. Pls refer the below link, you will have to alter the conditions little bit to suit your requirement though.

https://stackoverflow.com/questions/71174070/copying-single-csv-with-multiple-schema-in-adf

Please Upvote and Accept as answer if the reply was helpful, this will be helpful to other community members.

Answer 2

Hi pmscorca,

Thank you for posting query in Microsoft Q&A Platform.

You can achieve this, using data flows in Azure data factory or writing PySpark code in Azure Databricks or Azure Synapse analytics notebooks.

Below screenshots shows sample implementation details of similar case using mapping data flows. Here I am taking source data and partitioning based on dep column. Hence you can see folders and subfolders created with dep. In your case you can consider using year column.

You can achieve same using PySpark code partitionBy() function as well. Please check below video for same created by me.

partitionBy function in PySpark

Hope this helps. Please let me know how it goes or if any further queries.

---

Please consider hitting Accept Answer button. Accepted answers help community as well.