ADF Copy activity output into multiple parquet snappy file?

Question

In ADF copy activity output, pipeline is creating very large parquet file. Will it possible to split the output into multiple small parquet snappy files, so that synapse external table can use parallelism (polybase)? like Similar to DBrick data frame, where it writes into multiple small parquet files. Kindly advise how to achieve in ADF copy activity ?

Answer 1

Hi @ever ,

Welcome to Microsoft Q&A Platform.

There is no dynamic way to split the files in a copy activity in data factory. However, the same can be achieved by defining rules - specific year range or specific set of records in the table identified by a column value. These rules can be maintained in a config table/file and Lookup activity can be used to retrieve them prior to sending this information to a foreach activity that has copy activity. The query parameterization at source can be used to filter the data from single SQL table according to config table rules and loads into multiple files parallelly.

As per our recent engagement with Azure data factory product team, they mentioned that this requirement aligns perfectly with their ongoing work item - "New property maxRowsPerFile to split and write to multiple smaller files". This will take 4-6months to be available for use.

Hope this helps! Please let us know if our understanding is incorrect or for further queries and we will be glad to assist.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Answer 2

Hi @ever ,

Want to share the new update from ADF team which we were discussing in August. Please check this link that gives more details on this property and kindly let us know if it helps in implementing your requirement.

Thanks for your patience!