How to control the files number created in sink when partitioning on key column in ADF

Kobi Averbuch 26 Reputation points Microsoft Employee
2021-11-23T12:43:23.583+00:00

When I partition that data in sink (based on key) I get 12 files in each partition

  1. How can I control the number of files created? what is the logic there?
  2. Is there any support for the notion of row groups for the parquet files created?
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,226 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Kobi Averbuch 26 Reputation points Microsoft Employee
    2021-11-24T16:41:25.537+00:00
    1. I am partitioning by a key
    2. Parquet has a notion of row group, internal partitioning of the data into group within each file. i think that from a performance point of view the way they are organized will make a difference
    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.