What is the purpose of Optimize tab in dataflow transformations other than sink?

Kobi Averbuch 26 Reputation points
2021-11-23T07:48:19.46+00:00

The Optimize tab exists in many transformations types. I understand the purpose in sink as we want to partitions the results, but why does it do in other intermediate transformations?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,305 questions
No comments
{count} votes

1 answer

Sort by: Most helpful
  1. MarkKromer-MSFT 5,136 Reputation points Microsoft Employee
    2021-11-23T09:22:55.523+00:00

    At any point inside your data flow, you can tell ADF to repartition data during each transformation step. ADF will honor that setting and repart data at each step. The best scenario for doing this is when you sort. If you wish to maintain that sorting throughout your data flow, set the Sort transformation to single partition.