What is the purpose of Optimize tab in dataflow transformations other than sink?

Kobi Averbuch 26 Microsoft Employee

The Optimize tab exists in many transformations types. I understand the purpose in sink as we want to partitions the results, but why does it do in other intermediate transformations?

1 answer

MarkKromer-MSFT 5,206 Reputation points Microsoft Employee

2021-11-23T09:22:55.523+00:00

At any point inside your data flow, you can tell ADF to repartition data during each transformation step. ADF will honor that setting and repart data at each step. The best scenario for doing this is when you sort. If you wish to maintain that sorting throughout your data flow, set the Sort transformation to single partition.
Please sign in to rate this answer.
Kobi Averbuch 26 Reputation points Microsoft Employee

2021-11-23T12:21:00.27+00:00

what are the scenarios that can benefit for the other types of partitioning? also, is each step stored on disk, or kind of partitioned in memory...?

Mark Kromer MSFT 1,146 Reputation points

2021-11-23T19:53:12.7+00:00

You may sometimes wish to limit the number of partitions for certain operations, or partition the data manually using hashes, if you know your data very well and can avoid hotspots. In most cases, we recommend leaving it to default (use current partitioning) which also allows ADF to scale out the data partitions when you increase the number of cores in your Azure IR. ADF is utilizing Spark underneath, so all of the fundamental Spark principles will apply. That is, ADF will utilize data frames in memory and will access the data that is distributed across partitions on worker nodes.

Saurabh Sharma 23,791 Reputation points Microsoft Employee

2021-11-29T20:21:18.593+00:00

@Kobi Averbuch ,

Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

Thanks
Saurabh
Sign in to comment

Share via

What is the purpose of Optimize tab in dataflow transformations other than sink?

1 answer