This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 80%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKA%3C/text%3E%3C/svg%3E)
The Optimize tab exists in many transformations types. I understand the purpose in sink as we want to partitions the results, but why does it do in other intermediate transformations?
At any point inside your data flow, you can tell ADF to repartition data during each transformation step. ADF will honor that setting and repart data at each step. The best scenario for doing this is when you sort. If you wish to maintain that sorting throughout your data flow, set the Sort transformation to single partition.
what are the scenarios that can benefit for the other types of partitioning? also, is each step stored on disk, or kind of partitioned in memory...?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
You may sometimes wish to limit the number of partitions for certain operations, or partition the data manually using hashes, if you know your data very well and can avoid hotspots. In most cases, we recommend leaving it to default (use current partitioning) which also allows ADF to scale out the data partitions when you increase the number of cores in your Azure IR. ADF is utilizing Spark underneath, so all of the fundamental Spark principles will apply. That is, ADF will utilize data frames in memory and will access the data that is distributed across partitions on worker nodes.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.
Thanks
Saurabh