COPY Activity DIU's not using as per settings

Muralikrishna Pirla 216 Reputation points
2021-09-17T08:35:56.307+00:00

Hello Team,

I have a copy activity which extracts the data from Azure PostgreSQL to Data lake Gen1(CSV), this pipeline extracts more than one billion records daily to data lake, to improve the performance we have increased the DIU's to 30 and Copy parallel property to 8, however the pipeline is using only 4 DIU's and Copy parallels 1, I am wondering why ADF is not using all the DIU's though we set them explicitly.

Can anyone show some light here....

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,537 questions
Azure Database for PostgreSQL

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2021-09-20T18:15:14.457+00:00

    Hello @Muralikrishna Pirla ,
    Thanks for the ask and using Microsoft Q&A platform .
    I think you should look at the how the data is partioned on the source side . Have you tried to set the partition and see if that helps .?

    133645-image.png

    Please do read about the best practices here : https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-sql-database?tabs=data-factory#parallel-copy-from-sql-database

    Please do let me know how it goes .
    Thanks
    Himanshu

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
      • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of [Q&A Volunteer Moderators][50]
    0 comments