Azure SQL to Dataverse - performance issue with dataflows

Christopher Mühl 106 Reputation points
2022-05-02T19:22:48.107+00:00

Hello community,

I want to transfer data from an Azure SQL to Dataverse via Azure Data Factory. I have 2 questions about this:

Which connector is the latest and which one should be used for data transfer?
a) Dynamics 365 b) Dynamics CRM c) Dataverse

My first attempts were made with the Dataverse connector and I noticed the following:
First I tried a DataFlow with first of all no transformations, only source and sink. With this I could process about 20 rows per second or about 80k rows per hour. After that I ran into a connection reset error.

After this, I tried a copy activity. I was able to transfer 100k lines in under 9 minutes. The Linked Service was the same.

In my case I need a few data transformations, but also e.g. Conditions before the data is transferred to Dataverse. Therefore I would prefer the way over the dataflows.

How does this performance difference come about if I use the same connector? Is there anything I can do to improve the performance in the dataflow? In the CopyActivity I still had settings like BatchSize. In the DataFlow I do not have this option.

Many thanks in advance!
Christopher

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,131 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
    2022-05-04T20:34:32.693+00:00

    Hello @Christopher Mühl ,
    Thanks for the question and using MS Q&A platform.

    As we understand the ask here is if we can do to improve the performance in the dataflow, please do let us know if its not accurate.
    I think you can definitely try to update the compute type & Core count and you should see performance benefits .

    198988-image.png

    Please do let me if you have any queries.
    Thanks
    Himanshu


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators