Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
Azure Synapse Analytics to Dynamics 365 Migration Very slow speed copy activity !!!!!!!!!
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 43%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shreyash Choudhary
126
Reputation points
Hi,
My source is Azure Synapse analytics dedicated sql pool, target is Dynamics 365 crm, using mainly copy activity in pipeline, i have 4 batches of pipeline in source sql query [where column_rownum between 1 and 724,005 order by column_rownum ], 2nd batch with [ 724,006 and 1448010 ] 3rd batch [1448010 and 2172105] , 4th batch [where column_rownum >2172105 ].
copy activity settings :
max concurrent connection: 32
batch sizze:300
degree of copy parallelism: 8
Max data integration unit : 32
Below are the screenshots of 1 of the pipeline[batch] copy execution
it got timed out. it has to load 750,000 approx. records each batch but failed also it took 12+ hours for loading 550,000+ records
how to increase speed of copy drastically. it is very very very slow...
also in source table we have used hash distribution,and record count of table is 2896019, we are using 4 spn for 4 batches
Azure Synapse Analytics
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Microsoft 365 and Office | Install, redeem, activate | For business | Windows
Microsoft 365 and Office | Install, redeem, activate | For business | Windows
Processes in Microsoft 365 for setting up Office apps, redeeming product keys, and activating licenses.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu 32,535 Reputation points Microsoft External Staff Moderator2023-09-06T12:02:29.4433333+00:00 @Shreyash Choudhary Welcome to Microsoft Q&A platform and thanks for posting your question here.
I understand that you're looking to improve the performance of copying data from Azure Synapse Analytics to Dynamics 365. Please correct me if I've misunderstood your concern.
You've already set your Data Integration Unit (DIU) 32, which is great.
Here are some troubleshooting steps, explained in a more straightforward manner, that you can follow to enhance the performance of your Self-hosted Integration Runtime (IR) when working with copy activities:
- Check if the machine running your Self-hosted IR has a low-latency connection to the source and destination data stores. You can use a tool like Azure Speed Test to measure the latency. Lower latency is better. In case you are using self-hosted IR, kindly try to distribute the load on multiple machines.
Supporting document: Creating self-hosted IR - Ensure that the machine hosting the Self-hosted IR has sufficient outbound bandwidth for efficient data transfer and writing. If your data stores are in Azure, you can check the upload speed using Azure Speed Test.
- Dynamics 365 has throttling limits that can affect the copy speed. Make sure that you are not hitting these limits and that you are following best practices for data migration to Dynamics 365.
I hope this helps you achieve better performance in your data copying tasks!
- Check if the machine running your Self-hosted IR has a low-latency connection to the source and destination data stores. You can use a tool like Azure Speed Test to measure the latency. Lower latency is better. In case you are using self-hosted IR, kindly try to distribute the load on multiple machines.
-
Smaran Thoomu 32,535 Reputation points Microsoft External Staff Moderator
2023-09-07T11:45:40.69+00:00 We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
-
Smaran Thoomu 32,535 Reputation points Microsoft External Staff Moderator
2023-09-08T12:29:13.29+00:00 @Shreyash Choudhary Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
-
Shreyash Choudhary 126 Reputation points
2023-09-08T12:40:41.2133333+00:00 Hi,
No this solution is not helpfull,
i have same pipeline with different entity and much higher record count 1.8 million per batch got completed in under 4.5 hours, same shir is used
-
Smaran Thoomu 32,535 Reputation points Microsoft External Staff Moderator
2023-09-11T07:35:33.3+00:00 @Shreyash Choudhary I agree that this issue seems unusual, and I wasn't able to reproduce it. For a more in-depth investigation and immediate assistance with this matter, if you have a support plan, you can file a support ticket. If you don't have a support plan, I can enable a one-time free support request for your subscription. Please do let us know.
-
Smaran Thoomu 32,535 Reputation points Microsoft External Staff Moderator
2023-09-12T11:23:15.2733333+00:00 @Shreyash Choudhary We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sign in to comment
Sign in to answer