Salesforce to SQL via Data Factory -- Speed

Question

Salesforce to SQL via Data Factory -- Speed

Bakar, Cem 281

I am using a regular Salesforce connector to pull a table and writing the content onto a SQL database azure. All works. However, ~2GB data file takes +10 hours to be transferred.

.

Bakar, Cem 281 Reputation points

2021-01-11T19:06:36.25+00:00

I am using a predefined template for copying data. I have played with Degree of copy parallelism and Degree of copy parallelism but the result did not changed.
HarithaMaddi-MSFT 10,146 Reputation points

2021-01-12T06:32:56.55+00:00

Hi @Bakar, Cem ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

Please share your pipeline run id and if Self-hosted integration runtime is used, please share report id as well as below

In the integration runtime configuration manager, please click on diagnostics icon, reproduce the error and then click on ‘Send logs’.

You will be able to see report id, please share that with us to enable viewing logs.

Looking forward for your response to investigate further.
Bakar, Cem 281 Reputation points

2021-01-13T01:46:52.147+00:00

Hi @HarithaMaddi-MSFT ,
Thank you for your reply and welcoming. I can't locate the diagnostics icon/ integration runtime configuration manager but I have run one of the run ids.
I hope it helps!

PI: 2e0a5939-607c-4284-824d-bad8a5bc5d45
HarithaMaddi-MSFT 10,146 Reputation points

2021-01-15T08:39:03.73+00:00

Hi @Bakar, Cem ,

Thanks for sharing the run id. I will reach to product teams to investigate further and will get back to you as soon as I hear from them.

Stay tuned!

Accepted answer

0 additional answers

Your answer

Bakar, Cem 281 Reputation points

2021-01-11T19:06:36.25+00:00

I am using a predefined template for copying data. I have played with Degree of copy parallelism and Degree of copy parallelism but the result did not changed.
HarithaMaddi-MSFT 10,146 Reputation points

2021-01-12T06:32:56.55+00:00

Hi @Bakar, Cem ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

Please share your pipeline run id and if Self-hosted integration runtime is used, please share report id as well as below

In the integration runtime configuration manager, please click on diagnostics icon, reproduce the error and then click on ‘Send logs’.

You will be able to see report id, please share that with us to enable viewing logs.

Looking forward for your response to investigate further.
Bakar, Cem 281 Reputation points

2021-01-13T01:46:52.147+00:00

Hi @HarithaMaddi-MSFT ,
Thank you for your reply and welcoming. I can't locate the diagnostics icon/ integration runtime configuration manager but I have run one of the run ids.
I hope it helps!

PI: 2e0a5939-607c-4284-824d-bad8a5bc5d45
HarithaMaddi-MSFT 10,146 Reputation points

2021-01-15T08:39:03.73+00:00

Hi @Bakar, Cem ,

Thanks for sharing the run id. I will reach to product teams to investigate further and will get back to you as soon as I hear from them.

Stay tuned!

Answer 1

HarithaMaddi-MSFT 10,146

Hi @Bakar, Cem ,

Thanks for your patience. Product team confirmed that this occurrence is a transient rare Azure IR crash issue and rerun of activity should fix this. Using activity timeout will help avoid the stuck issue in future. Please let us know if you are seeing this issue again and we can work with them to investigate further.

--

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
Want a reminder to come back and check responses? Here is how to subscribe to a notification.

HarithaMaddi-MSFT 10,146 Reputation points

2021-01-19T12:41:59.867+00:00

Hi @Bakar, Cem ,

We have not received a response from you. Please suggest if above suggested details are helpful. Otherwise, let us know and we will continue to engage with you on the query.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members.
Bakar, Cem 281 Reputation points

2021-01-19T13:00:45.337+00:00

Thank you for your reply. I wan't able to improve the amount of time it takes but I have tried different connections on Data Bricks. Which made a difference. Thank you for your help.

Share via

Salesforce to SQL via Data Factory -- Speed

0 additional answers

Your answer