Take too long time to Load Data into Azure synapse.

Question

Take too long time to Load Data into Azure synapse.

EUNSIL kim 50

In Synapse pipeline, time to load data is taking too long.

This issue's happened after resolving MTBJ-PCZ

Before MTBJ-PCZ issue, it takes only 10 minutes.

Before

But now, It takes almost 2 hours.

After

You can find detail infomations images above .

Working duration (Queue + PreCopyScript + Transfer) is around 8 minutes, but total duration is 1:38:18.

Is anyone can explain about 1:30:00 (1:38:18 - 0:00:08)??

Smaran Thoomu 24,260 Reputation points Microsoft External Staff Moderator

2023-12-21T13:57:36.4333333+00:00

@EUNSIL kim We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Accepted answer

0 additional answers

Your answer

Smaran Thoomu 24,260 Reputation points Microsoft External Staff Moderator

2023-12-21T13:57:36.4333333+00:00

@EUNSIL kim We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Smaran Thoomu 24,260 Microsoft External Staff Moderator

Hi @EUNSIL kim

Welcome to the MS Q&A platform.

I understand that you're facing an issue where the time to load data in Synapse pipeline is taking too long. The difference between the working duration and total duration is the time taken for the data to be staged and committed to the destination. This time can vary depending on the size of the data, the number of files, and the destination.

Regarding the issue that started after resolving MTBJ-PCZ, it's possible that the resolution of that issue caused changes to the pipeline or the data that are causing the longer load times. It's also possible that the data being loaded has changed, resulting in a larger dataset or more complex data structure.

To investigate further, you can check the pipeline settings and the data being loaded to see if there are any changes that could be causing the longer load times. You can also check the logs and metrics for the pipeline to see if there are any errors or performance issues that could be causing the delay. It's also possible that there are other factors at play, such as network latency or resource contention.

To learn more about the pipeline and data loading process, you can refer to the Azure Synapse documentation. Here's a link to the documentation: https://docs.microsoft.com/en-us/azure/synapse-analytics/data-flow/data-flow-copy-activity

I hope this helps! Let me know if you have any further questions.

EUNSIL kim 50 Reputation points

2023-12-22T00:32:11.7266667+00:00

Hi @Smaran Thoomu . Thank you for your answer.

I checked your answer now, I didn't change any pipeline settings and data size is not dramatically increasing.

But, After 2 days from I posted this question, today this activity's duration time is decreased.

I'm monitoring for more days, then I'll write a comment on this post.
Smaran Thoomu 24,260 Reputation points Microsoft External Staff Moderator

2023-12-26T05:06:14.4+00:00

EUNSIL kim You're welcome! I'm glad to hear that the duration time has decreased. It's possible that there was a temporary issue or resource contention that caused the longer load times, and it has since been resolved.

Please feel free to write a comment on this post if you have any further questions or if you notice any changes in the duration time. I'll be happy to assist you further.
Please do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
EUNSIL kim 50 Reputation points

2023-12-29T01:08:43.91+00:00

@Smaran Thoomu This issue seems to be resolved. Pipeline's duration has been reduced in half.

Share via

Take too long time to Load Data into Azure synapse.

0 additional answers

Your answer