Dataflow takes long time

Question

Dataflow takes long time

atarantin 1

Hello,

I have a a pipeline with 1 dataflow. It takes between 2 and 3 minutes to run it and I do not understand why it takes so much time.

For instance, a run took 2 minutes 05.

The data flow inside it took 02 minutes 04.

There are 5 steps in my dataflow:

Take files from an azure storage container
Flatten the files
I do a derived column
I set the upsert for the database
I insert data to cosmos db

If I take a look to the detailed metrics, each step of the dataflow took 10s (with a cluster startup time to 1s 479ms for the dataflow).

I am already using an integration runtime with a TTL.

So do you have an idea on how my pipeline/dataflow can be so long while the detailed dataflow metrics are so short?

MarkKromer-MSFT 5,226 Reputation points Microsoft Employee Moderator

2021-08-13T21:15:21.377+00:00

Look at the monitoring view of the data flow activity. You can click on each transformation to see how long each step took so you can identify your bottleneck. Since you are writing data to CosmosDB, I would focus on the Sink transformation and look to see how long the Sink took to serialize the documents. You may be getting throttled by the throughput setting on your target collection.
atarantin 1 Reputation point

2021-08-16T05:50:31.743+00:00

In the monitoring of the data flow (when I click on the glasses) the "sink processing time" is 11s.
MarkKromer-MSFT 5,226 Reputation points Microsoft Employee Moderator

2021-08-16T15:55:48.737+00:00

Use the bottom panel of your monitoring view to sort by processing time and find the longest processing steps
atarantin 1 Reputation point

2021-08-17T16:50:09.443+00:00

IIt is the sink processing time which is the highest one.
ShaikMaheer-MSFT 38,551 Reputation points Microsoft Employee Moderator

2021-08-19T05:06:20.097+00:00

Hi @atarantin ,

You mentioned "each step of dataflow took 10s", and you have 5 steps in your data flow, So if we sum it your execution time should be 50s to 60s only right?

Are you trying to say your execution time of data flow is actually less but data flow activity itself shows over all duration as 2mins 5s?

You mentioned you already mentioned TTL. What is the value you mentioned there? Kindly check below to know more details about TTL and its billing.
https://learn.microsoft.com/en-us/answers/questions/38550/azure-data-flow-activity-time.html

Did you checked Quick re-use box under TTL settings?

between, Click here to know more details about monitoring data flows. Click here to know more details about Mapping data flows performance and tuning guide.
atarantin 1 Reputation point

2021-08-19T05:54:53.95+00:00

Yes exactly, the whole pipeline takes 2 min 5s and the dataflow itself, only 50 to 60s.

The TTL is set to 10 minutes because the dataflow should run every 5 minutes. And the box "quick re-use" is checked.

II have done some modifications around the dataflow in order to pre filter my data before processing and now I have a dataflow which takes 1s/2s itself and the whole processing time for the pipeline is 20s.

Do you know why I have a so high overhead? (20s vs 1s)
ShaikMaheer-MSFT 38,551 Reputation points Microsoft Employee Moderator

2021-08-19T10:57:05.953+00:00
Hi @atarantin ,

Is there any other activities in pipeline or only Data flow activity?

Is it debug run or trigger run?

If its trigger run, then please mention type of trigger as well?

It would be more helpful if you can share your Pipeline run and activity runs execution's screenshot to understand issue better. Thank you.
ShaikMaheer-MSFT 38,551 Reputation points Microsoft Employee Moderator

2021-08-23T06:44:37.117+00:00

Hi @atarantin ,

Following up to see if you get chance to share details on above requested asks? Thank you.
ramreddy thaviti 0 Reputation points

2024-01-05T12:34:08+00:00

i am also facing same issue, all activities are executed in 9min but processing time is 17mins why? i have checked all monitor logs already.

1 answer

Your answer

MarkKromer-MSFT 5,226 Reputation points Microsoft Employee Moderator

2021-08-13T21:15:21.377+00:00

Look at the monitoring view of the data flow activity. You can click on each transformation to see how long each step took so you can identify your bottleneck. Since you are writing data to CosmosDB, I would focus on the Sink transformation and look to see how long the Sink took to serialize the documents. You may be getting throttled by the throughput setting on your target collection.
atarantin 1 Reputation point

2021-08-16T05:50:31.743+00:00

In the monitoring of the data flow (when I click on the glasses) the "sink processing time" is 11s.
MarkKromer-MSFT 5,226 Reputation points Microsoft Employee Moderator

2021-08-16T15:55:48.737+00:00

Use the bottom panel of your monitoring view to sort by processing time and find the longest processing steps
atarantin 1 Reputation point

2021-08-17T16:50:09.443+00:00

IIt is the sink processing time which is the highest one.
ShaikMaheer-MSFT 38,551 Reputation points Microsoft Employee Moderator

2021-08-19T05:06:20.097+00:00

Hi @atarantin ,

You mentioned "each step of dataflow took 10s", and you have 5 steps in your data flow, So if we sum it your execution time should be 50s to 60s only right?

Are you trying to say your execution time of data flow is actually less but data flow activity itself shows over all duration as 2mins 5s?

You mentioned you already mentioned TTL. What is the value you mentioned there? Kindly check below to know more details about TTL and its billing.
https://learn.microsoft.com/en-us/answers/questions/38550/azure-data-flow-activity-time.html

Did you checked Quick re-use box under TTL settings?

between, Click here to know more details about monitoring data flows. Click here to know more details about Mapping data flows performance and tuning guide.
atarantin 1 Reputation point

2021-08-19T05:54:53.95+00:00

Yes exactly, the whole pipeline takes 2 min 5s and the dataflow itself, only 50 to 60s.

The TTL is set to 10 minutes because the dataflow should run every 5 minutes. And the box "quick re-use" is checked.

II have done some modifications around the dataflow in order to pre filter my data before processing and now I have a dataflow which takes 1s/2s itself and the whole processing time for the pipeline is 20s.

Do you know why I have a so high overhead? (20s vs 1s)
ShaikMaheer-MSFT 38,551 Reputation points Microsoft Employee Moderator

2021-08-19T10:57:05.953+00:00

Hi @atarantin ,

Is there any other activities in pipeline or only Data flow activity?

Is it debug run or trigger run?

If its trigger run, then please mention type of trigger as well?

It would be more helpful if you can share your Pipeline run and activity runs execution's screenshot to understand issue better. Thank you.
ShaikMaheer-MSFT 38,551 Reputation points Microsoft Employee Moderator

2021-08-23T06:44:37.117+00:00

Hi @atarantin ,

Following up to see if you get chance to share details on above requested asks? Thank you.
ramreddy thaviti 0 Reputation points

2024-01-05T12:34:08+00:00

i am also facing same issue, all activities are executed in 9min but processing time is 17mins why? i have checked all monitor logs already.

Answer 1

Jesse 1

I have the same problem. I'm running the pipeline in debug mode using the data flow debug session. The cluster startup time is 39s and the Data flow has a processing time of 55s. Still the Data flow pipeline activity takes 6 min to complete. Where does this overhead come from and how can I influence it?

Ljiljana Pintarić 1 Reputation point

2023-10-04T13:03:41.5433333+00:00

I have the same problem.

Share via

Dataflow takes long time

1 answer

Your answer