Azure Data Factory DataFlow activities stuck in queued status

Question

Azure Data Factory DataFlow activities stuck in queued status

Sakaar Mishra 11

Starting from 1st Aug we noticed dataflow taking more than 2 hours to execute where initially it stays in Queued state for more than 90 minutes.
Tried the below options:

Tried executing the dataflow with the snapshot of data before 1st Aug when it was running well in time (20 minutes) however it's going in Queued state now.
Incresing IR's compute power and running the dataflow again.
Runing the dataflow on memory optimized IR.
When investigating on DataFlow details all the Sorce transformation itself in Queued state and not progressing.

Unable to find much information when a mapping dataflow goes in Queued state ? The dataflow is currently been ran with same resources and with exact same snapshot of data but behaving differntly post 1st Aug.

Appreciate any recomendation on this behaviour.

AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2022-08-16T16:20:22.44+00:00

Hi @Sakaar Mishra ,
Just checking if the issue got resolved for you?
If the below answer helped, kindly do consider clicking Accept Answer as accepted answers help community as well. If you have any further query do let us know.
Sakaar Mishra 11 Reputation points

2022-08-17T11:09:43.117+00:00

@AnnuKumari-MSFT - Azure service ticket been raised as below, awaiting response from the support team.
SR Number 2208170030000005

Thank You

1 answer

Your answer

AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2022-08-16T16:20:22.44+00:00

Hi @Sakaar Mishra ,
Just checking if the issue got resolved for you?
If the below answer helped, kindly do consider clicking Accept Answer as accepted answers help community as well. If you have any further query do let us know.
Sakaar Mishra 11 Reputation points

2022-08-17T11:09:43.117+00:00

@AnnuKumari-MSFT - Azure service ticket been raised as below, awaiting response from the support team.
SR Number 2208170030000005

Thank You

Answer 1

AnnuKumari-MSFT 34,566 Microsoft Employee Moderator

Hi @Sakaar Mishra ,

Welcome to Microsoft Q&A platform and thanks for posting your question here .

As I understand your issue, your dataflow remains in queued state for long time since past few days. Please let me know if my understanding is incorrect.

Could you please help us with the following details:

Would you be able to tell what is the number of records or the amount of data being processed before and after 1st aug.
Is this happening with other dataflows as well or only a particular dataflow is being affected?

Looking forward to your response. Thankyou

Sakaar Mishra 11 Reputation points

2022-08-11T10:38:48.64+00:00
Hi @AnnuKumari-MSFT

Thanks for revreting back, yes your understanding as dataflow remians in queued state for long starting from 1st Aug.
Below are requested details:

There had not been any increase in the amount of data been processed before and after 1st of Aug, its processing around 700K records. I even tried running the dataflow with same data as of before 1st Aug and ended up with dataflow in queued state for around 90 minutes.

This is happening only with single dataflow and other dataflows are not impacted.

Thank You
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2022-08-12T05:10:15.903+00:00

Hi @Sakaar Mishra ,

Thanks for the required details.

I understand you have tried with the snapshot of data before 1 Aug. However, it's not getting through. Kindly try to split the file into smaller parts and try processing it individually.

I feel this issue needs a deeper investigation. I would recommend you to please file a support Request so that a Support engineer can take a deeper look into the logs with your runID's. After filing the request kindly do share the SR# with us for tracking it internally.

To know more about how to create a support , kindly visit : Create an Azure support request

And if you don't have a support plan, please let us know here.
Olufemi obembe 0 Reputation points

2023-04-16T16:06:40.67+00:00

I am relatively new at using Azure data flow. I am also trying with just over 200 records but it is taking forever running the data flow. I cannot seem to understand what the problem is.
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2023-04-24T10:34:16.2166667+00:00

Olufemi obembe , Kindly create a new thread as it's an old one

Share via

Azure Data Factory DataFlow activities stuck in queued status

1 answer

Your answer