Share via

ADF Data Flow won't start when used in a pipeline

Sven Peeters 66 Reputation points
2022-02-17T16:06:21.6+00:00

Hi,

I created a dataflow with parameters that does execute properly when tested with the "Data Preview" button.
However, when I use this dataflow inside a pipeline it remains in queued stage forever.
I double checked the parameters that are passed, they seam correct.
The run id of the actvity is 849e7c59-8233-4b01-8ddf-318298b753e8

How can be troubleshoot this kind of error?

175503-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,625 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. MartinJaffer-MSFT 26,236 Reputation points
    2022-02-18T23:45:47.43+00:00

    Hello and welcome to Microsoft Q&A @Sven Peeters .

    If I understand correctly, your dataflow previews fine, but when you try to pipeline it, gets stuck in queued state. Can you tell me a few more details?

    • was it the first activity in the pipeline? If not, did earlier activities complete?
    • how long was it in queued state? 5 minutes? 1 hour?
    • Did it finish acquiring compute?

    There is a significant chance I will have to pass this off to support where they can do a deeper investigation than I can here.

  2. Sven Peeters 66 Reputation points
    2022-03-11T11:27:01.423+00:00

    Hi,

    This has been solved now by support, appearantly needed 8+8 cores if you include a merge operation in your data flow

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.