Why a Pipeline keeps stuck in queued status forever in Azure Data Factory?

Alan Gutierrez Encizo 41 Reputation points
2021-11-25T18:30:31.327+00:00

I've a Pipeline to execute several Data Flows with in Azure Data Factory, some weeks ago it was running properly and lasted around 25 minutes to finish, I've had to make a little adjustment in one filter (specifically, it was in the last and only Data Flow that keeps in queued forever, this Data Flow inserts into SQL DB). After the adjustment, without explanation everytime Pipeline is running it just keeps static on queued status and never ends:

152638-hours-pipeline.png

I tried to delete the filter where was the adjustment, recreate the Pipeline, and as you see guys, is still not working, the last Data flow in the Pipeline keeps in the same status (30 to 40 hours waiting on queued and nothing happen while the other pipelines are running well):

152667-nothing.png

Do you know, guys if is there any configuration that I'm missing? here is what I've configured in my pipeline:

152668-configuration.png

Note: I'm just working with 500,000 records, and even though I upgraded the Core Count configuration to 8 Driver Cores but I got next error:

152734-captura.png

Thanks by the way, best regards

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,640 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.