Azure Synapse Pipeline Queue Wait

최 선경 0 Reputation points
2023-01-18T06:58:43.1866667+00:00

It's a pipeline that used to run normally, and it's a pipeline that runs multiple times a day.

It continued to run normally, but only some cases remain in a waiting state.

can i know the cause?

User's image

Success Case

User's image

Fail Case

User's image

The same thing is done, but in the Fail Case nothing is done when I check the details.

Other error messages also don't exist, making it difficult to determine.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,186 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Bhargava-MSFT 31,201 Reputation points Microsoft Employee
    2023-01-25T17:33:13.1366667+00:00

    Hello @최 선경 ,

    Welcome to the MS Q&A platform.

    Given the pipeline runs are in "waiting state", I suspect the issue could be that some resources are not responding.

    If you are reading the data from different sources, ensure there is no issue with the source/destination end.

    Ex: if you are reading from SQL source, make sure there are no open transactions/blocking, etc.

    From the monitoring tab, find the pipeline stuck in "waiting state". You can determine at what activity level the pipeline got stuck.

    Also, you can change the concurrency level settings.

    For example: When you set the concurrency 1, only one instance will be allowed to run at a time. When the scheduled trigger runs again and tries to run the pipeline, If the pipeline is already running, the next invocation will be queued. It will start after finishing the current instance.

    And set the Timeout property of the Stored Proc and Lookup activities to an hour or some value you are comfortable with. The default value is 7 days, meaning the activity waits 7 days before giving up and failing. Reducing this value will cause new executions to expire sooner.

    I hope this helps. Please let me know if you have any further questions.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.