Notebook execution is hanging when triggered using synapse pipeline

Question

Notebook execution is hanging when triggered using synapse pipeline

K Kulendra Singh (Harman Connected Services Inc) 5 Microsoft External Staff

My spark notebook is extracting data from data lake & is triggered through notebook activity from a synapse pipeline, it was working fine till 27th April, execution started hanging after that for long time, when executed manually notebook executes without any issues. Can anyone help what could be the issue here ?

Pilladi Padma Sai Manisha 7,900 Reputation points Microsoft External Staff Moderator

2026-04-28T12:53:44.5233333+00:00

Hi K Kulendra Singh (Harman Connected Services Inc),
Thanks for reaching out , I understand how concerning it is when a pipeline suddenly starts hanging while the same notebook works fine manually.

Based on what you described, this is typically not related to the notebook logic itself. When a notebook is triggered from a Synapse pipeline, it runs in a new Spark session, and this can sometimes get delayed or stuck due to resource allocation or service-side conditions. [Azure syna...rosoft Q&A | Learn.Microsoft.com]

In similar scenarios, we’ve seen cases where Spark session creation is delayed or cannot get resources, and the pipeline remains in an “In Progress” state without progressing, while manual runs succeed because they start or reuse sessions differently.

I would recommend first checking the pipeline run from Monitor → Apache Spark Applications and verifying whether the job is stuck in a “Starting” or “Queued” state. If it is, this usually indicates a resource allocation or backend scheduling delay. In such cases, cancelling the current run and triggering the pipeline again often resolves the issue, especially if it is transient.

It is also a good idea to review the Spark pool capacity and current workload. If there are multiple concurrent notebooks running, the pool may be experiencing contention, which can delay new session creation. Reducing concurrency or ensuring sufficient node capacity can help avoid this behavior.

To prevent pipelines from staying stuck indefinitely, you can configure timeout and retry policies on the Notebook activity. Additionally, adding a small “fail-fast” check at the beginning of the notebook, such as running a simple Spark action, can help detect resource allocation issues early instead of waiting for a long time.

This difference in behavior between manual and pipeline execution is expected because pipeline execution depends on Spark session provisioning and scheduling, which can introduce delays even when the notebook code itself is working correctly. [Azure syna...rosoft Q&A | Learn.Microsoft.com]

For more details, you can refer to the following documentation: https://learn.microsoft.com/en-us/azure/synapse-analytics/synapse-notebook-activity https://learn.microsoft.com/en-us/azure/data-factory/data-factory-troubleshoot-guide https://learn.microsoft.com/en-us/azure/synapse-analytics/known-issues [github.com] [docs.azure.cn] [learn.microsoft.com]

Since your notebook works fine manually and only hangs when triggered via pipeline, this strongly points to a Spark session or resource allocation issue on the service side rather than a problem in the code.

If the issue continues even after retries, please share the pipeline run ID and Spark application ID so we can help investigate further.

Happy to assist further if needed.
Pilladi Padma Sai Manisha 7,900 Reputation points Microsoft External Staff Moderator

2026-04-29T12:11:03.2733333+00:00

Hi K Kulendra Singh (Harman Connected Services Inc),
I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
K Kulendra Singh (Harman Connected Services Inc) 5 Reputation points Microsoft External Staff

2026-04-29T12:51:13.96+00:00

Hi @Pilladi Padma Sai Manisha when notebook is triggered through pipeline it was not in Starting or Queued Status, it was in "Running" status, when I looked through the stages found failure related executor allocation, which got resolved after re-run & enabling dynamic allocation with 2-4 & was still facing the issue.

Normally it takes ~8 mins to complete, it was stuck for more than 7 hrs & have to cancel it.

Pipeline run id is "c43db4ea-e85a-4d7a-b231-accbc84368be"

We face similar issue for another notebook & pipeline Run Id "9b5eee0d-7fbf-4582-af12-7003ce4578ff"

later we disabled to Auto Pause setting for spark pool yesterday.

Everything started working fine from today, not sure what caused this issue.

If you can look into this, would be great help, as we keep facing this issue.