pipelines stuck on preparing stage forever.

Antara Das 376

Unable to execute pipelines on the STANDARD_D16S_V3. It is stuck on the prepare stage forever. But the same pipeline executed without trouble 2 weeks ago.
Also now AutoML are stuck on 'Not Started' stage.
The compute instances and compute clusters were again recreated, just to ensure if there were some issues with our previous computes. We have faced the issue where suddenly the computes have slowed down but this time they just dont get ahead of the prepare stage.
To be noted, this occurs only in case of pipelines and automl.
At the same time we are getting billed for the compute resources.
What can be possibly wrong here?

romungi-MSFT 43,696 Reputation points Microsoft Employee

2021-09-02T06:19:45.597+00:00

@Antara Das I think in the prep stage the snapshot download might be an issue in a pipeline. You could compare the logs of current run and runs couple of weeks ago to confirm this.
To get a better understanding of the slowness in your subscription I would recommend to raise a support issue from Azure portal or post feedback from Azure ML portal from the top right hand corner by using the smiley icon.

Share via

pipelines stuck on preparing stage forever.