Share via

no way to stop when an ADF/Synapse pipeline goes into endless loop

Felix Huang 1 Reputation point
2022-12-14T02:44:40.67+00:00

Short question is that how to stop it when a pipeline goes endless loop (EG a bug in the code).

this is easy if the pipeline is run under debug mode. You can just click the "Cancel Debug" button. things are different when the pipeline is run on trigger mode. that is the pipeline is kicked off by a trigger. Yesterday I had an incident that a pipeline that is kicked off by a blobevents trigger that caused chain reaction for more triggered pipelines per minute that would never stop. So the simple question is that is there any way to stop/reset Azure data factory (or Synapse) when something goes terribly wrong?

A bit more details about the incident. Basically a pipeline is configured to be triggered by a blobevents trigger. that is, the pipeline will run if a blob storage folder has a new blob / file added. But the part of the pipeline run due to wrong configuration, a dozen more files generated on the same folder. Therefore trigger a dozen more running instances of the pipeline. You can imagine one instance triggers 12 instances then each of the instance will trigger another 12 instances. You can see that from the pipeline runs screen. though from that screen, I chose all pipelines and clicked cancel, there were still more pipeline instances kept being added to the queue every single minute. I did refresh the screen and kept cancelling the instances many times. no luck.

I did try delete the whole storage account / folder. but still no stopping. Here comes the silly question. There should be a decent way to stop it right? But ADF (and Synapse), not like SSIS service or SQL server service that can be stopped and reboot, it does not seem to have a concept that it can be stopped or reset.

Any advice will be greatly appreciated.

Azure Synapse Analytics
Azure Synapse Analytics

An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

{count} votes

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.