Orphaned Azure ML jobs

Question

I've just noticed that, in one of our Azure ML workspaces, we have about 20 jobs listed in Running status for over 60 days. They clearly aren't actually running - they're not using any compute resources, so I think this is really just a nuisance, but - if I try to cancel them, they say they're in a terminal state and can't be cancelled; if I try to delete them, they say they're running and can't be deleted. They're listed as "Active Runs" on the compute resource they theoretically WOULD have used, but as I said, they aren't actually causing any nodes to be spun up. Any advice on how to clean up these orphaned jobs? Thanks!

Answer 1

Hello @Gabe Mangiante

Thanks for the clear write-up and you're right that the two contradictory errors are the interesting part. They're the signature of a stale record rather than a live job.

There's a published limit that settles this: Azure ML enforces a maximum job lifetime of 21 days (7 days for jobs on low-priority nodes), and the documentation notes that "data for jobs not completed within the maximum lifetime isn't accessible." Your jobs are showing Running at 60+ days roughly three times past that ceiling. Whatever they were doing, the service itself no longer considers them live work.

That's consistent with everything else you're seeing: the cancel path evaluates the job as already terminal and refuses; the delete path reads the last-written status of Running and refuses. Two checks, two different answers, and no actual compute behind either. Your read that this is a display/metadata nuisance rather than a running workload is correct.

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-quotas?view=azureml-api-2

Worth trying first:- CLI rather than the studio

The CLI occasionally succeeds where the portal refuses, so it's a cheap thing to rule out:

Please try this on one job first and send me the exact error text and any correlation/request ID it returns — that detail materially speeds up the backend investigation.

Two notes so you don't lose time on dead ends:

• Archiving won't help here. az ml job archive operates on jobs in a terminal state, so it will most likely return the same conflict. Worth thirty seconds to confirm, not worth working around.
• Please ignore older forum guidance built on SDK/CLI v1 (run.complete(), run.fail(), and similar). SDK v1 reached end of support on 30 June 2026 and CLI v1 on 30 September 2025, so those workarounds no longer apply.

If the CLI returns the same conflict

Azure ML doesn't expose a customer-facing force-terminate or force-delete for jobs stuck in a non-terminal state — the control plane is waiting on a termination signal that will never arrive. Clearing the record has to be done server-side.

I can raise that for you directly from this case. Please send:

• Subscription ID, resource group, workspace name, and region
• The affected job names / run IDs (the full list of ~20 is fine)
• The compute target name they're attributed to
• The exact error text from the CLI attempt above

One thing worth checking in the meantime

Since they're showing under Active Runs on the compute target, please confirm the cluster is genuinely at zero nodes (Compute → your cluster → Nodes). If node count is 0 and new jobs are scheduling normally, this is cosmetic and safe to leave until we clear it — and you're not being billed for compute, since no nodes are allocated. If instead you see the cluster refusing to scale down, or new jobs queueing behind these phantom entries, tell me straight away and I'll treat it as impacting rather than cosmetic.

For the future

Since the 21-day ceiling is what these jobs sailed past, adding explicit termination conditions prevents a repeat:

• Set a job-level timeout in your job YAML
• For AutoML, use timeout and maxConcurrentTrials in limits
• For sweep jobs, configure an early-termination policy (bandit / median-stopping)

Try the CLI cancel when you get a moment and send over the details above. I'll take it from there.

Thanks,
Manish.