Custom Vision Stuck on 'Training'

Question

I cannot train any models on Custom Vision. They stay stuck on 'Training' regardless if I choose quick or advanced training, and it stays stuck even after the maximum training time. This problem started a couple days ago, and I haven't had any problems in the past training with the same data/ methods.

This is an example of the iteration details while the training job is in progress:

Iteration status: Training
Iteration Name: Iteration 1
Iteration training time (mins): 0
Iteration reserved budget time (hours): 12
Iteration training type: Advanced
Iteration classification type: None
Iteration error details: None

Yesterday I set a training job for 12 hours and it still had 0 mins for training time after 12 hours, so I had to manually delete the iteration.

I am used both the python SDK and azure custom vision GUI to start the training job, with the same results.

Answer 1

@Hazel Erickson A backend issue caused training jobs to fail or they got stuck. This issue is now fixed.
All jobs that were pending should be in failed or success state now.