quick training stuck for hours in custom vision

Question

quick training stuck for hours in custom vision

Peeyush Tuli 1

Hi, the quick training on a classification project is stuck for hours, usually it completes in minutes. I have no option to cancel it. Is there any way to troubleshoot this? Thanks.

Rohit Mungi 49,216 Reputation points Microsoft Employee Moderator

2021-07-26T07:03:52.907+00:00

@Peeyush Tuli Is the issue still pending with the training status stuck? If Yes, then it might be required to check the status of your training request from backend by a service engineer to sort the issue through Azure support channel.
Peeyush Tuli 1 Reputation point

2021-07-26T11:07:29.453+00:00

@Rohit Mungi the training status is still stuck on that project. I took help of APIs to delete the stuck iteration and then train again, but it would get stuck each time. Finally i had to export and import to a new project(using API calls) where it has finally worked again. I think the custom vision UI should have these features. If its open source, i would be more than happy to contribute as it seems like angular.
Hazel Erickson 1 Reputation point

2021-07-26T22:40:44.953+00:00

I am having the same problem, I cannot train any new models on Custom Vision. They stay on 'Training' regardless if I choose quick or advanced training, and it stays stuck even after the maximum training time.

For example, this is some of the iteration details when the training job has been in progress for over 30 mins:
Iteration status: Training
Iteration Name: Iteration 1
Iteration training time (mins): 0
Iteration reserved budget time (hours): 12
Iteration training type: Advanced
Iteration classification type: None
Iteration error details: None

Yesterday I set a training job for 12 hours and it still had 0 mins for training time after 12 hours, so I had to manually delete the iteration.

2 answers

Your answer

Rohit Mungi 49,216 Reputation points Microsoft Employee Moderator

2021-07-26T07:03:52.907+00:00

@Peeyush Tuli Is the issue still pending with the training status stuck? If Yes, then it might be required to check the status of your training request from backend by a service engineer to sort the issue through Azure support channel.
Peeyush Tuli 1 Reputation point

2021-07-26T11:07:29.453+00:00

@Rohit Mungi the training status is still stuck on that project. I took help of APIs to delete the stuck iteration and then train again, but it would get stuck each time. Finally i had to export and import to a new project(using API calls) where it has finally worked again. I think the custom vision UI should have these features. If its open source, i would be more than happy to contribute as it seems like angular.
Hazel Erickson 1 Reputation point

2021-07-26T22:40:44.953+00:00

I am having the same problem, I cannot train any new models on Custom Vision. They stay on 'Training' regardless if I choose quick or advanced training, and it stays stuck even after the maximum training time.

For example, this is some of the iteration details when the training job has been in progress for over 30 mins:
Iteration status: Training
Iteration Name: Iteration 1
Iteration training time (mins): 0
Iteration reserved budget time (hours): 12
Iteration training type: Advanced
Iteration classification type: None
Iteration error details: None

Yesterday I set a training job for 12 hours and it still had 0 mins for training time after 12 hours, so I had to manually delete the iteration.

Answer 1

Gutierrez, Gabriel 1

This issue is happening today. Started a quick training and it's been more than 6 hours that is running. There should be an option to cancel ongoing trainings/

0 comments

Answer 2

Rohit Mungi 49,216 Microsoft Employee Moderator

@Peeyush Tuli A backend issue caused training jobs to fail or they got stuck. This issue is now fixed.
All jobs that were pending should be in failed or success state now. Thanks!!

0 comments