Training a custom object identification model exceeding the set time budget

Question

I labeled my training images from the vision studio and uploaded them to a blobstorage, then tried to train a custom object identification model using the coco file. However, training time budget set to 1 hour was exceeded and training seems to be taking forever. I have attempted it multiple times, but with no success.

I followed this tutorial from Microsoft Learn: https://microsoftlearning.github.io/mslearn-ai-vision/Instructions/Exercises/02-image-classification.html

This is the dashboard of my custom models for reference:

Answer 1

@Nair, Indrajith (Allianz Technology) Apologies for the late reply. We appreciate your patience on this.

I have got an update from the Product Owners on this. Here is the root cause analysis and mitigation taken to fix this issue.

Cause:

The retry logic had a bug that, sometimes when the training job fails, the retry counter is reset so the job will be infinitely retried.

Mitigation: 

1. We have fixed the retry counter to catch that kind of failure and stop retrying.
2. A hotfix has been deployed.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

**

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.