Custom Vision model with a budget of 3 hours has been training for 5 days.

Charles Thomas 30

Several models with datasets of around 1400-1500 images and a budget of 3 hours have been training for 5 days. Is it possible to find out if something is stuck or do we simply need to wait longer?

Using the standard tier.

navba-MSFT 17,125 Reputation points Microsoft Employee

2024-04-09T05:23:31.9566667+00:00
@Charles Thomas Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

I have sent you a private message in this thread. Please share the below details:

Region in which your resource is based?

Subscription ID

Azure Custom Vision resource URI in below format:
/subscriptions/XXXXXXX/resourceGroups/XXXX/providers/Microsoft.CognitiveServices/accounts/XXXXXX

Type of Project ? Is it Object Detection ? Project name: Project ID: Training Name:

Once I get these details from you, I will be checking internally for any known issues, if more details are required we will reach out to you for more information.
Charles Thomas 30 Reputation points

2024-04-09T19:40:52.9666667+00:00

Trying to delete.
JC-Vision 0 Reputation points

2024-04-09T23:19:02.94+00:00

Hello,

I am also having a similar issue. I am unable to train any new custom vision AI models using Vision Studio. I was able to successfully train two custom models before Friday April 5, 2024. However, since then I have attempted to train three custom models and all of them have stayed in "Training" for way past the specified 1-hour training session. This includes one custom model that was stuck in training for three days.

I am using the free-tier subscription, so I am aware that I have limited capabilities, but I have not come across a resource that explains why my models are not training successfully. Any help is appreciated. Thank you. I am writing this as a comment because I am unable to ask a question due to receiving, "We encountered an unexpected error. Please try again later. If this issue continues, please contact site support." each time I try to submit my question.
JC-Vision 0 Reputation points

2024-04-09T23:22:16.35+00:00

Test Comment
Charles Thomas 30 Reputation points

2024-04-10T01:15:24.22+00:00

@JC-Vision That's interesting that you said you were successful before Friday April 5. That's roughly around the time when we experienced problems as well. Prior to that we succesfully trained 7 models which all seem to work well. The training was also completed within the expected time range.
navba-MSFT 17,125 Reputation points Microsoft Employee

2024-04-10T06:17:47.7666667+00:00

@Charles Thomas Thanks for sharing the details over private message. I have asked for some more details. Please reply once you get a chance.
TimboBaggins 25 Reputation points

2024-04-11T20:50:23.0533333+00:00

Hi All,

Just to add to this, I am also experiencing the same issue.. I am happy to read your comments, as it's been driving me mad! I have a relatively small dataset, 55 training images, approx 200 products (its a product recogniser model) I have been trying to training with a budget of 16hours and its been going now for over 24! The coco file has been verified and I have successfully trained other models using the same process.

I will raise a separate question on this, hopefully Microsoft will address and fix the issue with training!
Charles Thomas 30 Reputation points

2024-04-11T21:10:43.8866667+00:00

Will post updates here when I get them. Currently they are looking at the back end. No errors in the logs.
navba-MSFT 17,125 Reputation points Microsoft Employee

2024-04-12T02:40:57.1766667+00:00

@Charles Thomas Many customers have reported this issue. I have reached out to the Product Owners in the background and they are looking into this. I will keep you posted once I hear from them.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Charles Thomas 30 Reputation points

2024-04-12T16:23:01.0533333+00:00

A workaround that was provided by a support engineer: This problem does not seem to exist in the West US 2 region. If you create a Vision resource in that region, models should train normally. I haven't completed one yet but am just now starting a model in that region.
TimboBaggins 25 Reputation points

2024-04-12T18:11:23.3266667+00:00

Thank you Charles, that's a great tip.. I will try that and report back here if any success. Appreciate you sharing this, I have heard nothing yet from support or product team.
navba-MSFT 17,125 Reputation points Microsoft Employee

2024-04-16T06:18:18.1133333+00:00

@TimboBaggins @Charles Thomas Apologies for the delay in getting back. The issue is still being investigated. The Product group team is working on this on priority. I will update this thread once we have some progress done.
TimboBaggins 25 Reputation points

2024-04-16T10:09:50.5133333+00:00

@Charles Thomas

@navba-MSFT

Thanks both.. I can confirm that models trained on resources in West US 2 work fine and complete training as expected. I have now trained 2 new models using a new Vision resource and storage account based in West US 2. These have completed training and are now in use. Thank you @Charles Thomas for your suggestion!

@navba-MSFT Please keep us updated as to why this fails in East US region? As this is the region advised to be used in all the Vision documentation.
TimboBaggins 25 Reputation points

2024-04-16T10:12:51.36+00:00

Thanks both for your help. I can confirm that using resources in West US 2 works fine. I created a new resource group, vision resource and storage account all in West US 2 and have completed training 2 new models. these completed training as expected and are now in use.

Please keep us updated on the issue in East US.. thank you
navba-MSFT 17,125 Reputation points Microsoft Employee

2024-04-25T05:32:53.4733333+00:00

@Charles Thomas Just following up to check if the below answer helped. If that answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let me know. I would be happy to help.

Accepted answer

navba-MSFT 17,125 Reputation points Microsoft Employee

2024-04-23T04:48:58.51+00:00
@Charles Thomas @TimboBaggins Apologies for the late reply. We appreciate your patience on this.

I have got an update from the Product Owners on this. Here is the root cause analysis and mitigation taken to fix this issue.

Cause:

The retry logic had a bug that, sometimes when the training job fails, the retry counter is reset so the job will be infinitely retried.

Mitigation:

We have fixed the retry counter to catch that kind of failure and stop retrying.

A hotfix has been deployed.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

**

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

1 person found this answer helpful.
navba-MSFT 17,125 Reputation points Microsoft Employee

2024-04-25T05:33:16.94+00:00

@Charles Thomas Just following up to check if the above answer helped. If that answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let me know. I would be happy to help.

Konrad Gałczyński 0 Reputation points

2024-04-30T13:25:41.01+00:00

I am experiencing the same problem in West Europe even on very small datasets
Sign in to comment

Custom Vision model with a budget of 3 hours has been training for 5 days.

0 additional answers