Quick Training on Azure Custom Vision has not been updated since 3 days

Sanjit Verma 20

I am working on an Object detection project and I have been labeling images. I completed 4 iterations and the performance was gradually increasing as I kept adding images. But, on the 5th iteration..the performance page got stuck and has not been updated since the past 4 days. Could someone please help me with the same?

Thanks in advance.

Regards,
Sanjit H. Verma

navba-MSFT 24,905 Reputation points Microsoft Employee

2024-08-13T02:19:42.8266667+00:00
@Sanjit Verma Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

.

I have sent you a private message in this thread. Please share the below details:

Region in which your resource is based?

Resource Tier (free or S0) ?

Subscription ID

Azure Custom Vision resource URI in below format: /subscriptions/XXXXXXX/resourceGroups/XXXX/providers/Microsoft.CognitiveServices/accounts/XXXXXX

Type of Project ? Is it Object Detection ? Project name: Project ID: Training Name:

.

Once I get these details from you, I will be checking internally for any known issues, if more details are required we will reach out to you for more information.
Sanjit Verma 20 Reputation points

2024-08-13T02:35:07.65+00:00

Hello @navba-MSFT ,
Thank you for your response.
Here are the details:
The region for my resource is: East US
Pricing tier is Free
Subscription ID is:11461231-eb8c-4ba7-86d6-650f2df24d1d

Resource URI: /subscriptions/11461231-eb8c-4ba7-86d6-650f2df24d1/resourceGroups/OpenAIPOCresourcegroup/providers/Microsoft.CognitiveServices/accounts/PriceTagObserva

Type of Project: Object Detection
Project Name is: PriceTag
Project ID: 50749b4d-cfa3-4600-9fda-1cfb5d396471

Training Name: Iteration 5
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-08-14T04:04:55.23+00:00

@Sanjit Verma Thanks for sharing the details. I am checking this internally in our backend logs and get back once I have an update.
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-08-19T05:26:23.2066667+00:00

@Sanjit Verma I have involved the Product Owners and they are looking into this. I will get back to you once I hear from them.
Sanjit Verma 20 Reputation points

2024-08-19T19:28:31.27+00:00

@navba-MSFT Thank you for all your support. Please let me know as soon as you have an update as this is an essential project.
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-08-20T04:04:46.67+00:00

@Sanjit Verma Apologies for the late reply. I have heard back from the Product Owners.

Cause:

We have found that due to an internal failure the job was not able to complete.

.

Mitigation:
We are currently working to put the training in a failed state so that you will become unblocked.

ETA should be end of day tomorrow.

.

New Update

We have failed the iteration since there was an issue from our side. Please note you won't incur the billing for the failed iteration. Please feel free to start a new training.

Hope this helps.

**

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Sanjit Verma 20 Reputation points

2024-08-22T01:15:57.95+00:00

@navba-MSFT Thank you so much for resolving the issue. However, I am facing the same issue again. Iteration 5 was shown as failed and I could train the model for iteration 6. but, as I added more images and started iteration 7, it is again stuck since almost a day
Sanjit Verma 20 Reputation points

2024-08-22T01:19:25.9433333+00:00

Hello @navba-MSFT

Thank you so much for your help. However, I am facing the same issue again. After iteration 5, I could successfully move to iteration 6 but as I kept on adding more images and trained model for iteration 7..it has not been updated since yesterday. Please help me with the same. Also, can you tell me what could be done to avoid such errors?
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-08-22T04:37:41.3433333+00:00

@Sanjit Verma Thanks for getting back. I have shared these details again with the Product Owners and following up with them to unblock you.
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-08-23T02:10:18.1933333+00:00

@Sanjit Verma I believe you might be reaching the limit for free tier.

https://azure.microsoft.com/en-us/pricing/details/cognitive-services/custom-vision-service/

Could you please try to upgrade to the Standard tier and then test ?
Sanjit Verma 20 Reputation points

2024-08-23T03:36:00.0133333+00:00

Hi @navba-MSFT thanks a lot for your response. Could I switch the tier for an ongoing project too? and how would I stop training even after switching?
Sanjit Verma 20 Reputation points

2024-08-26T18:06:28.51+00:00

Hey @navba-MSFT ,
I changed the Tier from Free to Standard but my project is still stuck on Iteration 7.
Could you please unlock this for me? This is an essential project and these errors are causing a lot of inconvenience as the page/training gets stuck for days.
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-08-27T03:18:48.2633333+00:00

@Sanjit Verma Thanks for your reply.

It could be that you had exceeded the free tier limit which ends up with a hanging training. Now post upgrading to the Standard tier, I am hoping such issues shouldn't occur.

I have shared these details and asked the Product Owners to mark that iteration 7 as failed.

Please note, We would likely to require some time for a thorough investigation given the current bandwidth limitation.
Sanjit Verma 20 Reputation points

2024-09-03T20:55:56.25+00:00

Hi @navba-MSFT ,
I still have no updates on the web page..Iteration 7 is still stuck and I am not able to train the model further. This is indeed very disappointing and is making it tough to progress with the Custom Vision Model.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-09-04T06:45:11.1766667+00:00

@Sanjit Verma Please accept our sincere apologies for the delay in getting to the root of this. The Product Owners were having bandwidth limitations and were held up on other high severity production down issues. Post raising the severity of this issue, this issue was looked into on priority.

.

Current status:

With the help of Product Owners, the iteration 7 has been marked as failed. This should unblock you.

.

Hope this helps.

**

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-09-05T03:33:46.6166667+00:00

@Sanjit Verma Just a quick follow-up to check if you had a chance to look at my below comment and test it again.
Sanjit Verma 20 Reputation points

2024-09-05T20:11:53.7166667+00:00

@navba-MSFT Hello, Thank you so much for your help. The iteration 7 has been assigned as failed now. I will go ahead and start the training process again.
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-09-09T04:07:51.37+00:00

@Sanjit Verma A quick follow-up to check if you had a chance to look at my below findings. Hope this helps. Please do not forget to "Accept the answer” and “up-vote” if the below information provided helps you, this can be beneficial to other community members.

Accepted answer

navba-MSFT 24,905 Reputation points Microsoft Employee

2024-09-06T02:47:29.93+00:00

@Sanjit Verma Thanks for the update.

More information about the issue:

I looked deeper into the diagnostic logs. Your iteration 5 failed while 6 succeeded and then 7 failed. All trainings happen with ProductOnShelves domain. The difference is that you labeled images with

[Auto-Generated] Other Products within 5 and 7. This is the reason training failed. The service should actually reject this kind of label request, while to mitigate the issue, please don't label images with

[Auto-Generated] Other Products.

Please let me know the training process goes.

Hope this helps.

**

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
navba-MSFT 24,905 Reputation points Microsoft Employee

2024-09-09T04:07:10.9233333+00:00

@Sanjit Verma A quick follow-up to check if you had a chance to look at my above findings. Hope this helps. Please do not forget to "Accept the answer” and “up-vote” if the above information provided helps you, this can be beneficial to other community members.

Sanjit Verma 20 Reputation points

2024-09-09T23:24:17.78+00:00

Hi @navba-MSFT
Thank you very much for your support. Your advice on not having [Auto-Generated] Other Products as labels works and I am able to train the models without any issues.
Thank you once again!
Yours sincerely,

Sanjit H. Verma
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Quick Training on Azure Custom Vision has not been updated since 3 days

0 additional answers

Your answer