Cannot train custom ai vision model as the model hangs during training

Konrad Gałczyński 20

I am using Cognitive Service deployed in West Europe with tier S0 and would like to train my custom AI vision model for image classification. I have successfully prepared training data with Azure Machine Learning Studio and have started model training in Vision Studio with a time budget equal to 2h and 1h. Both jobs are stuck in Training state with no progress visible (one job was stuck for over 60h and the second for over 15h).

Is it a known problem? what should I do to proceed with training?
Thanks!

dupammi 8,535 Reputation points Microsoft Vendor

2024-04-30T08:46:27.9433333+00:00

Hi @Konrad Gałczyński

Thank you for the question.

I understand that you are experiencing issues while training a custom AI vision model for image classification using Cognitive Service deployed in West Europe with tier S0.

Have you checked the logs for any errors or warnings during the training process? Additionally, another suggestion would be to try training the model with a smaller dataset to see if the issue persists. Check if you already approached the Quota& limit of your resources. Also, please try training a new model and see if it works.

If the issue is still pending with the training status stuck, it might be required to check the status of your training request from the backend by a service engineer to sort the issue through Azure support channel. You can contact Microsoft Azure support for further assistance.

Thank you.
Konrad Gałczyński 20 Reputation points

2024-04-30T13:32:18.33+00:00

I have checked logs and see nothing suspicious there, I have also tried training new model on dataset with only 10 items but results are the same: taking far longer than budget and not completing but hanging.
I have also reviewed quotas and limits but I am far from reaching them.

I have found following issues reported recently:
https://learn.microsoft.com/en-us/answers/questions/1650725/custom-vision-model-with-a-budget-of-3-hours-has-b
https://learn.microsoft.com/en-us/answers/questions/1655467/training-a-custom-object-identification-model-exce

These are exactly my case but it looks like hotfix mentioned in these issues was not deployed to West Europe. Could you please help with this?
dupammi 8,535 Reputation points Microsoft Vendor

2024-04-30T14:03:51.2366667+00:00

Hi @Konrad Gałczyński

We apologize for any inconvenience caused. Yes, a similar issue has been reported by few customers. We have already contacted the product team and they were actively looking into the issue. please allow some time for this to be addressed.

Thank you.

Accepted answer

dupammi 8,535 Reputation points Microsoft Vendor

2024-05-02T06:15:37.38+00:00

Hi @Konrad Gałczyński

Apologies for the late reply. We appreciate your patience on this.

I received confirmation from the product owners that the fix for the issue you were experiencing has been deployed in West Europe. Therefore, I would like to suggest that you retry training your Custom AI Vision model for image classification in West Europe to check if it works.

Latest Update: I tried custom vision image classification in West Europe, and it worked.

Thank you.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Cannot train custom ai vision model as the model hangs during training

0 additional answers

Your answer