Fine-tuned model stuck on deployment at Creating

CogniTech 15

We have fine-tuned and deployed a number of models now, but the one we have made today is stuck in Creating when trying to deploy, it has been stuck here for over an hour and a half and we have also deleted the deployment and tried to deploy again, same issue, it just gets stuck at creating. This is holding back our work can anyone assist with this?

CogniTech 15 Reputation points

2024-07-14T02:04:00.56+00:00

This is still ongoing for over 2 and a half hours. Can someone assist please.
CogniTech 15 Reputation points

2024-07-14T04:56:25.7633333+00:00

It is now over 5 hours and still not created. In addition we have about 20 notifications that a model is being deployed, we clear those notifications and they all come back the next time we refresh the page.
CogniTech 15 Reputation points

2024-07-14T09:19:54.6533333+00:00

The problem is here: deployment:3 Uncaught SyntaxError: Illegal return statement (at deployment:3:7) Can you please fix your UI
CogniTech 15 Reputation points

2024-07-14T09:57:42.71+00:00
This is the offending code:

https://oai.azure.com/portal/1183e2b5dcd5490a8080c95e85692c75/deployment

Can you PLEASE fix this

if (currentUrl.includes("https://localhost:")) { return; }
CogniTech 15 Reputation points

2024-07-14T20:08:03.27+00:00

Can someone please update, this is still broken
CogniTech 15 Reputation points

2024-07-15T02:46:31.9233333+00:00

Can someone please resolve this issue?
CogniTech 15 Reputation points

2024-07-15T02:49:07.64+00:00

Now there is a new error!
AshokPeddakotla-MSFT 34,111 Reputation points

2024-07-15T03:40:40.7133333+00:00

CogniTech Greetings!

Could you also share the region where deployment is failing?

I'm checking on this issue internally and update you as earliest with findings.

Appreciate your time and patience.
CogniTech 15 Reputation points

2024-07-15T05:12:19.95+00:00

Thank you, East US 2 is the only region that allows fine-tuning, all of the others always say its not possible due to heavy load.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

1 answer

AshokPeddakotla-MSFT 34,111 Reputation points

2024-07-19T12:07:38.9233333+00:00

CogniTech I have checked internally and confirm that this issue is resolved. To give more context, it happened due to the job allocated only one node resource, causing jobs that require multiple node resources for fine-tune model training to wait for resources indefinitely.

Our team has rolled back this configuration and issue is resolved.

I would suggest you, kindly check now and update if you are still seeing this issue.

Thanks for your time and patience on this issue.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Fine-tuned model stuck on deployment at Creating

1 answer

Your answer