Fine-tuned model stuck on deployment at Creating

Question

Fine-tuned model stuck on deployment at Creating

CogniTech 20

We have fine-tuned and deployed a number of models now, but the one we have made today is stuck in Creating when trying to deploy, it has been stuck here for over an hour and a half and we have also deleted the deployment and tried to deploy again, same issue, it just gets stuck at creating. This is holding back our work can anyone assist with this?

CogniTech 20 Reputation points

2024-07-14T02:04:00.56+00:00

This is still ongoing for over 2 and a half hours. Can someone assist please.
CogniTech 20 Reputation points

2024-07-14T04:56:25.7633333+00:00

It is now over 5 hours and still not created. In addition we have about 20 notifications that a model is being deployed, we clear those notifications and they all come back the next time we refresh the page.
CogniTech 20 Reputation points

2024-07-14T09:19:54.6533333+00:00

The problem is here: deployment:3 Uncaught SyntaxError: Illegal return statement (at deployment:3:7) Can you please fix your UI
CogniTech 20 Reputation points

2024-07-14T09:57:42.71+00:00
This is the offending code:

https://oai.azure.com/portal/1183e2b5dcd5490a8080c95e85692c75/deployment

Can you PLEASE fix this

if (currentUrl.includes("https://localhost:")) { return; }
CogniTech 20 Reputation points

2024-07-14T20:08:03.27+00:00

Can someone please update, this is still broken
CogniTech 20 Reputation points

2024-07-15T02:46:31.9233333+00:00

Can someone please resolve this issue?
CogniTech 20 Reputation points

2024-07-15T02:49:07.64+00:00

Now there is a new error!
AshokPeddakotla-MSFT 36,021 Reputation points Moderator

2024-07-15T03:40:40.7133333+00:00

CogniTech Greetings!

Could you also share the region where deployment is failing?

I'm checking on this issue internally and update you as earliest with findings.

Appreciate your time and patience.
CogniTech 20 Reputation points

2024-07-15T05:12:19.95+00:00

Thank you, East US 2 is the only region that allows fine-tuning, all of the others always say its not possible due to heavy load.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

1 answer

Your answer

CogniTech 20 Reputation points

2024-07-14T02:04:00.56+00:00

This is still ongoing for over 2 and a half hours. Can someone assist please.
CogniTech 20 Reputation points

2024-07-14T04:56:25.7633333+00:00

It is now over 5 hours and still not created. In addition we have about 20 notifications that a model is being deployed, we clear those notifications and they all come back the next time we refresh the page.
CogniTech 20 Reputation points

2024-07-14T09:19:54.6533333+00:00

The problem is here: deployment:3 Uncaught SyntaxError: Illegal return statement (at deployment:3:7) Can you please fix your UI
CogniTech 20 Reputation points

2024-07-14T09:57:42.71+00:00

This is the offending code:

https://oai.azure.com/portal/1183e2b5dcd5490a8080c95e85692c75/deployment

Can you PLEASE fix this

if (currentUrl.includes("https://localhost:")) { return; }
CogniTech 20 Reputation points

2024-07-14T20:08:03.27+00:00

Can someone please update, this is still broken
CogniTech 20 Reputation points

2024-07-15T02:46:31.9233333+00:00

Can someone please resolve this issue?
CogniTech 20 Reputation points

2024-07-15T02:49:07.64+00:00

Now there is a new error!
AshokPeddakotla-MSFT 36,021 Reputation points Moderator

2024-07-15T03:40:40.7133333+00:00

CogniTech Greetings!

Could you also share the region where deployment is failing?

I'm checking on this issue internally and update you as earliest with findings.

Appreciate your time and patience.
CogniTech 20 Reputation points

2024-07-15T05:12:19.95+00:00

Thank you, East US 2 is the only region that allows fine-tuning, all of the others always say its not possible due to heavy load.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Answer 1

AshokPeddakotla-MSFT 36,021 Moderator

CogniTech I have checked internally and confirm that this issue is resolved. To give more context, it happened due to the job allocated only one node resource, causing jobs that require multiple node resources for fine-tune model training to wait for resources indefinitely.

Our team has rolled back this configuration and issue is resolved.

I would suggest you, kindly check now and update if you are still seeing this issue.

Thanks for your time and patience on this issue.

0 comments

Share via

Fine-tuned model stuck on deployment at Creating

1 answer

Your answer