Failed Status Custom Model Deployment

Question

Failed Status Custom Model Deployment

Eric Halsey 26

User's image

Adding training and validation JSONL works fine. Errors after about 5 min when deploying the model. Can't seem to find much detail for the error. Will attempt to try model deployment via the CLI to see if it works that way. User's image

Ramr-msft 17,826 Reputation points

2023-04-17T10:37:38.1666667+00:00

@Anonymous Thanks, Can you please add more details about the region that you are using for the finetune. Please use a resource in a region which supports fine-tune of the Azure OpenAI model. For fine tuning please check the supported regions in the following link: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability
Eric Halsey 26 Reputation points

2023-04-17T20:54:49.43+00:00

@Ramr-msft thank you for your quick response. RG is East US.

Base model is curie
Wang, Zhijun 5 Reputation points

2023-04-18T08:34:22.67+00:00

I have the same issue after a long time waiting from deployment.
Nanthini Ganesan 20 Reputation points

2023-04-18T15:02:17.57+00:00
Hi team,

Facing the similar issue, Model trained successfully but the deployment gets failed. Please help us to resolve the issue, it would be helpful.

Tried babbage and curie models for fine tuning

Region used - East US
Michelle-0463 20 Reputation points

2023-04-18T15:50:25.28+00:00

I am also facing the same issue. Has any progress been made on this? The custom model has been train successfully and sits within the US East region so it is deployable, but when I try to deploy it fails like described above.
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-18T18:30:56.6466667+00:00

Hello everyone, this seems to be a regional issue for deployment and we have escalated this issue to product team for investigation and fix. We are sorry for all inconveniences. Will update here if there is any news I get for product team. Thanks for your understanding. Regards, Yutong
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-18T19:11:31.56+00:00

Hello @Michelle-0463 @Nanthini Ganesan @Wang, Zhijun @Anonymous

Thanks for your reporting, I just got an update that this issue has been fixed, could you please retry again? Please let me know if you still see this issue.

Sorry for all inconveniences and thanks for your supporting again.

Regards, Yutong
Michelle-0463 20 Reputation points

2023-04-18T19:22:53.81+00:00

@YutongTie-MSFT I just tried deploying my model and its still failing.
Eric Halsey 26 Reputation points

2023-04-18T20:28:54.0933333+00:00

@YutongTie-MSFT same for me. Did not resolve the issue.
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-18T20:30:35.1233333+00:00

Thanks for your confirmation, we have informed product team for further investigation. Will let you know once we get any news.
Wang, Zhijun 5 Reputation points

2023-04-19T01:14:43.28+00:00

Still error occur with shorter time waiting than last time.
BTW, could not find any information about the error, could the product show some more information to user while error happened?
Ramr-msft 17,826 Reputation points

2023-04-19T06:58:37.56+00:00

@Anonymous @Wang, Zhijun @Michelle Morales @Nanthini Ganesan Thanks for the details, we ran out of GPU quota so you might be failing now, GPUs might be freed by other customers then you would be able to create the deployments. Will update further on this case.
John - Robocore 0 Reputation points

2023-04-19T07:39:55.6966667+00:00

Hi, we have the same issue. just want to know where can you get the statusMessage?
Wang, Zhijun 5 Reputation points

2023-04-19T08:35:35.14+00:00

When will be fine for this issue? Thanks.
Ekin Olkan 0 Reputation points

2023-04-19T08:59:26.3633333+00:00
Hello everyone, I am also getting the same error. The model trained succesfully. But when I try to deploy the model its failed. Also, I tried to deploy use AzureOpenAI portal and its give me the same error.

The other thing is the error message not clear. Can you also provide more specific message or error code rather than this message below. Thank you,

Checking for deployment status. Deployment deployment_ID is with status: failed
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-19T19:34:31.6566667+00:00

Hello @Michelle Morales @Nanthini Ganesan @Wang, Zhijun @Eric Halsey @Ekin Olkan @John - Robocore Thanks for reaching out to us again, the issue has been identified as capacity issue in East US; currently fine tuning jobs are failing due to lack of capacity. the team is working to open up west Europe and south central regions for using. We will let you know once the opening has been done soon. Regards, Yutong
Michelle-0463 20 Reputation points

2023-04-20T13:43:51.4966667+00:00

@YutongTie-MSFT Thank you!
Wang, Zhijun 5 Reputation points

2023-04-21T03:16:23.7466667+00:00

Hello @Michelle Morales , are you deploy successfully?
Michelle-0463 20 Reputation points

2023-04-21T09:30:58.1633333+00:00

@Wang, Zhijun no, deployment is still failing.
Shaaz 0 Reputation points

2023-04-21T14:44:29.2166667+00:00

I have been trying to deploy finetuned models with Ada and Babbage and the service looks like hit the capacity? Deployment Keep Failing even though the training was successful. Please advise.
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-21T17:19:53.81+00:00

Hello everyone @Michelle Morales @Nanthini Ganesan @Wang, Zhijun @Eric Halsey @Ekin Olkan @John - Robocore @Shaaz Quick update - West Europe and South Central US is usable now since East US is running out of GPU. I will post a short announcement later, I hope this helps.

Brian Anderson 0

My deployment of a customized model is failing this morning in region S. Central US.
Model is based on ada. The model customization succeeded and this is the first deploy attempt.
Error message is unhelpful.

"properties": {
        "statusMessage": "{\"status\":\"Failed\",\"error\":{\"code\":\"ResourceOperationFailure\",\"message\":\"The resource operation completed with terminal provisioning state 'Failed'.\",\"details\":[{\"code\":\"InternalServerError\",\"message\":\"Internal Server Error.\"}]}}",
        "eventCategory": "Administrative",

Accepted answer

1 additional answer

Your answer

Ramr-msft 17,826 Reputation points

2023-04-17T10:37:38.1666667+00:00

@Anonymous Thanks, Can you please add more details about the region that you are using for the finetune. Please use a resource in a region which supports fine-tune of the Azure OpenAI model. For fine tuning please check the supported regions in the following link: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability
Eric Halsey 26 Reputation points

2023-04-17T20:54:49.43+00:00

@Ramr-msft thank you for your quick response. RG is East US.

Base model is curie
Wang, Zhijun 5 Reputation points

2023-04-18T08:34:22.67+00:00

I have the same issue after a long time waiting from deployment.
Nanthini Ganesan 20 Reputation points

2023-04-18T15:02:17.57+00:00

Hi team,

Facing the similar issue, Model trained successfully but the deployment gets failed. Please help us to resolve the issue, it would be helpful.

Tried babbage and curie models for fine tuning

Region used - East US
Michelle-0463 20 Reputation points

2023-04-18T15:50:25.28+00:00

I am also facing the same issue. Has any progress been made on this? The custom model has been train successfully and sits within the US East region so it is deployable, but when I try to deploy it fails like described above.
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-18T18:30:56.6466667+00:00

Hello everyone, this seems to be a regional issue for deployment and we have escalated this issue to product team for investigation and fix. We are sorry for all inconveniences. Will update here if there is any news I get for product team. Thanks for your understanding. Regards, Yutong
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-18T19:11:31.56+00:00

Hello @Michelle-0463 @Nanthini Ganesan @Wang, Zhijun @Anonymous

Thanks for your reporting, I just got an update that this issue has been fixed, could you please retry again? Please let me know if you still see this issue.

Sorry for all inconveniences and thanks for your supporting again.

Regards, Yutong
Michelle-0463 20 Reputation points

2023-04-18T19:22:53.81+00:00

@YutongTie-MSFT I just tried deploying my model and its still failing.
Eric Halsey 26 Reputation points

2023-04-18T20:28:54.0933333+00:00

@YutongTie-MSFT same for me. Did not resolve the issue.
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-18T20:30:35.1233333+00:00

Thanks for your confirmation, we have informed product team for further investigation. Will let you know once we get any news.
Wang, Zhijun 5 Reputation points

2023-04-19T01:14:43.28+00:00

Still error occur with shorter time waiting than last time.
BTW, could not find any information about the error, could the product show some more information to user while error happened?
Ramr-msft 17,826 Reputation points

2023-04-19T06:58:37.56+00:00

@Anonymous @Wang, Zhijun @Michelle Morales @Nanthini Ganesan Thanks for the details, we ran out of GPU quota so you might be failing now, GPUs might be freed by other customers then you would be able to create the deployments. Will update further on this case.
John - Robocore 0 Reputation points

2023-04-19T07:39:55.6966667+00:00

Hi, we have the same issue. just want to know where can you get the statusMessage?
Wang, Zhijun 5 Reputation points

2023-04-19T08:35:35.14+00:00

When will be fine for this issue? Thanks.
Ekin Olkan 0 Reputation points

2023-04-19T08:59:26.3633333+00:00

Hello everyone, I am also getting the same error. The model trained succesfully. But when I try to deploy the model its failed. Also, I tried to deploy use AzureOpenAI portal and its give me the same error.

The other thing is the error message not clear. Can you also provide more specific message or error code rather than this message below. Thank you,

Checking for deployment status. Deployment deployment_ID is with status: failed
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-19T19:34:31.6566667+00:00

Hello @Michelle Morales @Nanthini Ganesan @Wang, Zhijun @Eric Halsey @Ekin Olkan @John - Robocore Thanks for reaching out to us again, the issue has been identified as capacity issue in East US; currently fine tuning jobs are failing due to lack of capacity. the team is working to open up west Europe and south central regions for using. We will let you know once the opening has been done soon. Regards, Yutong
Michelle-0463 20 Reputation points

2023-04-20T13:43:51.4966667+00:00

@YutongTie-MSFT Thank you!
Wang, Zhijun 5 Reputation points

2023-04-21T03:16:23.7466667+00:00

Hello @Michelle Morales , are you deploy successfully?
Michelle-0463 20 Reputation points

2023-04-21T09:30:58.1633333+00:00

@Wang, Zhijun no, deployment is still failing.
Shaaz 0 Reputation points

2023-04-21T14:44:29.2166667+00:00

I have been trying to deploy finetuned models with Ada and Babbage and the service looks like hit the capacity? Deployment Keep Failing even though the training was successful. Please advise.
YutongTie-MSFT 53,966 Reputation points Moderator

2023-04-21T17:19:53.81+00:00

Hello everyone @Michelle Morales @Nanthini Ganesan @Wang, Zhijun @Eric Halsey @Ekin Olkan @John - Robocore @Shaaz Quick update - West Europe and South Central US is usable now since East US is running out of GPU. I will post a short announcement later, I hope this helps.
Brian Anderson 0 Reputation points

2023-05-08T16:52:42.9633333+00:00

My deployment of a customized model is failing this morning in region S. Central US.
Model is based on ada. The model customization succeeded and this is the first deploy attempt.
Error message is unhelpful.

"properties": { "statusMessage": "{\"status\":\"Failed\",\"error\":{\"code\":\"ResourceOperationFailure\",\"message\":\"The resource operation completed with terminal provisioning state 'Failed'.\",\"details\":[{\"code\":\"InternalServerError\",\"message\":\"Internal Server Error.\"}]}}", "eventCategory": "Administrative",

Answer 1

YutongTie-MSFT 53,966 Moderator

Hello everyone @Michelle Morales @Nanthini Ganesan @Wang, Zhijun @Eric Halsey @Ekin Olkan @John - Robocore @Shaaz

Thank you for your patience. I have received an update from the product team regarding the issue you experienced. It appears that the root cause of the problem was a temporary shortage of available GPUs in the EastUS region. However, the team is actively working to free up resources, when there is one freed up their GPU, new deployments possible again.

We understand that this situation has been inconvenient for you, and we sincerely apologize for any frustration it may have caused. To minimize the impact of this issue, the product team has decided to open up additional deployment options in both the West Europe and South Central US regions. We hope that this will provide you with greater flexibility and enable you to continue using our services without further interruption.

Thank you for your understanding, and please do not hesitate to reach out to us if you have any further questions or concerns.

Regards, Yutong

-Please kindly accept the answer if you feel helpful and vote 'Yes' to support the community, thanks a lot.

Michelle-0463 20 Reputation points

2023-04-24T12:15:38.2+00:00

Thank you @YutongTie-MSFT ! Is there any way to move a custom model across regions? Or download the fine-tuned model and upload it into the other region? That way I do not need to fine-tune it again?
Kuluru Vineeth Kumar Reddy 0 Reputation points

2023-05-03T06:37:11.0833333+00:00

@YutongTie-MSFT Failing to create custom models and deployments in either West Europe or South Central US region is it due to shortage of GPU in both the regions as mentioned above? or is there something we are missing while creating, please assist, thanks.
Eric Halsey 26 Reputation points

2023-05-04T04:07:33.98+00:00

@YutongTie-MSFT any insight when we'll be able to fine tune the other base model types? Primarily interested in Davinci.

Answer 2

fongnx 5

My resource group locate at South Central US but still failed every deployment.

Share via

Failed Status Custom Model Deployment

1 additional answer

Your answer