Phi-3 Serverless Deployment Just Stopped working

Amiya Patanaik 25

I have been using multiple serverless deployments of foundation models on Azure AI Studio (East-2) specifically. All of a sudden the Phi-3 128K deployment stopped working, both through Langchain as well as in the playground within Azure AI Studio. When queried through langchain or CURL directly, the request times out after long while. When using the playground I get: Request failed with status code 424. Clear the output to start a new dialog.

The endpoint is working because if I supply a wrong key, it immediately responds with an invalid key response. I tried deleting the endpoint and redeploying it. Still does not work. It only happens with Phi-3 128K instruct model.

This is the endpoint: https://Phi-3-mini-128k-instruct-hxaff-serverless.eastus2.inference.ai.azure.com/v1/chat/completions

Screenshot 2024-06-26 at 1.50.35 PM

navba-MSFT 19,405 Reputation points Microsoft Employee

2024-06-26T09:11:36.57+00:00

@Amiya Patanaik Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

.

Could you please collect browser HAR traces at the time of the issue and share the traces here ?

I have also asked for a few details in the private message. Please reply once you get a chance.

Awaiting your reply.
Amiya Patanaik 25 Reputation points

2024-06-26T12:01:42.63+00:00

I have shared more details and HAR traces privately.
Amiya Patanaik 25 Reputation points

2024-06-26T13:29:12.7833333+00:00

I deployed a Phi-3 Medium to the same AI studio, after 5 minutes of deploying the model, I still cannot interact with it. On playground, I get:
Request failed with status code 400. Clear the output to start a new dialog.

This is only happening with Phi models.
navba-MSFT 19,405 Reputation points Microsoft Employee

2024-06-26T13:35:42.57+00:00

@Amiya Patanaik Thanks for sharing the details. Could you please confirm if this model was deployed using Serverless API or with Managed Compute ?

Do you see this issue with both the deployment options ?

Awaiting your reply.
Amiya Patanaik 25 Reputation points

2024-06-26T13:40:38.02+00:00

It is serverless API
Amiya Patanaik 25 Reputation points

2024-06-26T13:41:46.3766667+00:00

It is deployed using Serverless API
navba-MSFT 19,405 Reputation points Microsoft Employee

2024-06-26T15:07:11.6766667+00:00

@Amiya Patanaik Thanks for sharing the details. I will check this at my end and get back to you.
Rithika Chowta 0 Reputation points

2024-06-27T09:50:56.3+00:00

@navba-MSFT I have the same issue, please update as soon as there are any findings
Amiya Patanaik 25 Reputation points

2024-06-27T17:46:34.9033333+00:00

It now magically started working! No idea what was wrong!

Accepted answer

navba-MSFT 19,405 Reputation points Microsoft Employee

2024-06-28T04:34:13.2333333+00:00

@Amiya Patanaik I am glad that it is working fine now. I had involved the Product Owners to look into this issue in the background and address this.

.

If you encounter the same issue in future, please feel free to reach out to me. I would be happy to assist.

**

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
Amiya Patanaik 25 Reputation points

2024-06-28T04:41:52.7966667+00:00

Thank you. I have accepted the answer. But it would be helpful to understand what caused the issue. I'm not comfortable using the Phi models in production knowing they may again go down unexpectedly.

navba-MSFT 19,405 Reputation points Microsoft Employee

2024-06-28T04:45:33.2066667+00:00

@Amiya Patanaik I am waiting for the public facing RCA (Root Cause Analysis) from the Product Owners. I will keep you posted once I hear from them.

Patrick Godwin 0 Reputation points Microsoft Employee

2024-06-28T21:04:45.8266667+00:00

For what it's worth, I'm still running into this issue on fresh serverless deployments of Phi-3 128k

I seem to be able to get some completions back from the API if I don't use the "/v1" endpoint on the inference URL, but the performance is not very good.

I also should note that I do not observe this with Phi-3 4k serverless.

navba-MSFT 19,405 Reputation points Microsoft Employee

2024-06-29T05:11:55.0266667+00:00

@Amiya Patanaik The summary of the issue is that some of the client requests are getting timed out. However, they are not completely cancelled on the server causing more requests to be processed. This resulted in longer processing times overall and more requests were getting timed out. The issue was then fixed. Also if you encounter the same issue again, from your client side (client application) while invoking the endpoint, you could try to increase the timeout to a higher value. Hope this helps.

@Patrick Godwin Since you are internal to Microsoft, I am looping you with the Product Owners so that they can assist you directly.
Sign in to comment

Share via

Phi-3 Serverless Deployment Just Stopped working

0 additional answers