How to know azure open ai model deployments stored in cpu or gpu?

Question

How to know azure open ai model deployments stored in cpu or gpu?

Sathyadharani K 5

Please share the architecture of azure open ai model deployments and how to know that model deployments will stored in which hardware configuration**(CPU or GPU details) ?**

Sathyadharani K 5 Reputation points

2023-08-30T06:10:02.03+00:00

Please share details as soon as possible. Thanks in advance.

1 answer

Your answer

Sathyadharani K 5 Reputation points

2023-08-30T06:10:02.03+00:00

Please share details as soon as possible. Thanks in advance.

Answer 1

romungi-MSFT 48,766 Microsoft Employee

@Sathyadharani K The information about the exact infrastructure used for model hosting and deployment is confidential and is not disclosed in public domain. To summarize, deployments use both CPU and GPU depending on the type of model used for deployments. Thanks!!

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Saravanan Ramu 0 Reputation points

2023-08-30T23:37:28.1+00:00
@romungi-MSFT Thanks! Your response is helpful. There are 2 follow up questions

We understand that models are hosted in confidential infrastructure not disclosed in public domain. Is every model instance or deployment hosted in a separate dedicated infrastructure or is there a shared infrastructure within Microsoft that commonly hosts all instances?

When building applications based on Azure OpenAI, how do we guarantee performance/responsiveness for varying loads/usage patterns, if we do not have control over selection of the hosting hardware?

We are very new to Azure OpenAI and are exploring it to fulfill a customer requirement. Please excuse if the above questions are trivial.
romungi-MSFT 48,766 Reputation points Microsoft Employee

2023-09-01T07:43:59.18+00:00

The service is built on a shared infrastructure, similar to other Azure services and the same principles of isolation are applied for Azure OpenAI resources. You also have an option to isolate your resource by using private endpoints.

You can manage the quota limits of your deployments based on the usage and some of the best practices are highlighted in the rate limit best practices to avoid failures, the application should be able to implement retry logic and avoid sharp changes in workloads to ensure there are no quota related errors. You always have an option to request additional quota from the quotas screen through Azure support case if needed. Thanks!!
romungi-MSFT 48,766 Reputation points Microsoft Employee

2023-09-03T16:30:31.6633333+00:00

Saravanan Ramu I hope the above information helped. Thanks!!

Share via

How to know azure open ai model deployments stored in cpu or gpu?

1 answer

Your answer