API for gpt-4-1106-preview extremely slow

Question

When we do API calls for the gpt-4-1106-preview model, the average response time is around 60 seconds. When we use the chat GUI in the Azure AI studio on the same model, with the same parameters, the response takes 10 - 20 seconds? What can we do to speed up the API? We already tried to tune the temperature, max_tokens and top_p parameters and to minimize the content filters, but they all make no significant difference.

Example API call:

time curl -X POST -H "Content-Type: application/json" -H "api-key: XXX" -d '{
  "messages": [
    {
      "role": "user",
      "content": "What does a cow eat?"
    }
  ],
  "model": "gpt-4-1106-preview",
  "stream": true,
  "temperature": 0.7,
  "frequency_penalty": 0,
  "presence_penalty": 0
}' "https://XXX-sweden.openai.azure.com/openai/deployments/gpt-4-1106-preview/chat/completions?api-version=2023-09-01-preview"


.....

data: [DONE]


real	1m7,174s
user	0m0,079s
sys	0m0,024s

Answer 1

Same for me. My RAG app that I'm using in my project became unusable. GPT-3.5 is not an option because the quality is much worse. It used to be fine last year but some time in January it started to degrade. If this isn't fixed, I need to explore other LLM options which isn't easy as my employer has strict compliance requirements. Setting the max_tokens to a low value helps but my RAG app does not allow me to set this parameter.

Answer 2

The same is for us. we are considering moving to a different model provider bc the long latency is straining the usage..

Answer 3

Same here, please address the issue.

Answer 4

Same here. extremely slow and unusable.

Answer 5

I can confirm this as well. Same prompt using OpenAI directly is a few seconds. Op Azure I get timeouts exceeding 100 seconds. Using both gpt4 preview versions