Why is the data returned from openai azure endpoint different than in docs?

Question

Why is the data returned from openai azure endpoint different than in docs?

Nico Herrera 20

Hello, Im using https://{endpoint}/openai/deployments/{deployment-id}/chat/completions?api-version=2024-06-01, but my response looks like

{
  id: 'xxxx',
  model: 'xxxx',
  created: 1727883132,
  object: 'extensions.chat.completion.chunk',
  choices: [
    { index: 0, delta: [Object], end_turn: false, finish_reason: null }
  ],
  system_fingerprint: 'xxxx'
}

instead of it looking like

{
  "body": {
    "id": "chatcmpl-7R1nGnsXO8n4oi9UPz2f3UHdgAYMn",
    "created": 1686676106,
    "choices": [
      {
        "index": 0,
        "finish_reason": "stop",
        "message": {
          "role": "assistant",
          "content": "Content of the completion [doc1].",
          "context": {
            "citations": [
              {
                "content": "Citation content.",
                "title": "Citation Title",
                "filepath": "contoso.txt",
                "url": "https://contoso.blob.windows.net/container/contoso.txt",
                "chunk_id": "0"
              }
            ],
            "intent": "dog care"
          }
        }
      }
    ],
    "usage": {
      "completion_tokens": 557,
      "prompt_tokens": 33,
      "total_tokens": 590
    }
  }
}

Why am i getting a different response?

0 comments

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Sina Salam 31,456 Volunteer Moderator

Hello Nico Herrera,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

The reason is because you're receiving the response in streaming mode instead of a complete response. Check your code and disable Streaming Mode by setting the stream parameter in the request body to false (the default).

   {
     "messages": [
       {
         "role": "user",
         "content": "Your input here"
       }
     ],
     "max_tokens": 100,
     "stream": false
   }

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Nico Herrera 20 Reputation points

2024-10-02T23:21:43.2566667+00:00

I see! That does get me the response I need, however no streaming is a bummer, especially for large responses. Is there no way to get streaming with citations (like in openai studio)?
Sina Salam 31,456 Reputation points Volunteer Moderator

2024-10-03T01:54:40.3733333+00:00

Hello Nico Herrera,

Thank you for your feedback.

If the problem statement YOU posted is true, why can't you just change the settings as I said, except you have other solution you've discovered.

Success
Nico Herrera 20 Reputation points

2024-10-03T01:57:16.0266667+00:00

i dont follow, i changed the settings (set stream to false), which got me the citations. I'm asking if that is currently the only way to get citations to show? I need to know that it is not possible to stream a response and get citations at the same time.
Rohit Mungi 49,216 Reputation points Microsoft Employee Moderator

2024-10-03T06:44:24.82+00:00

@Nico Herrera If you are using the .net SDK, see the approach mentioned in these threads.

https://github.com/Azure/azure-sdk-for-net/issues/44415

https://github.com/Azure/azure-sdk-for-net/issues/43226
Sina Salam 31,456 Reputation points Volunteer Moderator

2024-10-03T11:12:20.61+00:00

Hi, @Nico Herrera

Okay.

Currently, it is not possible to stream a response and get citations at the same time. When streaming responses, the citations are not included due to the way the data is processed and delivered : https://community.openai.com/t/how-to-streaming-the-assistant-api-output-and-output-citation-at-the-same-time/953769. To receive citations, you need to disable streaming and use the non-streaming response method : https://community.openai.com/t/how-to-streaming-the-assistant-api-output-and-output-citation-at-the-same-time/953769
Nico Herrera 20 Reputation points

2024-10-03T14:34:16.5233333+00:00

Thanks for the information everyone, that helps, and Romungi I'm currently using javascript. Is this something that will be in the works soon (citations with streaming)?
Rohit Mungi 49,216 Reputation points Microsoft Employee Moderator

2024-10-07T04:37:39.16+00:00

@Nico Herrera I am not sure about JS, but JS Azure OpenAI client library is being deprecated and the guidance is to use the OpenAI JS client which supports Azure OpenAI too. See the migration guide here. The official JS repo for OpenAI is here.