How to fix streaming Azure AI responses from sending incomplete data?

suresh chengalvala 0 Reputation points
2024-06-08T19:17:30.1233333+00:00

I'm streaming chat completion responses from a serverless deployment of the Phi3 mini model from the Azure AI Hub. Though I have the issue of random characters or words being left out in the response. The API works just fine when tested with streaming disabled. I have tried the SSEParser from (https://github.com/thivy/azure-openai-js-stream) though it suffers the same issue. This is the result of a simple script that logs the parsed chunks and logs chunks that failed to parse.


data: {"id":"cmpl-6efec8012ebb48768a85a7489211369f","object":"
AP PHYSICS 2 (3 Sources for Physics Lesson
Key Topics to Study:
Introduction todata: {"id":"cmpl-6efec8012ebb48768a85a7489211369f","
ources: Descriptions of point, line, and surface sources, field patterns, and direction of fields
Radio Waves: Generation, propagation, and applications of radio waves
data: {"id":"cmpl-6efec8012ebb48768a85a7489211369f","object":"chat.co
 and Wave Propagation: Transmission modes, reflection, refraction, and absorption in various media
Study Tips:
Understanding Source Types:data: {"id":"cmpl-6efec8012ebb48768a85a7489
iarize yourself with diagrams of different source types and their field patterns.
Practice Problems: Work through problems related to field strength, intensity, and coverage areas ofdata: {"id":"cmpl-6efec8012e
 sources.
Visual Aids: Use animations and simulations to visualize wave propagation and interactions with various media.
Resources:
Textbook: Review the physics ofdata: {"id":"cmpl-6efec8012ebb48768a85a7489
 and sources chapter in your AP Physics textbook.
OpenStax: Access "AP Physics 2: Physics for College Students" by OpenStax for free, whichdata: {"id":"cmpl-6efec8012ebb48768a85a7489211369f","obje       
 a range of problems and explanations.
MIT OpenCourseWare: Explore the "AP Physics 2: Physics for College Students" course materials on Mdata: {"id":"cmpl-6efec8012ebb487
's open courseware website, which includes video lectures and problem sets.
Remember to align your study efforts with the particular questions and concepts that will likely appear on AP Physics 2 exam, with an emphasis on understanding the principles of wave sources, as these are fundamental to many AP Physics topics.

As you can see there are chunks of the streamed response that are incomplete, revealing the underlying problem on why random characters or words are being left out. Is there any way to fix this issue? Is it caused by Azure or is it something that needs to be handled when streaming responses, if so, are there any resources I can look at to do such thing?

Related info:

Request formation


      const headers = {
        "Content-Type": "application/json",
        "Authorization": "Bearer " + env.AI_KEY
      }
  
      const body = {
        "messages": [
          {
            "role": "user",
            "content": "..."
          },
          {
            "role": "user",
            "content": course + " " + name
          }
        ],
        "max_tokens": 1024,
        "temperature": 0.8,
        "top_p": 1,
        "stream": true
      }
  
      const response = await fetch(endpoint, {
        method: "POST",
        headers: headers,
        body: JSON.stringify(body)
      })

Reader

      const SSEEvents = {
        onError: (error: any) => {
          console.error(error);
          controller.error(error)
        },
        onData: (data: string) => {
          const queue = new TextEncoder().encode(data);
          process.stdout.write(queue)
          controller.enqueue(queue);
        },
        onComplete: () => {
          //console.log("Done")
          controller.close()
        },
      };
      const decoder = new TextDecoder();
      const sseParser = new SSEParser(SSEEvents);
  
      const reader = response.body?.getReader()
      
      while (true) {

        const book = await reader?.read();

        if (book?.done) break;

        const chunk = decoder.decode(book?.value);
        try {
          sseParser.parseSSE(chunk);
        } catch (error) {
          console.log(chunk)
          //console.log(error)
        }
        
      }

      controller.close()
    }

Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
2,482 questions
Azure AI services
Azure AI services
A group of Azure services, SDKs, and APIs designed to make apps more intelligent, engaging, and discoverable.
2,530 questions
{count} votes