the response not stream when using java to invoke the api

Jiang Xiaoqiang 0 Reputation points
2023-07-07T08:28:11.1033333+00:00

when I am using this java code to invoke the api, I found the response is not stream, it waiting for 30+ seconds and return all content onece:

public void getChatCompletion(SseEmitter emitter, String prompt, String azureOpenaiKey) {
        String endpoint = "https://cv.openai.azure.com/";
        String deploymentOrModelId = "cv-ai";
        OpenAIClient client = new OpenAIClientBuilder()
                .endpoint(endpoint)
                .credential(new AzureKeyCredential(azureOpenaiKey))
                .buildClient();
        List<ChatMessage> chatMessages = new ArrayList<>();
        chatMessages.add(new ChatMessage(ChatRole.SYSTEM).setContent("You are a helpful assistant."));
        chatMessages.add(new ChatMessage(ChatRole.USER).setContent(prompt));
        ChatCompletionsOptions options = new ChatCompletionsOptions(chatMessages);
        options.setStream(true);
        options.setModel("gpt-3.5-turbo-0613");
        IterableStream<ChatCompletions> chatCompletions = client.getChatCompletionsStream(deploymentOrModelId, options);
        chatCompletions.forEach(completions->{
            try {
                SseChatService.getInstance().appendMsg(completions, RequestContext.getRequestId());
                emitter.send(completions);
            } catch (Exception e) {
                log.error("send emit message error", e);
            }
        });
    }

Am I missing something? what should I to do make it as a stream response and improve the user experience?

Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
3,662 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.