the response not stream when using java to invoke the api
Jiang Xiaoqiang
0
Reputation points
when I am using this java code to invoke the api, I found the response is not stream, it waiting for 30+ seconds and return all content onece:
public void getChatCompletion(SseEmitter emitter, String prompt, String azureOpenaiKey) {
String endpoint = "https://cv.openai.azure.com/";
String deploymentOrModelId = "cv-ai";
OpenAIClient client = new OpenAIClientBuilder()
.endpoint(endpoint)
.credential(new AzureKeyCredential(azureOpenaiKey))
.buildClient();
List<ChatMessage> chatMessages = new ArrayList<>();
chatMessages.add(new ChatMessage(ChatRole.SYSTEM).setContent("You are a helpful assistant."));
chatMessages.add(new ChatMessage(ChatRole.USER).setContent(prompt));
ChatCompletionsOptions options = new ChatCompletionsOptions(chatMessages);
options.setStream(true);
options.setModel("gpt-3.5-turbo-0613");
IterableStream<ChatCompletions> chatCompletions = client.getChatCompletionsStream(deploymentOrModelId, options);
chatCompletions.forEach(completions->{
try {
SseChatService.getInstance().appendMsg(completions, RequestContext.getRequestId());
emitter.send(completions);
} catch (Exception e) {
log.error("send emit message error", e);
}
});
}
Am I missing something? what should I to do make it as a stream response and improve the user experience?
Sign in to answer