Azure OpenAI stream with functions doesn't work - chunks distribution in the end of completion on using "function_call"

Question

Azure OpenAI stream with functions doesn't work - chunks distribution in the end of completion on using "function_call", with both gpt3.5 and gpt4 chat models, using last GPT models and last Azure OpenAI version.

Distribution of stream chunks is NORMAL when using text completion, but always IN THE END OF COMPLETION when using FUNCTION, while it works OK directly with "OpenAI" for same models. We're testing directly using HTTP API, so it's not sdk-related problem.

# NO FUNCTION, GPT4, NORMAL DISTRIBUTION
curl -s -N --http2 "$API_BASE/openai/deployments/$API_DEPLOYMENT/chat/completions?api-version=2023-07-01-preview" -H 'Content-Type: application/json' -H "api-key: $API_KEY" --data '{"messages": [{"role": "user", "content": "generate text about weather, at least 1024 words!"}], "max_tokens": 128, "stream": true, "n": 1, "temperature": 0.7, "functions": [{"name": "format_message", "description": "", "parameters": {"type": "object", "properties": {"text": {"title": "Text", "type": "string"}}}}], "function_call": "none"}' | while IFS= read -r line; do date; done | uniq -c
      2 Fri Oct 20 11:51:11 AM EEST 2023
     56 Fri Oct 20 11:51:12 AM EEST 2023
     54 Fri Oct 20 11:51:13 AM EEST 2023
     52 Fri Oct 20 11:51:14 AM EEST 2023
     56 Fri Oct 20 11:51:16 AM EEST 2023
     44 Fri Oct 20 11:51:17 AM EEST 2023

# FUNCTION, GPT4, DISTRIBUTION AT END (after 9s)
curl -s -N --http2 "$API_BASE/openai/deployments/$API_DEPLOYMENT/chat/completions?api-version=2023-07-01-preview" -H 'Content-Type: application/json' -H "api-key: $API_KEY" --data '{"messages": [{"role": "user", "content": "generate text about weather, at least 1024 words!"}], "max_tokens": 128, "stream": true, "n": 1, "temperature": 0.7, "functions": [{"name": "format_message", "description": "", "parameters": {"type": "object", "properties": {"text": {"title": "Text", "type": "string"}}}}], "function_call": {"name": "format_message"}}' | while IFS= read -r line; do date; done | uniq -c
      2 Fri Oct 20 11:51:32 AM EEST 2023
    164 Fri Oct 20 11:51:41 AM EEST 2023
     98 Fri Oct 20 11:51:42 AM EEST 2023

# NO FUNCTION, GPT3.5, NORMAL DISTRIBUTION
curl -s -N --http2 "$API_BASE/openai/deployments/$API_DEPLOYMENT/chat/completions?api-version=2023-07-01-preview" -H 'Content-Type: application/json' -H "api-key: $API_KEY" --data '{"messages": [{"role": "user", "content": "generate text about weather, at least 1024 words!"}], "max_tokens": 128, "stream": true, "n": 1, "temperature": 0.7, "functions": [{"name": "format_message", "description": "", "parameters": {"type": "object", "properties": {"text": {"title": "Text", "type": "string"}}}}], "function_call": "none"}' | while IFS= read -r line; do date; done | uniq -c
      2 Fri Oct 20 11:54:36 AM EEST 2023
     84 Fri Oct 20 11:54:37 AM EEST 2023
     44 Fri Oct 20 11:54:38 AM EEST 2023
    114 Fri Oct 20 11:54:39 AM EEST 2023
     20 Fri Oct 20 11:54:40 AM EEST 2023
     
# FUNCTION, GPT3.5, DISTRIBUTION AT END (after 4s)
curl -s -N --http2 "$API_BASE/openai/deployments/$API_DEPLOYMENT/chat/completions?api-version=2023-07-01-preview" -H 'Content-Type: application/json' -H "api-key: $API_KEY" --data '{"messages": [{"role": "user", "content": "generate text about weather, at least 1024 words!"}], "max_tokens": 128, "stream": true, "n": 1, "temperature": 0.7, "functions": [{"name": "format_message", "description": "", "parameters": {"type": "object", "properties": {"text": {"title": "Text", "type": "string"}}}}], "function_call": {"name": "format_message"}}' | while IFS= read -r line; do date; done | uniq -c
      2 Fri Oct 20 11:54:58 AM EEST 2023
    262 Fri Oct 20 11:55:02 AM EEST 2023

Answer 1

@Victor Gavro Thanks for the question, yes content filtering can impact this for streaming, especially for Asian languages. We are working on options to improve this, and in the meantime you can apply for content moderation adjustments.