Azure OpenAI GPT 4o-mini-batch. Batch request with grounded data (Azure Search) always returns empty response. No errors

Question

Hi,

I would like to utilize the Azure OpenAI batch jobs.

My application uses Azure OpenAI for enterprise data. So my grounded data is taken from Azure Search (index)

I use GPT 4o-mini. While I make regular REST API calls on the LLM model, I get proper response. Now, I tried with batch process newly available on GPT 4o-mini in East US region.

When I make a call without (Azure search) data_sources tag, I get proper response. But when I include (Azure search) data_sources tag to generate response for my query, I always get empty response.

Here is my request.jsonl

{"custom_id":"task-0","method":"POST","url":"/chat/completions","body":{"model":"gpt-4o-mini-batch","messages":[{"role":"system","content":"You are an AI assistant that answers questions based only on the provided documents."},{"role":"user","content":"Tell me the names of sisters of Diana"}],"top_p":1,"frequency_penalty":0,"max_tokens":1024,"presence_penalty":0,"semantic_configuration":{"name":"my-semantic-config"},"temperature":0,"data_sources":[{"type":"azure_search","parameters":{"endpoint":"https://XXXXXXXXXXXXX.search.windows.net","scope":{"in_scope":true},"index_name":"XXXXXXXXXXXX","key":"XXXXXXXXXXXXXXX","role_information":"You are an AI assistant that helps people find information."}}]}}

Code:

import os
from openai import AzureOpenAI
import dotenv
dotenv.load_dotenv()
    
client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_KEY"),  
    api_version="2024-07-01-preview",
    azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
    )

currentDirectory = os.getcwd()
# Upload a file with a purpose of "batch"
file = client.files.create(
  file=open(currentDirectory +"/resources/requestJSON.jsonl", "rb"), 
  purpose="batch"
)

print(file.model_dump_json(indent=2))
file_id = file.id

# Wait until the uploaded file is in processed state
import time
import datetime

status = "pending"
while status != "processed":
    time.sleep(5)
    file_response = client.files.retrieve(file_id)
    
    status = file_response.status
    print(f"{datetime.datetime.now()} File Id: {file_id}, Status: {status}")

# Submit a batch job with the file
batch_response = client.batches.create(
    input_file_id=file_id,
    endpoint="/chat/completions",
    completion_window="24h",
)

# Save batch ID for later use
batch_id = batch_response.id
print("BatchId-->" + batch_id)
print(batch_response.model_dump_json(indent=2))

status = "validating"
while status not in ("completed", "failed", "canceled"):
    time.sleep(60)
    batch_response = client.batches.retrieve(batch_id)
    status = batch_response.status
    print(f"{datetime.datetime.now()} Batch Id: {batch_id},  Status: {status}")

import json

if status == "completed":
    file_response = client.files.content(batch_response.output_file_id)
    raw_responses = file_response.text.strip().split('\n')  

    for raw_response in raw_responses:  
        json_response = json.loads(raw_response)  
        formatted_json = json.dumps(json_response, indent=2)  
        print(formatted_json)  
elif status == "failed":
    print(f"{datetime.datetime.now()} Batch Id: {batch_id},  Status: {status}")

Can you help to understand if something is incorrect.

Note: My index in Azure Search has embeddings also.

Answer 1

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/batch?tabs=standard-input&pivots=programming-language-python

Here is the answer. Batch process is not yet supported for grounding data.