Azure OpenAi API - Java vs Python inconsistency

Question

I using Azure openAI to create an artificial intelligence that creates JSON files. I originally wrote this in python but the requirements of the project changed requiring it to be in Java. Both have the same model and storage configuration but when testing with the same prompt, the python creates a JSON every time and the Java never has created one. I have checked and double checked every part of my program and they seem to be identical except for syntax. In debugging it seems like my system message is not being sent properly.

I would love to see the backend on the Azure side to see what it receives from me but I am unsure if that is possible. Any suggestions on how to debug or fix this inconsistency? Thank you!

Java code:

this.chatMessages = new ArrayList<>(); this.chatMessages.add(new ChatRequestSystemMessage(messageSystem));

this.chatMessages.add(new ChatRequestUserMessage(message));

this.searchConfiguration= new AzureSearchChatExtensionConfiguration(
new AzureSearchChatExtensionParameters(Chat.searchEndpoint, Chat.searchIndexName)
.setSemanticConfiguration(Chat.searchIndexName + "-semantic-configuration")
.setQueryType(AzureSearchQueryType.VECTOR_SEMANTIC_HYBRID)
.setFieldsMapping(
new AzureSearchIndexFieldMappingOptions()
.setContentFieldsSeparator("\n")
.setContentFields(List.of("chunk"))
.setFilepathField("title")
.setTitleField("file_name")
.setUrlField("experiment_id")
.setVectorFields(List.of("text_vector")))
.setInScope(true)
.setRoleInformation(messageSystem)
.setStrictness(2)
.setTopNDocuments(5)
.setAuthentication(new OnYourDataApiKeyAuthenticationOptions(Chat.searchKey))
.setEmbeddingDependency(new OnYourDataDeploymentNameVectorizationSource("text-embedding-ada-002"))
);

ChatCompletionsOptions options = new ChatCompletionsOptions(this.chatMessages)
                                 .setDataSources(List.of(Chat.searchConfiguration))
                                 .setModel(Chat.openaiDeployment)
                                 .setTemperature(0.0)
                                 .setTopP(1.0)
                                 .setMaxTokens(800);

this.currentResponse = this.openaiClient.getChatCompletions(Chat.openaiDeployment, options);
 

Python Code:

  self._message_history = [{"role": "system", "content": self._message_system}]

    self._message_history.append({"role": "user", "content": message})

    self._message_latest =

    self._openai_client.chat.completions.create(
        messages=self._message_history,
        model=self._openai_deployment,
        extra_body={
            "data_sources": [  
                {
                    "type": "azure_search",
                    "parameters": {
                        "endpoint": self._search_endpoint,
                        "index_name": self._search_index_name,
                        "semantic_configuration": self._search_index_name + "-semantic-configuration",
                        "query_type": "vectorSemanticHybrid",
                        "fields_mapping": {
                            "content_fields_separator": "\n",
                            "content_fields": [
                                "chunk"
                            ],
                            "filepath_field": "title",
                            "title_field": "file_name",
                            "url_field": "experiment_id",
                            "vector_fields": [
                                "text_vector"
                            ]
                        },
                        "in_scope": True,
                        "role_information": self._message_system,
                        "filter": None,
                        "strictness": 2,
                        "top_n_documents": 5,
                        "authentication": {
                            "type": "api_key",
                            "key": self._search_key
                        },
                        "embedding_dependency": {
                            "type": "deployment_name",
                            "deployment_name": "text-embedding-ada-002"
                        },
                        "key": self._search_key,
                        "indexName": self._search_index_name
                    }
                }
            ],
        },
        temperature=0,
        top_p=1,
        max_tokens=800,
        stop=None,
    )

Answer 1

Hi Jack Campbell

1. Use Azure Application Insights

Azure Application Insights allows you to monitor and track API requests for various Azure services, including Cognitive Services like OpenAI. It can provide telemetry on your API calls, If you already have Application Insights configured in your Azure environment, you can use it to track and debug requests to Azure OpenAI.

Set up Application Insights in your Azure portal.

Instrument your code to send telemetry for your application (this might involve wrapping your API calls with telemetry calls).

• Use the Metrics Explorer or Logs (KQL queries) to analyze incoming requests to your OpenAI service.

You can follow the official guide to set up Application Insights for monitoring your APIs:

• Azure Application Insights Documentation

2. Monitor API Usage in Azure Portal

The Azure Portal provides some diagnostic features where you can monitor and inspect your API usage, This won't give you full visibility into the request payload itself, but it will provide insights into any errors or issues that might be happening at the service level.

Go to Azure Portal > Cognitive Services > Azure OpenAI.

Check Metrics and Diagnose and Solve Problems.

This will help you understand if there’s an underlying service issue causing the problem (e.g., rate-limiting, resource availability, etc.).

In the code you have shared :

1. Check Redundant Keys in Java

In your Java code, it looks like you may be passing redundant parameters, like key and indexName, which might not cause issues in Python but could potentially cause serialization errors in Java.Remove any redundant or unnecessary parameters and ensure that each parameter is only specified once, as this might break serialization in Java.

2. Verify data_sources Structure

In Python, you're passing the data_sources as part of the extra_body

In Java Ensure that Chat.searchConfiguration is correctly set up to match the expected data structure of the API. You might want to verify that AzureSearchChatExtensionConfiguration is being serialized correctly to the API format and contains all necessary fields (like type, parameters, etc.).

3.Inspect System Message Handling

You mentioned that your system message might not be sent properly. This could be related to the order of messages or serialization issues. In both the Python and Java code, you're adding a system message first. Ensure that both systems (Python and Java) are consistently serializing and sending the system message. Check if there's a mismatch in the way role or content is being handled. You could also log the final message list to see if the system message is correctly placed before the user message.

You can mark it 'Accept Answer' if this helped you

Regards,

Vishvani

Answer 2

Hello Jack !

Thank you for posting on Microsoft Learn.

What I have noticed in Python, you pass data_sources using extra_body, but in Java you do:

.setDataSources(List.of(Chat.searchConfiguration))

This is correct only if you're using Azure Java SDK that supports Assistants + Search Extensions, like com.azure.ai.openai.

However, some earlier or community SDK don’t serialize that into the correct payload, especially with nested fields like embedding_dependency.

So try logging the raw request body sent to Azure (if your Java SDK allows request interceptors or HttpPipeline logging) including :

"data_sources": [
  {
    "type": "azure_search",
    "parameters": { ... }
  }
]

Also, you have key and indexName twice, it could be tolerated in Python (OpenAI ignores redundant keys), but in Java this could silently break request serialization especially if your SDK serializes parameters via POJOs.