How to improve response consistency when using Azure OpenAI with enterprise data

prathap s 20 Reputation points
2025-09-09T17:26:45.01+00:00

I’m experimenting with Azure OpenAI and grounding it with my company’s internal data through Cognitive Search. While the setup works, I’m noticing that the responses are sometimes inconsistent —for example, the model answers correctly for one user query but gives a generic or incomplete response for another query that’s worded slightly differently.

I’ve already tried adjusting chunk sizes and prompt wording, but the issue still persists. Are there recommended techniques or configurations in Azure OpenAI (e.g., embeddings tuning, hybrid search, semantic ranker, or system prompts) that can help improve the consistency and reliability of responses when working with enterprise knowledge bases?

Foundry Tools
Foundry Tools

Formerly known as Azure AI Services or Azure Cognitive Services is a unified collection of prebuilt AI capabilities within the Microsoft Foundry platform

0 comments No comments

Answer accepted by question author
Azar 31,280 Reputation points MVP Volunteer Moderator
2025-09-09T17:33:09.1533333+00:00

Hi there prathap s

The key is to improve retrieval and prompt design. Use hybrid search with semantic ranker so the model always gets the most relevant chunks. Keep your documents split into smaller, meaningful sections to avoid noise, and add clear instructions in the system prompt to only answer from context (or say “I don’t know”). Also, log what’s being passed into the model—if bad context goes in, that’s where to fix things. This usually clears up the grounding issues.

If this helps kindly accept the answer thanks.

Was this answer helpful?

0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.