APIM not supported for knowledge base LLM in Agentic Retrieval

Question

Hi Team,

We have implemented Agentic Retrieval in our solution, For POC we used AzureOpenAI API's directly to configure Knowledge Base LLM and our implementation was working perfectly fine. Our organization has this policy that all requests to AzureOpenAI API's should route through APIM only so we can't use AzureOpenAI API or foundry resource directly while creating the knowledge base, we will have to go through APIM only.
so we created the managed identity of our Azure AI Search and provided the access to identity on APIM with API Management Service Reader Role.

We were able to create knowledge base successfully but when we are invoking the knowledge base, we are getting 401 error.
This is how we are creating the knowledgebase, here the endpoint is APIM endpoint.

We found in one of the documentations that APIM is not supported for knowledgebase LLM, but we don't have any other alternative, we need answer synthesis with at least low reasoning effort, so LLM is necessary.

Can you please let us know if you have any solution for this?

Answer 1

Hi @Sharma, Jyoti ,

Thanks for reaching out in Microsoft Q&A forum,

your Agentic Retrieval setup and the requirement to route all Azure OpenAI traffic through Azure API Management (APIM). From what you’ve described, your proof of concept worked when the Knowledge Base LLM was configured directly against the native Azure OpenAI/Foundry endpoint, but you now see 401 Unauthorized errors when the LLM ResourceUri is changed to an APIM endpoint. This behavior is consistent with a current product limitation rather than an RBAC or APIM policy misconfiguration.learn.microsoft+3

Why APIM fails for the Knowledge Base LLM?

In Azure AI Search, an Agentic Retrieval knowledge base is designed to call a supported Azure OpenAI or Foundry model endpoint directly using the search service’s managed identity and role assignment.learn.microsoft+2

• The knowledge base can be created successfully even when the LLM endpoint is set to APIM, because the create operation mainly validates configuration shape and does not fully exercise the LLM path.learn.microsoft+1
• During retrieval, answer synthesis, and low/medium reasoning effort, the service performs internal LLM calls as part of the agentic pipeline. These calls expect a native Azure OpenAI/Foundry endpoint and a supported auth model, not an APIM proxy.learn.microsoft+1
• APIM is not currently documented or supported as the endpoint for the LLM used by an Agentic Retrieval knowledge base, so changing APIM roles or policies alone is unlikely to resolve the 401 Unauthorized you are seeing. The supported configuration explicitly requires the Azure AI Search managed identity to have the Cognitive Services User role on the OpenAI/Foundry resource and call it directly.learn.microsoft+2

Given this limitation and your organization’s APIM requirement, the realistic options are:

1. Request a policy exception for the managed service-to-service LLM call
   • Allow the Azure AI Search managed identity to call the Azure OpenAI/Foundry resource directly, with the Cognitive Services User role and no API keys where appropriate.learn.microsoft+2
   • Secure the OpenAI/Foundry resource using:
   • managed identity and RBAC,
   • private network access (Private Endpoint / VNet integration), and
   • disabled key authentication if your security policy permits.learn.microsoft+1
   • This approach keeps the internal LLM traffic within a tightly controlled perimeter while preserving built‑in Agentic Retrieval behavior (query planning, answer synthesis, low/medium reasoning).learn.microsoft+1
2. Use the knowledge base without its LLM and generate answers in your application through APIM
   • Configure retrieval reasoning effort to minimal, which explicitly disables LLM-based query planning and processing. In this mode, the knowledge base issues direct text and vector searches against your knowledge sources and returns extractive results only.docs.azure
   • Your application can then:
   • call the knowledge base retrieve API (or standard AI Search queries) to get the relevant documents/snippets, and
   • call Azure OpenAI through APIM to perform answer generation using those retrieved results as context.
   • This pattern fully preserves the APIM requirement for all model calls, but you won’t have built‑in query planning or answer synthesis inside the knowledge base; all reasoning and generation are handled in your app.

Kindly let us know if the above helps or you need further assistance on this issue.

Please do not forget to and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hello, Welcome to MS Q&A

Based on my interpretation , Azure AI Search Knowledge Bases (Agentic Retrieval) currently expect the LLM to be a Microsoft Foundry/Azure OpenAI resource, accessed using the Search service's Managed Identity with the Cognitive Services User role. There is no documented support for configuring an APIM endpoint as the Knowledge Base LLM endpoint. Although APIM is supported for client applications calling Azure OpenAI, this support does not extend to the internal LLM calls made by Knowledge Bases for Answer Synthesis. Therefore, routing Knowledge Base LLM traffic through APIM is not a supported architecture today, which is likely the reason for the 401 error.

Microsoft Learn references:

• Query a Knowledge Base using Agentic Retrieval (Prerequisites, Managed Identity, LLM configuration)
• Agentic Retrieval Quickstart (Knowledge Base architecture and Answer Synthesis using Azure OpenAI/Foundry)

Any other solution what I can suggest

Option 1: Build a custom RAG pipeline

Instead of using the built-in Knowledge Base Answer Synthesis:

• Azure AI Search for retrieval
• Azure OpenAI via APIM for generation
• Optional orchestration with Azure AI Foundry Agent Service or Semantic Kernel

This provides the most flexibility and fully complies with the APIM requirement.

---

Option 2: Request Microsoft support

Since this appears to be a product limitation, ask Microsoft whether:

• APIM support for Knowledge Base LLM endpoints is on the roadmap.
• There is a private preview or undocumented supported configuration for APIM.

Pls check and let us know if any further ques

Thanks
Deepanshu