Tutorial: Use Azure Managed Redis as a semantic cache

S’applique à: ✅ Azure Managed Redis

In this tutorial, you learn how to use Azure Managed Redis as the Redis endpoint for a semantic cache in large language model (LLM) applications. Semantic caching is an application or gateway pattern, not a separate Azure Managed Redis managed feature. Azure Managed Redis provides the Redis endpoint and RediSearch vector search capabilities that store and search cached prompts, embeddings, metadata, and responses.

In this tutorial, you learn how to:

Decide whether semantic caching should run in your application or in Azure API Management.
Configure prerequisites for Azure Managed Redis, RediSearch, embeddings, TLS, and authentication.
Implement a Python semantic cache with RedisVL.
Configure gateway-level semantic caching with Azure API Management policies.
Tune thresholds, TTLs, partitions, versions, and observability for production.
Understand cost, performance, and product-fit considerations.

How semantic caching works

Semantic caching reuses prior LLM responses when a new prompt is close enough in meaning to a cached prompt. A typical request flow is:

Receive a prompt from the application or API consumer.
Generate an embedding for the prompt using the same embedding model used by the cache.
Search cached prompt embeddings in Azure Managed Redis by vector similarity.
If the closest cache entry is within the configured threshold and matches the required tenant, user, model, and prompt-version partition, return the cached response.
If there's no acceptable match, call the LLM and store the prompt, response, embedding, metadata, and TTL in Redis.

Azure Managed Redis can support this pattern because RediSearch supports vector similarity search, metadata filtering, and vector indexes over Redis data structures. For more information, see vector search in Azure Managed Redis and Redis vector search concepts.

Choose an implementation path

Path	Use when	Where cache logic runs
App-level semantic cache with RedisVL	You own the application code and need custom thresholding, partitioning, metadata, or model-call logic.	Your application uses RedisVL `SemanticCache` with Azure Managed Redis as the Redis endpoint. RedisVL `SemanticCache` supports prompt lookup, response storage, TTLs, metadata, filters, and a Redis COSINE-distance threshold. For details, see the RedisVL SemanticCache API.
Gateway-level semantic cache with Azure API Management	You want centralized semantic caching for LLM APIs that pass through Azure API Management.	Azure API Management uses `llm-semantic-cache-lookup` before the backend call and `llm-semantic-cache-store` after the backend response. Azure Managed Redis is configured as the external cache. For details, see Enable semantic caching for LLM APIs in Azure API Management.

Prerequisites

An Azure subscription.
An Azure Managed Redis instance on a tier where RediSearch is supported. The Azure Managed Redis modules article lists module availability by tier.
The RediSearch module enabled when you create the Azure Managed Redis instance. You must enable modules at creation time and can't manually add modules to an existing Azure Managed Redis instance. For more information, see Use Redis modules with Azure Managed Redis.
An Azure Managed Redis configuration that uses the Enterprise clustering policy and NoEviction eviction policy for RediSearch. RediSearch requires both settings, as described in the RediSearch module requirements.
An embedding model or deployment. Use one embedding model and vector dimension for each semantic cache index. Redis vector indexes require query vectors to have the same dimensions as the indexed vector field, as described in Redis vector search concepts.
TLS and authentication configured for your clients. Use TLS (rediss:// or ssl=True) and either Microsoft Entra ID or approved secret storage. Don't hard-code access keys. For more information, see Microsoft Entra ID for authentication and TLS configuration.
For the RedisVL path: Python, redisvl, redis, and any packages required by your embedding and Redis authentication approach.
For the Azure API Management path: an API Management instance with an LLM API, an embeddings backend, managed identity authentication to the embeddings backend, and Azure Managed Redis configured as an external Redis-compatible cache. These prerequisites are described in Enable semantic caching for LLM APIs in Azure API Management.

Path 1: Build an app-level semantic cache with RedisVL

Use this path when your application should control cache lookup, miss handling, model calls, metadata, and cache storage. RedisVL SemanticCache connects to Redis and provides check and store operations for semantically similar prompts. The distance_threshold value uses Redis COSINE distance units from 0 to 2; lower values mean stricter matching, according to the RedisVL SemanticCache API.

Install the required Python packages:

pip install openai redis redis-entraid redisvl

Set AZURE_MANAGED_REDIS_HOST, AZURE_OPENAI_ENDPOINT, OPENAI_API_VERSION, AZURE_OPENAI_API_KEY, AZURE_OPENAI_EMBEDDING_DEPLOYMENT, and AZURE_OPENAI_CHAT_DEPLOYMENT in your environment. You can also set AZURE_MANAGED_REDIS_PORT if your instance doesn't use port 10000.

The following example passes a configured Redis client to RedisVL so you can use Azure Managed Redis TLS and Microsoft Entra ID authentication:

import os

from openai import AzureOpenAI
from redis import Redis
from redis_entraid.cred_provider import create_from_default_azure_credential
from redisvl.extensions.cache.llm import SemanticCache
from redisvl.utils.vectorize import AzureOpenAITextVectorizer

credential_provider = create_from_default_azure_credential(
    ("https://redis.azure.com/.default",)
)

redis_client = Redis(
    host=os.environ["AZURE_MANAGED_REDIS_HOST"],
    port=int(os.getenv("AZURE_MANAGED_REDIS_PORT", "10000")),
    ssl=True,
    credential_provider=credential_provider,
)

vectorizer = AzureOpenAITextVectorizer(
    model=os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"],
    api_config={
        "azure_endpoint": os.environ["AZURE_OPENAI_ENDPOINT"],
        "api_version": os.environ["OPENAI_API_VERSION"],
        # Load from a secret store or managed configuration. Don't hard-code keys.
        "api_key": os.environ["AZURE_OPENAI_API_KEY"],
    },
)

openai_client = AzureOpenAI(
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    api_version=os.environ["OPENAI_API_VERSION"],
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
)

cache = SemanticCache(
    name="semcache_support_promptv1",
    redis_client=redis_client,
    vectorizer=vectorizer,
    distance_threshold=0.15,  # Redis COSINE distance [0-2]; lower is stricter.
    ttl=3600,
)


def call_llm(prompt: str) -> str:
    completion = openai_client.chat.completions.create(
        model=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT"],
        messages=[{"role": "user", "content": prompt}],
    )
    response = completion.choices[0].message.content
    if response is None:
        raise RuntimeError("The chat completion didn't contain a response.")
    return response


def answer(prompt: str) -> str:
    hits = cache.check(prompt=prompt, num_results=1)
    if hits:
        return hits[0]["response"]

    response = call_llm(prompt)
    cache.store(
        prompt=prompt,
        response=response,
        metadata={
            "model_id": os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT"],
            "prompt_version": "support-v1",
        },
    )
    return response

For production, don't share one unpartitioned cache across all callers. Use separate cache indexes or RedisVL filterable fields for tenants, users, model IDs, and prompt versions. Redis supports vector searches combined with metadata filters, as described in Redis vector search concepts.

Path 2: Build a gateway-level semantic cache with Azure API Management

Use this path when LLM traffic flows through Azure API Management and you want a centralized policy-based cache. Azure API Management semantic caching uses a configured external cache and an embeddings backend. The lookup policy checks for semantically similar cached responses before forwarding to the LLM backend, and the store policy caches eligible responses after the backend returns. For full setup steps, see Enable semantic caching for LLM APIs in Azure API Management.

The llm-semantic-cache-lookup policy runs in the inbound section and requires an embeddings backend. The score-threshold ranges from 0.0 to 1.0; lower values require higher semantic similarity, according to the llm-semantic-cache-lookup policy reference. The llm-semantic-cache-store policy runs in the outbound section and uses duration as the cache entry TTL in seconds, according to the llm-semantic-cache-store policy reference.

<policies>
    <inbound>
        <base />
        <llm-semantic-cache-lookup
            score-threshold="0.05"
            embeddings-backend-id="embeddings-backend"
            embeddings-backend-auth="system-assigned"
            ignore-system-messages="true"
            max-message-count="10">
            <vary-by>@(context.Subscription.Id)</vary-by>
        </llm-semantic-cache-lookup>
        <rate-limit calls="10" renewal-period="60" />
    </inbound>
    <outbound>
        <llm-semantic-cache-store duration="300" />
        <base />
    </outbound>
</policies>

Use <vary-by> to partition cached responses by subscription, tenant, user group, or another boundary that prevents cross-user data exposure. The Azure API Management policy reference recommends using vary-by to control cross-user access to cache entries in the llm-semantic-cache-lookup policy.

Recommended design choices

Tune the similarity threshold

Start with a strict threshold, evaluate results, and loosen only when your test set shows that cached responses remain correct. RedisVL distance_threshold uses Redis COSINE distance units [0-2], where lower values are stricter. Azure API Management score-threshold uses a 0.0 to 1.0 range, where lower values also require higher semantic similarity. Don't copy threshold values between RedisVL and Azure API Management without validating their units and behavior. For RedisVL threshold details, see the RedisVL SemanticCache API. For Azure API Management threshold details, see llm-semantic-cache-lookup.

Set TTL by use case

Set shorter TTLs for volatile or high-risk content and longer TTLs for stable content. For example:

Use case	Example TTL
Product availability, pricing, or operational status	Seconds to minutes
Policy, compliance, or legal assistance	Minutes to hours, with extra review
Product FAQ or public documentation answers	Hours to days

The Azure API Management store policy sets TTL with the duration attribute, as described in llm-semantic-cache-store. RedisVL SemanticCache supports a default ttl and per-entry TTL overrides, as described in the RedisVL SemanticCache API.

Partition and version cache entries

Partition cache entries by tenant, user group, or subscription before reusing responses. Also include model ID, embedding model ID, prompt template version, tool configuration version, and safety-policy version in the cache index name, metadata, key prefix, or API Management vary-by value. Reusing a cache entry after a model or prompt-template change can return stale or incompatible responses.

Decide when not to cache

Don't use semantic caching for prompts or responses that are highly personalized, regulated, safety-critical, time-sensitive, or dependent on side effects from tools or transactions. Because semantic caching returns responses based on similarity rather than exact prompt equality, Azure API Management warns that cached responses can be incorrect, outdated, or unsafe for the current request and should be evaluated carefully for the workload. For more information, see llm-semantic-cache-lookup usage notes.

Monitor cache behavior

Track cache hit rate, miss rate, lookup latency, backend LLM latency, token and cost reduction, threshold-distance distribution, stale-response reports, safety escalations, Redis memory use, key count, write failures, and cache operation errors. For gateway-level caching, use Azure API Management tracing and metrics. For app-level caching, log lookup outcome, threshold, distance or score when available, partition ID, model version, prompt version, and whether the response came from cache.

Understand cost, performance, and product fit

When semantic caching works well, it can reduce repeated backend LLM calls, token processing, and end-to-end response latency by returning a stored response for an identical or semantically similar prompt. RedisVL describes semantic caching as a way to reduce API costs and latency by avoiding redundant LLM calls. Azure API Management semantic caching is also designed to reduce bandwidth and backend processing requirements and lower latency for API consumers. The benefit is highest for workloads with repeated, stable questions and responses, such as product FAQs, policy guidance, support triage, documentation assistants, and generated summaries over stable inputs.

Semantic caching also adds work to every eligible request. The application or gateway still needs to create an embedding for the incoming prompt, perform a Redis vector lookup, apply partition and threshold rules, and store cache misses. On a cache hit, the LLM call is avoided. On a miss, the request pays the cache lookup overhead plus the normal model call. Plan for the extra embedding cost, Redis memory, index size, and lookup latency, and measure the net effect with production telemetry rather than assuming savings from cache hit rate alone.

Semantic caching is LLM agnostic in the sense that Redis stores prompt embeddings, responses, and metadata instead of depending on one chat model provider. The cache can sit in front of different LLM APIs, including model APIs exposed through Microsoft Foundry or other API-compatible providers. However, a semantic cache index must use the same embedding model, vector dimensions, and distance behavior for both stored prompts and lookup prompts. Also partition or version the cache by chat model, prompt template, tool behavior, and safety policy so a response generated under one contract isn't reused under another.

Use semantic caching where you control the model-call boundary:

Microsoft AI product area	Fit for semantic caching with Azure Managed Redis
Microsoft 365 Copilot and low-code/no-code Copilot Studio agents	Usually not the best fit for app-level RedisVL caching because these experiences emphasize managed orchestration and low-code authoring. If the agent calls your own API, consider caching behind that API boundary rather than inside the Copilot authoring experience.
Azure API Management in front of LLM APIs	A good fit when you want centralized, policy-based semantic caching without changing application code. Use `llm-semantic-cache-lookup` and `llm-semantic-cache-store` with Azure Managed Redis as the external cache.
Microsoft Foundry custom apps and agents	Often the best fit when you build custom AI applications, agents, tools, or orchestration in code and need control over thresholds, TTLs, partitions, model versions, telemetry, and safety rules. Use RedisVL in the app layer or Azure API Management at the API boundary.

Validate before production

Before enabling semantic caching for production traffic:

Build a test set of prompts that should match and prompts that must not match.
Run the test set through your selected cache path.
Review false positives, stale answers, latency, and token savings.
Tighten thresholds, partitions, TTLs, or no-cache rules as needed.
Repeat validation whenever the embedding model, chat model, prompt template, tool behavior, or safety policy changes.

Commentaires

Cette page a-t-elle été utile ?

Last updated on 2026-09-16