Semantic Reranker in Azure Cosmos DB for NoSQL (preview)

The Semantic Reranker uses an AI model to score and reorder the results from a query based on relevancy to the provided user search phrase or context. By integrating the reranker directly with Azure Cosmos DB, developers can apply reranking on any query results retrieved from any container (for example, using vector, full-text, or hybrid search), while using the latest Azure Cosmos DB SDKs with minimal code changes.

The Semantic Reranker service uses the Microsoft AI Semantic Ranker model, developed internally by Microsoft and available today in Azure AI Search's Semantic Ranker.

Why use Semantic Reranker?

When evaluating a search or retrieval system, four primary metrics define the end-user experience:

• Accuracy / Recall: How closely the retrieved results match the ground truth. For example, approximate vector search (DiskANN) vs exact search (flat)
• Latency: The total time taken to return results from query submission to response
• Relevancy: How well the retrieved documents match the user's intent. For example, a query for "best surf spots in Hawaii", a curated surf guide is more relevant than a general Hawaii travel article

The goal of the Semantic Reranker is to improve relevancy of the ordering of search results compared to the user's question or other context. The AI-powered semantic understanding can provide higher quality results near the top of your search, enhancing end-user experience and enriching agents with more relevant data in Retrieval Augmented Generation (RAG). However, it might not provide benefits across all workloads. Calling the Semantic Reranker also increases the overall latency of the search by introducing another network call and inference time for the model to generate relevancy scores. It's important to evaluate relevance both before and after applying the Semantic Reranker to determine whether it delivers measurable improvements to your search and retrieval scenarios given the increase in latency.

• Enhanced result quality: AI-powered semantic understanding can provide higher quality results near the top of your search, enhancing end-user experience and enriching agents with more relevant data in Retrieval Augmented Generation (RAG)
• Seamless integration in Cosmos DB SDKs: Works with existing vector, full-text, and hybrid search queries
• Minimal code changes: Easy to implement with the latest Azure Cosmos DB SDKs
• Flexible application: Can be applied to any query results from any container

Semantic Reranker output

When you call Semantic Reranker, the response can contain multiple fields, including:

• Original documents: The reranked results contain the same documents that your original query returned, but they can be reordered based on their semantic relevance to your provided context string. The document content and metadata stay the same.
• Relevance score: Each document in the reranked results includes a relevance score from 0 to 1 that indicates how semantically relevant the document is to your provided search terms or context. The higher the score, the higher the relevance. The AI model generates these scores and they help you understand which results are most relevant to the user's intent, the confidence level of semantic matching, and whether to filter results based on relevance.
• Inference latency: The time the service spends on the rerank request.
• Token usage: The number of tokens the reranking request consumes.

Set up Semantic Reranker

Use the Azure portal to enable, disable, and configure Semantic Reranker for a specific Azure Cosmos DB resource.

1. Semantic Reranker is in preview. You might need to run az provider register -n Microsoft.InferenceService by using the Azure CLI before enabling Semantic Reranker in the Azure portal.
2. Go to your Azure Cosmos DB account in the Azure portal.
3. In the resource menu, find the Semantic Reranker setup experience.
4. Review the preview information and enable the feature for your resource. You can return to this experience later to disable Semantic Reranker for the same resource.

1. Grant permissions to all users who need access to the Azure Cosmos DB account. This action creates the required role assignment for using Semantic Reranker.

   If you want to grant access to specific users, use Access Control (IAM) on the Azure Cosmos DB account and assign them the Semantic Reranker User role.

1. Role assignment changes can take a few minutes to propagate. Once the changes take effect, you can start using Semantic Reranker immediately.

RBAC roles for Semantic Reranker

Semantic Reranker uses account-level roles for feature configuration and runtime calls. Grant each identity only the role it needs.

Role name	Role identifier	Description
Inference Account Operator	360cab3d-4340-4c96-816d-682fbd52b3e2	Enables and disables Semantic Reranker on an Azure Cosmos DB account, but doesn't allow runtime reranker calls.
Inference Account Owner	76315a85-9e6b-4514-9780-868b1977b64e	Enables and disables Semantic Reranker on an Azure Cosmos DB account and allows runtime reranker calls.
Semantic Reranker User	6c74a7c5-4a87-40f9-bb03-61e49aecbc78	Allows an application, managed identity, service principal, or user to run Semantic Reranker queries against an Azure Cosmos DB account.

These roles control Semantic Reranker access only. If your application uses Microsoft Entra ID to query Azure Cosmos DB before reranking, also assign Azure Cosmos DB data plane permissions that can read items from the source container. Administrators who assign roles need Owner, User Access Administrator, or equivalent Microsoft Entra permissions.

Assign Semantic Reranker roles

You can assign Semantic Reranker roles in one of four ways:

• Use the Semantic Reranker portal blade to assign the Semantic Reranker User role to identities that already have query or reader roles on the Azure Cosmos DB account and need to call Semantic Reranker.
• Use Access control (IAM) on the Azure Cosmos DB account.
• Use Azure CLI.
• Use an Azure Resource Manager (ARM) template deployment with Bicep.

Assign the role by using Azure CLI

Assign the Semantic Reranker User role to the identity that calls Semantic Reranker. Use the CLI option when you want to assign the role to a user-assigned managed identity, system-assigned managed identity, service principal, or Microsoft Entra user without using the portal.

Before running these commands, replace the placeholder values with your Azure subscription, Azure Cosmos DB account, and principal details. The following example assigns the role at the Azure Cosmos DB account scope.

Assign the role to a managed identity or service principal

For a user-assigned managed identity, system-assigned managed identity, or service principal, use the object's principal ID.

az login

subscriptionId="<subscription-id>"
resourceGroupName="<resource-group-name>"
accountName="<azure-cosmos-db-account-name>"
principalId="<managed-identity-or-service-principal-object-id>"
roleDefinitionId="6c74a7c5-4a87-40f9-bb03-61e49aecbc78"
scope="/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DocumentDB/databaseAccounts/${accountName}"

az role assignment create \
  --assignee-object-id "$principalId" \
  --assignee-principal-type ServicePrincipal \
  --role "$roleDefinitionId" \
  --scope "$scope"

Assign the role to a Microsoft Entra user

For a Microsoft Entra user, get the user's object ID and use it as the principal ID.

az login

subscriptionId="<subscription-id>"
resourceGroupName="<resource-group-name>"
accountName="<azure-cosmos-db-account-name>"
userObjectId="<user-object-id>"
roleDefinitionId="6c74a7c5-4a87-40f9-bb03-61e49aecbc78"
scope="/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DocumentDB/databaseAccounts/${accountName}"

az role assignment create \
  --assignee-object-id "$userObjectId" \
  --assignee-principal-type User \
  --role "$roleDefinitionId" \
  --scope "$scope"

Assign the role by using Bicep

You can also create the role assignment by using an ARM template deployment with Bicep. The following resource definition assigns the Semantic Reranker User role at the resource group scope:

resource semanticRerankerRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(resourceGroup().id, 'semanticRerankerRoleAssignment')
  properties: {
    principalId: '<your principal id>'
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '6c74a7c5-4a87-40f9-bb03-61e49aecbc78')
    principalType: 'ServicePrincipal'
  }
}

Replace <your principal id> with the service principal identifier that should receive permissions. Because this resource definition uses the deployment resource group scope, the role assignment applies to the entire resource group where you deploy the template.

API parameters

Semantic Reranker takes a query context and an optional set of options that control output behavior. The .NET, Python, and Java SDKs use the same conceptual inputs with language-specific parameter names.

.NET

Parameter	Type	Required	Description
rerankContext	string	Yes	Query text or context used for semantic reranking.
documents	IEnumerable<string>	Yes	Candidate documents to rerank, serialized as strings.
options	Dictionary<string, dynamic>	No	Key-value options that modify reranking behavior and output.

Python

Parameter	Type	Required	Description
context	str	Yes	Query text or context used for semantic reranking.
documents	list[str]	Yes	Candidate documents to rerank, serialized as strings.
options	dict[str, object]	No	Key-value options that modify reranking behavior and output.

Java

Parameter	Type	Required	Description
rerankContext	String	Yes	Query text or context used for semantic reranking.
documents	List<String>	Yes	Candidate documents to rerank, serialized as strings.
options	Map<String, Object>	No	Key-value options that modify reranking behavior and output.

Supported options

Option	Type	Default	Description
return_documents	bool	true	If set to true, the reranking response includes the original documents. If set to false, the response doesn't include the original documents.
top_k	int	None	Limits the number of reranked results returned.
sort	bool	None	Sorts the response by reranking score when set to true.
document_type	string	None	Identifies the document format, such as json.
target_paths	string	None	Identifies the document field or fields to consider for reranking.

Use Semantic Reranker from an SDK

Use the supported SDKs to pass query results and a reranking context to Semantic Reranker. The reranking context should represent the user's query or task, while the documents should represent the candidate results returned from vector search, full-text search, hybrid search, or another query.

.NET

namespace DotNetReranker;

using Azure.Identity;
using Microsoft.Azure.Cosmos;
using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Threading.Tasks;

class Program
{
  static async Task Main(string[] args)
  {
    string cdbEndpoint = "https://mytestaccount.documents.azure.com:443/";
    string cdbDatabase = "testdatabase";
    string cdbContainer = "testcontainer";

    Environment.SetEnvironmentVariable(
      "AZURE_COSMOS_SEMANTIC_RERANKER_INFERENCE_ENDPOINT",
      "https://mytestaccount.eastus2.dbinference.azure.com",
      EnvironmentVariableTarget.Process);

    var tokenCredential = new DefaultAzureCredential(new DefaultAzureCredentialOptions()
    {
      ExcludeAzureCliCredential = false,
      ExcludeAzureDeveloperCliCredential = true,
      ExcludeAzurePowerShellCredential = true,
      ExcludeEnvironmentCredential = true,
      ExcludeInteractiveBrowserCredential = true,
      ExcludeSharedTokenCacheCredential = true,
      ExcludeVisualStudioCredential = true,
      ExcludeWorkloadIdentityCredential = true,
      ExcludeManagedIdentityCredential = false
    });

    CosmosClient client = new(cdbEndpoint, tokenCredential);
    Container container = client.GetDatabase(cdbDatabase).GetContainer(cdbContainer);

    QueryDefinition queryDefinition = new(
      "SELECT TOP 15 c.id, c.Title, c.Studio, c.Description, c.YearReleased " +
      "FROM c " +
      "WHERE FullTextContainsAny(c.Description, \"Sandra Bullock\", \"Johnny Depp\", \"comedy\") " +
      "ORDER BY RANK FullTextScore(c.Description, \"comedy\")");

    IList<string> results = new List<string>();

    using FeedIterator<Dictionary<string, dynamic>> resultSet =
      container.GetItemQueryIterator<Dictionary<string, dynamic>>(queryDefinition);

    while (resultSet.HasMoreResults)
    {
      foreach (Dictionary<string, dynamic> item in await resultSet.ReadNextAsync())
      {
        results.Add(JsonSerializer.Serialize(item));
      }
    }

    string rerankingContext = "audience rated PG-13 and public sentiment strong and positive";
    string documentType = "json";
    string targetPaths = "Description";

    SemanticRerankResult rerankedResults = await container.SemanticRerankAsync(
      rerankContext: rerankingContext,
      documents: results,
      options: new Dictionary<string, dynamic>
      {
        { "return_documents", true },
        { "top_k", 5 },
        { "sort", true },
        { "document_type", documentType },
        { "target_paths", targetPaths }
      });

    Console.WriteLine($"Full text search results count: {results.Count}");
    Console.WriteLine($"Reranker results count: {rerankedResults.RerankScores.Count}");
    Console.WriteLine($"Reranking context: {rerankingContext}");
    Console.WriteLine($"Latency details: Data preprocess time: {rerankedResults.Latency["data_preprocess_time"]}, Inference time: {rerankedResults.Latency["inference_time"]}, Postprocess time: {rerankedResults.Latency["postprocess_time"]}");
    Console.WriteLine($"Token usage details: {rerankedResults.TokenUsage["total_tokens"]}");
    Console.WriteLine($"Token usage details: {rerankedResults.TokenUsage["total_tokens"]}");
    Console.WriteLine($"{"DocumentId",-36}|{"Reranked order",-14}|{"FTS order",-9}|{"Reranking score",-15}");
    Console.WriteLine(new string('-', 77));

    for (int index = 0; index < rerankedResults.RerankScores.Count; index++)
    {
      RerankScore score = rerankedResults.RerankScores[index];
      IDictionary<string, object>? document = JsonSerializer.Deserialize<IDictionary<string, object>>(results[score.Index]);

      Console.WriteLine($"{document?["id"],-36}|{index,-14}|{score.Index,-9}|{score.Score,-15}");
    }
  }
}

Python

import json
import os

from azure.cosmos import CosmosClient
from azure.identity import DefaultAzureCredential

os.environ["AZURE_COSMOS_SEMANTIC_RERANKER_INFERENCE_ENDPOINT"] = \
  "https://mytestaccount.eastus2.dbinference.azure.com"

endpoint = "https://mytestaccount.documents.azure.com:443/"
database_name = "testdatabase"
container_name = "testcontainer"

credential = DefaultAzureCredential()
client = CosmosClient(endpoint, credential=credential)
database = client.get_database_client(database_name)
container = database.get_container_client(container_name)

query = """
SELECT TOP 15 c.id, c.Title, c.Studio, c.Description, c.YearReleased
FROM c
WHERE FullTextContainsAny(c.Description, "Sandra Bullock", "Johnny Depp", "comedy")
ORDER BY RANK FullTextScore(c.Description, "comedy")
"""

documents = list(container.query_items(
    query=query,
    enable_cross_partition_query=True,
))

reranking_context = "audience rated PG-13 and public sentiment strong and positive"
document_type = "json"
target_paths = "Description"

reranked_results = container.semantic_rerank(
  context=reranking_context,
  documents=[json.dumps(document) for document in documents],
  options={
    "return_documents": True,
    "top_k": 5,
    "sort": True,
    "document_type": document_type,
    "target_paths": target_paths,
  },
)

for score in reranked_results["Scores"]:
  print(f"index: {score['index']}, score: {score['score']}, document: {score['document']}")

print("Latency stats:")
print(
  f"Preprocess: {reranked_results['latency']['data_preprocess_time']}, "
  f"Inference: {reranked_results['latency']['inference_time']}, "
  f"Postprocess: {reranked_results['latency']['postprocess_time']}"
)

print("Token stats:")
print(f"Token usage: {reranked_results['token_usage']}")

Java

package com.azure.cosmos.semanticrerank;

import com.azure.core.credential.TokenCredential;
import com.azure.cosmos.CosmosAsyncClient;
import com.azure.cosmos.CosmosAsyncContainer;
import com.azure.cosmos.CosmosClientBuilder;
import com.azure.cosmos.models.SemanticRerankResult;
import com.azure.cosmos.models.SemanticRerankScore;
import com.azure.identity.DefaultAzureCredentialBuilder;

import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class SemanticRerankSample {
  public static void main(String[] args) {
    String endpoint = System.getenv("COSMOS_ENDPOINT");
    String databaseName = System.getenv("COSMOS_DATABASE");
    String containerName = System.getenv("COSMOS_CONTAINER");
    String inferenceEndpoint = System.getenv("AZURE_COSMOS_SEMANTIC_RERANKER_INFERENCE_ENDPOINT");

    if (endpoint == null || endpoint.isEmpty()) {
      System.err.println("Set the COSMOS_ENDPOINT environment variable.");
      return;
    }

    if (databaseName == null || databaseName.isEmpty()) {
      System.err.println("Set the COSMOS_DATABASE environment variable.");
      return;
    }

    if (containerName == null || containerName.isEmpty()) {
      System.err.println("Set the COSMOS_CONTAINER environment variable.");
      return;
    }

    if (inferenceEndpoint == null || inferenceEndpoint.isEmpty()) {
      System.err.println("Set the AZURE_COSMOS_SEMANTIC_RERANKER_INFERENCE_ENDPOINT environment variable.");
      return;
    }

    TokenCredential credential = new DefaultAzureCredentialBuilder().build();

    CosmosAsyncClient client = new CosmosClientBuilder()
      .endpoint(endpoint)
      .credential(credential)
      .buildAsyncClient();

    try {
      runSemanticRerankDemo(client, databaseName, containerName);
    } finally {
      client.close();
    }
  }

  static void runSemanticRerankDemo(CosmosAsyncClient client, String databaseName, String containerName) {
    CosmosAsyncContainer container = client.getDatabase(databaseName).getContainer(containerName);

    List<String> documents = Arrays.asList(
      "Berlin is the capital of Germany.",
      "Paris is the capital of France.",
      "Madrid is the capital of Spain.",
      "Rome is the capital of Italy.",
      "London is the capital of England."
    );

    String rerankContext = "What is the capital of France?";

    SemanticRerankResult basicResult = container.semanticRerank(rerankContext, documents, null).block();
    printResults(basicResult);

    Map<String, Object> options = new HashMap<>();
    options.put("return_documents", true);
    options.put("top_k", 3);
    options.put("sort", true);

    SemanticRerankResult customResult = container.semanticRerank(rerankContext, documents, options).block();
    printResults(customResult);
  }

  private static void printResults(SemanticRerankResult result) {
    if (result == null) {
      return;
    }

    System.out.println("Semantic Rerank Results:");

    List<SemanticRerankScore> scores = result.getScores();

    if (scores != null && !scores.isEmpty()) {
      for (int index = 0; index < scores.size(); index++) {
        SemanticRerankScore score = scores.get(index);
        System.out.printf("[%d] index=%d, score=%.7f%n", index, score.getIndex(), score.getScore());

        if (score.getDocument() != null) {
          System.out.println("document: " + score.getDocument());
        }
      }
    }

    Map<String, Object> latency = result.getLatency();

    if (latency != null) {
      System.out.println("Latency:");
      latency.forEach((key, value) -> System.out.printf("%s: %s%n", key, value));
    }

    Map<String, Object> tokenUsage = result.getTokenUsage();

    if (tokenUsage != null) {
      System.out.println("Token usage:");
      tokenUsage.forEach((key, value) -> System.out.printf("%s: %s%n", key, value));
    }
  }
}

Limitations

• Semantic Reranker supports a maximum of 50 documents per rerank call.
• A single context-document pair supports up to 2,048 tokens.

Pricing

Semantic Reranker pricing is $1 USD for 1,000 rerank calls. Regional and cloud pricing might vary. For full details, see the Azure Cosmos DB pricing page.

Supported SDKs

The Azure Cosmos DB .NET, Python, and Java SDKs support Semantic Reranker. For .NET, use the latest preview version of the Azure Cosmos DB .NET SDK. Use the SDK integration to send retrieved documents and user context to the reranker, then use the reranked response in your search, RAG, or agent workflow.

Related content

• Vector Search in Azure Cosmos DB for NoSQL
• Full-text search in Azure Cosmos DB for NoSQL
• Hybrid search in Azure Cosmos DB for NoSQL