Citiți în limba engleză Editare

Partajați prin

Vector store in Azure Cosmos DB for MongoDB vCore

  • Articol
  • Se aplică pentru:
    ✅ MongoDB vCore

Use the Integrated Vector Database in Azure Cosmos DB for MongoDB (vCore) to seamlessly connect your AI-based applications with your data that's stored in Azure Cosmos DB. This integration can include apps that you built by using Azure OpenAI embeddings. The natively integrated vector database enables you to efficiently store, index, and query high-dimensional vector data that's stored directly in Azure Cosmos DB for MongoDB (vCore), along with the original data from which the vector data is created. It eliminates the need to transfer your data to alternative vector stores and incur additional costs.

What is a vector store?

A vector store or vector database is a database designed to store and manage vector embeddings, which are mathematical representations of data in a high-dimensional space. In this space, each dimension corresponds to a feature of the data, and tens of thousands of dimensions might be used to represent sophisticated data. A vector's position in this space represents its characteristics. Words, phrases, or entire documents, and images, audio, and other types of data can all be vectorized.

How does a vector store work?

In a vector store, vector search algorithms are used to index and query embeddings. Some well-known vector search algorithms include Hierarchical Navigable Small World (HNSW), Inverted File (IVF), DiskANN, etc. Vector search is a method that helps you find similar items based on their data characteristics rather than by exact matches on a property field. This technique is useful in applications such as searching for similar text, finding related images, making recommendations, or even detecting anomalies. It is used to query the vector embeddings (lists of numbers) of your data that you created by using a machine learning model by using an embeddings API. Examples of embeddings APIs are Azure OpenAI Embeddings or Hugging Face on Azure. Vector search measures the distance between the data vectors and your query vector. The data vectors that are closest to your query vector are the ones that are found to be most similar semantically.

In the Integrated Vector Database in Azure Cosmos DB for MongoDB (vCore), embeddings can be stored, indexed, and queried alongside the original data. This approach eliminates the extra cost of replicating data in a separate pure vector database. Moreover, this architecture keeps the vector embeddings and original data together, which better facilitates multi-modal data operations, and enables greater data consistency, scale, and performance.

Azure Cosmos DB for MongoDB (vCore) provides robust vector search capabilities, allowing you to perform high-speed similarity searches across complex datasets. To perform vector search in Azure Cosmos DB for MongoDB, you first need to create a vector index. Cosmos DB currently supports three types of vector indexes:

  • DiskANN (Recommended): Ideal for large-scale datasets, leveraging SSDs for efficient memory usage while maintaining high recall in approximate nearest-neighbor (ANN) searches.
  • HNSW: Suited for moderate-sized datasets needing high recall, with a graph-based structure that balances accuracy and resource efficiency.
  • IVF: Uses clustering to optimize search speed in expansive datasets, focusing searches within targeted clusters to accelerate performance.

DiskANN indexes are available on M40 tiers and above. To create the DiskANN index, set the "kind" parameter to "vector-diskann" following the template below:

JavaScript 
{ 
    "createIndexes": "<collection_name>",
    "indexes": [
        {
            "name": "<index_name>",
            "key": {
                "<path_to_property>": "cosmosSearch"
            },
            "cosmosSearchOptions": { 
                "kind": "vector-diskann", 
                "dimensions": <integer_value>,
                "similarity": <string_value>,
                "maxDegree" : <integer_value>, 
                "lBuild" : <integer_value>, 
            } 
        } 
    ] 
}
Field Type Description
index_name string Unique name of the index.
path_to_property string Path to the property that contains the vector. This path can be a top-level property or a dot notation path to the property. Vectors must be a number[] to be indexed and used in vector search results. Using another type, such as double[], prevents the document from being indexed. Non-indexed documents won't be returned in the result of a vector search.
kind string Type of vector index to create. The options are vector-ivf, vector-hnsw, and vector-diskann.
dimensions integer Number of dimensions for vector similarity. DiskANN supports up to 2000 dimensions, with future support planned for 40,000+.
similarity string Similarity metric to use with the index. Possible options are COS (cosine distance), L2 (Euclidean distance), and IP (inner product).
maxDegree integer Maximum number of edges per node in the graph. This parameter ranges from 20 to 2048 (default is 32). Higher maxDegree is suitable for datasets with high dimensionality and/or high accuracy requirements.
lBuild integer Sets the number of candidate neighbors evaluated during DiskANN index construction. This parameter, which ranges from 10 to 500 (default is 50), balances accuracy and computational overhead: higher values improve index quality and accuracy but increase build time

Perform a vector search with DiskANN

To perform a vector search, use the $search aggregation pipeline stage, and query with the cosmosSearch operator. DiskANN allows high-performance searches across massive datasets with optional filtering such as geospatial or text-based filters.

JavaScript 
{
  "$search": {
    "cosmosSearch": {
      "path": "<path_to_property>",
      "query": "<query_vector>",  
      "k": <num_results_to_return>,  
      "filter": {"$and": [
        { "<attribute_1>": { "$eq": <value> } },
        {"<location_attribute>": {"$geoWithin": {"$centerSphere":[[<longitude_integer_value>, <latitude_integer_value>], <radius>]}}}
      ]}
    }
  }
},
Field Type Description
lSearch integer Specifies the size of the dynamic candidate list for search. The default value is 40, with a configurable range from 10 to 1000. Increasing the value enhances recall but may reduce search speed.
k integer Defines the number of search results to return. The k value must be less than or equal to lSearch.

Enabling DiskANN on a new cluster

To enable DiskANN Vector Index on a newly provisioned Azure Cosmos DB for MongoDB (vCore) cluster, follow these steps to perform a cluster-level registration via the Azure CLI:

  1. Log in to Azure CLI
Bash 
az login
  1. Retrieve the current settings for the feature flags on your cluster. This ensures you retain any existing flags while adding the new feature.
Bash 
az resource show --ids "/subscriptions/<sub id>/resourceGroups/<resource group name>/providers/Microsoft.DocumentDB/mongoClusters/<resource name of your Cosmos DB for MongoDB cluster>" --api-version <cluster's api version>
  1. Add the DiskANNIndex flag to the list of preview features without removing any existing ones.
Bash 
az resource patch --ids "/subscriptions/<sub id>/resourceGroups/<resource group name>/providers/Microsoft.DocumentDB/mongoClusters/<resource name of your Cosmos DB for MongoDB cluster>" --api-version <cluster's api version> --properties "{\"previewFeatures\": [\"GeoReplicas\", \"DiskANNIndex\"]}"

Example using a DiskANN Index with Filtering

Add vectors to your database

To use vector search with geospatial filters, add documents that include both vector embeddings and location coordinates. You can create the embeddings by using your own model, Azure OpenAI Embeddings, or another API (such as Hugging Face on Azure).

Python 
from pymongo import MongoClient

client = MongoClient("<your_connection_string>")
db = client["test"]
collection = db["testCollection"]

documents = [
    {"name": "Eugenia Lopez", "bio": "CEO of AdventureWorks", "is_open": 1, "location": [-118.9865, 34.0145], "contentVector": [0.52, 0.20, 0.23]},
    {"name": "Cameron Baker", "bio": "CFO of AdventureWorks", "is_open": 1, "location": [-0.1278, 51.5074], "contentVector": [0.55, 0.89, 0.44]},
    {"name": "Jessie Irwin", "bio": "Director of Our Planet initiative", "is_open": 0, "location": [-118.9865, 33.9855], "contentVector": [0.13, 0.92, 0.85]},
    {"name": "Rory Nguyen", "bio": "President of Our Planet initiative", "is_open": 1, "location": [-119.0000, 33.9855], "contentVector": [0.91, 0.76, 0.83]}
]

collection.insert_many(documents)

Create a DiskANN vector index

The following example demonstrates how to set up a DiskANN vector index with filtering capabilities. This includes creating the vector index for similarity search, adding documents with vector and geospatial properties, and indexing fields for additional filtering.

Python 
db.command({
    "createIndexes": "testCollection",
    "indexes": [
        {
            "name": "DiskANNVectorIndex",
            "key": {
                "contentVector": "cosmosSearch"
            },
            "cosmosSearchOptions": {
                "kind": "vector-diskann",
                "dimensions": 3,
                "similarity": "COS",
                "maxDegree": 32,
                "lBuild": 64
            }
        },
        { 
            "name": "is_open",
            "key": { 
                "is_open": 1 
            }      
        },
        {
            "name": "locationIndex",
            "key": {
                "location": 1
            }
        }
    ]
})

This command creates a DiskANN vector index on the contentVector field in exampleCollection, enabling similarity searches. It also adds:

  • An index on the is_open field, allowing you to filter results based on whether businesses are open.
  • A geospatial index on the location field to filter by geographic proximity.

To find documents with similar vectors within a specific geographic radius, specify the queryVector for similarity search and include a geospatial filter.

Python 
query_vector = [0.52, 0.28, 0.12]
pipeline = [
    {
        "$search": {
            "cosmosSearch": {
                "path": "contentVector",
                "vector": query_vector,
                "k": 5,
                "filter": {
                    "$and": [
                        {"is_open": {"$eq": 1}},
                        {"location": {"$geoWithin": {"$centerSphere": [[-119.7192861804, 34.4102485028], 100 / 3963.2]}}}
                    ]
                }
            }
        }
    }
]

results = list(collection.aggregate(pipeline))
for result in results:
    print(result)

In this example, the vector similarity search returns the top k closest vectors based on the specified COS similarity metric, while filtering results to include only open businesses within a 100-mile radius.

Python 
[
  {
    similarityScore: 0.9745354109084544,
    document: {
      _id: ObjectId("645acb54413be5502badff94"),
      name: 'Eugenia Lopez',
      bio: 'CEO of AdventureWorks',
      is_open: 1,
      location: [-118.9865, 34.0145],
      contentVector: [0.52, 0.20, 0.23]
    }
  },
  {
    similarityScore: 0.9006955671333992,
    document: {
      _id: ObjectId("645acb54413be5502badff97"),
      name: 'Rory Nguyen',
      bio: 'President of Our Planet initiative',
      is_open: 1,
      location: [-119.7302, 34.4005],
      contentVector: [0.91, 0.76, 0.83]
    }
  }
]

This result shows the top similar documents to queryVector, constrained to a 100-mile radius and open businesses. Each result includes the similarity score and metadata, demonstrating how DiskANN in Cosmos DB for MongoDB supports combined vector and geospatial queries for enriched, location-sensitive search experiences.

Get vector index definitions

To retrieve your vector index definition from the collection, use the listIndexes command:

JavaScript 
db.exampleCollection.getIndexes();

In this example, vectorIndex is returned with all the cosmosSearch parameters that were used to create the index:

JavaScript 
[
  { v: 2, key: { _id: 1 }, name: '_id_', ns: 'test.exampleCollection' },
  {
    v: 2,
    key: { vectorContent: 'cosmosSearch' },
    name: 'vectorSearchIndex',
    cosmosSearch: {
      kind: <index_type>, // options are `vector-ivf`, `vector-hnsw`, and `vector-diskann`
      numLists: 3,
      similarity: 'COS',
      dimensions: 3
    },
    ns: 'test.exampleCollection'
  }
]

Filtered vector search (preview)

You can now execute vector searches with any supported query filter such as $lt, $lte, $eq, $neq, $gte, $gt, $in, $nin, and $regex. Enable the "filtering vector search" feature in the "Preview Features" tab of your Azure Subscription. Learn more about preview features here.

First, you'll need to define an index for your filter in addition to a vector index. For example, you can define the filter index on a property

JavaScript 
db.runCommand({ 
     "createIndexes": "<collection_name",
    "indexes": [ {
        "key": { 
            "<property_to_filter>": 1 
               }, 
        "name": "<name_of_filter_index>" 
    }
    ] 
});

Next, you can add the "filter" term to your vector search as shown below. In this example the filter is looking for documents where the "title" property is not in the list of ["not in this text", "or this text"].

JavaScript 

db.exampleCollection.aggregate([
  {
      '$search': {
          "cosmosSearch": {
              "vector": "<query_vector>",
              "path": <path_to_vector>,
              "k": num_results,
              "filter": {<property_to_filter>: {"$nin": ["not in this text", "or this text"]}}
          },
          "returnStoredSource": True }},
      {'$project': { 'similarityScore': { '$meta': 'searchScore' }, 'document' : '$$ROOT' }
}
]);

Important

While in preview, filtered vector search may require you to adjust your vector index parameters to achieve higher accuracy. For example, increasing m, efConstruction, or efSearch when using HNSW, or numLists, or nProbes when using IVF, may lead to better results. You should test your configuration before use to ensure that the results are satisfactory.

Use LLM Orchestration tools

Use as a vector database with Semantic Kernel

Use Semantic Kernel to orchestrate your information retrieval from Azure Cosmos DB for MongoDB vCore and your LLM. Learn more here.

https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/connectors/memory/azure_cosmosdb

Use as a vector database with LangChain

Use LangChain to orchestrate your information retrieval from Azure Cosmos DB for MongoDB vCore and your LLM. Learn more here.

Use as a semantic cache with LangChain

Use LangChain and Azure Cosmos DB for MongoDB (vCore) to orchestrate Semantic Caching, using previously recorded LLM responses that can save you LLM API costs and reduce latency for responses. Learn more here

Features and limitations

  • Supported distance metrics: L2 (Euclidean), inner product, and cosine.
  • Supported indexing methods: IVFFLAT, HNSW, and DiskANN (Preview)
  • Indexing vectors up to 2,000 dimensions in size.
  • Indexing applies to only one vector per path.
  • Only one index can be created per vector path.

Summary

This guide demonstrates how to create a vector index, add documents that have vector data, perform a similarity search, and retrieve the index definition. By using our integrated vector database, you can efficiently store, index, and query high-dimensional vector data directly in Azure Cosmos DB for MongoDB vCore. It enables you to unlock the full potential of your data via vector embeddings, and it empowers you to build more accurate, efficient, and powerful applications.

Related content

Next step

Create a lifetime free-tier vCore cluster for Azure Cosmos DB for MongoDB