Problem with Azure Cosmos DB for MongoDB vCore using a DiskANN Index with Filtering

Question

Problem with Azure Cosmos DB for MongoDB vCore using a DiskANN Index with Filtering

Anonymous

According to: https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search?WT.mc_id=Portal-Microsoft_Azure_Resources&tabs=diskann there should be possible to use vector search with DiskANN Index with Filtering. There is even an example (that I have followed). Apart from following the example I have also (to my knowledge) followed all pre-requisites (e.g., M40 cluster tier, enabling of DiskANN on the cluster, enabled the "filtering vector search" feature in the "Preview Features" tab on my Azure Subscription).

What works: vector search using DiskANN (which verifies that most things are configures set-up correctly)

What does NOT work: filtering

Again; I have enabled the "filtering vector search" feature in the "Preview Features" tab on my Azure Subscription and verified that it is "registered".

I get the following error message:

pymongo.errors.OperationFailure: $filter is not supported for vector search yet., full error: {'ok': 0.0, 'errmsg': '$filter is not supported for vector search yet.', 'code': 115, 'codeName': 'CommandNotSupported'}

I am little bit confused, given that I have followed the documentation (see link above) which states that filtering is available (in preview).

Can anyone help me with this problem?

Mahesh Kurva 10,520 Reputation points Microsoft External Staff Moderator

2025-01-06T22:06:53.83+00:00

Hi @Jesper Echardt,

Just checking in to see if the below answer helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Anonymous

Hi,

vector_store_filter_test.txt
I have run multiple trials for this (with the same error output that the $filter is not supported for vector search yet.), but the last trial is using the python code base from https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search?WT.mc_id=Portal-Microsoft_Azure_Resources&tabs=diskann together with my own credentials (that I am confident are working) and Azure Cosmos DB for MongoDB vCore.

Apart from my own credentials/account (e.g., model key and connections strings etc) I am also using a Azure Cosmos DB for MongoDB vCore with the python example code (which should be the db that the code base is applicable for) that is configured in accordance to the link above (e.g., M40 cluster tier, enabling of DiskANN on the cluster, enabled the "filtering vector search" feature in the "Preview Features" tab on my Azure Subscription).

I have verified that vector search is working for my Cosmos DB for MongoDB vCore

When I check the JSON view (for the db) I see the following:

{
    "apiVersion": "2024-06-01-preview",
    "id": "...",
    "name": "mongodb...",
    "type": "microsoft.documentdb/mongoclusters",
    "location": "swedencentral",
    "tags": {},
    "properties": {
        "provisioningState": "Succeeded",
        "clusterStatus": "Ready",
        "administratorLogin": "cosmos",
        "serverVersion": "7.0",
        "nodeGroupSpecs": [
            {
                "name": "",
                "kind": "Shard",
                "sku": "M40",
                "diskSizeGB": 128,
                "enableHa": false,
                "nodeCount": 1
            }
        ],
        "connectionString": "...",
        "earliestRestoreTime": "2024-12-05T11:03:39Z",
        "privateEndpointConnections": [
            {
                "id": "...",
                "systemData": {
                    "createdAt": "2024-12-06T14:44:02.092Z",
                    "createdBy": "...",
                    "createdByType": "Application",
                    "lastModifiedAt": "2024-12-06T14:44:02.092Z",
                    "lastModifiedBy": "...",
                    "lastModifiedByType": "Application"
                },
                "properties": {
                    "privateEndpoint": {
                        "id": "/subscriptions/..."
                    },
                    "groupIds": [
                        "MongoCluster"
                    ],
                    "privateLinkServiceConnectionState": {
                        "status": "Approved",
                        "description": "Auto-approved",
                        "actionsRequired": "None"
                    }
                }
            }
        ],
        "publicNetworkAccess": "Enabled",
        "previewFeatures": [
            "GeoReplicas",
            "DiskANNIndex"
        ],
        "replica": {
            "role": "Primary",
            "replicationState": "Active"
        },
        "infrastructureVersion": "2.0"
    },
    "systemData": {
        "createdAt": "2024-12-02T10:54:05.368Z",
        "createdBy": "...",
        "createdByType": "User",
        "lastModifiedAt": "2024-12-15T18:49:14",
        "lastModifiedBy": "wh_lev_jesec@...",
        "lastModifiedByType": "User"
    }
}

One reflection, I am not seeing anything related to "filter" here for the "previewFeatures", but I see the "filtering vector search" feature in the "Preview Features" tab on my Azure Subscription and verified that it is "registered".

I have attached the latest code I run (again, based on the Microsoft python example base but with my own credentials and Cosmos DB for MongoDB vCore) as txt-file (it is a Jupyter Notebook file, so you need to change the file extension to ipynb to be able to run it. This though I was not allowed to upload that kind of file).

Anonymous

I copied out the code from the Jupyter Notebook to make it easier for you. I am only using the Notebook when debugging. All steps, but the last works (I.e., performing the vector search with filter fails. However, I can do the vector search successfully without the filter applied)

import os
import numpy as np
from openai import AzureOpenAI
from pymongo import MongoClient

# Get the API key, endpoint, and deployment name from environment variables
api_key = os.environ['WH_AZURE_OPENAI_API_KEY']
api_endpoint = os.environ['WH_AZURE_OPENAI_ENDPOINT']
# Define the GPT model to use
gpt_model_deployment = os.environ['WH_AZURE_OPENAI_DEPLOYMENT']
# Instantiate the AzureOpenAI client
client = AzureOpenAI(
    api_key=api_key,
    azure_endpoint=api_endpoint,
    api_version="2024-08-01-preview"  # works for gpt-4o
)

# Retrieve credentials for Cosmos db (MongoDB)
# connection string, including username, and password
connection_string = os.environ['WH_AI_DB_CONNECTION_STRING']
# Attempt to establish connection to Cosmos MongoClient
try:
    cosmos_mongo_client = MongoClient(connection_string)
# The ping command is lightweight and does not require auth.
    cosmos_mongo_client.admin.command('ping')
    print("✅ Cosmos MongoDB connected!")
    try:
        # successful connection; print server info
        info = cosmos_mongo_client.server_info()
        print(f"✅ Server info: {info}")
    except Exception as exc:
        print(f"❌ Error reading database server info: {exc}")
        raise exc
except Exception as exc:
	print(f"❌ Database not available, error: {exc}")
# Connect to willhem database in the Azure Cosmos MongoDB
try:
    db = cosmos_mongo_client['test3']
    existing_collections = db.list_collection_names()
    # first_doc = collection.find_one()
except Exception as exc:
	existing_collections = ["<error>"]
	# first_doc = {"<error>": "<error>"}


# Add vectors to your database #
collection = db["testCollection"]
documents = [
    {"name": "Eugenia Lopez", "bio": "CEO of AdventureWorks", "is_open": 1,
        "location": [-118.9865, 34.0145], "contentVector": [0.52, 0.20, 0.23]},
    {"name": "Cameron Baker", "bio": "CFO of AdventureWorks", "is_open": 1,
        "location": [-0.1278, 51.5074], "contentVector": [0.55, 0.89, 0.44]},
    {"name": "Jessie Irwin", "bio": "Director of Our Planet initiative", "is_open": 0,
        "location": [-118.9865, 33.9855], "contentVector": [0.13, 0.92, 0.85]},
    {"name": "Rory Nguyen", "bio": "President of Our Planet initiative", "is_open": 1,
        "location": [-119.0000, 33.9855], "contentVector": [0.91, 0.76, 0.83]}
]
collection.insert_many(documents)

# Create a DiskANN vector index #
db.command({
    "createIndexes": "testCollection",
    "indexes": [
        {
            "name": "DiskANNVectorIndex",
            "key": {
                "contentVector": "cosmosSearch"
            },
            "cosmosSearchOptions": {
                "kind": "vector-diskann",
                "dimensions": 3,
                "similarity": "COS",
                "maxDegree": 32,
                "lBuild": 64
            }
        },
        {
            "name": "is_open",
            "key": {
                "is_open": 1
            }
        },
        {
            "name": "locationIndex",
            "key": {
                "location": 1
            }
        }
    ]
})

# Perform a Vector Search #
query_vector = [0.52, 0.28, 0.12]
pipeline = [
    {
        "$search": {
            "cosmosSearch": {
                "path": "contentVector",
                "vector": query_vector,
                "k": 5,
                "filter": {
                    "$and": [
                        {"is_open": {"$eq": 1}},
                        {"location": {"$geoWithin": {"$centerSphere": [
                            [-119.7192861804, 34.4102485028], 100 / 3963.2]}}}
                    ]
                }
            }
        }
    }
]
results = list(collection.aggregate(pipeline))
for result in results:
    print(result)

So when the filter is used for the search I get the following error message with the code above:

OperationFailure: $filter is not supported for vector search yet., full error: {'ok': 0.0, 'errmsg': '$filter is not supported for vector search yet.', 'code': 115, 'codeName': 'CommandNotSupported'}

Anonymous

2025-01-08T14:24:37.6233333+00:00

I accidentally answered your request for seeing the code earlier in the thread (so see comments and code there).

3 answers

Your answer

Mahesh Kurva 10,520 Reputation points Microsoft External Staff Moderator

2025-01-06T22:06:53.83+00:00

Hi @Jesper Echardt,

Just checking in to see if the below answer helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Anonymous

2025-01-08T14:24:37.6233333+00:00

I accidentally answered your request for seeing the code earlier in the thread (so see comments and code there).

Answer 1

Anonymous

Yes, as I have stated previously in the thread, I have read the documentation (and to my knowledge I have also followed all the requirements and pre-requisites)).

Also, as can be seen in the provided code (by me) in this thread/errand, I have defined an index with the db.command (see details earlier in the thread).

Vivek 0 Reputation points

2025-01-13T17:28:49.4266667+00:00

Facing the exact same issue and getting the same error. All pre requisities of M40 cluster, index creation etc thoughly checked. we keep getting the error raise OperationFailure(errmsg, code, response, max_wire_version) pymongo.errors.OperationFailure: $filter is not supported for vector search yet., full error: {'ok': 0.0, 'errmsg': '$filter is not supported for vector search yet.', 'code': 115, 'codeName': 'CommandNotSupported'} Please provide a resolution for this issue. Link to others also facing this issue:

https://github.com/langchain-ai/langchain/issues/23963#issuecomment-2463634463
Oury Ba-MSFT 21,151 Reputation points Microsoft Employee Moderator

2025-01-14T22:52:56+00:00

Jesper Echardt Thank you for reaching out.

I would suggest raising a support ticket so we can further investigate this issue.

Please let us know if you don't have a support plan. Please do share the support ticket number once opened.

Regards,

Oury
Mahesh Kurva 10,520 Reputation points Microsoft External Staff Moderator

2025-01-16T15:30:10.09+00:00

Hi Jesper Echardt,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Mahesh Kurva 10,520 Reputation points Microsoft External Staff Moderator

2025-01-17T17:51:21.22+00:00

Hi Jesper Echardt,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 2

Sajeetharan 2,261 Microsoft Employee

Have you checked the documentation?

To use filtered vector search effectively, you first need to define an index for your filter alongside the vector index.

db.runCommand({ "createIndexes": "<collection_name", "indexes": [ { "key": { "<property_to_filter>": 1 }, "name": "<name_of_filter_index>" } ] });

Anonymous

2025-01-09T10:16:11.02+00:00

See answer below

Answer 3

Hi @Jesper Echardt,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

As I understand, you are encountering an error related to filtering when attempting to use vector search with DiskANN on Azure Cosmos DB.

Here are a few steps you can take to troubleshoot this issue:

Double-check that the “filtering vector search” feature is fully registered and active in your Azure Subscription. Sometimes, there might be a delay in the feature becoming fully operational.

For more information, please refer the document: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/preview-features?tabs=azure-portal

Ensure you are using the latest versions of pymongo and any related Azure SDKs. Updates might include bug fixes or enhancements that address this issue.
Revisit the documentation to ensure that all steps and configurations are correctly followed. Sometimes, small details can be missed.
Since this feature is in preview, there might be known issues or limitations that are not fully documented. Reaching out to Azure support request.

For more information, please refer the document: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/preview-features?tabs=azure-portal#register-preview-feature

Hope this helps. Do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Anonymous

2025-01-07T09:01:31.0366667+00:00

Hi,

Still not working.

I have double-checked and confirmed that “filtering vector search” feature is fully registered and active in my Azure Subscription

I have verified that I have pymongo 4.10.1 installed (latest to my knowledge)
I have updated openai package (do not think that is related, but it is the only thing where I see Azure specifics in the packages used, I.e. "from openai import AzureOpenAI") to latest.

I have revisited the documentation (I.e., https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search?WT.mc_id=Portal-Microsoft_Azure_Resources&tabs=diskann) and I cannot see anything that I have missed.

Still get this error message:

pymongo.errors.OperationFailure: $filter is not supported for vector search yet., full error: {'ok': 0.0, 'errmsg': '$filter is not supported for vector search yet.', 'code': 115, 'codeName': 'CommandNotSupported'}
Mahesh Kurva 10,520 Reputation points Microsoft External Staff Moderator

2025-01-07T22:59:47.7466667+00:00

Hi @Jesper Echardt,

To better understand the issue, could you please share the code you are using for the vector search with filtering? This will help in identifying any potential issues or areas that might need adjustment.

Share via

Problem with Azure Cosmos DB for MongoDB vCore using a DiskANN Index with Filtering

3 answers

Your answer