Add an AI Search index resource to a Databricks app

Add AI Search indexes as Databricks Apps resources to enable semantic search and similarity-based retrieval in your applications. AI Search indexes store and query high-dimensional vector embeddings, powering use cases like retrieval-augmented generation (RAG), semantic search, and recommendation systems.

Databricks AI Search was formerly known as Databricks Vector Search.

Privilege requirements

To access an AI Search index, the app's service principal must have the USE CATALOG privilege on the parent catalog, the USE SCHEMA privilege on the parent schema, and the SELECT privilege on the index. When you add the index resource, Azure Databricks automatically grants these privileges to the app's service principal.

For this automatic granting to succeed, one of the following must be true for each privilege:

  • For USE CATALOG: Either all account users have the USE CATALOG privilege on the catalog, or you have the MANAGE privilege on the catalog.
  • For USE SCHEMA: Either all account users have the USE SCHEMA privilege on the schema, or you have the MANAGE privilege on the schema.
  • For SELECT: Either all account users have the SELECT privilege on the index, or you have the MANAGE privilege on the index.

If the index uses a Databricks-managed embedding model on a provisioned throughput or custom serving endpoint, also add that endpoint as an app resource with Can query permission, plus any separate query embedding endpoint (model_endpoint_name_for_query). get_index() reads these endpoints and fails without access. Pay-per-token Foundation Model APIs endpoints, such as databricks-gte-large-en, don't need an app resource unless your account restricts foundation models with Unity Catalog permissions, in which case the service principal needs EXECUTE on the model. See Set up a service principal.

For more information about querying AI Search indexes with these permissions, see How to query an AI Search index.

See Unity Catalog privileges reference.

Add an AI Search index resource

Before you add an AI Search index as a resource, review the app resource prerequisites.

  1. In the App resources section when you create or edit an app, click + Add resource > Vector search index.
  2. Choose an AI Search index from the available indexes in your workspace. The index must already exist in Unity Catalog.
  3. Select the permission level for your app:
    • Can select: Grants the app permission to query the AI Search index for similarity searches. Corresponds to the SELECT privilege.
  4. (Optional) Specify a custom resource key, which is how you reference the index in your app configuration. The default key is vector-search-index.

Note

AI Search indexes are Unity Catalog tables with type TABLE_ONLINE_VECTOR_INDEX_REPLICA or TABLE_ONLINE_VECTOR_INDEX_DIRECT. When you select an AI Search index, you're selecting a specially configured table that supports semantic search operations.

Environment variables

When you deploy an app with an AI Search index resource, Azure Databricks exposes the full three-level name through environment variables that you can reference using the valueFrom field.

Example configuration:

env:
  - name: VECTOR_SEARCH_INDEX
    valueFrom: vector-search-index # Use your custom resource key if different

Using the index in your application:

Query the index with the databricks-ai-search Python SDK and the app's service principal credentials. When the workspace meets the index URL requirements, the SDK sends queries to the index URL, which can use the extra capacity of an endpoint with a target queries per second (QPS). Queries sent with WorkspaceClient().vector_search_indexes.query_index use the workspace query URL, which is capped at a few tens of QPS.

Add databricks-ai-search version 0.81 or above to your app's dependencies. See Manage dependencies for a Databricks app.

import os
from databricks.ai_search.client import AISearchClient

# DATABRICKS_HOST can be set without a scheme, and the SDK requires one.
host = os.environ["DATABRICKS_HOST"]
if not host.startswith(("https://", "http://")):
    host = f"https://{host}"

# Pass the app's service principal credentials so queries use the index URL.
client = AISearchClient(
    workspace_url=host,
    service_principal_client_id=os.environ["DATABRICKS_CLIENT_ID"],
    service_principal_client_secret=os.environ["DATABRICKS_CLIENT_SECRET"],
)

# Create the index object when the app starts and reuse it for every query.
index = client.get_index(index_name=os.environ["VECTOR_SEARCH_INDEX"])

results = index.similarity_search(
    query_text="What is machine learning?",  # Or pass query_vector for self-managed embeddings.
    columns=["id", "text"],
    num_results=10,
)

# Each row lists values in the same order as the manifest columns.
column_names = [column["name"] for column in results["manifest"]["columns"]]
for row in results["result"]["data_array"]:
    print(dict(zip(column_names, row)))

Replace columns with the columns of your index. To reach higher throughput, set a target QPS on a standard endpoint.

For more information, see Use environment variables to access resources.

Remove an AI Search index resource

When you remove an AI Search index resource from an app, the app's service principal loses access to the index. The index itself remains unchanged and continues to be available for other users and applications that have appropriate permissions.

Best practices

Consider the following when you work with AI Search index resources:

  • Make sure the app's service principal has access to underlying data sources if the index references other tables.
  • Monitor query performance and adjust index configuration or embedding models if response times degrade.
  • Consider index refresh schedules to keep embeddings synchronized with source data.
  • Use appropriate similarity metrics (cosine, euclidean, dot product) based on your embedding model.
  • An endpoint can be online before its indexes are ready to query. To check an index without blocking, read ready and detailed_state in the status field that index.describe() returns. To wait for an index, call index.wait_until_ready(), which polls every 30 seconds and raises an error if the index goes offline or the timeout passes.
  • Sidst opdateret den