How to configure Azure Cosmos DB for NoSQL global secondary indexes (preview)

APPLIES TO: NoSQL

Important

Azure Cosmos DB for NoSQL global secondary indexes are currently in preview. You can enable this feature by using the Azure portal and the feature can't be disabled. This preview is provided without a service-level agreement. At this time, we don't recommend that you use global secondary indexes for production workloads. Certain features of this preview might not be supported or might have constrained capabilities. For more information, see the supplemental terms of use for Microsoft Azure previews.

Global secondary indexes provide a powerful way to optimize query performance and simplify application logic by storing your data with a different partition key and/ or data model. This article describes how to create global secondary indexes and how to use them to handle cross-partition queries efficiently.

Prerequisites

• An existing Azure Cosmos DB account.
  • If you have an Azure subscription, create a new account.
  • If you don't have an Azure subscription, create a free account before you begin.
  • Alternatively, you can try Azure Cosmos DB free before you commit.

Enable global secondary indexes

The global secondary index feature needs to be enabled for your Azure Cosmos DB account before provisioning a builder or creating index containers.

Azure portal

1. Sign in to the Azure portal.

2. Go to your Azure Cosmos DB for NoSQL account.

3. In the resource menu, select Settings.

4. Navigate to the Features page. Then select Global Secondary Index for NoSQL API (preview) and Enable.

   

Azure CLI

Use the Azure CLI to enable the global secondary index feature either by using a native command or a REST API operation on your Azure Cosmos DB for NoSQL account.

1. Sign in to the Azure CLI.

   az login
   

   Note

   This requires the Azure CLI, see how to install the Azure CLI.

2. Define the variables for the resource group and account name of your existing Azure Cosmos DB for NoSQL account.

   # Variable for resource group name
   $resourceGroupName="<resource-group-name>"
   
   # Variable for account name
   $accountName="<account-name>"
   
   # Variable for Azure subscription
   $subscriptionId="<subscription-id>"
   

3. Create a new JSON file named capabilities.json by using the capabilities manifest.

   {
     "properties": {
       "enableMaterializedViews": true
     }
   }
   

   Note

   The property name to enable global secondary indexes is enableMaterializedViews, which is the former feature name. Enabling materialized views will enable global secondary indexes.

4. Get the identifier of the account and store it in a shell variable named $accountId.

   $accountId="/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.DocumentDB/databaseAccounts/$accountName"
   

5. Enable the preview global secondary index feature for the account by using the REST API and az rest with an HTTP PATCH verb.

   az rest \
       --method PATCH \
       --uri "https://management.azure.com/$accountId/?api-version=2022-11-15-preview" \
       --body @capabilities.json
   

Warning

The global secondary index feature can't be disabled on an account once enabled, however the global secondary builder and index containers themselves can be deprovisioned.

Create a global secondary index builder

After the global secondary index feature is enabled for your account, you'll see a new page in the Settings section of the Azure portal for Global Secondary Index Builder. You must provision a builder before creating global secondary index containers in your account. The builder is responsible for automatically hydrating data in the index containers and keeping them in sync with source containers. Learn more about options for provisioning the global secondary index builder.

Azure portal

1. Sign in to the Azure portal.

2. Go to your Azure Cosmos DB for NoSQL account.

3. In the resource menu, select Global Secondary Index Builder.

4. On the Global Secondary Index Builder page, configure the SKU and the number of instances for the builder.

   Note

   This resource menu option and page appear only when the global secondary index feature is enabled for the account.

5. Select Save.

Azure CLI

1. Define the variables for the resource group and account name of your existing Azure Cosmos DB for NoSQL account.

   # Variable for resource group name
   $resourceGroupName="<resource-group-name>"
   
   # Variable for account name
   $accountName="<account-name>"
   
   # Variable for Azure subscription
   $subscriptionId="<subscription-id>"
   

2. Create a new JSON file named builder.json by using the builder manifest. Update the instanceCount and instanceSize as needed.

   {
     "properties": {
       "serviceType": "materializedViewsBuilder",
       "instanceCount": 1,
       "instanceSize": "Cosmos.D4s"
     }
   }
   

   Note

   The service type is materializedViewsBuilder, which is the former feature name. Creating this resource will provision a global secondary index builder.

3. Get the identifier of the account and store it in a shell variable named $accountId.

   $accountId="/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.DocumentDB/databaseAccounts/$accountName"
   

4. Enable the global secondary index builder for the account using the REST API and az rest with an HTTP PUT verb:

   az rest \
       --method PUT \
       --uri "https://management.azure.com$accountId/services/materializedViewsBuilder/?api-version=2022-11-15-preview" \
       --body @builder.json
   

5. Wait for a couple of minutes, and then check the status by using az rest again with the HTTP GET verb. The status in the output should now be Running.

   az rest \
       --method GET \
       --uri "https://management.azure.com$accountId/services/materializedViewsBuilder/?api-version=2022-11-15-preview"
   

Create a global secondary index

After the global secondary index feature is enabled and the builder is provisioned, you can create index containers.

Create a source container

Global secondary indexes store a copy of data from the source container. Before creating a global secondary index, create the source container that your index container will be built from. If you already have a source container in your Azure Cosmos DB account, you can skip these steps.

1. Use the Azure portal, the Azure SDKs, the Azure CLI, or the REST API to create a source container that has /customerId as the partition key path. Name this source container gsi-src.

   Note

   The /customerId field is used only as an example in this article. For your own containers, select a partition key that works for your solution.

2. Insert a few items in the source container. To follow the examples that are shown in this article, make sure that the items have customerId and emailAddress fields. A sample item might look like this:

   {
     "id": "eaf0338e-2b61-4163-822f-7bef75bf51de",
     "customerId": "36c7cc3d-1709-45c6-819f-10e5586a6cb7",
     "emailAddress": "justine@contoso.com",
     "name": "Justine"
   }
   

   Note

   In this example, you populate the source container with sample data before adding an index container. You can also create a global secondary index from an empty source container.

Create a global secondary index container

Once the source container is created, you can create a global secondary index container using the Azure portal or Azure CLI.

Azure portal

1. Navigate to Data Explorer in your Azure Cosmos DB account. Select your source container, gsi-src in this example, and select New Global Secondary Index from the drop-down.

   

2. The source container ID will be populated for you. In the Index container id field, enter gsi-target.

3. In the Global secondary index definition field, enter SELECT c.customerId, c.emailAddress FROM c.

4. In the Partition key field, enter /emailAddress.

   

5. Configure any other container settings you'd like and select OK to create the global secondary index container.

6. After the global secondary index container is created, the builder automatically syncs data from the source container. Try executing create, update, and delete operations in the source container. You'll see the same changes propagated to items in the index container.

Azure CLI

1. Create a global secondary index named gsi-target with a partition key path that is different from the source container. For this example, specify /emailAddress as the partition key path for the gsi-target container.

   1. Create a definition manifest for a global secondary index and save it in a JSON file named gsi-definition.json:

      {
        "location": "West US 2",
        "tags": {},
        "properties": {
          "resource": {
            "id": "gsi-target",
            "partitionKey": {
              "paths": [
                "/emailAddress"
              ]
            },
            "materializedViewDefinition": {
              "sourceCollectionId": "gsi-src",
              "definition": "SELECT c.customerId, c.emailAddress FROM c"
            }
          },
          "options": {
            "autoscaleSettings": {
              "maxThroughput": 1000
            }
          }
        }
      }        
      

   Important

   In the template, the property for defining global secondary indexes is materializedViewDefinition, which is the former name for this feature.

   Notice that the partition key path is set as /emailAddress. The sourceCollectionId defines the source container and the definition contains a query to determine the data model of the index container. The global secondary index source container and definition query can't be changed once created. Learn more about defining global secondary indexes and the query constraints.

2. Next, make a REST API call to create the global secondary index as defined in the gsi-definition.json file. Use the Azure CLI to make the REST API call.

   1. Create a variable for the name of the global secondary index and source database name:

      # This should match the resource ID you defined in your json file
      $gsiContainerName = "gsi-target"
      
      # Database name for the source and index containers
      $databaseName = "<Database that contains source container>"
      
      # Azure Cosmos DB account name
      $accountName = "<Azure Cosmos DB account name>"
      
      # Resource name for your Azure Cosmos DB account
      $resourceGroupName = "<Resource group for Azure Cosmos DB account>"
      
      # Subscription id for your Azure Cosmos DB account
      $subscriptionId = "<Subscription id>"
      

   2. Construct the resource ID using these variables.

      $accountId = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.DocumentDB/databaseAccounts/$accountName"
      

   3. Make a REST API call to create the global secondary index:

      az rest \
          --method PUT \
          --uri "https://management.azure.com$accountId/sqlDatabases/$databaseName/containers/$gsiContainerName/?api-version=2022-11-15-preview" \
          --body @gsi-definition.json \
          --headers content-type=application/json \
          --output json \
          --verbose
      

   4. Check the status of the global secondary index container creation by using the REST API:

      az rest \
          --method GET \
          --uri "https://management.azure.com$accountId/sqlDatabases/$databaseName/containers/$gsiContainerName/?api-version=2022-11-15-preview" \
          --headers content-type=application/json \
          --query "{mvCreateStatus: properties.Status}"
      

3. After the global secondary index is created, the builder automatically syncs data from the source container. Try executing create, update, and delete operations in the source container. You'll see the same changes propagated to items in the index container.

Query data from global secondary indexes

In this example, we have a source container partitioned on customerId and a global secondary index container partitioned on emailAddress. Without the index container, queries that only include the emailAddress would be cross-partition, but now they can use be executed against the global secondary index instead to increase efficiency.

Querying data from global secondary indexes is similar to querying data from any other container. You can use the Azure portal, Azure SDKs, or REST API to query data in global secondary indexes.

.NET

Container container = client.GetDatabase("gsi-db").GetContainer("gsi-target");

FeedIterator<MyClass> myQuery = container.GetItemQueryIterator<MyClass>(new QueryDefinition("SELECT * FROM c WHERE c.emailAddress = 'justine@contoso.com'"));

Java

CosmosAsyncDatabase container = client.getDatabase("gsi-db");
CosmosAsyncContainer container = database.getContainer("gsi-target");

CosmosPagedFlux<MyClass> pagedFluxResponse = container.queryItems(
        "SELECT * FROM c WHERE c.emailAddress = 'justine@contoso.com'", null, MyClass.class);

Node.js

const database = client.database("gsi-db");
const container = database.container("gsi-target");

const querySpec = {
    query: "SELECT * FROM c WHERE c.emailAddress = 'justine@contoso.com'"
 };
const { resources: items } = await container.items
    .query(querySpec)
    .fetchAll();

Python

database = client.get_database_client("gsi-db")
container = database.get_container_client("gsi-target")

query = "SELECT * FROM c WHERE c.emailAddress = 'justine@contoso.com'"
container.query_items(
    query=query
)

Next steps

Data modeling and partitioning Global secondary index overview