High availability with Spring Data Cosmos

This Spring Boot 4 sample enables threshold-based availability (hedging) and the per-partition circuit breaker (PPCB) on one Cosmos client. It saves users with different lastName partition keys, runs synchronous and reactive cross-partition queries by firstName, verifies the results, and deletes only its own users.

Set up and run

Prerequisites

  • JDK 17 or later and Maven.
  • An Azure Cosmos DB for NoSQL account, its global endpoint, and Direct-mode/HTTPS connectivity.
  • Session-or-stronger account consistency: Session, Bounded Staleness, or Strong. Immediate queries can miss recent writes on Eventual or Consistent Prefix accounts; a client setting cannot upgrade weaker account consistency.
  • Multiple readable regions for regional read recovery; multiple writable regions for write-side PPCB. The client does not change the account's regional configuration.
  • Azure CLI for resource provisioning and local Microsoft Entra ID (AAD) authentication.

1. Create the database and container

Provision the resources once with an administrative identity. The application does not create them:

az login
az account set --subscription "<subscription-id>"
$resourceGroup = "<resource-group>"
$accountName = "<account-name>"
$databaseName = "cosmos-high-availability-sample"
$accountId = az cosmosdb show --resource-group $resourceGroup --name $accountName --query id -o tsv

az cosmosdb sql database create --resource-group $resourceGroup --account-name $accountName --name $databaseName
az cosmosdb sql container create --resource-group $resourceGroup --account-name $accountName `
    --database-name $databaseName --name highAvailabilityUsers --partition-key-path /lastName --throughput 400

For a serverless account, omit --throughput. An administrator can perform this management-plane step separately from running the application.

2. Configure the account and authentication

Set these environment variables, or put the same names in cosmos-v4.properties in your user home directory, outside the checkout. Environment variables override the imported file.

ACCOUNT_HOST=https://<account-name>.documents.azure.com:443/
COSMOS_DATABASE=cosmos-high-availability-sample
PREFERRED_LOCATIONS=East US,West US

Replace the account and region names with your own. Preferred regions control routing order; they are optional for the application, but the live test below requires exactly two distinct regions.

With no account key configured, the application uses DefaultAzureCredential, which can use the Azure CLI login from step 1. Assign the application's identity the Cosmos DB Built-in Data Contributor data-plane role on the sample container:

$principalId = az ad signed-in-user show --query id -o tsv
az cosmosdb sql role assignment create --resource-group $resourceGroup --account-name $accountName `
    --principal-id $principalId `
    --role-definition-id "$accountId/sqlRoleDefinitions/00000000-0000-0000-0000-000000000002" `
    --scope "$accountId/dbs/$databaseName/colls/highAvailabilityUsers"

Role assignment requires management-plane permissions and can take time to propagate. Azure management-plane Contributor alone does not grant item/query access. For a service principal or managed identity, use that identity's object ID instead of the developer's. DefaultAzureCredential supports both.

Remove ACCOUNT_KEY from both the environment and imported file: a nonempty key takes precedence. The pre-created container and autoCreateContainer=false avoid resource-creation calls that this data-plane role does not authorize.

Account key (optional)

If key authentication is enabled on the account, add this value to the external file or environment:

ACCOUNT_KEY=<account-key>

Never commit credentials or pass keys in command-line arguments.

3. Build and run

From cosmos/azure-spring-data-cosmos/cosmos-high-availability-sample:

mvn clean verify
mvn spring-boot:run

Expected output:

Synchronous cross-partition query returned 2 users.
Reactive cross-partition query completed with 2 users.

mvn clean verify runs only credential-free unit tests. The application waits for both query results, then closes its Spring context and Cosmos client. See cleanup for resource ownership.

Configure availability

Both repositories share the client configured in SampleAppConfiguration.java. Override its defaults with environment variables or the external properties file:

Setting Variable Default
SDK query-page timeout COSMOS_END_TO_END_TIMEOUT PT10S
Initial hedging threshold COSMOS_AVAILABILITY_THRESHOLD PT5S
Subsequent hedging step COSMOS_AVAILABILITY_THRESHOLD_STEP PT0.1S
PPCB read failure count COSMOS_PPCB_READ_FAILURE_THRESHOLD 10
PPCB write failure count COSMOS_PPCB_WRITE_FAILURE_THRESHOLD 5

Durations use ISO-8601 syntax. The timeout must exceed the initial threshold; PPCB counts must be at least 10 for reads and 5 for writes.

The timeout and hedging portion of the builder, using these defaults:

import com.azure.cosmos.CosmosClientBuilder;
import com.azure.cosmos.CosmosEndToEndOperationLatencyPolicyConfigBuilder;
import com.azure.cosmos.ThresholdBasedAvailabilityStrategy;
import com.azure.identity.DefaultAzureCredentialBuilder;
import java.time.Duration;

CosmosClientBuilder builder = new CosmosClientBuilder()
    .endpoint(System.getenv("ACCOUNT_HOST"))
    .credential(new DefaultAzureCredentialBuilder().build())
    .directMode()
    .endToEndOperationLatencyPolicyConfig(
        new CosmosEndToEndOperationLatencyPolicyConfigBuilder(Duration.ofSeconds(10))
            .availabilityStrategy(new ThresholdBasedAvailabilityStrategy(
                Duration.ofSeconds(5), Duration.ofMillis(100)))
            .build());

Return this builder from the Spring @Bean rather than building another client. The full configuration also sets preferred regions, selects authentication, and enables PPCB before client construction.

Mechanism Behavior
Hedging Starts parallel requests in alternate eligible regions after a threshold; can add RU/network cost.
PPCB Tracks terminal failures for a physical partition in a region and avoids unhealthy locations on subsequent requests; initial requests can still fail.

PPCB's COSMOS.PARTITION_LEVEL_CIRCUIT_BREAKER_CONFIG setting is process-wide: configure it before creating any clients, not per request. To experiment without explicit PPCB, omit enablePartitionLevelCircuitBreaker(); to omit the explicit hedging/timeout policy, remove endToEndOperationLatencyPolicyConfig(...). Other SDK defaults can still apply.

Test regional Gateway recovery

The opt-in GatewayRecoveryIT.java establishes healthy queries, then simulates a Gateway outage during steady-state operation. It uses the real sample configuration with a fresh client per case:

Injected fault Target Repository APIs
Connection delay All Gateway requests to the first preferred region Synchronous and reactive
Response delay All Gateway requests to the first preferred region Synchronous and reactive

This models an outage of the regional Gateway path.

After completing setup, set PREFERRED_LOCATIONS to exactly two distinct account regions and run:

mvn verify -P live-tests

Failsafe runs these tests only with live-tests. Missing live prerequisites cause failures, not skips.

Assertions require Gateway fault hits and traffic to the healthy second region, the expected user IDs and completion within the SDK timeout plus a five-second test allowance, and diagnostics identifying the runtime SDK. The query's CosmosDiagnosticsContext must also record both preferred regions as contacted. The test logs these regions and the observed Gateway operation types to show the recovery path. Faults affect only the test client and are disabled before cleanup. The test library is test-scoped and excluded from the runnable application.

These cases demonstrate the specified recovery scenarios, not every outage mode or PPCB opening. They do not distinguish hedging from ordinary retry-based recovery. Live Entra ID execution has not been validated; configuration/authentication selection is covered by unit tests.

Cleanup

The application and live tests delete only their uniquely named users, tolerate missing items, and preserve original failures if cleanup also fails. Both deletes are attempted even after an ambiguous write. The application closes the client, but the pre-created database/container remain and can incur charges. Remove only dedicated sample resources, never shared data.

Resources