Monitor Azure Cosmos DB

Article
11/16/2022
11 minutes to read

APPLIES TO: NoSQL MongoDB Cassandra Gremlin Table

When you have critical applications and business processes relying on Azure resources, you want to monitor those resources for their availability, performance, and operation. This article describes the monitoring data generated by Azure Cosmos DB databases and how you can use the features of Azure Monitor to analyze and alert on this data.

You can monitor your data with client-side and server-side metrics. When using server-side metrics, you can monitor the data stored in Azure Cosmos DB with the following options:

Monitor from Azure Cosmos DB portal: You can monitor with the metrics available within the Metrics tab of the Azure Cosmos DB account. The metrics on this tab include throughput, storage, availability, latency, consistency, and system level metrics. By default, these metrics have a retention period of seven days. To learn more, see the Monitoring data collected from Azure Cosmos DB section of this article.
Monitor with metrics in Azure monitor: You can monitor the metrics of your Azure Cosmos DB account and create dashboards from the Azure Monitor. Azure Monitor collects the Azure Cosmos DB metrics by default, you will not need to explicitly configure anything. These metrics are collected with one-minute granularity, the granularity may vary based on the metric you choose. By default, these metrics have a retention period of 30 days. Most of the metrics that are available from the previous options are also available in these metrics. The dimension values for the metrics such as container name are case-insensitive. So you need to use case-insensitive comparison when doing string comparisons on these dimension values. To learn more, see the Analyze metric data section of this article.
Monitor with diagnostic logs in Azure Monitor: You can monitor the logs of your Azure Cosmos DB account and create dashboards from the Azure Monitor. Data such as events and traces that occur at a second granularity are stored as logs. For example, if the throughput of a container changes, the properties of an Azure Cosmos DB account are changed, and these events are captured within the logs. You can analyze these logs by running queries on the gathered data. To learn more, see the Analyze log data section of this article.
Monitor programmatically with SDKs: You can monitor your Azure Cosmos DB account programmatically by using the .NET, Java, Python, Node.js SDKs, and the headers in REST API. To learn more, see the Monitoring Azure Cosmos DB programmatically section of this article.

The following image shows different options available to monitor Azure Cosmos DB account through Azure portal:

Monitoring options available in Azure portal

When using Azure Cosmos DB, at the client-side you can collect the details for request charge, activity ID, exception/stack trace information, HTTP status/sub-status code, diagnostic string to debug any issue that might occur. This information is also required if you need to reach out to the Azure Cosmos DB support team.

Monitor overview

The Overview page in the Azure portal for each Azure Cosmos DB account includes a brief view of the resource usage, such as total requests, requests that resulted in a specific HTTP status code, and hourly billing. This information is helpful, however only a small amount of the monitoring data is available from this pane. Some of this data is collected automatically and is available for analysis as soon as you create the resource. You can enable other types of data collection with some configuration.

What is Azure Monitor?

Azure Cosmos DB creates monitoring data using Azure Monitor, which is a full stack monitoring service in Azure that provides a complete set of features to monitor your Azure resources in addition to resources in other clouds and on-premises.

If you're not already familiar with monitoring Azure services, start with the article Monitoring Azure resources with Azure Monitor, which describes the following concepts:

What is Azure Monitor?
Costs associated with monitoring
Monitoring data collected in Azure
Configuring data collection
Standard tools in Azure for analyzing and alerting on monitoring data

The following sections build on this article by describing the specific data gathered from Azure Cosmos DB and providing examples for configuring data collection and analyzing this data with Azure tools.

Azure Cosmos DB insights

Azure Cosmos DB insights is a feature based on the workbooks feature of Azure Monitor and uses the same monitoring data collected for Azure Cosmos DB described in the sections below. Use Azure Monitor for a view of the overall performance, failures, capacity, and operational health of all your Azure Cosmos DB resources in a unified interactive experience, and use the other features of Azure Monitor for detailed analysis and alerting. To learn more, see the Explore Azure Cosmos DB insights article.

Note

When creating containers, make sure you don’t create two containers with the same name but different casing. That’s because some parts of the Azure platform are not case-sensitive, and this can result in confusion/collision of telemetry and actions on containers with such names.

Monitoring data

Azure Cosmos DB collects the same kinds of monitoring data as other Azure resources, which are described in Monitoring data from Azure resources. See Azure Cosmos DB monitoring data reference for a detailed reference of the logs and metrics created by Azure Cosmos DB.

The Overview page in the Azure portal for each Azure Cosmos DB database includes a brief view of the database usage including its request and hourly billing usage. This is useful information but only a small amount of the monitoring data available. Some of this data is collected automatically and available for analysis as soon as you create the database while you can enable more data collection with some configuration.

Overview page

Collection and routing

Platform metrics and the Activity log are collected and stored automatically, but can be routed to other locations by using a diagnostic setting.

Resource Logs aren't collected and stored until you create a diagnostic setting and route them to one or more locations.

See Create diagnostic setting to collect platform logs and metrics in Azure for the detailed process for creating a diagnostic setting using the Azure portal and some diagnostic query examples. When you create a diagnostic setting, you specify which categories of logs to collect.

The metrics and logs you can collect are discussed in the following sections.

Analyzing metrics

Azure Cosmos DB provides a custom experience for working with metrics. You can analyze metrics for Azure Cosmos DB with metrics from other Azure services using Metrics explorer by opening Metrics from the Azure Monitor menu. See Getting started with Azure Metrics Explorer for details on using this tool. You can also check out how to monitor server-side latency, request unit usage, and normalized request unit usage for your Azure Cosmos DB resources.

For a list of the platform metrics collected for Azure Cosmos DB, see Monitoring Azure Cosmos DB data reference metrics article.

All metrics for Azure Cosmos DB are in the namespace Azure Cosmos DB standard metrics. You can use the following dimensions with these metrics when adding a filter to a chart:

CollectionName
DatabaseName
OperationType
Region
StatusCode

For reference, you can see a list of all resource metrics supported in Azure Monitor.

View operation level metrics for Azure Cosmos DB

Sign in to the Azure portal.
Select Monitor from the left-hand navigation bar, and select Metrics.
From the Metrics pane > Select a resource > choose the required subscription, and resource group. For the Resource type, select Azure Cosmos DB accounts, choose one of your existing Azure Cosmos DB accounts, and select Apply.
Next you can select a metric from the list of available metrics. You can select metrics specific to request units, storage, latency, availability, Cassandra, and others. To learn in detail about all the available metrics in this list, see the Metrics by category article. In this example, let's select Request units and Avg as the aggregation value.

In addition to these details, you can also select the Time range and Time granularity of the metrics. At max, you can view metrics for the past 30 days. After you apply the filter, a chart is displayed based on your filter. You can see the average number of request units consumed per minute for the selected period.

Add filters to metrics

You can also filter metrics and the chart displayed by a specific CollectionName, DatabaseName, OperationType, Region, and StatusCode. To filter the metrics, select Add filter and choose the required property such as OperationType and select a value such as Query. The graph then displays the request units consumed for the query operation for the selected period. The operations executed via Stored procedure aren't logged so they aren't available under the OperationType metric.

Add a filter to select the metric granularity

You can group metrics by using the Apply splitting option. For example, you can group the request units per operation type and view the graph for all the operations at once as shown in the following image:

Add apply splitting filter

Analyzing logs

Data in Azure Monitor Logs is stored in tables where each table has its own set of unique properties.

All resource logs in Azure Monitor have the same fields followed by service-specific fields. The common schema is outlined in Azure Monitor resource log schema. For a list of the types of resource logs collected for Azure Cosmos DB, see Monitoring Azure Cosmos DB data reference.

The Activity log is a platform that provides insight into subscription-level events. You can view it independently or route it to Azure Monitor Logs, where you can do much more complex queries using Log Analytics.

Azure Cosmos DB stores data in the following tables.

Table	Description
AzureDiagnostics	Common table used by multiple services to store Resource logs. Resource logs from Azure Cosmos DB can be identified with `MICROSOFT.DOCUMENTDB`.
AzureActivity	Common table that stores all records from the Activity log.

Sample Kusto queries

Prior to using Log Analytics to issue Kusto queries, you must enable diagnostic logs for control plane operations. When enabling diagnostic logs, you will select between storing your data in a single AzureDiagnostics table (legacy) or resource-specific tables.

When you select Logs from the Azure Cosmos DB menu, Log Analytics is opened with the query scope set to the current Azure Cosmos DB account. Log queries will only include data from that resource.

Important

If you want to run a query that includes data from other accounts or data from other Azure services, select Logs from the Azure Monitor menu. For more information, see Log query scope and time range in Azure Monitor Log Analytics.

Here are some queries that you can enter into the Log search search bar to help you monitor your Azure Cosmos DB resources. The exact text of the queries will depend on the collection mode you selected when you enabled diagnostics logs.

AzureDiagnostics table (legacy)
Resource-specific table

To query for all control-plane logs from Azure Cosmos DB:

AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="ControlPlaneRequests"

To query for all data-plane logs from Azure Cosmos DB:

AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="DataPlaneRequests"

To query for a filtered list of data-plane logs, specific to a single resource:
```
AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="DataPlaneRequests"
| where Resource=="<account-name>"
```
Important

In the AzureDiagnostics table, many fields are case-sensitive and uppercase including, but not limited to; ResourceId, ResourceGroup, ResourceProvider, and Resource.

To get a count of data-plane logs, grouped by resource:

AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="DataPlaneRequests"
| summarize count() by Resource

To generate a chart for data-plane logs, grouped by the type of operation:

AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="DataPlaneRequests"
| summarize count() by OperationName
| render columnchart

To query for all control-plane logs from Azure Cosmos DB:
```
CDBControlPlaneRequests
```
To query for all data-plane logs from Azure Cosmos DB:
```
CDBDataPlaneRequests
```
To query for a filtered list of data-plane logs, specific to a single resource:
```
CDBDataPlaneRequests
| where AccountName=="<account-name>"
```

To get a count of data-plane logs, grouped by resource:

CDBDataPlaneRequests
| summarize count() by AccountName

To generate a chart for data-plane logs, grouped by the type of operation:

CDBDataPlaneRequests
| summarize count() by OperationName
| render piechart

These examples are just a small sampling of the rich queries that can be performed in Azure Monitor using the Kusto Query Language. For more information, see samples for Kusto queries.

Alerts

Azure Monitor alerts proactively notify you when important conditions are found in your monitoring data. They allow you to identify and address issues in your system before your customers notice them. You can set alerts on metrics, logs, and the activity log. Different types of alerts have benefits and drawbacks

For example, the following table lists few alert rules for your resources. You can find a detailed list of alert rules from the Azure portal. To learn more, see how to configure alerts article.

Alert type	Condition	Description
Rate limiting on request units (metric alert)	Dimension name: StatusCode, Operator: Equals, Dimension values: 429	Alerts if the container or a database has exceeded the provisioned throughput limit.
Region failed over	Operator: Greater than, Aggregation type: Count, Threshold value: 1	When a single region is failed over. This alert is helpful if you didn't enable service-managed failover.
Rotate keys(activity log alert)	Event level: Informational, Status: started	Alerts when the account keys are rotated. You can update your application with the new keys.