Monitor Azure Cosmos DB

Article
03/30/2023
11 minutes to read

APPLIES TO: NoSQL MongoDB Cassandra Gremlin Table

When you have critical applications and business processes that rely on Azure resources, you want to monitor those resources for availability, performance, and operation. This article describes the monitoring data generated by Azure Cosmos DB databases and how you can use the features of Azure Monitor to analyze and alert on this data.

You can monitor your data with client-side and server-side metrics. When using server-side metrics, you can monitor the data stored in Azure Cosmos DB with the following options:

Monitor from Azure Cosmos DB portal: You can monitor with the metrics available within the Metrics tab of the Azure Cosmos DB account. The metrics on this tab include throughput, storage, availability, latency, consistency, and system level metrics. By default, these metrics have a retention period of seven days. To learn more, see the Monitoring data collected from Azure Cosmos DB section of this article.
Monitor with metrics in Azure monitor: You can monitor the metrics of your Azure Cosmos DB account and create dashboards from the Azure Monitor. Monitor collects the Azure Cosmos DB metrics by default. You don't need to explicitly configure anything. These metrics are collected with one-minute granularity. The granularity might vary based on the metric you choose. By default, these metrics have a retention period of 30 days.

Most of the metrics that are available from the previous options are also available in these metrics. The dimension values for the metrics, such as container name, are case insensitive. Use case insensitive comparison when doing string comparisons on these dimension values. To learn more, see the Analyze metric data section of this article.
Monitor with diagnostic logs in Azure Monitor: You can monitor the logs of your Azure Cosmos DB account and create dashboards from the Monitor. Data such as events and traces that occur at a second granularity are stored as logs. For example, if the throughput of a container changes, the properties of an Azure Cosmos DB account change. The logs capture these events. You can analyze these logs by running queries on the gathered data. To learn more, see the Analyze log data section of this article.
Monitor programmatically with SDKs: You can monitor your Azure Cosmos DB account programmatically by using the .NET, Java, Python, Node.js SDKs, and the headers in REST API. To learn more, see the Monitoring Azure Cosmos DB programmatically section of this article.

The following image shows different options available to monitor Azure Cosmos DB account through the Azure portal:

Diagram shows monitoring options available in Azure portal.

On the client-side, you can collect the details for request charge, activity ID, exception and stack trace information, HTTP status and substatus code, and diagnostic string to debug issues that might occur. You need this information to reach out to the Azure Cosmos DB support team.

Monitor overview

The Overview page in the Azure portal for each Azure Cosmos DB account includes a brief view of the resource usage, such as total requests, requests that resulted in a specific HTTP status code, and hourly billing. This information is helpful. It's only a small amount of the monitoring data is available from this pane. Some of this data is collected automatically. It's available for analysis as soon as you create the resource. You can enable other types of data collection after some configuration.

What is Azure Monitor?

Azure Cosmos DB creates monitoring data using Azure Monitor. Monitor is a full stack monitoring service in Azure that provides a complete set of features to monitor your Azure resources in addition to resources in other clouds and on-premises.

If you're not already familiar with monitoring Azure services, start with Monitoring Azure resources with Azure Monitor, which describes the following concepts:

What is Azure Monitor?
Costs associated with monitoring
Monitoring data collected in Azure
Configuring data collection
Standard tools in Azure for analyzing and alerting on monitoring data

The following sections build on this article. They describe the specific data gathered from Azure Cosmos DB and provide examples for configuring data collection and analyzing this data with Azure tools.

Azure Cosmos DB insights

Azure Cosmos DB insights is a feature based on the workbooks feature of Azure Monitor and uses the same monitoring data collected for Azure Cosmos DB described in the following sections. Use Monitor for a view of the overall performance, failures, capacity, and operational health of all your Azure Cosmos DB resources in a unified interactive experience. Use the other features of Monitor for detailed analysis and alerting. To learn more, see Explore Azure Cosmos DB insights.

Note

When creating containers, make sure you don't create two containers with the same name but different casing. Some parts of the Azure platform are not case sensitive. This situation can result in confusion or collision of telemetry and actions on containers with such names.

Monitoring data

Azure Cosmos DB collects the same kinds of monitoring data as other Azure resources, which are described in Monitoring data from Azure resources. For a detailed reference of the logs and metrics created by Azure Cosmos DB, see Azure Cosmos DB monitoring data reference.

The Overview page in the Azure portal for each Azure Cosmos DB database includes a brief view of the database usage including its request and hourly billing usage. This information is useful. It's only a small amount of the monitoring data available. Some of this data is collected automatically and available for analysis as soon as you create the database. You can enable more data collection after some configuration.

Screenshot shows the Cosmos DB Overview page.

Collection and routing

Platform metrics and the Activity log are collected and stored automatically. Use a diagnostic setting to route it to other locations.

Resource logs aren't collected and stored until you create a diagnostic setting and route them to one or more locations.

To learn about the process to create a diagnostic setting using the Azure portal and diagnostic query examples, see Create diagnostic setting to collect platform logs and metrics in Azure. When you create a diagnostic setting, you specify which categories of logs to collect.

The following sections discuss the metrics and logs you can collect.

Analyzing metrics

Azure Cosmos DB provides a custom experience for working with metrics. You can analyze metrics for Azure Cosmos DB with metrics from other Azure services using Metrics explorer by opening Metrics from the Azure Monitor menu. For more information about this tool, see Getting started with Azure Metrics Explorer.

You can also monitor server-side latency, request unit usage, and normalized request unit usage for your Azure Cosmos DB resources.

For a list of the platform metrics collected for Azure Cosmos DB, see Monitoring Azure Cosmos DB data reference metrics.

All metrics for Azure Cosmos DB are in the namespace Azure Cosmos DB standard metrics. You can use the following dimensions with these metrics when adding a filter to a chart:

CollectionName
DatabaseName
OperationType
Region
StatusCode

You can see a list of all resource metrics supported in Azure Monitor.

View operation level metrics for Azure Cosmos DB

Sign in to the Azure portal.
Select Monitor from the left navigation bar, and select Metrics.
In the Select a scope pane, select a Subscription. You can narrow the scopes by Resource type and Locations. Select Azure Cosmos DB accounts to quickly find your account.
Select an Azure Cosmos DB account, and choose Apply.
Next, select a metric from the list of available metrics. You can select metrics specific to request units, storage, latency, availability, Cassandra, and others. For more information about the available metrics, see Metrics by category. In this example, select Total request units and Avg as the aggregation value.

In addition to these details, you can also select the Time range and Time granularity of the metrics. At the most, you can view metrics for the past 30 days.

After you apply the filter, the Azure portal displays a chart based on your filter. You can see the average number of request units consumed per minute for the selected period.

Add filters to metrics

You can also filter metrics and the chart displayed by properties:

CapacityType
CollectionName
DatabaseName
OperationType
Region
Status
StatusCode

To filter the metrics, select Add filter and choose the required property, such as OperationType. Then select a value, such as Query.

The graph then displays the request units consumed for the query operation for the selected period. The operations run by using a stored procedure aren't logged. They aren't available under the OperationType metric.

Screenshot shows the Add filter option.

Group metrics

You can group metrics by using the Apply splitting option. For example, you can group the request units per operation type and view the graph for all the operations at once, as shown in the following image:

Screenshot shows the Apply splitting filter.

Analyzing logs

Data in Azure Monitor Logs is stored in tables. Each table has its own set of unique properties.

All resource logs in Azure Monitor have the same fields followed by service-specific fields. The common schema is outlined in Azure Monitor resource log schema. For the types of resource logs collected for Azure Cosmos DB, see Monitoring Azure Cosmos DB data reference.

The Activity log is a platform that provides insight into subscription-level events. You can view it independently or route it to Monitor Logs, where you can do much more complex queries using Log Analytics.

Azure Cosmos DB stores data in the following tables.

Table	Description
AzureDiagnostics	Common table used by multiple services to store Resource logs. Resource logs from Azure Cosmos DB can be identified with `MICROSOFT.DOCUMENTDB`.
AzureActivity	Common table that stores all records from the Activity log.

Sample Kusto queries

Prior to using Log Analytics to issue Kusto queries, you must enable diagnostic logs for control plane operations. When you enable diagnostic logs, select between storing your data in a single AzureDiagnostics table (legacy) or resource-specific tables.

When you select Logs from the Azure Cosmos DB menu, Log Analytics is opened with the query scope set to the current Azure Cosmos DB account. Log queries only include data from that resource.

Important

If you want to run a query that includes data from other accounts or data from other Azure services, select Logs from the Azure Monitor menu. For more information, see Log query scope and time range in Azure Monitor Log Analytics.

Here are some queries that you can enter into the Log search search bar to help you monitor your Azure Cosmos DB resources. The exact text of the queries depends on the collection mode you selected when you enabled diagnostics logs.

AzureDiagnostics table (legacy)
Resource-specific table

To query for all control-plane logs from Azure Cosmos DB:

AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="ControlPlaneRequests"

To query for all data-plane logs from Azure Cosmos DB:

AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="DataPlaneRequests"

To query for a filtered list of data-plane logs, specific to a single resource:

AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="DataPlaneRequests"
| where Resource=="<account-name>"

Important

In the AzureDiagnostics table, many fields are case sensitive and uppercase including, but not limited to ResourceId, ResourceGroup, ResourceProvider, and Resource.

To get a count of data-plane logs, grouped by resource:

AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="DataPlaneRequests"
| summarize count() by Resource

To generate a chart for data-plane logs, grouped by the type of operation:

AzureDiagnostics
| where ResourceProvider=="MICROSOFT.DOCUMENTDB"
| where Category=="DataPlaneRequests"
| summarize count() by OperationName
| render columnchart

To query for all control-plane logs from Azure Cosmos DB:

CDBControlPlaneRequests

To query for all data-plane logs from Azure Cosmos DB:

CDBDataPlaneRequests

To query for a filtered list of data-plane logs, specific to a single resource:

CDBDataPlaneRequests
| where AccountName=="<account-name>"

To get a count of data-plane logs, grouped by resource:

CDBDataPlaneRequests
| summarize count() by AccountName

To generate a chart for data-plane logs, grouped by the type of operation:

CDBDataPlaneRequests
| summarize count() by OperationName
| render piechart

These examples are just a small sampling of the rich queries that can be performed in Monitor using the Kusto Query Language. For more information, see samples for Kusto queries.

Alerts

Azure Monitor alerts proactively notify you when Monitor finds important conditions in your monitoring data. Alerts allow you to identify and address issues in your system before your customers notice them. You can set alerts on metrics, logs, and the activity log. Different types of alerts have benefits and drawbacks

For example, the following table lists few alert rules for your resources. You can find a detailed list of alert rules from the Azure portal. To learn more, see how to configure alerts article.

Alert type	Condition	Description
Rate limiting on request units (metric alert)	Dimension name: StatusCode, Operator: Equals, Dimension values: 429	Alerts if the container or a database has exceeded the provisioned throughput limit.
Region failed over	Operator: Greater than, Aggregation type: Count, Threshold value: 1	When a single region is failed over. This alert is helpful if you didn't enable service-managed failover.
Rotate keys (activity log alert)	Event level: Informational, Status: started	Alerts when the account keys are rotated. You can update your application with the new keys.