How to instrument error budgets (SRE concept) in Azure monitor

Question

How to instrument error budgets (SRE concept) in Azure monitor

Dinesh Dharmawardena 1

My company is using Azure monitor as our monitoring platform compared to some of the other competing products such as new relic or datadog. The problem is Azure monitor unlike those products does not natively support SRE concepts. I can still leverage the platform to implement SLO alerts and SLIs but I'm struggling with how to implement Error budgets, especially considering the error budget needs to be persisted.

SwathiDhanwada-MSFT 18,996 Reputation points Moderator

2022-05-23T04:19:42.25+00:00

@Dinesh Dharmawardena Welcome to Microsoft Q & A Community Forum. As per my understanding, error budgets calculate the time the services can be downtime without any consequences. To retrieve this information, you can create Service health alerts and as well as resource-level health activity log alerts. And you can retrieve all the alerts triggered in your subscription using this PowerShell Sample.

In Azure, you have different ways to monitor the resources based on the service. To get an overview on how to monitor your resources for reliability, you can refer this documentation.
Luis Arocas del Castillo 1 Reputation point

2022-08-12T18:31:24.683+00:00

I would like to implement SLO alerting for my Application gateways. I would like to compute my error rate by dividing: 5XXErrors/TotalRequests. This way I can tell the error rate of my application and alert when my application is generating too many 5XX errors relative to the number of total requests.

As of now, I don't see a way to generate a dashboard in which I can divide 5XX Errors by the TotalRequests. Therefore I don't see a way to alert based on SLOs.

I can get both metrics separately (5XX errors and TotalRequets) but I cannot do the division to get the error rate. Is there any way to accomplish this?
Luis Arocas del Castillo 1 Reputation point

2022-08-12T18:32:09.85+00:00

I would like to implement SLO alerting for my Application gateways. I would like to compute my error rate by dividing: 5XXErrors/TotalRequests. This way I can tell the error rate of my application and alert when my application is generating too many 5XX errors relative to the number of total requests.

As of now, I don't see a way to generate a dashboard in which I can divide 5XX Errors by the TotalRequests. Therefore I don't see a way to alert based on SLOs.

I can get both metrics separately (5XX errors and TotalRequets) but I cannot do the division to get the error rate. Is there any way to accomplish this?
Maxim Sergeev 6,586 Reputation points Microsoft Employee

2022-08-12T20:24:35.817+00:00

Hi there,

@Luis Arocas del Castillo in this case you need to export metrics somewhere outside the Metrics Storage to make this aggregation and calculate your results.
For example, use DiagnosticLogs to export all metrics to LogAnalytics worskspace, query the results and send alerts as required.

1 answer

Your answer

SwathiDhanwada-MSFT 18,996 Reputation points Moderator

2022-05-23T04:19:42.25+00:00

@Dinesh Dharmawardena Welcome to Microsoft Q & A Community Forum. As per my understanding, error budgets calculate the time the services can be downtime without any consequences. To retrieve this information, you can create Service health alerts and as well as resource-level health activity log alerts. And you can retrieve all the alerts triggered in your subscription using this PowerShell Sample.

In Azure, you have different ways to monitor the resources based on the service. To get an overview on how to monitor your resources for reliability, you can refer this documentation.
Luis Arocas del Castillo 1 Reputation point

2022-08-12T18:31:24.683+00:00

I would like to implement SLO alerting for my Application gateways. I would like to compute my error rate by dividing: 5XXErrors/TotalRequests. This way I can tell the error rate of my application and alert when my application is generating too many 5XX errors relative to the number of total requests.

As of now, I don't see a way to generate a dashboard in which I can divide 5XX Errors by the TotalRequests. Therefore I don't see a way to alert based on SLOs.

I can get both metrics separately (5XX errors and TotalRequets) but I cannot do the division to get the error rate. Is there any way to accomplish this?
Luis Arocas del Castillo 1 Reputation point

2022-08-12T18:32:09.85+00:00

I would like to implement SLO alerting for my Application gateways. I would like to compute my error rate by dividing: 5XXErrors/TotalRequests. This way I can tell the error rate of my application and alert when my application is generating too many 5XX errors relative to the number of total requests.

As of now, I don't see a way to generate a dashboard in which I can divide 5XX Errors by the TotalRequests. Therefore I don't see a way to alert based on SLOs.

I can get both metrics separately (5XX errors and TotalRequets) but I cannot do the division to get the error rate. Is there any way to accomplish this?
Maxim Sergeev 6,586 Reputation points Microsoft Employee

2022-08-12T20:24:35.817+00:00

Hi there,

@Luis Arocas del Castillo in this case you need to export metrics somewhere outside the Metrics Storage to make this aggregation and calculate your results.
For example, use DiagnosticLogs to export all metrics to LogAnalytics worskspace, query the results and send alerts as required.

Answer 1

Maxim Sergeev 6,586 Microsoft Employee

@Dinesh Dharmawardena

Regarding your scenario it depends on what telemetry you want to use. If ApplicationInsights, go to kusto, find right metrics (calls, dependencies, uris, "functions") and measure it as required
For example,

You can do the same for platform metrics with the similar approach

Share via

How to instrument error budgets (SRE concept) in Azure monitor

1 answer

Your answer