High availability with Azure Managed Prometheus

Aoustin, Rod 0

Hi,

I'm having trouble understanding how Azure ensures high availability with the Prometheus managed service.

Every search I've performed related to high availability (HA) with "Azure Monitor," "Azure Metrics," or "Azure Prometheus" redirects me to this page:

https://learn.microsoft.com/en-us/azure/azure-monitor/logs/availability-zones

This page explains that to support more than one zone, we need to create a dedicated cluster, and this feature is only supported in a limited number of regions.

However, this page specifically refers to Azure Monitor Logs. Does this information also apply to metrics?

I'm trying to understand how to safeguard my Prometheus metrics and ensure the availability of my Prometheus endpoints in two different scenarios:

If one zone goes down.
If an entire region goes down.

Thank you!

Rod

Monalla-MSFT 12,861 Reputation points

2023-08-04T14:56:38.71+00:00

@Aoustin, Rod - Just checking in to see if the below answer helped. If this answers your query, do click "Accept as Answer" and click "Yes" for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

1 answer

Monalla-MSFT 12,861 Reputation points

2023-07-28T14:41:19.58+00:00

@Aoustin, Rod - Welcome to Microsoft Q&A and thanks for reaching out to us.

Prometheus metrics is built on the same back end as Azure Monitor Metrics and has high availability built in. It's not something that we document though because it maybe something we charge for later. All our metrics have resiliency built in, but there is no current offering for failover of Azure Monitor workspace.

Hope this helps. and please feel free to reach out if you have any further questions.

If the above response was helpful, please feel free to "Accept as Answer" and click "Yes" so it can be beneficial to the community.
Please sign in to rate this answer.
Aoustin, Rod 0 Reputation points

2023-07-28T14:54:31.75+00:00

when you say "high availability", does it mean cross zone?

Monalla-MSFT 12,861 Reputation points

2023-07-31T17:40:14.2633333+00:00

Aoustin, Rod - Yes.

Aoustin, Rod 0 Reputation points

2023-09-18T09:19:17.32+00:00

thanks. this link:

https://techcommunity.microsoft.com/t5/azure-observability-blog/introducing-azure-monitor-managed-service-for-prometheus/ba-p/3600185

says:

"Data is then routed to our highly available ingestion endpoint and then stored in our Prometheus-compatible timeseries database. The data is automatically stored in the region you selected when enabling Managed Prometheus as well as in a paired region in the same Geo. When querying the data from our Prometheus compatible HTTP API, we automatically route the queries to the primary data store and will automatically move to the secondary store as needed."

so metrics are also replicated cross region (within same geo). correct?
That being said if the primary region is down, I will not be able to reach the prometheus endpoint because it is regional (so it can't move to secondary)? is there a manual procedure to recover those metrics from the secondary region? like creating another workspace in the secondary region with the option to select the replicated metrics so I can query them? if not then what's the point of this replication cross region. is it only used internally in case only the data store is not available in the region but the (prometheus) backend service is still up and running?

I'm looking at a DR scenario. worst case I will duplicate the workspace (+managed grafana) in a secondary region and push metrics from AKS to those 2 clusters, so if primary is downed I can just switch to the grafana on the secondary region. but don't want to replicate metrics if that's something which is already done as per my understanding from the link above.

another confusing link:

https://learn.microsoft.com/en-us/azure/azure-monitor/logs/availability-zones

or here:

https://learn.microsoft.com/en-us/azure/reliability/migrate-monitor-log-analytics

"Azure Monitor support for availability zones requires a Log Analytics workspace linked to an Azure Monitor dedicated cluster. Dedicated clusters are a deployment option that enables advanced capabilities for Azure Monitor Logs including availability zones."

so it sounds like standard workspace is not HA and stays within 1 zone. The zonal availability can be overcome with a "dedicated Azure Monitor cluster" . which I understand being much more expensive ("Clusters require a minimum ingestion commitment of 500 GB per day.") and limited to some regions: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/availability-zones#data-resilience---supported-regions

thanks,

Rod
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

High availability with Azure Managed Prometheus

1 answer

Your answer