Azure Managed Prometheus - Custom Scrape from IaaS VM's

Question

Hi All,

I am trying to get my head around the Azure Managed Prometheus, when setup on the AKS cluster using Grafana I can see the basic data for the AKS cluster. I am looking at ways to use the managed Prometheus but with custom scrapes against IaaS VM's for monitoring of other applications using various node exporters etc.

I've setup previously the scrape configs on Prometheus configuration file when running on a VM, but can't get my head around where to do it in Azure managed version.

Any pointers?

Answer 1

To set up custom scrapes for IaaS VMs using Azure Managed Prometheus, you can utilize Prometheus exporters to collect metrics from your VMs. Here are the steps you can follow:

1. Configure Exporters: Ensure that you have the necessary Prometheus exporters installed on your IaaS VMs. Common exporters include Node Exporter for system metrics and others depending on the applications you want to monitor.
2. Create a Scrape Configuration: In Azure Managed Prometheus, you can create a scrape configuration using a ConfigMap. This configuration will define how Prometheus collects metrics from your VMs. You can refer to the example provided in the context for customizing your scrape config.
3. Use Custom Resource Definitions (CRDs): Managed Prometheus automatically deploys custom resource definitions (CRDs) for pod monitors and service monitors. You can configure these CRDs to include your IaaS VM metrics collection.
4. Deploy the Configuration: Once you have your scrape configuration ready, you can deploy it using kubectl. For example, you can create a ConfigMap that references your scrape configuration file and apply it to your AKS cluster.
5. Verify Metric Collection: After deploying the configuration, ensure that the metrics from your IaaS VMs are being collected correctly. You may need to check the Azure Monitor workspace to confirm that the data is being ingested.

For detailed guidance on customizing the collection and using CRDs, refer to the Azure documentation on Prometheus exporters and managed service for Prometheus.

---

References:

• Use Prometheus exporters for common workloads with managed service for Prometheus
• Enable Azure Monitor managed service for Prometheus

Answer 2

Hello Toby Glenn, it sounds like you're trying to configure Azure Managed Prometheus for custom scrapes from IaaS VMs to monitor additional applications using node exporters. That can be a bit tricky since the setup differs from the traditional Prometheus configuration you're familiar with.

In Azure Managed Prometheus, the way you define scrape configurations isn't done through a configuration file like in self-hosted Prometheus. Instead, you typically use Azure Monitor to manage your resources. Here are a few pointers to get you started:

1. Connect Your IaaS VMs to Azure Monitor: First, ensure your IaaS VMs are connected to Azure Monitor. You'll want to enable monitoring on these VMs by installing the Azure Monitor agent if you haven't already.
2. Use Azure Monitor for containers: Given that you're starting with AKS, you'll likely want to leverage Azure Monitor for containers. The data collected from your VMs will be sent to Azure Monitor, which can work in conjunction with Prometheus.
3. Configure Custom Scrapes: For custom scraping, you might need to configure a custom Prometheus datasource in Grafana that points to your managed Prometheus endpoint. Custom scraping generally requires setting up your node exporters and ensuring that they can be accessed from your AKS cluster.
4. Documentation: Here’s some useful documentation that can guide you in this process:
   • Azure Managed Prometheus Overview
   • Monitor containers with Azure Monitor
   • Azure Monitor for VMs

Follow-Up Questions:

1. Have you already configured Azure Monitor on your IaaS VMs?
2. Are you using any specific node exporters for your applications?
3. What challenges are you facing when trying to set up the custom scrapes?

I hope this gives you a better understanding of how to approach your setup! If you have more specifics about your current configuration or any errors you're encountering, feel free to share!