In Azure Monitor then Metrics, you can select Azure VMs as scope and pull a list of pre-set metrics i.e. CPU percentage, memory available, etc..... But it lacks certain metrics such as the network bandwidth usage, or disk free space, etc. So what I did was I created a Data Collection Rule, in which Data Source is from VMs metrics and added those network bandwidth usage/disk free space metrics etc.... and destination was set to Azure Metrics. The rule was created successfully and I was also able to associate all Azure VMs to the rule. Everything looked good. But when I went back to the Monitor/Metrics, I still don't see those needed metrics to show up from the list. Did I do it right or I missed something? Thanks,

Hi PerhapsCloud , Thanks for reaching out to Microsoft Q&A. The metrics are not showing up in the metrics explorer could be due to a few reasons, consider the following points; Data Collection Rule Latency: After creating a DCR, there can be some delay before the data starts flowing into Azure Monitor and becomes visible in Metrics Explorer. Wait for a bit and then refresh the metrics dashboard to see if the new metrics appear. Correct Namespace and Metric Type: Ensure that when you go back to Metrics Explorer, you are selecting the correct namespace for the custom metrics. Sometimes custom metrics can appear under a different namespace or category than the default "VM metrics." Check if there are new metric namespaces like 'CustomMetrics' or 'GuestMetrics', depending on what was set in the DCR. Metrics Aggregation: Depending on the type of metric you're collecting (such as network bandwidth or disk space), these metrics may be aggregated over a certain time period. Check if you can modify the aggregation settings (average, minimum, maximum) in the Metrics Explorer to display these metrics properly. Data Collection Rule Configuration: Double-check the configuration of the DCR, especially the metric destination (Azure Metrics). Ensure that the rules for collecting network bandwidth usage or disk free space are properly set and mapped to the right destination. Check Logs and Diagnostic Settings: Use Log Analytics or check diagnostic logs to verify if data is being collected as expected. If metrics are not showing up in Metrics Explorer, they may still be visible in logs, which can help debug the issue. You can verify these steps and see if the metrics show up after allowing some time for the data to propagate. If you're still not seeing the desired metrics, there might be an issue with the scope or permissions of the Data Collection Rule, and you may need to review that configuration. Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

How to add metrics to the pre-set metrics in Metrics Explorer or Monitor/Metrics

Accepted answer

Vinodh247 21,881 Reputation points

2024-09-11T14:01:09.98+00:00
Hi PerhapsCloud,

Thanks for reaching out to Microsoft Q&A.

The metrics are not showing up in the metrics explorer could be due to a few reasons, consider the following points;

Data Collection Rule Latency: After creating a DCR, there can be some delay before the data starts flowing into Azure Monitor and becomes visible in Metrics Explorer. Wait for a bit and then refresh the metrics dashboard to see if the new metrics appear.

Correct Namespace and Metric Type: Ensure that when you go back to Metrics Explorer, you are selecting the correct namespace for the custom metrics. Sometimes custom metrics can appear under a different namespace or category than the default "VM metrics." Check if there are new metric namespaces like 'CustomMetrics' or 'GuestMetrics', depending on what was set in the DCR.

Metrics Aggregation: Depending on the type of metric you're collecting (such as network bandwidth or disk space), these metrics may be aggregated over a certain time period. Check if you can modify the aggregation settings (average, minimum, maximum) in the Metrics Explorer to display these metrics properly.

Data Collection Rule Configuration: Double-check the configuration of the DCR, especially the metric destination (Azure Metrics). Ensure that the rules for collecting network bandwidth usage or disk free space are properly set and mapped to the right destination.

Check Logs and Diagnostic Settings: Use Log Analytics or check diagnostic logs to verify if data is being collected as expected. If metrics are not showing up in Metrics Explorer, they may still be visible in logs, which can help debug the issue.

You can verify these steps and see if the metrics show up after allowing some time for the data to propagate. If you're still not seeing the desired metrics, there might be an issue with the scope or permissions of the Data Collection Rule, and you may need to review that configuration.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.
Please sign in to rate this answer.
PerhapsCloud 41 Reputation points

2024-09-15T19:14:13.44+00:00

Oh wow, thanks for all the details!! This makes sense of how the custom metrics should have worked so I accepted it, though my custom metrics still not showing up after following the guidances. I have waited for at least 3 days but I will wait for a few more days for. The customized metrics that I chose to collect through the DCR were Network Bandwidth Usage and LogicalDisk % Free Space, none of which showed up. Also you mentioned that search in the LA Workspace for the metrics, here is the confusion part for me. When I created the DCR, I chose the Azure Monitor Metrics as the destination, didn't choose the Log Analytics Workspace. So I was a bit of confused where these so-called Metrics go to store themselves as clearly I didn't have to specify which LA Workspace as the destination. Thanks again.

Vinodh247 21,881 Reputation points

2024-09-16T06:58:28.1033333+00:00

When I created the DCR, I chose the Azure Monitor Metrics as the destination, didn't choose the Log Analytics Workspace. So I was a bit of confused where these so-called Metrics go to store themselves

Where Custom Metrics Are Stored:

When you select Azure Monitor Metrics as the destination, the custom metrics should be stored in the Metrics Database of Azure Monitor, not in a Log Analytics Workspace (LA Workspace). This is an important distinction:

Azure Monitor Metrics Database: This is where platform metrics and custom metrics are stored when you choose "Azure Monitor Metrics" as the destination.

Log Analytics Workspace (LA Workspace): This is a separate destination used for log data, custom log queries, and other diagnostic data (e.g., activity logs).

If you didn't choose an LA Workspace as the destination, your metrics will not be stored there, so you won't find them in LA Workspace queries. Instead, the metrics should appear in Metrics Explorer under Azure Monitor, where you can query them as metrics and visualize them as charts, graphs etc.

Hth!

PerhapsCloud 41 Reputation points

2024-09-22T07:13:27.75+00:00

Wondering where is that Azure Monitor Metrics Database and if there is a way I can do a query against to it like how we query agaist to a log analytics workspace?

Also I should have mentioned that the custom performance metric that I had to add wasn't in the list of the Custom Metrics, I actually had to add one because it is missing from the Custom Metrics list. For example the Custom Metrics list only contains \LogicalDisk(_Total)% Free Space, but I needed the \LogicalDisk(*)% Free Space which I manually added to the list. Wondering if that would cause the issue as it's not natively from the list?? Thanks again.

PerhapsCloud 41 Reputation points

2024-09-22T07:15:16.08+00:00

hello

Vinodh247 21,881 Reputation points

2024-09-22T10:05:02.9433333+00:00

The Azure Monitor Metrics database is a special time-series database built to store and manage platform and custom metrics in Azure. Unlike Log Analytics, you cannot directly query the Monitor Metrics database using a language like KQL as you would for log data. Instead, you query it through the Metrics Explorer or through the Azure Monitor REST API.

The fact that you manually added \LogicalDisk(*)% Free Space to the DCR is likely the root cause of the metrics not appearing. Azure Monitor’s custom metrics list is limited to what’s natively supported. in my opinion, if the metric is not on the pre-defined list of Azure Monitor-supported metrics, even if you add it manually, it may not be collected because:

Custom metrics collection in Azure Monitor is somewhat restricted to metrics it officially supports.

Some performance counters, like wildcard entries (ex: \LogicalDisk(*)), might not be fully compatible with how Azure Monitor collects metrics. The wildcard (*) could complicate the collection since Azure Monitor might not know how to handle the variable instances.

PerhapsCloud 41 Reputation points

2024-09-22T16:27:14.5066667+00:00

Thanks a lot again. What I was trying to do is to monitor the free space on all the drives across all the VMs. Hoped that I could use the custom performance metrics to deliver to the Monitor Metrics where I can set up an alert to monitor. But since the Monitor Metrics don't seem to pick up the custom metrics, I guess I will have to destinate the custom metrics to the Log Analytics where I can set up an alert for disk space monitoring?

There IS a pre-defined custom metrics, which is called \logicalDisk(_Total)% Free Space, but that gives a sum of all drives free space, otherwise it would be perfect.

The VM Insights if turned on does collect logical drive free spaces, but it seems an extra cost to turn on the Insights for every VM and plus it collects a whole lot stuff whereas I just needed the free space metrics.

Also do you know if the cost goes up if I collect more custom metrics and destinate to Monitor Metrics? I know the more I collect and send to log analytics the more cost it will be, but is it the same if sending to the Monitor Metrics that the cost would go up as well? Thanks again.

Vinodh247 21,881 Reputation points

2024-09-23T05:22:39.44+00:00

Using Log Analytics for Drive-Specific Metrics:

Since azure monitor metrics cannot pick up wildcard-based custom metrics like \LogicalDisk(*)% Free Space, sending these metrics to log analytics is a good alternative. you can then use kusto queries to create alerts based on the specific drive's free space and avoid using the predefined \logicalDisk(_Total)% Free Space, which sums all drives.

Cost Considerations:

azure monitor metrics costs are based on the number of custom metrics you collect and store, though the pricing is typically lower than storing log data in log analytics.

custom metrics in azure monitor: there is a cost for each metric time series you collect and retain (ex., for disk space per drive on multiple vm’s). if you collect many custom metrics, the cost can increase, though it's usually less expensive than logs.

Log Analytics: costs are based on data volume (gb’s of data ingested and retained). if you collect detailed performance data for multiple vms, especially at a granular level (per drive, per metric), costs can rise quickly, especially as the data volume increases.

Cost for VM Insights:

VM Insights provides a more comprehensive view of vm performance, including disk space, but as you mentioned, it collects a lot of data you may not need, which increases costs due to the volume of ingested data.

Since you only need free space metrics, turning on vm insights for all vms may not be the most cost-effective solution unless you require detailed performance monitoring across other parameters too.

Hope this info answers your doubts

PerhapsCloud 41 Reputation points

2024-09-23T06:12:59.5666667+00:00

Thank you so much. I will then go with destinating it to the log analytics. I have another question if you don't mind.

I'm trying to create an alert on all the VMs in a resource group, using the Metrics with a pre-defined metric which is called Memeory Available (Preview). The resource group contains Azure VMs as well as a few ARC-enabled VMs from on-prem. When I chose the scope for the alert creation, I could only select Azure VMs in the resource group, the ARC-enabled VMs in the same resource group don't show somehow when defining the scope for the alert. Is that because metrics aren't supported on Arc-enabled VMs? I tried to validate it by navigating to the Arc-enaled VM and from there select the Metrics and I could actually select and see the chart of Memory Available (Preview) on the Arc machine. So it is availabe as a metrics to the Arc machine, but why when creating an alert on the metric the Arc machine won't show? Did I miss anything? Thanks a lot !

PerhapsCloud 41 Reputation points

2024-09-29T18:20:28.87+00:00

Atter some extensive digging, finally dicovered that although the AMA agent was provisioned successfully as it is shown in the Portal, the agent is not actually running on the VMs. When I try to start it, It gives the error:

ErrorCode:1100 ERROR: Invalid number of commands to the extension. 1
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to add metrics to the pre-set metrics in Metrics Explorer or Monitor/Metrics

0 additional answers

Your answer