Metrics sent to Azure Monitor are not visible in Log Analytics (even after turning on Diagnostic Setting)

Question

I am sending custom metrics to azure monitor through my code through REST APIs. However I dont see those metrics being pushed to Log Analytics

My code snippet looks like this

metric_data = {
    "time": date,
    "data": {
        "baseData": {
            "metric": "PipelineSucceeded",
            "namespace": metric_namespace,
            "dimNames": ["BuildId"],
            "series": [
                {
                    "dimValues": [build_id],
					# 0 or 1 based on the boolean
                    "min":  pipeline_succeeded_value,
                    "max":  pipeline_succeeded_value,
                    "sum":  pipeline_succeeded_value,
                    "count": 1
                }
            ]
        }
    }
}


.
.
url = f"https://{location}/monitoring.azure.com/{resource_id}/metrics"
.
.
response = requests.post(url, json=metric_data, headers=headers)

resource_id points to the log analytics workspace and I am able to successfully push the metrics to the /metrics endpoint and I am able to view it in Azure Monitoring/metrics.

Things to consider:

1. There are no tables associated that store these data in either Metrics > Tables or Log Analytics > Tables and I am also not able to view any of the metrics I pushed with KQL because I dont know which tables it is storing them in
2. After listing all tables in KQL I still did not find the custom metrics anywhere
3. I have Monitoring Contributor and Monitoring Metrics Publisher Role assigned to me

Questions:

1. I also have a diagnostic setting enabled that should send all Logs to Log Analytics workspace but that did not work. I have it enabled for more than 10 hours. How do I make it work?
2. Can I see the source of the data (my metrics) all in one place without having to aggregate it? Are there other ways to do so?

Answer 1

Make sure you have the necessary permissions to view the logs in Log Analytics. You should have roles like Monitoring Contributor or Monitoring Metrics Publisher assigned.

https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-troubleshoot

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful **

Answer 2

Hello @Amit Rokade

Welcome to Microsoft Q&A Forum, Thanks for asking questions here...!

It sounds like you are successfully sending custom metrics to Azure Monitor, but you are not seeing them in Log Analytics. below are troubleshooting steps for this issue

1. Check the diagnostic settings for your resource. Make sure that the diagnostic setting is configured to send metrics to Log Analytics. You can do this by going to the "Diagnostic settings" blade for your resource and checking the "Metrics" section.
2. Check the Log Analytics workspace for your resource. Make sure that the workspace is configured to receive metrics from your resource. You can do this by going to the "Advanced settings" blade for your workspace and checking the "Data" section.
3. Check the Log Analytics query language (KQL) for your workspace. Make sure that you are using the correct syntax to query custom metrics. You can find more information on how to query custom metrics in the Azure Monitor documentation.

Regarding your questions:

1. If you have a diagnostic setting enabled to send logs to Log Analytics, it should start sending logs immediately. If you are not seeing logs in Log Analytics, you may need to check the diagnostic setting configuration to make sure it is set up correctly.
2. You can see the source of your custom metrics in Azure Monitor by going to the "Metrics" blade for your resource and selecting the "Custom" tab. This will show you a list of all custom metrics that have been sent to Azure Monitor, along with their source and other metadata. You can also use KQL to query custom metrics and aggregate them as needed. For more details please go through attached link : https://learn.microsoft.com/en-us/kusto/query/?view=microsoft-fabric https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-custom-overview

I hope this has been helpful! Your feedback is important so please take a moment to accept answers.
If you still have questions, please let us know what is needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!