After migrating to Open Telemetry the Monitor in the VM Blade gives me Access Denied

Question

Good day to you. When I saw the the call to action on the portal to migrate to the Open Telemetry our virtual machines, I was happy to do it. It was giving me an option to run parallel with the log-based legacy option, so I kept this one also.

I created an Azure Monitoring Workspace in the same region where the test machine was located, and then onboarded a single machine over the UI. Worked pretty well, I got metrics on the VM View/Monitor blade, and I also played around to extend the metrics with a few more, and everything seemed fine.

After that I cleaned up everything, and started from scratch. Then, I created two AMW in two separate regions, and I created two DCRs with proper/internal naming conventions in the same regions, and I assigned all of our machines with scripts. While I got metrics in the AMW, the VM view / Monitor blade gave me an error: "Access Denied: You do not have the necessary permissions to view the data in this section. Please contact your administrator to request access." I have PIM activated Global Admin, Global Resource owner permissions, Global reader assigned permanently, and I also assigned Monitor Contributor to myself. The error is the same, after 24 hours.

So I thought, might be subscription issue. So I cleaned up everything again, created the two AMW in separate regions, I also created ~11 DCRs per our subscriptions and assigned the machines. And the error is the same. "Access Denied: You do not have the necessary permissions to view the data in this section. Please contact your administrator to request access."

Some "visual" explanations of the structure:

- Tenant
|- Subscription (Cloud Infra)
 |- Resource Group for AMW 1 (WEU)
  |- AMW WEU
 |- Resource Group for AMW 2 (GWC)
  |- AMW GWC
|- Subscription (Customer 1)
 |- Resource Group - Infra (WEU)
  |- DCR (WEU)
 |- Resource Group - VMs (WEU)
  |- Virtual Machine (WEU)
|- Subscription (Customer 2)
 |- Resource Group - Infra (GWC)
  |- DCR (GWC)
 |- Resource Group - VMs (GWC)
  |- Virtual Machine (WEU)

To be honest, it drives me crazy. I have metrics in AMW, so they are arriving properly, but not having the same metrics in VM view/Monitor blade - so that's not ok. If I do the clickety-clickety over the UI, it works, but it generates some random named DCRs per machine and that's not feasible.

I would like to ask you, if you can point out any problem in my configuration if I made any mistake, and happy to provide further information if needed. Thank you.

Answer 1

Hello Csaba Sari,

It looks like you're working with a complex Azure monitoring setup using Data Collection Rules (DCRs) and OpenTelemetry (Otel) metrics. Based on the details you’ve shared, here's a summary of what might be happening and some potential steps to resolve or further investigate the situation:

1. Role and Permissions Issue:

Initially, you encountered an "Access Denied" error, which suggests there might have been some permission issues with the "Monitoring Contributor" role not being fully propagated yet. Even though you added the necessary roles, sometimes changes like this can take a while to fully reflect across the platform.

The "Access Denied" message is now gone, which means that permission propagation likely completed. However, you still face issues with the metrics not displaying.

1. Metrics Display in AMW vs. Insights:

You mentioned that you can see metrics when you go directly to the Azure Monitor Workspace (AMW), but not in the "Insights" tab. This suggests that there may be a difference in the way these two views are pulling or visualizing data.

The "Insights" blade may still be in preview and could have some limitations or bugs. This might explain why metrics are visible in one view but not the other.

1. Missing Metrics in Insights:

Since you're still seeing "No data returned matching the selected filters" in the "Insights" blade, it’s possible the system isn't recognizing or properly pulling the Otel data for display.

One thing to check is whether the correct filters or time window are selected in the Insights view. Sometimes, the data might not be immediately visible depending on the configuration.

1. Preview Features:

As you've noted, this setup seems to be quite "preview" (in a testing or early access phase), so it's possible there are bugs or limitations in the user interface or data syncing between different sections of Azure Monitor.

It’s worth considering that the Insights view might not yet support all the same capabilities or data that AMW can show, especially since Otel metrics are somewhat newer in Azure Monitor.

Next Steps

1. Double-Check Filters: Ensure that the filters applied in the "Insights" view (e.g., time range, metrics selected) are correct. Sometimes missing data is due to overly restrictive filters.
2. Check Otel Metrics Settings: In AMW, verify the setup for the Otel-based metrics (especially the ones you're seeing) to ensure that they are being collected correctly and there are no missing data points.
3. Check Preview Limitations: Since the "Insights" section is still in preview, it might not support all metrics or data collection sources yet, or there could be some bugs. You could check the Azure documentation for any updates on supported data sources or known limitations.

Let me know if you need more help as you investigate further!

 

Answer 2

hey hey,

what you are seeing is usually not a VM permission issue but a data permission issue, VM Monitor blade reads metrics from the Azure Monitor Workspace, not directly from the VM, even if you are Global Admin or Monitor Contributor, you still need data access on the workspace itself.

Check the Azure Monitor Workspace and open Access control. Make sure you have Monitoring Reader or Monitoring Contributor assigned at the workspace level, if the workspace uses data plane RBAC, you must have explicit data access there. Being Global Admin in Entra does not automatically grant data access to Azure Monitor Workspace. Check whether the DCR is actually linked to the correct workspace and region that the VM Monitor blade expects. The portal UI creates additional hidden DCR associations when you onboard through the click method. When you create DCRs manually, the VM blade may not recognise the association in the same way.

If metrics are visible inside the workspace but not in the VM blade, that usually means the workspace RBAC is missing or the VM is not properly associated with the workspace for the Monitor view scenario. I would first verify workspace level RBAC, then verify DCR association to the VM, and finally confirm that the VM and workspace are in supported regions for that Monitor blade experience.

This is almost always an RBAC scope issue at workspace level rather than a telemetry problem.

rgds,

Alex