Virtual Machine: Disk Latency Regression — Impacting 25 Linux VMs

Question

Problem description

We are experiencing a significant disk latency regression on our Azure Linux virtual machines that started between September 24 and September 27, 2026. Approximately 25 production VMs are affected, with OS disk latency increasing to about 100–200 milliseconds, leading to application performance slowing down by 2–3 times. The burst credit metrics for the OS disks, which previously showed activity, now remain at zero during this period.

Environment

Azure Virtual Machines running Linux in West Europe and East US 2 regions. The affected disks include StandardSSD_LRS, Premium SSD, and Standard SSD, all using NVMe controllers with ReadOnly host caching.

What I've already tried

I have reviewed performance metrics from September 16–22 and October 3–9, noting increased latency and decreased burst credit metrics during the regression. I have performed inventory-wide performance comparisons, kernel rollback testing, compared with newly provisioned VMs, upgraded disk SKUs, checked VM limits, and conducted platform health checks. I also collected guest logs and system diagnostics, but no definitive cause has been identified.

Current status

I am seeking assistance to understand the root cause of this latency regression and to identify platform-side issues or configuration changes that may have contributed. I would like guidance on the next steps for investigation and resolution.

Answer 1

Likely cause: because this started in a narrow time window across many VMs and multiple disk SKUs/regions, this is more likely a throughput/IO-path constraint at VM+disk combination level (or host/caching interaction) than a single-guest regression.

1. Validate effective limits at both layers (VM and disk), not disk alone:
   • VM-level IOPS/throughput caps can throttle all attached disks.
   • Disk SKU limits and bursting behavior must be checked together.
   Reference: https://learn.microsoft.com/en-us/azure/virtual-machines/disks-performance
2. Correlate per-VM metrics during the regression window:
   • OS disk Read/Write latency
   • VM uncached IOPS/throughput utilization
   • queue depth / await / %util in guest (iostat)
   If VM-level utilization is near cap while disk-level headroom exists, the bottleneck is likely VM-side.
3. Test one controlled mitigation branch:
   • Branch A: temporarily move one representative VM to a size with higher uncached disk throughput/IOPS.
   • Branch B: for one VM, test host caching mode change (ReadOnly → None) on OS disk in maintenance window.
   • Branch C: place workload on a dedicated data disk (or striped set) to isolate OS-disk bottleneck.
4. Because this affects ~25 production VMs across regions, open a support request for backend platform correlation and include:
   • impacted VM list + regions
   • exact onset window (Sept 24–27)
   • before/after latency metrics and burst-credit behavior
   • one VM where mitigation branch changed outcome
   Support process: https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

Confirmed facts from your post:

• Cross-VM impact (~25 Linux VMs)
• Latency increased to ~100–200 ms
• Burst metrics changed behavior in the same period
• No conclusive cause after guest/kernel/SKU checks

Possible cause (not yet confirmed): VM-level disk bandwidth ceiling or host-path behavior change interacting with NVMe + ReadOnly cache pattern.

Validation checks:

• Latency returns near pre-regression baseline on test VM.
• VM uncached throughput utilization drops below sustained cap.
• Application response times recover proportionally.

Risk/side effect notes:

• VM resize and caching changes can require restart and may cause short downtime.
• Caching changes can alter consistency/performance characteristics; test on one VM first.

“If you find this answer helpful, please upvote it. If this resolves your issue, please mark it as the accepted answer so that it can help others in the community.”