Share via

Elevated error rates - 11/16 - Mitigated

  • Article

We’ve confirmed that all systems are back to normal as of 2018/11/08 17:30 UTC. Our logs show the incident started on 2018/11/09 09:08 UTC and that during the 8 hours and 22 minutes that it took to resolve the issue, users have noticed elevated error rates.

  • Root Cause: The failure was due to background activity which caused cache invalidations
  • Chance of Re-occurrence: Low
  • Lessons Learned: We are working both minimizing resource-intensive activities in our post-deployment steps, and are also working targeting monitors specifically to detect post-deployment issues in the future.

We continue to investigate issues impacting several regions - West Europe, North and Central US. Root cause is not fully understood at this time, however we are progressing with the investigation on 1-2 hypothesis. The problem began at 8:45am UTC, was mitigated at 11:15, and recurred at 12:20pm UTC. We currently have no estimated time for resolution.

We continue to investigate issues and Root Cause. There is no currently estimated time for resolution.

We continue to investigate issues on Multiple scale units. Root cause is not fully understood at this time .The problem began at November 9th 2018 8:50 UTC. We currently have no estimated time for resolution.

We are actively investigating performance issues with Azure DevOps in multiple regions. Some customers may experience slower performance than usual while accessing their accounts.