Service resilience in Microsoft Teams

Microsoft Teams is engineered from the ground up for the cloud. It takes a cloud-first, mobile-first, cross-platform approach since inception. Teams is built with a highly resilient microservice architecture that splits tasks across multiple geo-redundant systems to ensure high availability and performance around the world. Microsoft Teams also leverages Azure's built-in resiliency, making it resistant to failures.

Service configuration

In geographies where Microsoft Teams is available, it's operated out of at least two geographically distant Azure regions, and more where possible. The service runs in an Active/Active/n configuration in which users are actively served from all available regions in the geography. Capacity management and planning is such that even in a geography running only the minimum two regions, each region can handle the full service load from the entire geography if one Azure region is unavailable.

The microservice architecture means that the Teams service is more reliable, as microservices run in Active/Active state but don't need to be located all within the same Azure regions at the same time. This design allows us to use capacity where it's available within a given geography, so that if a site becomes partially unavailable, we can migrate some microservices, without being forced to migrate all of them. Microservices also leverage Azure's native resiliency features.

Failover

We use failover operations as part of our normal day-to-day service management processes for load balancing and maintenance tasks, as well as for recovery during service incidents, so these actions are exercised regularly. The incident management team makes failover decisions if deemed appropriate, and automation carries out operations at scale with speed. We constantly work to improve failover procedures based on learnings from incident postmortems, and in the future are looking at ways to implement automated failover decisions based on machine learning.

Limiting impact of service incidents

Fault isolation

Microsoft Teams service infrastructure is highly segmented, which severely limits the scope of most types of impact. Teams is composed of over 140 microservices by design, in part to limit the impact of single component degradation. As such, most service incidents are limited to only a single component, or a small subset of users. This limited impact is true even of DNS incidents, as many Teams micro-services rely not only on Azure Traffic Manager (ATM), but also third-party (Akamai) for DNS and load balancing.

Graceful degradation

Microsoft Teams introduces several features to limit the real-world impact of service degradation, including that caused by third-party providers such as ISPs or even a customer's own internal network. Examples include:

• Keeping the media path short – Teams utilizes intelligent routing to reduce the number of network hops and network traversal distance, to reduce the number of potential failure points.
• Handling degradation gracefully – Teams has in-app user notifications when things aren't working great, and suggestions such as turning off video to improve performance if the connection is substandard. Teams prompts users to mute audio if their connection is causing feedback or noise on the line.
• Temporary network loss – Teams automatically reconnects to a meeting or call once a network connection is restored.
• Fall back to PSTN - Teams automatically gives you the option to rejoin via Public Switched Telephone Network (PSTN) if you're dropped from a call due to a bad network connection. This feature has the added benefit of requiring no conference numbers or passcodes, so it's even easier for users on the go to stay connected.
• Device fail & switch – Teams detects if an audio device is disconnected and attempts to switch to an alternate device if available.

Code deployment

Teams uses stepped rings of deployment when updating service-side components, starting with small groups of internal test users, followed by the wider Microsoft corporate network. At each step, engineers collect feedback on features and performance, and make every effort to detect any issues before deploying in production. When deploying into production, we start with a small subset of systems and then begin deploying in ever-widening rings until all users are connecting to systems running the latest code. Again, any reports of problems related to the update triggers an incident response, and may result in halting an update, rolling back to the last version, and removing degraded systems from active rotation until the problem is solved. Teams also has the capability to disable specific code paths if necessary, which allows us to quickly identify and disable problematic code and return the system to a healthy state without having to wait for a new update to roll out.