Feature Flighting in Exchange Server

Overview

Historically, before installing an Exchange update in production, organizations often deploy updates in a test environment to first validate the update before deploying it to their production environment. This is an important task, but it is also time-consuming, and it can slow down the deployment of important updates. Moreover, not all organizations have test environments.

Feature Flighting provides an additional way for administrators to test and roll out select new features across their Exchange Server organization. Feature Flighting is an optional cloud-based service for on-premises Exchange servers. It uses the Office Config Service (OCS) - the same endpoint used by the Emergency Mitigation service and Microsoft Office clients—to check for updates from Microsoft related to flighted features

With Feature Flighting, administrators can deploy updates immediately and control when a flighted feature is enabled in their environment. Feature Flighting also enables Microsoft to disable a flighted feature in case a significant issue is discovered after the update containing the flighted feature is released.

Feature Flighting does not apply to all new features and changes in future updates. The Exchange Server engineering team determines which features are distributed via Feature Flighting, and a living, detailed list of flighted features is maintained in this article. Using Feature Flighting is optional, but it is enabled by default. You can configure it or disable it by following the steps outlined below.

Flighted Features

The table provides a detailed list of all currently flighted features. This table is a living table; Microsoft updates the table with new flights and changes to the affected Ring at least 7 days before they become active. Microsoft may not be able to announce changes several days in advance if a feature needs to be disabled (Recalled) due to a regression or known issue.

Feature	Description	Rings	Admin Approval	Recalled	Start Build	End Build
PING.1.0	Heartbeat Probe	0,1,2	No	No	15.02.1748.005	N/A

How Feature Flighting works

Feature Flighting is an innovative method for managing features and changes in Exchange Server that allows Microsoft to introduce new features and changes for Exchange Server in an initially disabled state. Here's how it works:

1. Initial Introduction: When a flighted feature or change is introduced, it's included in the Exchange Server update (CU or SU) in a disabled state. This behavior ensures that the feature doesn't immediately affect the server's operation.
2. Controlled Enablement: Flighted features and changes can then be automatically enabled on all Exchange servers within a specific Ring. This phased approach allows Microsoft to monitor the feature's performance and gather telemetry data.
3. Evaluation and Deployment: Based on the collected telemetry (and any feedback we might receive on our blog, via support cases, or from other sources), Microsoft evaluates the feature's stability and effectiveness. If the feature performs well, it can then be gradually deployed to Exchange servers in other Rings.
4. Recall Mechanism: A key advantage of Feature Flighting is the ability to recall a feature or change if any issues or regressions are detected. This helps ensure that any problematic features can be disabled quickly to maintain the stability and reliability of the Exchange Server environment.

Note

Feature Flighting isn't available for Edge Transport servers.

This results into the following benefits:

• Enhanced Control: Administrators have more granular control over when and how new features are introduced.
• Improved Stability: By deploying features in a staged manner, potential issues can be identified and addressed before widespread deployment.
• Flexibility: The ability to recall features ensures that any regressions or known issues can be quickly mitigated.

Prerequisites

A new service called Microsoft Exchange Flighting service (MSExchangeFlighting) controls Feature Flighting. This service checks for and downloads Feature Flight Definitions (FFD) from OCS hourly and manages the activation and deactivation of flighted features based on the configured Ring.

Feature Flighting can't be used in air gapped environments or in environments without outbound connectivity to OCS. the Exchange Flighting service requires outbound connectivity to OCS to check for and download FFDs. You don't need to take any action if your servers can't reach OCS. Additionally, Feature Flighting is not yet available on Edge Transport servers.

We recommend checking our documentation and the Exchange Team Blog regularly to stay informed about known issues that may temporarily disable a feature as a mitigation measure.

Endpoint	Address	Port	Description
Office Config Service	officeclient.microsoft.com/*	443	Endpoint for the Microsoft Exchange Flighting Service to download FFDs

If you're using a network proxy to allow outbound connectivity, ensure that the InternetWebProxy is configured on each of your Exchange servers:

PowerShell

Set-ExchangeServer -Identity <ServerName> -InternetWebProxy <http://proxy.contoso.local:port>

You must also configure the proxy settings for WinHTTP, which is a component of Windows that handles HTTP requests for applications that don't use the WinINet API:

command

netsh winhttp set proxy <proxy.contoso.com:port>

In addition to outbound connectivity to OCS, the Flighting Service also needs outbound connectivity to various Certificate Revocation List (CRL) endpoints. This is required to verify the certificates used to sign the FFDs. We recommend letting Windows maintain the Certificate Trust List (CTL) on your machine. To enable Windows to maintain the CTL, ensure that the following URL is accessible from the computer where Exchange Server is installed:

Endpoint	Address	Port	Description
Certificate Trust List Download	ctldl.windowsupdate.com/*	80	Endpoint for downloading the Certificate Trust List

Rings

Feature Flighting uses a deployment mechanism called Rings that defines when, or in what sequence, a feature is enabled on a particular Exchange server. Every Exchange server running Exchange Server 2019 CU15 or later is automatically assigned to a Ring. The default assignment can be changed by an admin at any time. The following Rings are available:

Ring No.	Name	Default Ring	Description
0	Early Adopter Ring	No	This is the earliest Ring and it's intended to be used for testing new features. If an Exchange server is assigned to this Ring, flighted features introduced in an update are immediately enabled after the update is installed. This happens regardless of the Feature Classification.
1	Worldwide Ring	Yes	This is the default Ring assigned to all Exchange servers when the first build that supports Feature Flighting is installed (for example, CU15). Servers in this Ring receive new features as soon as Microsoft has confirmed that the features are ready for general availability.
2	Admin Action Ring	No	Exchange servers in this Ring don't automatically enable any new flighted features. This Ring allows admins to revert to previous experience (for example, roll back). Flighted features are shipped in a disabled state and must be enabled by the admin using Set-ExchangeFeature as explained in the Feature States section.

Important

Moving a server between Rings can result in certain features being enabled or disabled, depending on the Feature Status defined in the Flighting Service for the new Ring.

The following workflow outlines how the Ring determines whether a flighted feature should be enabled or not:

You can use Exchange Management Shell (EMS) to assign an Exchange server to a specific Ring. In this example, the Exchange server is assigned to Ring 0:

PowerShell

Set-ExchangeServer -Identity <ServerName> -RingLevel 0

Important

If you don't want Microsoft to automatically enable new features or make changes to your server via Feature Flighting, you must assign your Exchange servers to Ring 2. Stopping and/or disabling the Microsoft Exchange Flighting Service (MSExchangeFlighting) isn't supported.

The following example assigns a server to Ring 2:

PowerShell

Set-ExchangeServer -Identity <ServerName> -RingLevel 2

Feature Classification

There are two types of features that can be managed by the Flighting Service: features with prerequisites and features without prerequisites. The key distinction is that some features depend on certain prerequisites that must be fulfilled:

• Features with prerequisites:
  • These features need some preconditions to be met before they can be used. For example, all Exchange servers in the organization must be running a specific build.
  • Example: Certificate Signing of PowerShell serialization payloads depends on a valid Auth Certificate, which is available on all Exchange servers.
• Features without any prerequisites:
  • These features work out of the box without dependencies or prerequisites. All you need to do is install the update and enable the feature.
  • Example: Support for AES256-CBC-encrypted content doesn't have any dependencies or prerequisites.

For Exchange servers in Ring 1, features with prerequisites are assigned the FeaturesAwaitingAdminApproval state, as they need administrator approval before becoming active. In contrast, for Exchange servers in Ring 0, features with prerequisites are enabled without waiting for administrator approval.

Feature States

Each flighted feature is assigned a Feature State that indicates the current state of the feature. Feature States are an essential component of Feature Flighting. When a feature is flighted, it gets one or more of the following Feature States assigned:

Feature State	Feature Enabled	Description
FeaturesEnabled	Yes	This feature is enabled on the server.
FeaturesDisabled	No	Feature Flighting didn't enable this feature on the server either because Microsoft recalled it due to a regression or because the Exchange Server administrator blocked it (for example, because the server doesn't fulfill the required prerequisites yet).
FeaturesAwaitingAdminApproval	No	This feature requires explicit approval by an Exchange Server administrator.
FeaturesApproved	Yes	An Exchange Server administrator explicitly approved this feature, and it's now active.
FeaturesBlocked	No	This feature was explicitly blocked by an Exchange Server administrator and remains in disabled state.

For example, if a feature is waiting for approval, it has both the FeaturesAwaitingAdminApproval and FeaturesDisabled states assigned. Once the admin approves the feature, it's assigned the FeaturesApproved and FeaturesEnabled states.

Tip

If you run the latest version of the Exchange Server Health Checker script, it shows you information such as the server's assigned Ring, as well as features controlled by the Feature Flighting.

You can use EMS to query the feature states for a specific Exchange server:

PowerShell

Get-ExchangeServer -Identity <ServerName> | Format-List Features*

To get an overview of all available features, use the Get-ExchangeFeature cmdlet. The following command returns all features which are enabled:

PowerShell

Get-ExchangeFeature -Status "Enabled"

You can also use the -FeatureID parameter together with the name of a feature, to query its status and a short description:

PowerShell

Get-ExchangeFeature -FeatureID "PING.1.0"

After executing the previous command, the Exchange server will return a result like the following:

PowerShell

Server              FeatureID   RingLevel  Status                  Description
------              ---------   ---------  ------                  -----------
EXCH01              PING.1.0    1          Enabled                 Heartbeat Probe. Validates the Telemetry Channel

This information is important in case you want to approve a new feature or change that requires admin approval. You'll also need this information when you want to prevent a feature from becoming enabled.

Feature naming is standardized using the following format: <FeatureId>.<SettingId>.<Version>

This format allows features to have multiple settings, which are used to control it. Assume that there's a feature FeatureID=F4 which can be controlled via two setting overrides SettingId=1 and SettingId=2, you'll see the following flighting entries:

• F4.1.0
• F4.2.0

To approve a new feature awaiting admin approval, use Set-ExchangeFeature with the -Approve parameter. Once the feature is approved or blocked, it remains in an intermittent state until the next Feature Flighting cycle runs, which may take up to an hour at most:

PowerShell

Set-ExchangeFeature -Identity <ComputerName> -FeatureID @("F1.1.1") -Approve

It's also possible to approve multiple features at once:

PowerShell

Set-ExchangeFeature -Identity <ComputerName> -FeatureID @("F1.1.1", "F1.2.1", "F2.1.1") -Approve 

To prevent a feature from being enabled, use Set-ExchangeFeature with the -Block parameter. As shown in the previous example, it's also possible to block multiple features at once:

PowerShell

Set-ExchangeFeature -Identity <ComputerName> -FeatureID @("F1.1.1", "F1.2.1", "F2.1.1") -Block

Example: Lifecycle of a flighted feature

This section illustrates the workflow and lifecycle of a flighted feature called Feature1.

Contoso has two Exchange 2019 servers, EXCHPRD01 and EXCHPRD02 that are members of a database availability group (DAG) called DAGPRD01. They also have a third Exchange 2019 server, EXCHTST01 which is used for validating and testing new updates, and which isn't a member of the DAG.

The administrator first installs Exchange Server 2019 CU15 on EXCHTST01 and, after all tests have been successfully completed, the administrator installs CU15 on EXCHPRD01 and EXCHPRD02. The administrator assigns EXCHTST01 to Ring 0 so that it receives the flighted features and changes as soon as they become available.

First, they validate the current Ring assignment:

PowerShell

Get-ExchangeServer | Format-List Identity,RingLevel

Identity  : EXCHPRD01
RingLevel : 1

Identity  : EXCHPRD02
RingLevel : 1

Identity  : EXCHTST01
RingLevel : 1

Next they assign EXCHTST01 to a Ring 0:

PowerShell

Set-ExchangeServer -Identity EXCHTST01 -RingLevel 0

Confirm
Changing the RingLevel of the server will change the flighting of the features based on the RingLevel selected. Changes will apply once the flighting service applies these changes. Are you sure you want to continue?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"):

A few months later, Microsoft releases an SU for CU15. In the release notes for the SU, the administrator learns that the SU contains a new flighted feature, Feature1. After installing the SU on all servers, they want to confirm that Feature1 is enabled only on EXCHTST01 and not on the production servers EXCHPRD01 and EXCHPRD02:

PowerShell

Get-ExchangeServer | Get-ExchangeFeature

Server              FeatureID   RingLevel  Status                  Description
------              ---------   ---------  ------                  -----------
EXCHPRD01           PING.1.0    1          Enabled                 Heartbeat Probe. Validates the Telemetry Channel
EXCHPRD01           F1.1.0      1          Disabled                Feature1 introduces a new functionality
EXCHPRD02           PING.1.0    1          Enabled                 Heartbeat Probe. Validates the Telemetry Channel
EXCHPRD02           F1.1.0      1          Disabled                Feature1 introduces a new functionality
EXCHTST01           PING.1.0    0          Enabled                 Heartbeat Probe. Validates the Telemetry Channel
EXCHTST01           F1.1.0      0          Enabled                 Feature1 introduces a new functionality

On EXCHTST01, the administrator tests that their workflows and 3rd party applications aren't adversely affected by Feature1 and they learn that there's a problem introduced by Feature1. So, the administrator uses the following command to prevent Feature1 from becoming enabled in Ring 1 and to disable it in Ring 0 where it's already active:

PowerShell

Get-ExchangeServer | Set-ExchangeFeature -FeatureID "F1.1.0" -Block

Confirm
Are you sure you want to perform this action?
By running this cmdlet, the features will be updated on server "EXCHTST01".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"):

They confirm that Feature1 was successfully blocked:

PowerShell

Get-ExchangeServer | Get-ExchangeFeature

Server              FeatureID   RingLevel  Status                  Description
------              ---------   ---------  ------                  -----------
EXCHPRD01           PING.1.0    1          Enabled                 Heartbeat Probe. Validates the Telemetry Channel
EXCHPRD01           F1.1.0      1          Blocked                 Feature1 introduces a new functionality
EXCHPRD02           PING.1.0    1          Enabled                 Heartbeat Probe. Validates the Telemetry Channel
EXCHPRD02           F1.1.0      1          Blocked                 Feature1 introduces a new functionality
EXCHTST01           PING.1.0    0          Enabled                 Heartbeat Probe. Validates the Telemetry Channel
EXCHTST01           F1.1.0      0          Blocked                 Feature1 introduces a new functionality

Contoso contacts Microsoft Support to report the issue. Microsoft confirms the issue and plans to address it in the next update. After the update with the fix is released, Contoso deploys the update on their servers. The administrator tests their workflows and 3rd party applications and confirms that everything works as expected. The admin now wants to enable Feature1 on their production servers.

After learning from the release notes that Feature1 has dependencies, the administrator checks to see if admin approval is required:

PowerShell

Get-ExchangeServer -Identity EXCHPRD01 | Format-List Features*

FeaturesApproved              : {}
FeaturesAwaitingAdminApproval : {F1.1.0}
FeaturesEnabled               : {PING.1.0}
FeaturesBlocked               : {}
FeaturesDisabled              : {}

To approve and enable Feature1, the administrator runs the following command:

PowerShell

Get-ExchangeServer -Identity EXCHPRD* | Set-ExchangeFeature -FeatureID "F1.1.0" -Approve

Confirm
Are you sure you want to perform this action?
By running this cmdlet, the features will be updated on server "EXCHPRD01".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"):

Next, they validate that the feature was approved and enabled:

PowerShell

Get-ExchangeServer -Identity EXCHPRD01 | Format-List Features*

FeaturesApproved              : {F1.1.0}
FeaturesAwaitingAdminApproval : {}
FeaturesEnabled               : {PING.1.0, F1.1.0}
FeaturesBlocked               : {}
FeaturesDisabled              : {}

Once Microsoft considers all issues resolved, it enables the feature for all customers by default. This will happen in the next update, which enables the feature outside of feature flighting for all customers.

Diagnostic Data

When data sharing is enabled, Feature Flighting sends diagnostic data to the OCS. This data helps Microsoft to identify the saturation of flighted features. To learn more about what is collected and how to disable data sharing, see Diagnostic Data collected for Exchange Server.