Understanding back pressure
Back pressure is a system resource monitoring feature of the Microsoft Exchange Transport service that exists on Mailbox servers and Edge Transport servers. Back pressure detects when vital system resources, such as hard drive space and memory, are overused, and takes action to prevent the server from becoming completely overwhelmed and unavailable. For example, when a system resource utilization level on the Exchange server is determined to be too high, the server delays accepting new messages. If the resource utilization gets worse, the server stops accepting new messages to work exclusively on processing all existing messages, and might even stop processing outgoing messages. When the system resource utilization returns to an acceptable level, the Exchange server resumes normal operation by accepting new messages and processing outgoing messages.
Monitored resources
The following system resources are monitored by back pressure:
- DatabaseUsedSpace[%ExchangeInstallPath%TransportRoles\data\Queue]: Hard drive utilization for the drive that holds the message queue database.
- PrivateBytes: The memory that's used by the EdgeTransport.exe process.
- QueueLength[SubmissionQueue]: The number of messages in the Submission queue.
- SystemMemory: The memory that's used by all other processes.
- UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue]: Hard drive utilization for the drive that holds the message queue database transaction logs.
- UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data]: Hard drive utilization for the drive that's used for content conversion.
- UsedVersionBuckets[%ExchangeInstallPath%TransportRoles\data\Queue\mail.que]: The number of uncommitted message queue database transactions that exist in memory.
For each monitored system resource on a Mailbox server or Edge Transport server, the following levels of resource utilization or pressure are defined:
- Low or Normal: The resource isn't overused. The server accepts new connections and messages.
- Medium: The resource is slightly overused. Back pressure is applied to the server in a limited manner. Mail from senders in the organization's authoritative domains can flow. However, depending on the specific resource under pressure, the server uses tarpitting to delay server response or rejects incoming MAIL FROM commands from other sources.
- High: The resource is severely overused. Full back pressure is applied. All message flow stops, and the server rejects all new incoming MAIL FROM commands.
Transition levels define the low, medium and high resource utilization values depending on whether the resource pressure is increasing or decreasing. Typically, a resource utilization level that's lower than the original level is required as the resource utilization decreases. In other words, there really isn't a static value for low, medium and high resource pressure. You need to know if the utilization is increasing or decreasing before you can determine the next change in resource utilization level.
The following sections explain how Exchange handles the situation when a specific resource is under pressure.
Hard drive utilization for the drive that holds the message queue database
Resource: DatabaseUsedSpace[%ExchangeInstallPath%TransportRoles\data\Queue]
Description: Monitors the percentage of total drive space that's consumed by all files on the drive that holds the message queue database. Note that the message queue database file contains unused space, so an accurate description of the total drive space that's consumed by all files is drive size - free disk space - free space in the database.
To change the default location of the message queue database, see Change the location of the queue database.
Pressure transitions (%):
- LowToMedium: 96
- MediumToHigh: 99
- HighToMedium: 97
- MediumToLow: 94 Comments::
The default high level of hard drive utilization is calculated by using the following formula:
100 * (<hard drive size in MB> - 500 MB) / <hard drive size in MB>
This formula accounts for the fact that there's unused space in the message queue database
1 GB = 1024 MB. The result is rounded down to the nearest integer.
For example, if your message queue database is located on a 1 terabyte (TB) drive (1048576 MB), the high level of utilization is 100*(1048576-500)/1048576) or 99%.
As you can see from the formula and the rounding down behavior, the hard drive needs to be very small before the formula calculates a high utilization value that's less than 99%. For example, a 98% value for high utilization requires a hard drive of approximately 25 GB or less.
Memory used by the EdgeTransport.exe process
Resource: PrivateBytes
Description: Monitors the percentage of memory that's used by the EdgeTransport.exe process that's part of the Microsoft Exchange Transport service. This doesn't include virtual memory in the paging file, or memory that's used by other processes.
Pressure transitions (%):
- LowToMedium: 72
- MediumToHigh: 75
- HighToMedium: 73
- MediumToLow: 71
Comments:
By default, the high level of memory utilization by the EdgeTransport.exe process is 75 percent of the total physical memory or 1 terabyte, whichever is less. The results are always rounded down to the nearest integer.
Exchange keeps a history of the memory utilization of the EdgeTransport.exe process. If the utilization doesn't go down to low level for a specific number of polling intervals, known as the history depth, Exchange rejects incoming messages until the resource utilization goes back to the low level. By default, the history depth for EdgeTransport.exe memory utilization s 30 polling intervals.
Number of messages in the Submission queue
Resource: QueueLength[SubmissionQueue]
Description: Monitors the number of messages in the Submission queue. Typically, message enter the Submission queue from Receive connectors. For more information, see Mail flow and the transport pipeline. A large number of messages in the Submission queue indicates the categorizer is having difficulty processing messages.
Pressure transitions:
- LowToMedium: 9999
- MediumToHigh: 15000
- HighToMedium: 10000
- MediumToLow: 2000
Comments:
When the Submission queue is under pressure, the Exchange throttles incoming connections by delaying acknowledgement of incoming messages. Exchange reduces the rate of incoming message flow by tarpitting, which delays the acknowledgment of the SMTP MAIL FROM command to the sending server. If the pressure condition continues, Exchange gradually increases the tarpitting delay. After the Submission queue utilization returns to the low level, Exchange reduces the acknowledgment delay and eases back into normal operation. By default, Exchange delays message acknowledgments for 10 seconds when under Submission queue pressure. If the resource pressure continues, the delay is increased in 5-second increments up to 55 seconds.
Exchange keeps a history of Submission queue utilization. If the Submission queue utilization doesn't go down to the low level for a specific number of polling intervals, known as the history depth, Exchange stops the tarpitting delay and rejects incoming messages until the Submission utilization goes back to the low level. By default, the history depth for the Submission queue is in 300 polling intervals.
Memory used by all processes
Resource: SystemMemory
Description: Monitors the percentage of memory that's used by all processes on the Exchange server. This doesn't include virtual memory in the paging file.
Pressure transitions (%):
- LowToMedium: 88
- MediumToHigh: 94
- HighToMedium: 89
- MediumToLow: 84
Comments:
When the server reaches the high level of memory utilization, message dehydration occurs. Message dehydration removes unnecessary elements of queued messages that are cached in memory. Typically, complete messages are cached in memory for increased performance. Removal of the MIME content from these cached messages reduces the amount of memory that's used at the expense of higher latency, because the messages are now read directly from the message queue database. By default, message dehydration is enabled.
Hard drive utilization for the drive that holds the message queue database transaction logs
Resource: UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue]
Description: Monitors the percentage of total drive space that's consumed by all files on the drive that holds the message queue database transaction logs. To change the default location, see Change the location of the queue database.
Pressure transitions (%):
- LowToMedium: 89
- MediumToHigh: 99
- HighToMedium: 90
- MediumToLow: 80
Comments::
The default high level of hard drive utilization is calculated by using the following formula:
100 * (<hard drive size in MB> - 1152 MB) / <hard drive size in MB>
1 GB = 1024 MB. The result is rounded down to the nearest integer.
For example, if your queue database is located on a 1 terabyte (TB) drive (1048576 MB), the high level of utilization is 100*(1048576-1152)/1048576) or 99%.
As you can see from the formula and the rounding down behavior, the hard drive needs to be fairly small before the formula calculates a high utilization value that's less than 99%. For example, a 98% value for high utilization requires a hard drive of approximately 56 GB or less.
The %ExchangeInstallPath%Bin\EdgeTransport.exe.config application configuration file contains the DatabaseCheckPointDepthMax key that has the default value 384MB. This key controls the total allowed size of all uncommitted transaction logs that exist on the hard drive. The value of this key is used in the formula that calculates high utilization. If you customize this value, the formula becomes:
100 * (<hard drive size in MB> - Min(5120 MB, 3* DatabaseCheckPointDepthMax)) / <hard drive size in MB>
Note
The value of the DatabaseCheckPointDepthMax key applies to all transport-related Extensible Storage Engine (ESE) databases that exist on the Exchange server. On Mailbox servers, this includes the message queue database, and the sender reputation database. On Edge Transport servers, this includes the message queue database, the sender reputation database, and the IP filter database that's used by the Connection Filtering agent.
Hard drive utilization for the drive that's used for content conversion
Resource: UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data]
Description: Monitors the percentage of total drive space that's consumed by all files on the drive that's used for content conversion. The default location of the folder is %ExchangeInstallPath%TransportRoles\data\Temp and is controlled by the TemporaryStoragePath key in the %ExchangeInstallPath%Bin\EdgeTransport.exe.config application configuration file.
Pressure transitions (%):
- LowToMedium: 89
- MediumToHigh: 99
- HighToMedium: 90
- MediumToLow: 80
Comments:
The default high level of hard drive utilization is calculated by using the following formula:
100 * (<hard drive size in MB> - 500 MB) / <hard drive size in MB>
1 GB = 1024 MB. The result is rounded down to the nearest integer.
For example, if your message queue database is located on a 1 terabyte (TB) drive (1048576 MB), the high level of utilization is 100*(1048576-500)/1048576) or 99%.
As you can see from the formula and the rounding down behavior, the hard drive needs to be very small before the formula calculates a high utilization value that's less than 99%. For example, a 98% value for high utilization requires a hard drive of approximately 25 GB or less.
Number of uncommitted message queue database transactions in memory
Resource: UsedVersionBuckets[%ExchangeInstallPath%TransportRoles\data\Queue\mail.queue]
Description: Monitors the number of uncommitted transactions for the message queue database that exist in memory.
Pressure transitions:
- LowToMedium: 999
- MediumToHigh: 1500
- HighToMedium: 1000
- MediumToLow: 800
Comments::
A list of changes that are made to the message queue database is kept in memory until those changes can be committed to a transaction log. Then the list is committed to the message queue database itself. These outstanding message queue database transactions that are kept in memory are known as version buckets. The number of version buckets may increase to unacceptably high levels because of an unexpectedly high volume of incoming messages, spam attacks, problems with the message queue database integrity, or hard drive performance.
When version buckets are under pressure, the Exchange server throttles incoming connections by delaying acknowledgment of incoming messages. Exchange reduces the rate of incoming message flow by tarpitting, which delays the acknowledgment of the SMTP MAIL FROM command to the sending server. If the resource pressure condition continues, Exchange gradually increases the tarpitting delay. After the resource utilization returns to normal, Exchange gradually reduces the acknowledgement delay and eases back into normal operation. By default, Exchange delays message acknowledgments for 10 seconds when under resource pressure. If the pressure continues, the delay is increased in 5-second increments up to 55 seconds.
When the version buckets are under high pressure, the Exchange server also stops processing outgoing messages.
Exchange keeps a history of version bucket resource utilization. If the resource utilization doesn't go down to the low level for a specific number of polling intervals, known as the history depth, Exchange stops the tarpitting delay and rejects incoming messages until the resource utilization goes back to the low level. By default, the history depth for version buckets is in 10 polling intervals.
Actions taken by back pressure when resources are under pressure
The following table summarizes the actions taken by back pressure when a monitored resource is under pressure.
| Resource under pressure | Utilization level | Actions taken |
|---|---|---|
| DatabaseUsedSpace | Medium | Reject incoming messages from non-Exchange servers. Reject message submissions from the Pickup directory and the Replay directory. Message resubmission is paused. Shadow Redundancy rejects messages. For more information about Shadow Redundancy, see Shadow redundancy in Exchange Server. |
| DatabaseUsedSpace | High | All actions taken at the medium utilization level. Reject incoming messages from other Exchange servers. Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers. |
| PrivateBytes | Medium | Reject incoming messages from non-Exchange servers. Reject message submissions from the Pickup directory and the Replay directory. Message resubmission is paused. Shadow Redundancy rejects messages. Processing messages after a server or Transport service restart (also known as boot scanning) is paused. Start message dehydration. |
| PrivateBytes | High | All actions taken at the medium utilization level. Reject incoming messages from other Exchange servers. Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers. |
| QueueLength[SubmissionQueue] | Medium | Introduce or increment the tarpitting delay to incoming messages. If normal level isn't reached for the entire Submission queue history depth, take the following actions:
|
| QueueLength[SubmissionQueue] | High | All actions taken at the medium utilization level. Reject incoming messages from other Exchange servers. Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers. Flush enhanced DNS cache from memory. Start message dehydration. |
| SystemMemory | Medium | Start message dehydration. Flush caches. |
| SystemMemory | High | All actions taken at the medium utilization level. |
| UsedDiskSpace (message queue database transaction logs) | Medium | Reject incoming messages from non-Exchange servers. Reject message submissions from the Pickup directory and the Replay directory. Message resubmission is paused. Shadow Redundancy rejects messages. |
| UsedDiskSpace (message queue database transaction logs) | High | All actions taken at the medium utilization level. Reject incoming messages from other Exchange servers. Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers. |
| UsedDiskSpace (content conversion) | Medium | Reject incoming messages from non-Exchange servers. Reject message submissions from the Pickup directory and the Replay directory. |
| UsedDiskSpace (content conversion) | High | All actions taken at the medium utilization level. Reject incoming messages from other Exchange servers. Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers. |
| UsedVersionBuckets | Medium | Introduce or increment the tarpitting delay to incoming messages. If normal level isn't reached for the entire version bucket history depth, take the following actions:
|
| UsedVersionBuckets | High | All actions taken at the medium utilization level. Reject incoming messages from other Exchange servers. Reject message submissions from mailbox databases by the Microsoft Exchange Mailbox Transport Submission service on Mailbox servers. Stop processing outgoing messages. Remote delivery is paused. |
View back pressure resource thresholds and utilization levels
You can use the Get-ExchangeDiagnosticInfo cmdlet in the Exchange Management Shell to view the resources that are being monitored, and the current utilization levels. To learn how to open the Exchange Management Shell in your on-premises Exchange organization, see Open the Exchange Management Shell.
To view the back pressure settings on an Exchange server, run the following command:
[xml]$bp=Get-ExchangeDiagnosticInfo [-Server <ServerIdentity> ] -Process EdgeTransport -Component ResourceThrottling; $bp.Diagnostics.Components.ResourceThrottling.ResourceTracker.ResourceMeter
To see the values on the local server, you can omit the Server parameter.
Back pressure configuration settings in the EdgeTransport.exe.config file
All configuration options for back pressure are done in the %ExchangeInstallPath%Bin\EdgeTransport.exe.config XML application configuration file. However, few of the settings exist in the file by default.
Caution
These settings are listed only as a reference for the default values. We strongly discourage any modifications to the back pressure settings in the EdgeTransport.exe.config file. Modifications to these settings might result in poor performance or data loss. We recommend that you investigate and correct the root cause of any back pressure events that you may encounter.
General back pressure settings
| Key name | Default value |
|---|---|
| ResourceMeteringInterval | 00:00:02 (2 seconds) |
| DehydrateMessagesUnderMemoryPressure | true |
DatabaseUsedSpace settings
| Key name | Default value (%) |
|---|---|
| DatabaseUsedSpace.LowToMedium | 96 |
| DatabaseUsedSpace.MediumToHigh | 99 |
| DatabaseUsedSpace.HighToMedium | 97 |
| DatabaseUsedSpace.MediumToLow | 94 |
PrivateBytes settings
| Key name | Default value (%) |
|---|---|
| PrivateBytes.LowToMedium | 72 |
| PrivateBytes.MediumToHigh | 75 |
| PrivateBytes.HighToMedium | 73 |
| PrivateBytes.MediumToLow | 71 |
| PrivateBytesHistoryDepth | 30 |
QueueLength[SubmissionQueue] settings
| Key name | Default value |
|---|---|
| QueueLength[SubmissionQueue].LowToMedium | 9999 |
| QueueLength[SubmissionQueue].MediumToHigh | 15000 |
| QueueLength[SubmissionQueue].HighToMedium | 10000 |
| QueueLength[SubmissionQueue].MediumToLow | 2000 |
| SubmissionQueueHistoryDepth | 300 (after 10 minutes) |
SystemMemory settings
| Key name | Default value (%) |
|---|---|
| SystemMemory.LowToMedium | 88 |
| SystemMemory.MediumToHigh | 94 |
| SystemMemory.HighToMedium | 89 |
| SystemMemory.MediumToLow | 84 |
UsedDiskSpace settings (message queue database transaction logs)
| Key name | Default value (%) |
|---|---|
| UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue].LowToMedium | 89 |
| UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue].MediumToHigh | 99 |
| UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue].HighToMedium | 90 |
| UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data\Queue].MediumToLow | 80 |
Note
Values that contain only UsedDiskSpace (for example, UsedDiskSpace.MediumToHigh) apply to the message queue database transaction logs and to content conversion.
UsedDiskSpace settings (content conversion)
| Key name | Default value (%) |
|---|---|
| UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data].LowToMedium | 89 |
| UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data].MediumToHigh | 99 |
| UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data].HighToMedium | 90 |
| UsedDiskSpace[%ExchangeInstallPath%TransportRoles\data].MediumToLow | 80 |
| TemporaryStoragePath | %ExchangeInstallPath%TransportRoles\data\Temp |
UsedVersionBuckets settings
| Key name | Default value |
|---|---|
| UsedVersionBuckets.LowToMedium | 999 |
| UsedVersionBuckets.MediumToHigh | 1500 |
| UsedVersionBuckets.HighToMedium | 1000 |
| UsedVersionBuckets.MediumToLow | 800 |
| VersionBucketsHistoryDepth | 10 |
Back pressure logging information
The following list describes the event log entries that are generated by specific back pressure events in Exchange:
Event log entry for an increase in any resource utilization level
Event Type: Error
Event Source: MSExchangeTransport
Event Category: Resource Manager
Event ID: 15004
Description: Resource pressure increased from <Previous Utilization Level> to <Current Utilization Level>.
Event log entry for a decrease in any resource utilization level
Event Type: Information
Event Source: MSExchangeTransport
Event Category: Resource Manager
Event ID: 15005
Description: Resource pressure decreased from <Previous Utilization Level> to <Current Utilization Level>.
Event log entry for critically low available disk space
Event Type: Error
Event Source: MSExchangeTransport
Event Category: Resource Manager
Event ID: 15006
Description: The Microsoft Exchange Transport service is rejecting messages because available disk space is below the configured threshold. Administrative action may be required to free disk space for the service to continue operations.
Event log entry for critically low available memory
Event Type: Error
Event Source: MSExchangeTransport
Event Category: Resource Manager
Event ID: 15007
Description: The Microsoft Exchange Transport service is rejecting message submissions because the service continues to consume more memory than the configured threshold. This may require that this service be restarted to continue normal operation.