Exchange 2016 - Insufficient system storage

Question

Issue:
Insufficient system storage. The server response was: 4.3.1 Insufficient system resources (CheckpointDepth[C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que])

We have Exchange Server 2016 CU20 running on Windows server 12 r2 (Hyper- V)
C: drive 200GB with 80GB available space
E: drive (exchange database) 1.5TB with 1TB available space.
RAM: 48GB

We got the above error and mails stuck in the queue and won’t process/route. We checked the memory and CPU usage in a Task Manager and they were around 20% CPU and under 40% memory usage. We also checked the disk space on both C: and E: and they look normal as we stated above.
We have to reboot the Exchange server in order for mails to process again.

What could we do to prevent this issue from happening again in the future?

Your comment will be greatly appreciated.

Tee

Answer 1

You had too many uncommitted messages in the mail.que.
Its not a disk resource issue.

It may clear itself but you may need to adjust it for future large messaging queues

https://learn.microsoft.com/en-us/exchange/mail-flow/back-pressure?view=exchserver-2019#hard-drive-utilization-for-the-drive-that-holds-the-message-queue-database-transaction-logs

The default high level of hard drive utilization is calculated by using the following formula:

100 * (<hard drive size in MB> - 1152 MB) / <hard drive size in MB>

1 GB = 1024 MB. The result is rounded down to the nearest integer.

For example, if your queue database is located on a 1 terabyte (TB) drive (1048576 MB), the high level of utilization is 100*(1048576-1152)/1048576) or 99%.

As you can see from the formula and the rounding down behavior, the hard drive needs to be fairly small before the formula calculates a high utilization value that's less than 99%. For example, a 98% value for high utilization requires a hard drive of approximately 56 GB or less.

The %ExchangeInstallPath%Bin\EdgeTransport.exe.config application configuration file contains the DatabaseCheckPointDepthMax key that has the default value 384MB. This key controls the total allowed size of all uncommitted transaction logs that exist on the hard drive. The value of this key is used in the formula that calculates high utilization. If you customize this value, the formula becomes:

100 * (<hard drive size in MB> - Min(5120 MB, 3* DatabaseCheckPointDepthMax)) / <hard drive size in MB>

Also see:

https://techcommunity.microsoft.com/t5/exchange-team-blog/transport-database-understand-size-and-troubleshoot/ba-p/1113388