What Causes Exchange Disk I/O
Microsoft Exchange Server 2007 will reach end of support on April 11, 2017. To stay supported, you will need to upgrade. For more information, see Resources to help you upgrade your Office 2007 servers and clients.
Applies to: Exchange Server 2007, Exchange Server 2007 SP1, Exchange Server 2007 SP2, Exchange Server 2007 SP3
The server roles in Microsoft Exchange Server 2007 are Hub Transport and Edge Transport (collectively referred to as transport servers), Client Access, Unified Messaging, and Mailbox. Each server role has different storage requirements and different backup and restore requirements, in part because they each perform different functions:
Hub Transport and Edge Transport servers deliver:
Mail into and out of the organization.
Mail into and out of Mailbox servers.
Voice mail messages submitted by Unified Messaging servers.
Client Access servers are the client protocol servers for Exchange, providing Microsoft Outlook Web Access, Exchange ActiveSync, Outlook Anywhere, and other Internet protocols.
Unified Messaging servers provide Outlook Voice Access, as well as incoming fax support.
Mailbox servers, the heart of Exchange Server, are where user mailboxes and public folders are stored.
Mailbox clustering, or single copy cluster (SCC), uses the Cluster service in a shared disk active/passive configuration.
Continuous replication ships log files to an alternate location, which can be on a stand-alone server using local continuous replication (LCR), in a cluster using cluster continuous replication (CCR), or on a remote server using standby continuous replication (SCR).
Mailbox Server Role
The Exchange 2007 Mailbox server role is the core server role upon which all other server roles build. After you determine your mailbox profile, which includes user input/output (I/O) per second and capacity, you can begin to plan your deployment. How many users you place on an Exchange server is usually based on a balance between preventing a hardware bottleneck and providing the ability to back up and restore that data within your service level agreement (SLA).
Three storage requirements must be balanced to achieve a successful Exchange 2007 deployment. The first requirement is the transactional I/O, or the performance measured in latency for each I/O to be satisfied by the storage. The second requirement is backup and restore throughput, or how quickly your data can be moved to and from your backup medium. The third requirement is capacity, or making sure that you have enough space in the chosen redundant array of independent disks (RAID) configuration for your production logical unit numbers (LUNs) and on the target backup medium.
For information about how to size your disk I/O requirements using mailbox profiling, see Mailbox Server Storage Design. For example, you may want to place 3,000 users on a server with a 0.4 I/O per second (IOPS) profile with 2-gigabyte (GB) mailboxes. Your performance requirement would be 1,200 IOPS. You would have to make sure that you could back up and restore 6 terabytes of information. If your backup SLA is four hours, you would have to back up 1.5 terabytes of data in an hour or 417 megabytes (MB) per second. If your backup solution could only back up 300 MB per second, you would have to reduce the mailbox size or the number of users by 28 percent.
In Exchange 2000 Server, the best practice, which was influenced by virtual memory limitations, is to fill a storage group with five databases before creating another storage group. In Exchange Server 2003, those limitations were considerably reduced, and the best practice is to add an additional storage group for each new database until the maximum number of storage groups has been created. With Exchange 2007, the I/O footprint is reduced due to enhancements to the Extensible Storage Engine (ESE), the underlying database engine used by Exchange Server.
Core Extensible Storage Engine Enhancements
Exchange 2007 reduces the overall I/O footprint for Exchange Server because of several key design changes in ESE:
A 64-bit operating system and a 64-bit Exchange Server application enable a much larger database cache, increasing from 900 MB to potentially dozens of gigabytes, depending on total system memory.
Database read operations also benefit from many new cache optimizations. An I/O coalescing increase from 64 kilobytes (KB) to 1 MB further reduces disk I/O by increasing the opportunity to read and write larger I/O.
There is no streaming database file, and the installable file system (IFS) has been removed.
As a 64-bit application, Exchange 2007 does not have the virtual memory limitations of its 32-bit predecessors. Exchange 2007 Mailbox servers support up to 50 databases and 50 storage groups, and you can place up to five databases in a storage group. However, each Exchange 2007 Mailbox server can have a maximum of 50 databases.
Each storage group creates its own separate transaction log and is the basic unit for backup and restore. In the absence of cache pressure, the maximum amount of data that the ESE can write to the transaction log before it writes to the database is a cache called the checkpoint depth. Using one database in a storage group allocates the entire checkpoint depth to that database, increasing the likelihood that multiple updates to a database page will be done in the cache, and only the last update will be written to the database, thereby reducing I/O.
Exchange Mailbox Data Components
The following table describes Mailbox server role activities, and how each activity affects disk I/O.
Mailbox server role activities in Exchange 2007
| Activity | How activity affects disk I/O |
|---|---|
ESE database (.edb file) |
The Mailbox server stores all mail in an ESE database. The ESE database is randomly accessed and uses an 8-KB page size, although I/O coalescing can result in larger I/Os. For reliability, and in some cases for performance reasons, the database should be on disks that do not contain transaction logs. |
Transaction log files (.log files) |
All changes made to the database are first committed to the transaction log, which is a sequential write to the disk. The writes vary in size from 512 bytes to the log buffer size. |
Content indexing |
Content indexing is a random workload that should be placed on the same LUN as your database and will typically be about 5 percent of the database size. Because content indexing runs in the background, indexing messages as they arrive, the disk I/O impact is minimal. |
Paging |
If a process requests a page in memory and the system cannot find the page at the requested location, a page fault occurs. If the page is elsewhere in memory, the fault is a soft page fault. If the page must be retrieved from disk, the fault is a hard page fault. Most processors can handle large numbers of soft page faults without consequence. However, hard page faults can cause significant delays. Continuous high rates of disk paging indicate a memory shortage. |
Content conversion |
The native method to store data in Exchange uses MAPI messages that are encapsulated in Transport Neutral Encapsulating Format (TNEF). This allows MAPI messages to be transported over SMTP and provides MAPI messages for MAPI clients, such as Microsoft Outlook. Non-MAPI clients require messages in MIME format. This format requires Exchange to perform a content conversion process from TNEF / MAPI to MIME format. Most content conversion is performed on the Client Access servers and Hub Transport servers. However, legacy Web Distributed Authoring and Versioning (WebDAV) applications, such as Microsoft Entourage, access the Mailbox server directly. In this scenario, the content conversion process occurs directly on the Exchange 2007 Mailbox server. When a legacy WebDAV client requests data that must be converted on a Client Access server, the data is accessed from the Exchange 2007 Mailbox server by accessing the /Exchange virtual directory. (Some tools access the /ExAdmin virtual directory to access data.) The data is converted in the Tmp directory on the Mailbox server, and then sent to the Client Access server. The greatest usage of the Tmp directory often occurs after non-MAPI client users are moved to a new server. This behavior occurs because there may be a large amount of content conversion when users first connect to their mailboxes. To improve performance, the Tmp directory should not be on the same LUN as the page file or the operating system. |
Database maintenance |
The Exchange 2007 information store requires periodic online maintenance to be run against each database. The two tasks that affect disk I/O are the hard deletion of messages and mailboxes that are older than the configured retention policy and online defragmentation of the database. Because backup of a database will halt any online defragmentation of that database that might be taking place, care must be taken to give both backup and database maintenance exclusive windows of time to complete their tasks. |
Backup and restore |
The process of backing up data requires that data be read from database and transaction log file volumes. This additional I/O can affect user response times and should be avoided during business hours. The process of soft recovery requires that the ESE play back all of the transaction log files. This causes the I/O profile to be a sequential read stream. As a result, the recovery performance improves if the transaction log files are on a disk with fast sequential disk access. One way to avoid this is to use continuous replication, which enables you to offload Volume Shadow Copy Service(VSS)-based backups from the active copy of the database to the passive copy of the database. |
Zero out deleted database pages |
If you configure a Mailbox server to zero out deleted database pages, every time you delete an item from the database, multiple pages are deleted. Exchange will then overwrite the deleted pages with zeros. In the release to manufacturing (RTM) version of Microsoft Exchange Server 2007, this feature is only run during an online streaming backup, and it causes more physical disk I/O during backup. In Exchange Server 2007 Service Pack 1 (SP1), this feature can be enabled during the online maintenance window. |
In addition to database file access, there are other activities that result in disk I/O. The following table lists these additional activities and their effect on disk I/O.
Additional activities that affect disk I/O
| Activity | How activity affects disk I/O |
|---|---|
Number of items in a folder |
As the number of items in the core mailbox folders increases, the physical disk cost to perform some tasks also increases for users of Outlook in Online Mode. Indexes and searches are performed on the client when using Outlook in Cached Exchange Mode. Sorting your Inbox by size for the first time requires the creation of a new index, which will require many disk I/Os. Future sorts of the Inbox by size will be very inexpensive. There is a static number of indexes that you can have, so users that often sort their folders in many different ways could exceed this limit and cause additional disk I/O. |
BlackBerry |
For more information about BlackBerry and Exchange 2007, view the performance benchmarking guidance on the BlackBerry Enterprise Server for Microsoft Exchange web page on the Research In Motion (RIM) web site. Note The third-party Web site information in this topic is provided to help you find the technical information you need. The URLs are subject to change without notice. |
Public folders |
If public folders reside on the server, additional I/O loads are incurred. However, if no replicas of folder content exist on the server, the I/O generated from having a public folder database is inconsequential relative to the I/O generated by user mailbox access. |
Backup
Backing up mailboxes requires careful planning. The following sections provide some considerations for VSS and streaming online backups. There are compromises in every solution that affect variables such as cost, time, and reliability. Most administrators define a time for online database maintenance, defragmentation, and operating system maintenance. These activities compete with the backup time. Special care is needed to balance backup, maintenance, and production load. Larger mailboxes may make a daily full backup strategy impossible to complete within your SLA. A common strategy to lessen the impact of a full nightly backup is to perform weekly full backups and daily differential backups. With this strategy, you need to recover the full backup, and then recover the last differential backup.
Volume Shadow Copy Service
For details about VSS fundamentals and VSS best practices for Exchange 2003, we recommend that you read Best Practices for Using Volume Shadow Copy Service with Exchange Server 2003. In addition to the information covered in that article, there are two major VSS-related considerations in Exchange 2007 that must be considered:
Larger mailboxes
Ability to back up a CCR and LCR copy
Although VSS backups can be made on either the production or copy data, we recommend that you back up the copy and avoid an impact to the production physical disks.
With LCR, Exchange 2007 is replicating transaction logs to a separate disk on the same server. When using VSS clones on the copy, the copy storage should be configured on different physical disks so that the clone operation and checksum integrity do not affect the production physical disks. With VSS snapshots on the copy, the copy storage should be configured on different physical disks so that the checksum integrity and subsequent streaming to tape do not affect your production physical disks.
With CCR, Exchange 2007 is replicating transaction logs to a separate server. This server is a node in the cluster, but the target copy is not kept on shared storage. When using VSS clones, you can run the checksum integrity on the copy on the passive node using different physical disks, thereby isolating the backup process. With VSS snapshots, the checksum integrity and subsequent streaming to tape will not impact your production server or physical disks.
Streaming Online Backup
Unlike hardware-based VSS backups, where the data is typically moved within the storage appliance, when using streaming backups, the server is responsible for moving your data. The performance impact of the checksum integrity process is not an issue because every page is checked during the backup. In the case of concurrent backups, multiple streams can stress the limits of your backup media, whether it is gigabit Ethernet or Fibre Channel attached tape. For many customers, the backup SLA window, divided by the throughput-per-minute of their streaming backup medium, limits the permissible size of the storage group. For example, if you had a one hour SLA on your storage group, and you can back up 100 MB per second, your maximum storage group size would be 360 GB.
Client Access Server
The Client Access server offloads many stateless tasks from the Mailbox server (assuming the roles are installed on different physical servers), and provides a unified namespace where your users need only point to a single name regardless of which Mailbox server they are on. Internet protocols such as Internet Message Access Protocol 4 (IMAP4), Post Office Protocol 3 (POP3), and HTTP are serviced by the Client Access server. Outlook Anywhere, ActiveSync, Autodiscover service, Availability service, and Web services are other examples of features serviced by the Client Access server.
The Client Access server can be affected by CPU, memory, and network bottlenecks, yet it has a small disk I/O footprint. Simple Mail Transfer Protocol (SMTP) traffic, a potential disk I/O consideration in front-end servers running Exchange 2003 and Exchange 2000, is now owned exclusively by the Hub Transport servers and Edge Transport servers.
The following table describes Client Access server role activities, and how each activity affects disk I/O.
Client Access server role activities in Exchange 2007
| Activity | How activity affects disk I/O |
|---|---|
Protocol logging |
Protocol logging is a sequential write that, if enabled, causes a performance issue and consumes disk space to store the logs. When you keep a history of the protocol you choose to log, you can verify whether the protocol is performing its work as expected or is experiencing communications problems, and identify attacks from the Internet. |
Content conversion |
Content conversion for all Exchange 2007 protocols occurs on the Client Access server. Legacy WebDAV content conversion, for legacy Outlook Web Access clients, occurs on the Exchange 2003 Mailbox server. When a client requests data that must be converted on a Client Access server, the data is accessed from the Exchange 2003 Mailbox server, converted in the Mailbox server TMP folder, and sent to the Client Access server. To improve performance, the TMP folder should not be on the same LUN as the page file and operating system. |
Paging |
If a process requests a page in memory and the system cannot find the page at the requested location, a page fault occurs. If the page is elsewhere in memory, the fault is a soft page fault. If the page must be retrieved from disk, the fault is a hard page fault. Most processors can handle large numbers of soft page faults without consequence. However, hard page faults can cause significant delays. Continuous high rates of disk paging indicate a memory shortage. |
The scenario where disk I/O becomes an issue for Client Access servers is one where the user uses an Internet client to access mailbox data through either POP3 or IMAP4 protocols. Because the transport engine converts all incoming mail to MAPI, a POP3 or IMAP4 client needs that content converted back into Multipurpose Internet Mail Extensions (MIME) before sending it to the client. This conversion occurs on the Client Access server, and if the message is larger than 64 KB, the conversion occurs on disk. If a large percentage of the user base is using POP3 or IMAP4, the temporary folder where conversion occurs should be placed on a dedicated fast disk.
Transport Servers
The Hub Transport servers and Edge Transport servers are the bridgehead and gateway servers for Exchange 2007. Their primary mission is to send and receive mail. Many businesses will deploy a transport server into two groups:
Anti-spam and antivirus protection (Edge Transport server)
Routing (Hub Transport server)
The Edge Transport server's primary responsibility is to protect the Exchange infrastructure from incoming mail that contains spam or viruses. The Hub Transport server then categorizes the clean mail and delivers it to the correct Mailbox server. The storage impact to these servers fluctuates depending on the number of messages handled per second and the average size of those messages.
The following table describes Edge Transport server and Hub Transport server activities, and how each activity affects disk I/O.
Edge Transport and Hub Transport server role activities in Exchange 2007
| Activity | How activity affects disk I/O |
|---|---|
ESE database (mail.que file) |
Both the Exchange 2007 Edge Transport server and Hub Transport server store all mail in an ESE database. The ESE database is randomly accessed and uses an 8-KB page size. For reliability, and in some cases for performance reasons, the database should be on separate disks from the transaction logs. |
Transaction log files (.log files) |
All changes made to the database are first committed to the transaction log, which is a sequential write to the disk. The writes vary in size from 512 bytes to the log buffer size. |
Protocol logging and message tracking logs |
Message tracking and protocol logging is a sequential write that, if enabled, causes a disk performance issue and consumes disk space to store the logs. When you keep a history of the protocol you choose to log, you can verify whether the protocol is performing its work as expected or is experiencing communications problems, and identify attacks from the Internet. |
Content conversion |
On the Hub Transport server, incoming mail from the Internet is converted to MAPI prior to being delivered. This content conversion process occurs in the TMP folder. To improve performance, the TMP folder should not be on the same LUN as the paging file and the operating system. |
Paging |
If a process requests a page in memory and the system cannot find the page at the requested location, a page fault occurs. If the page is elsewhere in memory, the fault is a soft page fault. If the page must be retrieved from disk, the fault is a hard page fault. Most processors can handle large numbers of soft page faults without consequence. However, hard page faults can cause significant delays. Continuous high rates of disk paging indicate a memory shortage. |
Agents |
Customization of the transport server is done via code, known as agents, running in the common language runtime environment and triggered by an event. Some agents log data, which will be a disk performance hit and consume disk space to store the logs. |
Unified Messaging Server
For information about sizing Unified Messaging servers, see Determining the Number of Users an Exchange 2007 Unified Messaging Server Can Support.
Note
The content of each blog and its URL are subject to change without notice. The content within each blog is provided "AS IS" with no warranties, and confers no rights. Use of included script samples or code is subject to the terms specified in the Microsoft Terms of Use.