Troubleshooting Slow RPC Request Processing Issues
Microsoft Exchange Server 2007 will reach end of support on April 11, 2017. To stay supported, you will need to upgrade. For more information, see Resources to help you upgrade your Office 2007 servers and clients.
When you use Microsoft Office Outlook in MAPI mode, Outlook executes client operations as remote procedure calls (RPCs) between the client and the server. If the user is running in online mode, these RPC calls occur synchronously. Any delay by the server in fulfilling these synchronous requests directly affects the user experience and the responsiveness of Outlook. In contrast, most operations that are performed when you run in cached mode occur against the user’s local copy of the mailbox or are issued to the server in the form of asynchronous (background) RPCs. Generally, asynchronous RPCs do not affect the responsiveness or overall experience of the Outlook client itself.
When the Microsoft Exchange Information Store service first starts on a server, it registers with the RPC service and receives an allocation of 500 RPC threads. Clients connect and disconnect to individual threads as they perform daily operations. This includes reading and sending e-mail, creating appointments and tasks, and viewing calendars. The MSExchangeIS\RPC Requests performance counter indicates how many threads are currently in use (“owned” by clients). The MSExchangeIS\RPC Operations/sec performance counter reflects the number of operations the server received in the past second. If the number of RPC requests increases steadily over time, it is a clear indication that the server cannot process client operations quickly enough. When the MSExchangeIS\RPC Requests performance counter reaches 500, all RPC threads have been exhausted, and clients will be unable to submit new requests to the server until all operations on an existing thread have been completed and that thread is released.
Because the MSExchangeIS\RPC Operations/sec performance counter reflects a server’s current workload, it is extremely useful for identifying bottlenecks in Exchange processing, especially if administrators know the expected values for their server during peak periods and normal operations. Servers which perform acceptably when receiving 300 RPC operations per second may struggle to keep up with 1500 RPC operations per second. Administrators should always review the MSExchangeIS\RPC Operations/sec counter and correlate changes in this value with changes in the number of RPC requests.
If clients are complaining of poor Exchange performance, and MSExchangeIS\RPC Requests and MSExchangeIS\RPC Operations/sec are both low or zero, this is a clear indication that the bottleneck is not on the server itself. The issue here is that something external to the server is preventing information from reaching the server at all. Administrators should review Active Directory performance, network performance, client configuration, and other areas that may result in data not reaching the Exchange server.
If MSExchangeIS\RPC Requests is steadily increasing while MSExchangeIS\RPC Operations/sec remains fairly stable, this indicates that the server cannot handle the existing workload. Administrators should verify hardware components, including physical memory, storage, and processor capability, or decrease the number of users on the server.
If MSExchangeIS\RPC Requests is steadily increasing but MSExchangeIS\RPC Operations/sec steadily decreases, this indicates that the Exchange server is the source of the bottleneck. In this situation, something is preventing the Information Store from completing RPC operations, and the associated RPC threads remain allocated. As more and more threads are allocated, the server has fewer available threads for new operations, so the number of new operations decreases. If the server eventually reaches 500 outstanding RPC requests, new RPC operations will be refused. This is generally caused by either a serious physical resource shortage (memory or disk) or a processing issue within the Information Store or an integrated component (antivirus, journaling, and so on).
The following table includes the most important counters for troubleshooting and isolating RPC processing issues.
Performance counters for RPC processing
| Counter | Expected values |
|---|---|
MSExchangeIS\RPC Requests Indicates the number of MAPI RPC requests currently being serviced by the Microsoft Exchange Information Store service. The Microsoft Exchange Information Store service can support up to 500 RPC requests simultaneously before rejecting new client requests. |
|
MSExchangeIS\RPC Averaged Latency Indicates the latency in milliseconds, averaged across all RPC operations in the past 1024 RPC packets. See "Client Throttling" later in this topic for information about how clients are affected when overall server RPC average latency increases. |
|
MSExchangeIS\RPC Operations/sec Indicates the current number of RPC operations that are submitted to the Information Store each second. |
|
MSExchangeIS\RPC Num. of Slow Packets Indicates the number of RPC packets in the past 1024 that have latencies that are longer than two seconds. |
|
Client-Side Monitoring
Microsoft Office Outlook 2003 and Office Outlook 2007 include added client-side monitoring features. Client-side monitoring is used to find client errors and latency problems. You can enable client-side monitoring on an Exchange server by modifying the server's registry. When enabled, Outlook 2007 and Outlook 2003 clients send data to the server based on status and state of connection, which includes failed RPC requests and error conditions. This information is aggregated on the server and logged in event log entries on the server. For detailed steps on how to enable client-side monitoring in Outlook, see How to Enable Client-Side Monitoring.
Client Throttling
Exchange 2007 includes a new feature known as RPC Client Throttling which is available to help administrators manage their end user's performance experience. RPC Client Throttling was introduced to help prevent client applications from sending too many RPC operations per second to the Exchange server which could decrease overall server performance. These client applications include desktop search engines searching through every object inside a user's mailbox, custom applications written to manipulate data located in Exchange mailboxes, enterprise-class e-mail archiving products, and CRM enabled mailboxes with e-mail auto-tagging enabled. Client throttling enables Exchange to discover and help prevent server monopolization by a few users. When a client is identified by the Exchange server as causing a disproportionate effect on the server, the server will send a "back-off" request to the client to reduce the performance effect on the server. For detailed information about the client throttling features that are available with Exchange 2007, see Understanding Client Throttling.