MSExchangeGuru.com

So you got a call from help-desk stating they have a lot of users complaining they cannot access their mailbox or performance is terribly slow? Ok, What next?I blogged on the background process involved where exchange processes transactions and commit to the database.

Let’s take a look at troubleshooting this issue.

Once you ensure all mailbox stores are mounted, the next step would be to do this.

Start — Run — Perfmon

You see in the figure:

RPC requests –> Clients are requesting for data

RPC Operations/sec –>Server is acknowledging the client request.

Let’s look at how exchange prepares itself to handle client requests.

1. Microsoft Exchange Information Store service (store.exe) starts on a server
   
2. 500 RPC threads is allocated to the service upon registering with the RPC service
   
3. Clients connect and disconnect to individual RPC threads as they perform operations from their outlook like reading and sending e-mail, creating appointments and tasks, and viewing calendars.
   
4. The MSExchangeIS\RPC Requests performance counter indicates how many threads are currently in use (“owned” by clients).
   
5. The MSExchangeIS\RPC Operations/sec performance counter reflects the number of operations the server received in the past second.
   
6. If the number of RPC requests increases steadily over time, it is a clear indication that the server cannot process client operations quickly enough.
   
7. When the MSExchangeIS\RPC Requests performance counter reaches 500, all RPC threads have been exhausted, and clients will be unable to submit new requests to the server until all operations on an existing thread have been completed and that thread is released.
   

Right now, you are facing an issue mentioned in step7, maybe that’s why you’re reading this.

How to troubleshoot?

I am listing down the various conditions the 2 counters will be and what they mean.

MSExchangeIS\RPC Requests and MSExchangeIS\RPC Operations/sec are either low or at zero

• Bottleneck might not be on the server.
  
• Something external to the server is preventing information from reaching the server. Review Active Directory performance, network performance, client configuration etc.
  
• Get information about the GC the server is talking to using the NLTest command: nltest /SC_QUERY:<Type Domain name here>
  
• Reset the GC to another one using: nltest /sc_reset:<Type Domain name here>
  
• To get a list of DC’s for the domain, use: nltest /DCLIST:<Type Domain name here>
  

MSExchangeIS\RPC Requests is increasing rapidly while MSExchangeIS\RPC Operations/sec remains fairly stable (Horizontal line)

• The server is unable to handle the existing workload.
  
• Verify hardware components like physical memory, storage, and processor capability
• Open task manager and look at “All processes”. On an Exchange server, apart from the Store.exe process, nothing should consume memory or CPU no matter what
  
• Verify that the system drive (usually C Drive) has atleast 2-5GB free space
  
• Decrease the number of users on the server
  
• See if online defrag is getting completed on all database’s. Believe me, it plays a huge role when the DB is too large and has a lot of whitespace in it. The best way to identify this is to check eventvwr for events related to Online defragmentation like 700, 701, 702 etc.
  
• Run ExMon to see if it is a specific user is a top talker. In ExMon, you can again sort client requests by RPC requests
  

MSExchangeIS\RPC Requests is steadily increasing but MSExchangeIS\RPC Operations/sec steadily decreases

• Exchange server is the source of the bottleneck
  
• Verify that the system drive (usually C Drive) has atleast 2-5GB free space
• Open task manager and look at “All processes”. On an Exchange server, apart from the Store.exe process, nothing should consume memory or CPU no matter what
• Might be caused as something is preventing the Information Store from completing RPC operations, and the associated RPC threads remain allocated permanently
  
• More and more threads are allocated; the server has fewer available threads for new operations, so the number of new operations decreases.
  
• If the server eventually reaches 500 outstanding RPC requests, new RPC operations will be refused. This is generally caused by either a serious physical resource shortage (memory or disk) or a processing issue within the Information Store or an integrated component (antivirus, journaling, and so on).
  
• Run ExMon to see if it is a specific user is a top talker. In ExMon, you can again sort client requests by RPC requests
  
• See if online defrag is getting completed on all database’s. Believe me, it plays a huge role when the DB is too large and has a lot of whitespace in it. The best way to identify this is to check eventvwr for events related to Online defragmentation like 700, 701, 702 etc.

If ExMon shows you high RPC requests from one user mailbox, open his mailbox and look for corruption in calendar. Sort the calendar items by “list” and then by “size”. Any calendar item which is more than 8-10MB is not considered as a “calendar” item. Delete it and notify the organizer that you had to delete it since the item got corrupted.

Additionally, look for messages stuck in Outbox for that user. There might be a zip file sitting and looping again and again. Use this:

Get-Mailbox -ResultSize Unlimited | Get-MailboxFolderStatistics |Where-Object {$_.Name -eq “Outbox” -and $_.ItemsInFolder -gt ‘0’ } | Select-Object Identity, FolderType, ItemsinFolder, FolderSize | Export-CSV “C:\Outbox.csv”

This is a checklist of RPC counters and their expected values:

If you have a monitoring system in place, it is mandatory to tune it to throw alerts when the counters cross the allowed limit.

Ratish Nair
MVP Exchange
Team@ MSExchange

Keywords: Exchange server performance issues, troubleshoot performance issues with exchange, Exchange RPC request are spiking high, RPC operations not processed on exchange

Posted February 27th, 2011 under Exchange 2003, Exchange 2007. RSS 2.0 feed. Leave a response, or trackback.