MSExchangeGuru.com

Learn Exchange the Guru way !!!

 

Troubleshooting performance issues with Exchange when RPC request spike high

So you got a call from help-desk stating they have a lot of users complaining they cannot access their mailbox or performance is terribly slow? Ok, What next?I blogged on the background process involved where exchange processes transactions and commit to the database.

Let’s take a look at troubleshooting this issue.

Once you ensure all mailbox stores are mounted, the next step would be to do this.

Start — Run — Perfmon

You see in the figure:

RPC requests –> Clients are requesting for data

RPC Operations/sec –>Server is acknowledging the client request.

Let’s look at how exchange prepares itself to handle client requests.

  1. Microsoft Exchange Information Store service (store.exe) starts on a server
  2. 500 RPC threads is allocated to the service upon registering with the RPC service
  3. Clients connect and disconnect to individual RPC threads as they perform operations from their outlook like reading and sending e-mail, creating appointments and tasks, and viewing calendars.
  4. The MSExchangeIS\RPC Requests performance counter indicates how many threads are currently in use (“owned” by clients).
  5. The MSExchangeIS\RPC Operations/sec performance counter reflects the number of operations the server received in the past second.
  6. If the number of RPC requests increases steadily over time, it is a clear indication that the server cannot process client operations quickly enough.
  7. When the MSExchangeIS\RPC Requests performance counter reaches 500, all RPC threads have been exhausted, and clients will be unable to submit new requests to the server until all operations on an existing thread have been completed and that thread is released.

Right now, you are facing an issue mentioned in step7, maybe that’s why you’re reading this.

How to troubleshoot?

I am listing down the various conditions the 2 counters will be and what they mean.

MSExchangeIS\RPC Requests and MSExchangeIS\RPC Operations/sec are either low or at zero

  • Bottleneck might not be on the server.
  • Something external to the server is preventing information from reaching the server. Review Active Directory performance, network performance, client configuration etc.
  • Get information about the GC the server is talking to using the NLTest command: nltest /SC_QUERY:<Type Domain name here>
  • Reset the GC to another one using: nltest /sc_reset:<Type Domain name here>
  • To get a list of DC’s for the domain, use: nltest /DCLIST:<Type Domain name here>

MSExchangeIS\RPC Requests is increasing rapidly while MSExchangeIS\RPC Operations/sec remains fairly stable (Horizontal line)

  • The server is unable to handle the existing workload.
  • Verify hardware components like physical memory, storage, and processor capability
  • Open task manager and look at “All processes”. On an Exchange server, apart from the Store.exe process, nothing should consume memory or CPU no matter what
  • Verify that the system drive (usually C Drive) has atleast 2-5GB free space
  • Decrease the number of users on the server
  • See if online defrag is getting completed on all database’s. Believe me, it plays a huge role when the DB is too large and has a lot of whitespace in it. The best way to identify this is to check eventvwr for events related to Online defragmentation like 700, 701, 702 etc.
  • Run ExMon to see if it is a specific user is a top talker. In ExMon, you can again sort client requests by RPC requests

MSExchangeIS\RPC Requests is steadily increasing but MSExchangeIS\RPC Operations/sec steadily decreases

  • Exchange server is the source of the bottleneck
  • Verify that the system drive (usually C Drive) has atleast 2-5GB free space
  • Open task manager and look at “All processes”. On an Exchange server, apart from the Store.exe process, nothing should consume memory or CPU no matter what
  • Might be caused as something is preventing the Information Store from completing RPC operations, and the associated RPC threads remain allocated permanently
  • More and more threads are allocated; the server has fewer available threads for new operations, so the number of new operations decreases.
  • If the server eventually reaches 500 outstanding RPC requests, new RPC operations will be refused. This is generally caused by either a serious physical resource shortage (memory or disk) or a processing issue within the Information Store or an integrated component (antivirus, journaling, and so on).
  • Run ExMon to see if it is a specific user is a top talker. In ExMon, you can again sort client requests by RPC requests
  • See if online defrag is getting completed on all database’s. Believe me, it plays a huge role when the DB is too large and has a lot of whitespace in it. The best way to identify this is to check eventvwr for events related to Online defragmentation like 700, 701, 702 etc.

If ExMon shows you high RPC requests from one user mailbox, open his mailbox and look for corruption in calendar. Sort the calendar items by “list” and then by “size”. Any calendar item which is more than 8-10MB is not considered as a “calendar” item. Delete it and notify the organizer that you had to delete it since the item got corrupted.

Additionally, look for messages stuck in Outbox for that user. There might be a zip file sitting and looping again and again. Use this:

Get-Mailbox -ResultSize Unlimited | Get-MailboxFolderStatistics |Where-Object {$_.Name -eq “Outbox” -and $_.ItemsInFolder -gt ‘0’ } | Select-Object Identity, FolderType, ItemsinFolder, FolderSize | Export-CSV “C:\Outbox.csv”

This is a checklist of RPC counters and their expected values:

Counter Values
MSExchangeIS\RPC Requests lower than 70
MSExchangeIS\RPC Operations/sec Always higher than RPC Requests
MSExchangeIS\RPC Averaged Latency Less than 25 milliseconds
MSExchangeIS\RPC Num. of Slow Packets Less than 2

If you have a monitoring system in place, it is mandatory to tune it to throw alerts when the counters cross the allowed limit.

Ratish Nair
MVP Exchange
Team@ MSExchange

Keywords: Exchange server performance issues, troubleshoot performance issues with exchange, Exchange RPC request are spiking high, RPC operations not processed on exchange

21 Responses to “Troubleshooting performance issues with Exchange when RPC request spike high”

  1. Troubleshooting no Exchange 2010 com problemas de performance « Rodrigo Rodrigues .:. www.andersonpatricio.org Says:

    […] Veja aqui: http://msexchangeguru.com/2011/02/27/rpc-issues/ […]

  2. Shyam Seegu Says:

    This is really a Helpful article for me…Thankyou so much Ratish.:)

  3. Anand Kumar Deva Says:

    Great write up…Thanks Ratish

  4. Kanta Prasad Says:

    This is realy very cool article and in very easy language and steps for a reader.

    thanks again

  5. Shyam Says:

    Hi Ratish,

    here I have one doubt,can you please explain me….

    Suppose the issue has been Narrow down and we found that, The issue is not with the Mailbox..Issue is with the some third party application which is taking more Resorces of the server.

    Can you please help me out if this is the scenario.

    Thankyou.

    / Shyam

  6. Jeevananthan Says:

    HI Rathish,

    Good article about RPC request issue…

    Great Job!!!

    Thanks
    Jeevananthan S

  7. Babu B Says:

    Hi Ratish,

    Good article

    Babu B

  8. Francesco Says:

    Great, but you didnt mention a last scenario.. RPC Operations/sec is HIGH and RPC Request is low.

    I get upp to peaks of 700 sometimes on RPC Operations/sec.. but only 0-5 in RPC Requests
    Latency goes up to 50ms sometimes and sometimes together with the peak of RPC Operations/Sec.
    Slow packets sometimes are up to 4.

    Operations in Exmon varies from 1-3000…

    I would like to get more screens from you if possible and why my RPC Packets are soo much lower then Operation/Sec since you say it should be below 75 and I get 0-5..

  9. Perry J Says:

    Hey Francesco,

    I am getting exactly the same thing, i dont know if its a coincidence but when Blackberry went down across the world earlier this week i am getting extremely low perofrmance from exchange and the same symptoms as you.

    Did you find a solution?

  10. Gary J Says:

    I’m experiencing very very high RPC Operations during the backup window to the point where the server can become unresponsive and the cluster service will stop on the DAG member. RPC Operations hits between 1500-2000 at one point, drops down, and spikes back up to 1500 for a period of time before decreasing. The eseutil (consistency check) during the backup seems to be the culprit. I’m not sure what to do to resolve. Have 1 VM Cas/Hub, 1 VM MBX and then 1 VM Cas/Hub/MBX in remote site.

  11. George Ann Says:

    Terrific work… Thank you

  12. Carl Says:

    Thanks for the guide, really handy

  13. Gengaiyan Says:

    Thanks, Its very useful. Thank you very much.

  14. Coach Jon Says:

    You state that “On an Exchange server, apart from the Store.exe process, nothing should consume memory or CPU no matter what” In my mailbox server (DAG) the cpu is consumed by Indexing, Exchange Search, and Mailbox Assistant. The over all cpu climbs to about 65% during the business day.
    Any advice on trouble shooting the cpu usage of these processes?

  15. Herbert Says:

    thanks, Iam stumped here, I am getting spikes, where the rpc requests goes up usually mostly between 0 and 200. RPC operations also follow the same trend with a spike of usually between 15000 and 20000. Then for the next few miliseconds the everything is at zero. none of the lines for operations or requests levels out. this happens throughout the day. But not at night. What is happening, is the bottleneck on my network or server, i cannot make sense of this

  16. Ved Says:

    thanks,
    excellent info

  17. Exchange 2010–Nice links to articles showing how to simply troubleshoot Exchange 2010 performance issues - Microsoft Exchange pour Tous V2 - Microsoft Exchange made simple - Site Home - TechNet Blogs Says:

    […] http://msexchangeguru.com/2011/02/27/rpc-issues/ […]

  18. Anjaiah Says:

    good

  19. Ziad Says:

    Thank you very very much..It explains a lot, Keep up the good work!!

  20. Mithun Says:

    Thanks for the valuable information. This is really good for who are looking for some straight forward stuffs.. (y)

  21. Mogomotsi Says:

    Great article

Leave a Reply

ad

Categories

Archives