MSExchangeGuru.com

Microsoft Exchange Server versions 2007 and 2010 with HUB or EDGE transport server roles installed comes with a feature that facilitates monitoring of the system resources by the transport service. This feature is known as Back Pressure.

Backpressure can be used to monitor key resources like the hard disk space available, availability of memory, version buckets etc. The transport server temporarily prevents establishment of fresh connections and messages from some sources if the system resource utilization exceeds certain threshold. This can hence reduce the overloading of the transport service thus ensuring proper delivery of messages. The establishment of connections and messages are resumed once the system resource utilization is reduced to normal levels.

Nonetheless at some circumstances, manual recovery is required due to excessive overloading of server. Here we discuss the actions necessary under such circumstances.

Symptoms:

When the backpressure exceeds permissible values, you can see the termination of Inbound mail submission from other Hub Transport servers, the Internet, the Pickup directory, the Replay directory, and the Mailbox server on a Hub Transport server. These circumstances are characterized by any of the following events:

Log Name: Application
Source: MSExchangeTransport
Date: 4/17/2013 7:00:33 AM
Event ID: 15006
Task Category: ResourceManager
Level: Warning
Keywords: Classic
User: N/A
Computer: HUBServerName

Description:
The Microsoft Exchange Transport service is rejecting message submissions because the available disk space has dropped below the configured threshold.
Resource utilization of the following resources exceed the normal level:

Queue database logging disk space (“C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\”) = 99% [High] [Normal=95% Medium=97% High=99%]

Back pressure caused the following components to be disabled:
Inbound mail submission from Hub Transport servers
Inbound mail submission from the Internet
Mail submission from the Pickup directory
Mail submission from the Replay directory
Mail submission from Mailbox servers

Log Name: Application
Source: MSExchangeTransport
Date: 4/17/2013 7:00:34 AM
Event ID: 15004
Task Category: ResourceManager
Level: Warning
Keywords: Classic
User: N/A
Computer: HUBServerName
Description:

Resource pressure increased from Normal to Medium.
Resource utilization of the following resources exceed the normal level:
Version buckets = 123 [Medium] [Normal=80 Medium=120 High=200]

Back pressure caused the following components to be disabled:
Inbound mail submission from the Internet
Mail submission from the Pickup directory
Mail submission from the Replay directory
Mail delivery to remote domains

The following resources are in the normal state:
Queue database and disk space (“Q:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\mail.que”) = 3% [Normal] [Normal=95% Medium=97% High=99%]
Queue database logging disk space (“Q:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\”) = 4% [Normal][Normal=95% Medium=97% High=99%]
Private bytes = 16% [Normal] [Normal=71% Medium=73% High=75%]
Physical memory load = 40% [limit is 94% before message dehydration occurs.]

Troubleshooting Backpressure with Exchange:

High back pressure due to Low Disk space where we have queue database or queue database transaction logs path

In this case, you can see an increment in the resource pressure. For normal resource pressure, you can see that it has increased to medium and for medium resource pressure, to high.

Resources that are under pressure will be

Queue database/ Queue database logging path(Path: C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue\)= 96%/98% [Medium/High] [Normal=94% Medium=96% High=98%]

To resolve this issue, you can choose to open up some space on the drive that houses the queue or the queue database transaction logs.

How much free disk space is required refer article –

Understanding Back Pressure
http://technet.microsoft.com/en-us/library/bb201658(v=exchg.141).aspx

Option 2:

In some cases, you cannot make the necessary disk space adjustments. Then we can choose to change the location of the queue database to a disk drive with sufficient free space. It can be done by following the following steps:

1. Begin by terminating the transport service
   
2. Go to the original queue folder (the one containing all the queue database and logs) and copy the files to the new desired directory.
   
3. Open notepad and open the file C:\Program Files\Microsoft\Exchange Server\V14\Bin—EdgeTransport.exe.config
   
4. Edit the following Keys
   

<add key=”QueueDatabasePath” value=”C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue” />

<add key=”QueueDatabaseLoggingPath” value=”C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue” />

Note: The paths above need not be identical

High back pressure due to high version buckets.

In this case, you can see an increment in the resource pressure. For normal resource pressure, you can see that it has increased to medium and for medium resource pressure, to high.

Resources that are under more than normal pressure will be

: Version Buckets= 230 [Normal=80 Medium=120 High=200]

Version Buckets

Exchange stores the changes that are made to the message queue database till these actions can be saved permanently to a transaction log. Next these changes are added to the message queue database. Version Buckets refer to these message queue database transactions that are stored in the memory.

Because of various factors like the large size of the messages submitted to the transport, high incoming message numbers, spam messages etc, the number of version buckets may be considerably large.

Incoming mailflow will be stopped once the maximum limit of bucket versions are reached. You will also experience the following changes:

• Swift Increase in the transaction file resulting in faster occupancy of the drive space which in turn make the transport service to terminate every couple of minutes
  
• Transport service does not initiate or Transport service terminates frequently
  
• An exponential growth in the transport mail queue resulting in the crashing of transport service
  

Note: If the messages are of size less than 900 MB you will only see that the transport service goes back to back pressure.

Resolving the Issue

1. Open notepad and open the file C:\Program Files\Microsoft\Exchange Server\V14\Bin—EdgeTransport.exe.config
   
2. Edit the following Keys

<add key=”VersionBucketsHighThreshold” value=”200″ />

<add key=”VersionBucketsMediumThreshold” value=”120″ />

<add key=”VersionBucketsNormalThreshold” value=”80″ />

1. Find if any large mails are being blocked in the queue.
   
2. Navigate to configuration–>Hub Transport>Globalsettings and figure out the limit applied on message sizes within the organization. Ensure that the limit is not set to a high value or unlimited
   
3. In case you encounter 15004, fetch the tracking logs of the message. Make use of Process Tracking Log Explorer (PTL) so that you can examine the tracking logs of the message.
   

   PTLRefer-
   Updated Process Tracking Log (PTL) tool for use with Exchange 2007 and Exchange 2010
   http://blogs.technet.com/b/exchange/archive/2011/10/21/updated-process-tracking-log-ptl-tool-for-use-with-exchange-2007-and-exchange-2010.aspx
   

4. In case there are size constraints that has been established, open the edgetransport.exe.config. and check the following settings:
   

    <add key=”DatabaseMaxCacheSize” value=”134217728” />

New recommended setting depending upon physical memory on server, for example for 4 GBmemory:

    <add key=”DatabaseMaxCacheSize” value=”536870912” />

Save the settings and restart transport service

Refer Article:

Large Message Processing in Exchange, Part 1: Prevention and Planning
http://blogs.technet.com/b/exchange/archive/2009/07/07/3407776.aspx

High level of memory utilization by the EdgeTransport.exe process

Another major cause of Backpressure is the usage of memory by the process edgetransport.exe. The high memory utilization level for this executable is either the 75% of the net physical memory or 1 TB whichever the lesser.

For a medium memory utilization level, we can calculate this as 73 % of net physical memory or the 98% of High memory utilization whichever the lesser.

Normal memory utilization level is prescribed as 71 % of net physical memory or 96% percent of the high memory utilization level, whichever the lesser.

The following keys of the EdgeTransport.exe.config file may be edited so that the default values of high, normal and medium memory utilization may be overridden.

<add key=”PercentagePrivateBytesUsedHighThreshold” value=”0″ />valid range 3 to 100

<add key=”PercentagePrivateBytesUsedMediumThreshold” value=”0″ /> valid range 3 to 100 <high

<add key=”PercentagePrivateBytesUsedNormalThreshold” value=”0″ /> valid range 3 to 100 >medium

Resolving the Issue

1. All third party transport agents, Forefront transport agents, file level antivirus may be disabled. While disabling each of these, check for memory utilization reduction by the EdgeTransport.exe file.
2. Next try to rename the queue directory. If this do the trick, initiate transport service with old queue database and make use of Procdump to obtain the hang dump for EdgeTransport.exe. This can help us figure out the source of the issue in queue database.
   

If the memory utilization by different processes is high you may experience Backpressure. Here a high value refers to 94% of net physical memory.

Under such levels of memory utilization, something called messagedehydration occurs.

Once again, we can change the default values by editing the edgetransport.exe.configfile with the following keys.

<add key=”PercentagePhysicalMemoryUsedLimit” value=”94″ />

<add key=”DehydrateMessagesUnderMemoryPressure” value=”true” />

Resolving the Issue:

Find out the processes having high and abnormal memory usages from the task manager and work on them.

If all the processes are using optimum memory then try adding extra physical memory to the server.

Reference Articles:

Understanding Back Pressure –
http://technet.microsoft.com/en-us/library/bb201658%28v=EXCHG.80%29.aspx

Large Message Processing in Exchange – http://blogs.technet.com/b/exchange/archive/2009/07/07/3407776.aspx

Updated Process Tracking Log (PTL) tool – http://blogs.technet.com/b/exchange/archive/2011/10/21/updated-process-tracking-log-ptl-tool-for-use-with-exchange-2007-and-exchange-2010.aspx

Ratish Nair

Microsoft MVP | Exchange Server

Team @MSExchangeGuru

Posted July 29th, 2013 under Exchange 2007, Exchange 2010, Exchange 2013, Exchange Tools. RSS 2.0 feed. Leave a response, or trackback.