MSExchangeGuru.com

Learn Exchange the Guru way !!!

 

Preventing NDRs Storms in Exchange

A problem e-mail administrators face is users setting up wrong forwarding rules in their mailboxes to e-mail addresses that either doesn’t exist (this happens to me a lot!) or to mailboxes that are full and not accepting new e-mails.

Whenever an e-mail is sent to these users, it gets forwarded automatically by the rule in place to that non-existing address which generates a Non-Delivery Receipt (NDR). When this NDR is received, it gets forwarded to the same address again, which generates another NDR, and so on, and so on, going into an infinite loop. This is known as an NDR Storm.

Internally, Microsoft Exchange is able to detect this and automatically stops it. However, with e-mails coming from outside the organization, this is not the case due to multiple reasons…

What is an NDR?

An NDR is a message that a mail server sends to notify the sender when a problem occurs with the delivery of the e-mail. For example if the server resources are unavailable, the recipient’s mailbox is full (and it doesn’t accept more e-mails), the recipient is unknown, etc. Depending on the reason for the NDR, this message looks similar to this:

——————————————-

From: mailer-daemon

Sent: Wednesday, August 3, 2011 11:06:38 PM

To: Nuno Mota

Subject: There was an error sending your mail …

Auto forwarded by a Rule

I’m afraid I had problems forwarding your message. Full details follow:

Subject: ‘just testing a forwarding rule’

Date: ‘Wed, 3 Aug 2011 23:06:38 +0100 (BST)’

1 error(s):

SMTP Server rejected recipient (Error following RCPT command). It responded as follows: [550 unknown user nuno@msexchangeguru.com]

——————————————-

I have also attached the mail’s original headers. Sorry it didn’t work out.

What you can do is create a transport rule to BCC every e-mail sent to outside users and that contains, for example, the text “FW: There was an error sending your mail“, “FW: Mail delivery failed“, “FW: failure notice” in the subject to your Quarantine mailbox. This way, every time you see a huge number of NDRs for the same sender, all you have to do is go to the user’s mailbox and disable the forwarding rule (if that is the case of course).

This works fine, except for the times this happens during the night. You can easily get 2GB of NDRs in just a few hours…

So, another approach is necessary. You can change the transport rule to simply drop the e-mails, but this is not the ideal solution…

What about a script to analyse the transport logs regularly looking for a NDR Storm? If detected, the script can create a Transport Rule?

The following script does just that. It analyses the transport logs every 20 minutes and if it finds a user that forwarded 25 or more NDRs, it creates a transport rule just for that user to block those e-mails from going out and redirects them to the Quarantine mailbox so that the Administrator is aware.

When a rule is created, an e-mail notifying the administrator that a transport rule was created is sent as well. After this, all the administrator needs to do is wait for a couple of hours, login to the user’s mailbox, disable the rule and delete the transport rule. You should leave the rule in place for a couple of hours so that any delayed e-mails get blocked as well, otherwise it would start all over again.

The script by itself searches the logs once, creates the rule(s) if necessary and then terminates. If you want it to run regularly, please update AddMinutes(20) to how far back you want it to search, create a batch file with a code similar to

PowerShell.exe -PSConsoleFile “C:\Program Files\Microsoft\Exchange Server\Bin\ExShell.psc1” -Command “. ‘D:\Scripts\PreventNDRsStorm.ps1′”

And then have a Schedule Task running the batch file every X minutes (this is one of the ways of running PowerShell scripts from the command line or from batch files).

The syntax to create transport rules in Exchange 2007 is completely different from 2010. In 2007, you have to configure all the conditions and actions first (just like in the script below) and then build the transport rule with those conditions. In 2010, it is much easier and similar to any other Exchange command.

New-TransportRule (2010) http://technet.microsoft.com/en-us/library/bb125138.aspx

New-TransportRule (2007) http://technet.microsoft.com/en-us/library/bb125138(EXCHG.80).aspx

So, if you are running a 2010 environment, just replace the whole code to create the transport rule with

New-TransportRule “Prevent NDRs Storm – $strName” -Comments ” Prevent NDRs Storm” -From $strNDR.Name -SentToScope “NotInOrganization” -SubjectContainsWords “FW: There was an error sending your mail”, “FW: Mail delivery failed”, “FW: failure notice” -RedirectMessageTo “quarantine@letsexchange.com” -Enabled $True

You can also replace -RedirectMessageTo with –DeleteMessage If you want to simply drop the e-mail.

Note that when you introduce a 2010 Hub Transport server in a 2007 environment for the first time, all the transport rules get copied to 2010 so that all servers have the same rules. After that, you have to create rules manually in both environments as they don’t get replicated between one another!

So, if you are currently in a co-existence scenario, please take this into consideration and update the script to reflect it.

# Script: PreventNDRsStorm.ps1

# Get the date and time for 20 minutes ago. This will be the starting point to search the transport logs

$strStartFrom= (Get-Date).AddMinutes(20)

# Get all the users who forwarded non-delivered messages to external users in the last 20 minutes. Group them so we can analyse the total number of e-mails sent

$strNDRs= Get-TransportServer | Get-MessageTrackingLog -ResultSize Unlimited -Start $strStartFrom -EventId SEND | ? {($_.MessageSubject -match“FW: There was an error sending your mail”) -or ($_.MessageSubject -match
“FW: Mail delivery failed”) -or ($_.MessageSubject -match“FW: failure notice”) -or ($_.MessageSubject -match“FW: Delivery Status Notification(Failure)”)} | GroupSender

# If we didn’t find any, simply exit

If ($strNDRs-eq$null) { Exit }

# For each sender, check if they sent more than 25 e-mails

ForEach ($strNDRin$strNDRs)

{

# If they sent more than 25 e-mails (in the last 20 minutes) and the user is an internal user, create the transport rule and send an e-mail to the administrator

If (($strNDR.Count -ge 25) -and ($strNDR.Name -match“msexchangeguru”))

{

# Create the Transport Rule

# For every e-mail sent by that user


$condition1= Get-TransportRulePredicate From
$condition1.Addresses = @(Get-Mailbox $strNDR.Name)

# only when the e-mail is going Outside the organization


$condition2= Get-TransportRulePredicate SentToScope

$condition2.Scope = @(“NotInOrganization”)

# and only when the subject contains any of these phrases
$condition3= Get-TransportRulePredicate SubjectContains
$condition3.Words = @(“FW: There was an error sending your mail”, “FW: Mail delivery failed”, “FW: failure notice”)

# Redirect the FW e-mail to the Quarantine NDRs mailbox


$action= Get-TransportRuleAction RedirectMessage
$action.Addresses = @(Get-Mailbox Quarantine)

# Get the user’s alias from the e-mail address to create the transport rule with it
$strName= [regex]::split($strNDR.Name, “@”)[0]

# Create the Transport Rule itself
New-TransportRule -Name “Prevent NDRs Storm – $strName” -Comments “Prevent NDRs Storm” -Conditions @($condition1, $condition2, $condition3) -Actions @($action) -Enabled $True -Priority 0

# Send the E-mail to the administrator

$body=“”

$body+=“`n**********************************”

$body+=“`n* *”

$body+=“`n* WARNING: NDR Storm Prevented *”

$body+=“`n* *”

$body+=“`n**********************************”

$body+=“`n`nTransport Rule Created for “, $strNDR.Name

$body+=“`n`nFW e-mails: “, $strNDR.Count

$FromAddress=“ndr.storm@letsexchange.com”

$ToAddress=“nuno.mota@letsexchange.com, exchange.admin@letsexchange.com”

$MessageSubject=“NDRs Storm!”

$SendingServer=“HTCAS1”

$SMTPMessage=New-ObjectSystem.Net.Mail.MailMessage
$FromAddress, $ToAddress, $MessageSubject, $body

$SMTPClient=New-ObjectSystem.Net.Mail.SMTPClient

$SMTPClient.Send($SMTPMessage)

}

}

Nuno Mota

Team @MSExchangeGuru

17 Responses to “Preventing NDRs Storms in Exchange”

  1. Avinash Says:

    This is just great……Too usefull.

  2. Nuno Mota Says:

    Thank you Avinash! Hope it proves useful to you as well 🙂
    Just don’t forget to double-check the code as some of the formatting got lost in the post (extra new lines or spaces deleted for example).

  3. Ron Says:

    Absolutely brilliant post. I had this concept in mind for a long time but never knew how to achieve it. I think MS Should look at making this a kb. Thank you nuno.

  4. Rob Campbell Says:

    Interesting.

    I worked on a solution for detecting and shutting down general mail loops (not just NDR) by reading the message tracking logs, and building a hash table of sender/recipient addresses, and incrementing it for each email with the same sender and recipient.

    Any that exceeded a threshold were deemed to be a probable loop, and it was throttled by dropping the Exchange mailbox prohibitsendquota, and sending a notification email to the administrator and the Exchange mailbox. The mailbox could still receive new mail, but couldn’t reply, and that broke the loop.

  5. Nuno Mota Says:

    Thank you Ron! Hope it will help you.
    It can be much improved but I think it is a starting point. Let me know if you need any help with it.

  6. Nuno Mota Says:

    Hi Rob,
    That seems like an even better script! This one can be much improved, but I think it is a starting point. I didn’t improve it as it met our requirements at the time.
    I was also thinking of one that would look into senders (over a period of time) that send multiple e-mails to non-existing e-mail addresses (using a similar method as my script and your script). I think this would help prevent spam as well.

  7. Ritin Says:

    Very informative.

  8. Andy Says:

    Is there any method to do this in Exchange 2003?

  9. Nuno Mota Says:

    Hi Andy,
    There isn’t as far as I know… You could possibly develop your own routing agent but it will never be as easy as with Exchange 2007/2010…

  10. Oliver Weber Says:

    Thank you for the article…

    We are relatively new to Exchange2010 (have just migrated), so we faced our first NDR storm today…strange thing is that it happened internally.

    A user had set up a rule to forward all mail with a certain subject to a non-existing internal address…and the NDR did maintain that subject…and Exchange did not notice that, so we ended up with 38.000 NDRs in the users mailbox..

    Shouldn’t Exchange have detected and stopped that storm?

    Oliver

  11. Nuno Mota Says:

    Hi Oliver,

    Yes, Exchange should have stopped that! I have just tested your scenario and only one e-mail was sent – Exchange didn’t go into a loop. So not sure what happened in your case…
    Anything useful on the Message Tracking Logs? Did you have the script running?
    Note that Exchange will also prevent storms caused by AutoReply rules, but only between an internal recipient and sender.

    Regards,
    Nuno

  12. Oliver Weber Says:

    Hi Nuno,

    I don’t see anything unusual in the Tracking logs….each message create one “mailboxrule receive” entry, and then an “routing fail”…

    What happens when you test this scenario? Is there any indication where/when the loop is detected ans stopped?

    Here is what I did:
    set up a inbox rule in mailbox A, stating: “forward any mail that contains FOO in the subject to address@domain.com“. For this scenario, address@domain.com does not exists.

    When I then send a mail with subject FOO from mailbox b to mailbox a, the loop starts.

    a will forward the original message to address@domain.com, and will get back a NDR with subject “Undeliverable: FW: FOO”, which will trigger the rule again…

    Oliver

  13. Jamie Tsottles Says:

    I found this logic very useful. However when testing the “FW: Delivery Status Notification(Failure)” case, I was not getting any results. I am terrible with regular expression so it took me a while to figure this out. I wanted to post for anyone that is having the same trouble and maybe for those who did not test each string…
    You cannot use (at least in my version of powershell) parentheses within a string without escaping the (). In my case, the match looks like this:

    ($_.MessageSubject -match “FW: Delivery Status Notification\(Failure\)”)

  14. Chris Says:

    This article described my issue to a ‘T’. I’m having trouble running the script though. Powershell returns:
    Missing ‘in’ after variable in foreach loop.
    At C:\scripts\PreventNDRsStorm.ps1:16 char:19
    + ForEach ($strNDRin <<<< $strNDRs)
    + CategoryInfo : ParserError: (InToken:TokenId [], ParseException
    + FullyQualifiedErrorId : MissingInInForeach

    I've double checked line 16 of my script with what is posted about and I have the 'in' statement?

  15. Nuno Mota Says:

    Hi Chris,

    Can you please ensure you have spaces in between words/code? For example, “ForEach ($strNDRin$strNDRs)” is wrong as it should be “ForEach ($strNDR in $strNDRs)”
    Please let me know if this was the issue.

    Regards,
    Nuno

  16. Alex Says:

    This is brilliant, fantastic work. I have an issue running the script also.. I am getting the error “‘Get-TransportServer’ is not recognized as the name of a cmdlet, function, script file, or operable program” on compile.

    Any ideas?

  17. Nuno Mota Says:

    Hi Alex,

    Are you using the Exchange Management Shell to run the script or a “normal” Windows PowerShell console?
    What Exchange permissions does the account you are running the script under has?

    Regards,
    Nuno

Leave a Reply

ad

Categories

Archives