MSExchangeGuru.com

Learn Exchange the Guru way !!!

 

Troubleshooting Exchange ActiveSync and reading IIS logs

In this article I will try cover my way of troubleshooting Exchange ActiveSync issues from Server side and client side.

Recently I had an issue where users on one of the Exchange 2007 servers “ONLY” weren’t able to sync their mobile device. My way of troubleshooting flows this way which I will explain in detail:

  1. Isolate the issue be for user, device, server or organization wide
  2. Testexchangeconnectivity.com and a test mobile device
  3. Read IIS logs and look at the error’s if any
  4. Confirm Authentication settings on IIS VDir’s for CAS and MBX roles
  5. Look at CPU utilization for w3wp.exe on all CAS servers
  6. Read event logs and filter MSExchange ActiveSync event in 10xx series
  7. Confirm ADPermission for internet facing CAS servers on “ms-Exch-EPI-Token-Serialization”

Isolate the issue be for user, device, server or organization wide

This is the primary focus area or we will be going in circles. Let’s split this into 4 different scenario and the issue should be isolated to one of these:

  1. Issue with one single user only: First thing I would look for is to check the OMA AD property enabled or not. Navigate to Active Directory Users and Computers à Search for the user à Exchange Feature tab à


Ideally, if this all set to enable, the attribute “msexchOMAAdminWirelessEnable“will be <not Set> in ADSIEdit. If msexchOMAAdminWirelessEnable is set to a number, it will cause EAS issues.

msExchOmaAdminWirelessEnable = 1 = 001 = Option 1 and option 2 enabled.

msExchOmaAdminWirelessEnable = 2 = 010 = Option 2 and option 3 enabled (option 3 requires 2).

msExchOmaAdminWirelessEnable = 3 = 011 = Option 2 enabled.

msExchOmaAdminWirelessEnable = 4 =  Option 1 and 3 enabled and Option 2 disabled

msExchOmaAdminWirelessEnable = 5 = 101 = Option 1 enabled and Option 2,3 disabled

msExchOmaAdminWirelessEnable = 7 = 111 = All options disabled.

Let’s confirm that it is indeed set that way. Navigate to the user location in ADSIEdit and open his properties:


Have the user flip Wifi and 3G network to ensure the issue is not caused because of internet connection drop on the device or in other words, before proceeding have the user access a couple of websites on the mobile browser.

Once we ensure all is well with the user, navigate to ExRCA.com and run a test for ActiveSync and give the user credentials and click Next. If this shows a pass, I would put the blame on the device and move forward.

Note: ExRCA doesn’t work all the time, so have a test mobile device handy and the website is solely owned by Microsoft.

  1. Issue with a single device only: We get this a lot. Hard part is to convince the user that its his device broke and not the server. To prove this, run ExRCA.com and explain to the user that performing a test on this website it like configuring a mobile device. Additionally, configure another mobile device with the user credentials and confirm that you can sync the device.

    Issue in a device only scenario is mostly caused because of an outdated firmware or some applications conflicting with the default EMAIL app. Have the user update the firmware on the device and if needed backup Contacts, Photos, Notes etc and reset the device to factory settings.

  2. Issue with a single server: It is important for us to understand if the issue is happening for all users or if it is isolated for users on a single mailbox server or even a mailbox database.

    Test-ActiveSyncConnectivity cmdlet can come handy here:

This example tests the Exchange ActiveSync connectivity for the mailbox PaulS on the Client Access server computer CAS01.

Test-ActiveSyncConnectivity -ClientAccessServer contoso\CAS01 -URL “http://contoso.com/mail” -MailboxCredential (get-credential PaulS)

EXAMPLE 2: This example tests the Exchange ActiveSync connectivity for the mailbox PaulS using the Autodiscover URL.

Test-ActiveSyncConnectivity -UseAutodiscoverForClientAccessServer $true -URL “http://contoso.com/mail” -MailboxCredential (get-credential PaulS@contoso.com)

EXAMPLE 3: This example tests the Exchange ActiveSync connectivity for the mailbox PaulS.

Test-ActiveSyncConnectivity -AllowUnsecureAccess $true -URL “http://contoso.com/mail” -MailboxCredential (get-credential contoso\pauls)

Well, the truth is as and when users raise issue’s you will be able to diagnose if the issue is server side or affecting all users in the organization. If you think the issue is happening for users on one specific database, look for event 9667 – Database props quota error with AirSync named property in the description can be the culprit.

Read more about the same here:

Event id: 9667 – When database reaches maximum limit of named properties or replica identifiers: https://msexchangeguru.com/2009/09/04/event-id-9667/

  1. Issue where entire organization is down with no access to EAS: You would know if this happened. TestExchangeConnectivity.com/ExRCA.com won’t work and no users will have access to their email.

    In this case, check if you can access OWA if EAS and OWA servers are the same. If OWA works and EAS do not, the issue is not caused because of the internet facing CAS servers but specific to EAS protocol.

ExRCA.com/Testexchangeconnectivity.com and a test mobile device

A lot of people are not aware of ExRCA.com. This is a website from Microsoft where you can test external access for the following options:


Your troubleshooting should start with ExRCA.com and put the username and password for a test user. Again, don’t rely completely on ExRCA.com and have a test mobile device handy during times of troubleshooting. Post testing, it will return the HTTP status code for the test.

200                   – Authentication pass
400                    – Bad/invalid request
401 and 403       – Unauthorized/server refusing request
404                    – File not found
449                    – Retry
500                   – Server error
503                   – Service unavailable

If you are presented with a 500, 501 or 503 status code, we know it’s something to do with the server side and 400 series usually shows file not found un-authorized etc.

5 series or 4 series status codes means server error and authentication issues respectively.

How do we find out HTTP status codes for each and every connection made to the internet facing CAS servers – IIS Logs is where we will start to achieve this…

Read IIS logs and look at the errors if any

This is a must know-how for all Exchange pro’s. For every connection made by individual user or devices, there will be a respective IIS log generated on the internet facing CAS server. But how do we read it?

The default location of IIS log files:

In W2K3–> C:\WINDOWS\system32\LogFiles

In W2K8–> C:\inetpub\logs\LogFiles\W3SVC1

Try to sync your mobile device and make a note of the time. Let it error out. We need to perform the next step in all Internet facing CAS servers at the same time.

Open the latest IIS log file in the directory – in Notepad and change to Format à Wordwrap. Scroll to the extreme bottom. Do a Ctrl + F and type your username. In this case, it is RatishNair. Now I see the latest entry made by my mobile device:

2012-01-10 14:42:26 172.32.22.12 POST /Microsoft-Server-ActiveSync/default.eas User=ratishnair&DeviceId=Appl8xxxxx4S&DeviceType=iPhone&Cmd=FolderSync&Log=PrxFrom:10.123.33.88_Error:BackingOffMailboxServer_ 443 CONTOSO\CAS01$ 10.123.33.88Apple-iPhone3C1/901.405 503 0 0 765

Timezone showed in IIS Log is GMT

This log means that on the date shown, a POST command was issued to the server with ip 172.32.22.12 (non internet facing CAS Server) to the VDir /Microsoft-Server-ActiveSync/ for the user ratishnair with deviceID “Appl8xxxxx4S” from a DeviceType iPhone for FolderSync from another CAS server (internet facing) with ip 10.123.33.88 on port 443 from Machine CAS1 (10.123.33.88) and it was errored saying “Error:BackingOffMailboxServer” and the firmware version of the iPhone is iPhone4C1/901.405 which converts to iPhone4S running iOS 5.0.1 and the HTTP status code is 503 0 0 765

So here, the http status code is shown in 5xx series which means a server error. If it’s in the 4xx series, it could be an authentication issue and I would start by comparing authentication settings on the CAS as well as Mailbox servers.

Remember the chaos created by iOS4.0? So no matter what if your organization contains devices running iOS 4.0, have them “mandatory” update their Operating system to the latest version.

Official Microsoft and Apple documentation can be found here:

Unable to connect using Exchange ActiveSync due to Exchange resource consumption: http://support.microsoft.com/kb/2469722

iOS 4.0: Exchange Mail, Contacts, or Calendars may not sync after update: http://support.apple.com/kb/TS3398

So while troubleshooting EAS issues, you need to know the DeviceUserAgent for devices on iOS to know what version they are running. This is all I could collect.

 

iOS Ver. Device Type DeviceUserAgent
3 iPhone Apple-iPhone/701.341
3.1 iPhone Apple-iPhone/703.144
3.2 iPad Apple-iPad/702.367
3.0.1 iPhone Apple-iPhone/701.400
3.1.2 iPhone Apple-iPhone/704.11
3.1.3 iPhone Apple-iPhone/705.18
4 iPod Apple-iPod2C1/801.293
4 iPod Apple-iPod3C1/801.293
4 iPhone 3G Apple-iPhone1C2/801.293
4 iPhone 3GS Apple-iPhone2C1/801.293
4 iPhone 4 Apple-iPhone3C1/801.293
4.0.1 iPod Apple-iPod2C1/801.306
4.0.1 iPod Apple-iPod3C1/801.306
4.0.1 iPhone 3G Apple-iPhone1C2/801.306
4.0.1 iPhone 3GS Apple-iPhone2C1/801.306
4.0.1 iPhone 4 Apple-iPhone3C1/801.306
4.1 iPhone Apple-iPhone/508.11
5 iPhone 3GS Apple-iPhone2C1/901.334
5 iPhone4 Apple-iPhone3C1/901.334
5 iPhone 4S Apple-iPhone4C1/901.334
5 iPad Apple-iPad1C1/901.334
5 iPad 2 WiFi Apple-iPad2C1/901.334
5 iPad 2 GSM Apple-iPad2C2/901.334
5 iPad 2 CDMA Apple-iPad2C3/901.334
5.0.1 iPhone 3GS Apple-iPhone2C1/901.405
5.0.1 iPhone 4 GSM Apple-iPhone3C1/901.405
5.0.1 iPhone 4 CDMA Apple-iPhone3C3/901.405
5.0.1 iPhone 4S Apple-iPhone4C1/901.405
5.0.1 iPhone 4S Apple-iPhone4C1/901.406
5.0.1 iPad Apple-iPad1C1/901.334
5.0.1 iPad 2 WiFi Apple-iPad2C1/901.334
5.0.1 iPad 2 GSM Apple-iPad2C2/901.334
5.0.1 iPad 2 CDMA Apple-iPad2C3/901.334
5.1 iPhone 3GS Apple-iPhone2C1/902.176
5.1 iPhone 4 GSM Apple-iPhone3C1/902.176
5.1 iPhone 4 CDMA Apple-iPhone3C3/902.176
5.1 iPhone 4S Apple-iPhone4C1/902.179
5.1 iPad Apple-iPad1C1/902.176
5.1 iPad 2 WiFi Apple-iPad2C1/902.176
5.1 iPad 2 GSM Apple-iPad2C2/902.176
5.1 iPad 2 CDMA Apple-iPad2C3/902.176
5.1 iPad (3rd generation) WiFi Apple-iPad3C1/902.176
5.1 iPad (3rd generation) Verizon Apple-iPad3C2/902.176
5.1 iPad (3rd generation) AT&T / International Apple-iPad3C3/902.176
5.1.1 iPhone 3GS Apple-iPhone2C1/902.206
5.1.1 iPhone 4 GSM Apple-iPhone3C1/902.206
5.1.1 iPhone 4 CDMA Apple-iPhone3C3/902.206
5.1.1 iPhone 4S Apple-iPhone4C1/902.206
5.1.1 iPad Apple-iPad1C1/902.206
5.1.1 iPad 2 WiFi Apple-iPad2C1/902.206
5.1.1 iPad 2 GSM Apple-iPad2C2/902.206
5.1.1 iPad 2 CDMA Apple-iPad2C3/902.206
5.1.1 iPad (3rd generation) WiFi Apple-iPad3C1/902.206
5.1.1 iPad (3rd generation) Verizon Apple-iPad3C2/902.206
5.1.1 iPad (3rd generation) AT&T / International Apple-iPad3C3/902.206
6 iPhone 4 GSM Apple-iPhone3C1/1001.403
6 iPhone 4 CDMA Apple-iPhone3C3/1001.403
6 iPhone 4S Apple-iPhone4C1/1001.403
6 iPhone 5 GSM Apple-iPhone5C1/1001.405
6 iPhone 5 CDMA Apple-iPhone5C2/1001.405

If you want know details of Exchange ActiveSync devices syncing in your environment, use this cmdlet:

Run: $AdminSessionADSettings.ViewEntireForest = $true
or it won’t find users in other domains.

Get-Childitem C:\WINDOWS\system32\LogFiles\W3SVC1 | where-object {$_.lastwritetime -gt $DateToCompare} | ForEach { Export-ActiveSyncLog -FileName $_.FullName -OutputPath “C:\ActiveSync_Reporting” -OutputPrefix $_.Name.Replace(“.log”,”_”) -UseGMT:$true}

Get-CASMailbox -ResultSize unlimited -filter {HasActivesyncDevicePartnership -eq $True} | % {Get-ActiveSyncDeviceStatistics -Mailbox $_.identity} | ft Identity, DeviceType, DeviceID, DeviceUserAgent, LastSuccessSync >>”C:\ActiveSync_Reporting\EAS.txt”

Get ActiveSync device details from a list:

gc .\Users.txt | Get-CASMailbox | % {Get-ActiveSyncDeviceStatistics -Mailbox $_.identity} | ft Identity, DeviceType, DeviceID, DeviceUserAgent, LastSuccessSync >>./oo.txt

I have a separate article here:

Exchange 2007 ActiveSync reporting:
https://msexchangeguru.com/2010/05/20/e2k7-activesync-reporting/

Confirm Authentication settings on IIS VDir’s for CAS and MBX roles and recreate IIS Virtual directories

Client Access Role = IIS

CAS servers run completely on IIS service and it is all about virtual directories within. Any OWA, OA or EAS troubleshooting should start with ensuring authentication settings to be properly set on VDir’s in IIS. I would start with the article:

AutoDiscover Troubleshooting- Default authentication for Exchange VDir’s aka Virtual directories on CAS and Mailbox role:
https://msexchangeguru.com/2010/10/05/autodiscover/

Also, you don’t want to make the mistake of not checking if the Application pool for EAS is running or not:

  1. Click Start, click Administrative Tools, and then click Internet Information Services (IIS) Manager.
  2. Double-click to expand the server name, and then double-click to expand the Application Pools folder.
  3. Right-click MSExchangeSyncAppPool, and then see if it is Started or not

If the MSExchangeSyncAppPool is stopped, Exchange ActiveSync will be disabled for that server.

One of the main troubleshooting performed by PSS is to recreate the IIS virtual directories:

Get-ActiveSyncVirtualDirectory 
Remove-ActiveSyncVirtualDirectory -Identity "Contoso.com\Microsoft-Server-ActiveSync" 
New-ActiveSyncVirtualDirectory -WebSiteName "Contoso.com" 

In this article we have focused on the first 4 troubleshooting to be performed for EAS issues. To be continued…

Ratish Nair
MVP Exchange
Team@ MSExchangeGuru

Keywords: Exchange ActiveSync, troubleshooting Exchange ActiveSync, troubleshooting ActiveSync, IIS logs, Exchange 2007, Exchange 2010, ActiveSync not working, Exchange 2010, Http error for ActiveSync, Find all Exchange ActiveSync users.

18 Responses to “Troubleshooting Exchange ActiveSync and reading IIS logs”

  1. Shyam Madeti Says:

    Thanks heaps Ratish…

  2. Anita Says:

    Incredible write-up! The flow and explanation was amazing…

  3. Milind Naphade Says:

    Very well explained. Great write up Ratish.

  4. Exchange 2007 ActiveSync reporting « MSExchangeGuru.com Says:

    […] Troubleshooting Exchange ActiveSync and reading IIS logs « MSExchangeGuru.com Says: February 1st, 2012 at 9:29 am […]

  5. ActiveSync woes–“Cannot get mail” and the case of the endless re-sync | User Error Says:

    […] MSExchangeGuru.com: Troubleshooting Exchange ActiveSync and reading IIS logs […]

  6. Chris_P Says:

    Ratish or anyone else….

    Can you rewrite this against Exchange 2010? It seems to me that items have changed and I’m finding a lot of information stating that some of the new commands are failing.

    Here is my need, what end I’m working toward:
    The ActiveSync User and Device Lifecycle Reporting
    1. User added to the server, maybe which users were added to the system in the last week (7-days) with their associated device info.
    2. All Current Users Report (Username, Device Type, Device Info, LastSync Date, Policy Date, Policy Compliant?)
    3. Remote Wipe command sent
    • Who sent the Remote Wipe, when, for which user, which device?
    • Did the Device recieve it?
    • Was the Device wiped?
    4. Which were the Devices removed from the system in the last 7-days, and who removed them?

  7. Scott Mickelson Says:

    I am seeing the following error in the IIS Logs. Any tips, as my searches have found nothing on this error…
    Error:StoragePermanentMapiExceptionNotFound
    Thanks,
    -Scott

  8. PH D Says:

    Hello,

    We have been experiencing some strange behavior for a while now on our Exchange-Activesync infrastructure.
    we have several domains with the following names (i put some “x” for privacy):

    regions-xxx.org
    regions-toto.xxx.org
    siege.xxx.org
    siege-toto.xxx.org

    For some unknown reasons users on the “regions-xxx.org” domain have random blackout connections, where they can’t connect at all even though all other users on the other domains can. They all get the error authentication failed.
    But the domain is fine everyone can connect to there computers, or outlook, … It’s only the phones activesync that stops working.
    No logs are generated on the iis logs when the phones can’t connect, we only get the last successful ones.

    The blackout is totally random last time it lasted a week, then it started working again, and it stopped working yesterday again.

    Any ideas ?

  9. mike Says:

    We have created a service the is dedicated for monitoring the health of the ActiveSync service as well as for diagnosing ActiveSync related problems.
    You are welcome to give it a try.

  10. phunktional johnkey Says:

    Thanks mate! Been struggling to implement certificate based authentication for ActiveSync and this was very useful for troubleshooting especially the IIS Logs info.

    Cheers

  11. ActiveSync woes–“Cannot get mail” and the case of the endless re-sync | User Error Says:

    […] MSExchangeGuru.com: Troubleshooting Exchange ActiveSync and reading IIS logs […]

  12. Rosario Carcò Says:

    Does anybody know what (A)Conn%3a0%2cHangingConn means? I find it in the IIS-Logs of our Exchange CAS&HUB servers for iPhone, Android, etc. As we are testing a new load-balancer I do not know if this means the connection with the client is hanging or correctly established?

    Thanks a lot, Rosario

  13. Pushkar Says:

    Want some help with one issue: In our exchange 2010 environment intermittently User A’s email can be seen on User’s B device. Will provide more details as I am trying to capture logs real time.

    Please update if you have any inputs.

  14. Prabhat Nigam Says:

    This is not possible unless they have delegation configured.

  15. Pushkar Says:

    Hi Prabhat

    Thanks for the update, but I have checked for the mailbox delegation. No delegation rights have been given. Any other possibility which can cause this?

  16. How can I trace the source of repeated account lockout against Exchange server? – segmentfault Says:

    […] Log entry quoted from https://msexchangeguru.com/2012/02/01/exchange-activesync/ […]

  17. Sudi Krish Says:

    Hi Guys,

    Unable to sync a folder on windows 8.1, have 2 folders, one folder doesn’t sync, removed device partnership – no go

  18. Prabhat Nigam Says:

    Check if inheritance blocked for this user.

Leave a Reply

Categories

Archives

MSExchangeGuru.com