MSExchangeGuru.com

In this article I will try cover my way of troubleshooting Exchange ActiveSync issues from Server side and client side.

Recently I had an issue where users on one of the Exchange 2007 servers “ONLY” weren’t able to sync their mobile device. My way of troubleshooting flows this way which I will explain in detail:

1. Isolate the issue be for user, device, server or organization wide
   
2. Testexchangeconnectivity.com and a test mobile device
   
3. Read IIS logs and look at the error’s if any
   
4. Confirm Authentication settings on IIS VDir’s for CAS and MBX roles
   
5. Look at CPU utilization for w3wp.exe on all CAS servers
   
6. Read event logs and filter MSExchange ActiveSync event in 10xx series
   
7. Confirm ADPermission for internet facing CAS servers on “ms-Exch-EPI-Token-Serialization”
   

Isolate the issue be for user, device, server or organization wide

This is the primary focus area or we will be going in circles. Let’s split this into 4 different scenario and the issue should be isolated to one of these:

1. Issue with one single user only: First thing I would look for is to check the OMA AD property enabled or not. Navigate to Active Directory Users and Computers à Search for the user à Exchange Feature tab à
   

   
   

Ideally, if this all set to enable, the attribute “msexchOMAAdminWirelessEnable“will be <not Set> in ADSIEdit. If msexchOMAAdminWirelessEnable is set to a number, it will cause EAS issues.

msExchOmaAdminWirelessEnable = 1 = 001 = Option 1 and option 2 enabled.

msExchOmaAdminWirelessEnable = 2 = 010 = Option 2 and option 3 enabled (option 3 requires 2).

msExchOmaAdminWirelessEnable = 3 = 011 = Option 2 enabled.

msExchOmaAdminWirelessEnable = 4 =  Option 1 and 3 enabled and Option 2 disabled

msExchOmaAdminWirelessEnable = 5 = 101 = Option 1 enabled and Option 2,3 disabled

msExchOmaAdminWirelessEnable = 7 = 111 = All options disabled.

Let’s confirm that it is indeed set that way. Navigate to the user location in ADSIEdit and open his properties:

Have the user flip Wifi and 3G network to ensure the issue is not caused because of internet connection drop on the device or in other words, before proceeding have the user access a couple of websites on the mobile browser.

Once we ensure all is well with the user, navigate to ExRCA.com and run a test for ActiveSync and give the user credentials and click Next. If this shows a pass, I would put the blame on the device and move forward.

Note: ExRCA doesn’t work all the time, so have a test mobile device handy and the website is solely owned by Microsoft.

1. Issue with a single device only: We get this a lot. Hard part is to convince the user that its his device broke and not the server. To prove this, run ExRCA.com and explain to the user that performing a test on this website it like configuring a mobile device. Additionally, configure another mobile device with the user credentials and confirm that you can sync the device.
   

   Issue in a device only scenario is mostly caused because of an outdated firmware or some applications conflicting with the default EMAIL app. Have the user update the firmware on the device and if needed backup Contacts, Photos, Notes etc and reset the device to factory settings.
   

2. Issue with a single server: It is important for us to understand if the issue is happening for all users or if it is isolated for users on a single mailbox server or even a mailbox database.
   

   Test-ActiveSyncConnectivity cmdlet can come handy here:
   

This example tests the Exchange ActiveSync connectivity for the mailbox PaulS on the Client Access server computer CAS01.

Test-ActiveSyncConnectivity -ClientAccessServer contoso\CAS01 -URL “http://contoso.com/mail” -MailboxCredential (get-credential PaulS)

EXAMPLE 2: This example tests the Exchange ActiveSync connectivity for the mailbox PaulS using the Autodiscover URL.

Test-ActiveSyncConnectivity -UseAutodiscoverForClientAccessServer $true -URL “http://contoso.com/mail” -MailboxCredential (get-credential PaulS@contoso.com)

EXAMPLE 3: This example tests the Exchange ActiveSync connectivity for the mailbox PaulS.

Test-ActiveSyncConnectivity -AllowUnsecureAccess $true -URL “http://contoso.com/mail” -MailboxCredential (get-credential contoso\pauls)

Well, the truth is as and when users raise issue’s you will be able to diagnose if the issue is server side or affecting all users in the organization. If you think the issue is happening for users on one specific database, look for event 9667 – Database props quota error with AirSync named property in the description can be the culprit.

Read more about the same here:

Event id: 9667 – When database reaches maximum limit of named properties or replica identifiers: https://msexchangeguru.com/2009/09/04/event-id-9667/

1. Issue where entire organization is down with no access to EAS: You would know if this happened. TestExchangeConnectivity.com/ExRCA.com won’t work and no users will have access to their email.
   

   In this case, check if you can access OWA if EAS and OWA servers are the same. If OWA works and EAS do not, the issue is not caused because of the internet facing CAS servers but specific to EAS protocol.
   

ExRCA.com/Testexchangeconnectivity.com and a test mobile device

A lot of people are not aware of ExRCA.com. This is a website from Microsoft where you can test external access for the following options:

Your troubleshooting should start with ExRCA.com and put the username and password for a test user. Again, don’t rely completely on ExRCA.com and have a test mobile device handy during times of troubleshooting. Post testing, it will return the HTTP status code for the test.

200                   – Authentication pass
400                    – Bad/invalid request
401 and 403       – Unauthorized/server refusing request
404                    – File not found
449                    – Retry
500                   – Server error
503                   – Service unavailable

If you are presented with a 500, 501 or 503 status code, we know it’s something to do with the server side and 400 series usually shows file not found un-authorized etc.

5 series or 4 series status codes means server error and authentication issues respectively.

How do we find out HTTP status codes for each and every connection made to the internet facing CAS servers – IIS Logs is where we will start to achieve this…

Read IIS logs and look at the errors if any

This is a must know-how for all Exchange pro’s. For every connection made by individual user or devices, there will be a respective IIS log generated on the internet facing CAS server. But how do we read it?

The default location of IIS log files:

In W2K3–> C:\WINDOWS\system32\LogFiles

In W2K8–> C:\inetpub\logs\LogFiles\W3SVC1

Try to sync your mobile device and make a note of the time. Let it error out. We need to perform the next step in all Internet facing CAS servers at the same time.

Open the latest IIS log file in the directory – in Notepad and change to Format à Wordwrap. Scroll to the extreme bottom. Do a Ctrl + F and type your username. In this case, it is RatishNair. Now I see the latest entry made by my mobile device:

2012-01-10 14:42:26 172.32.22.12 POST /Microsoft-Server-ActiveSync/default.eas User=ratishnair&DeviceId=Appl8xxxxx4S&DeviceType=iPhone&Cmd=FolderSync&Log=PrxFrom:10.123.33.88_Error:BackingOffMailboxServer_ 443 CONTOSO\CAS01$ 10.123.33.88Apple-iPhone3C1/901.405 503 0 0 765

Timezone showed in IIS Log is GMT

This log means that on the date shown, a POST command was issued to the server with ip 172.32.22.12 (non internet facing CAS Server) to the VDir /Microsoft-Server-ActiveSync/ for the user ratishnair with deviceID “Appl8xxxxx4S” from a DeviceType iPhone for FolderSync from another CAS server (internet facing) with ip 10.123.33.88 on port 443 from Machine CAS1 (10.123.33.88) and it was errored saying “Error:BackingOffMailboxServer” and the firmware version of the iPhone is iPhone4C1/901.405 which converts to iPhone4S running iOS 5.0.1 and the HTTP status code is 503 0 0 765

So here, the http status code is shown in 5xx series which means a server error. If it’s in the 4xx series, it could be an authentication issue and I would start by comparing authentication settings on the CAS as well as Mailbox servers.

Remember the chaos created by iOS4.0? So no matter what if your organization contains devices running iOS 4.0, have them “mandatory” update their Operating system to the latest version.

Official Microsoft and Apple documentation can be found here:

Unable to connect using Exchange ActiveSync due to Exchange resource consumption: http://support.microsoft.com/kb/2469722

iOS 4.0: Exchange Mail, Contacts, or Calendars may not sync after update: http://support.apple.com/kb/TS3398

So while troubleshooting EAS issues, you need to know the DeviceUserAgent for devices on iOS to know what version they are running. This is all I could collect.

 

iOS Ver.	Device Type	DeviceUserAgent
3	iPhone	Apple-iPhone/701.341
3.1	iPhone	Apple-iPhone/703.144
3.2	iPad	Apple-iPad/702.367
3.0.1	iPhone	Apple-iPhone/701.400
3.1.2	iPhone	Apple-iPhone/704.11
3.1.3	iPhone	Apple-iPhone/705.18
4	iPod	Apple-iPod2C1/801.293
4	iPod	Apple-iPod3C1/801.293
4	iPhone 3G	Apple-iPhone1C2/801.293
4	iPhone 3GS	Apple-iPhone2C1/801.293
4	iPhone 4	Apple-iPhone3C1/801.293
4.0.1	iPod	Apple-iPod2C1/801.306
4.0.1	iPod	Apple-iPod3C1/801.306
4.0.1	iPhone 3G	Apple-iPhone1C2/801.306
4.0.1	iPhone 3GS	Apple-iPhone2C1/801.306
4.0.1	iPhone 4	Apple-iPhone3C1/801.306
4.1	iPhone	Apple-iPhone/508.11
5	iPhone 3GS	Apple-iPhone2C1/901.334
5	iPhone4	Apple-iPhone3C1/901.334
5	iPhone 4S	Apple-iPhone4C1/901.334
5	iPad	Apple-iPad1C1/901.334
5	iPad 2 WiFi	Apple-iPad2C1/901.334
5	iPad 2 GSM	Apple-iPad2C2/901.334
5	iPad 2 CDMA	Apple-iPad2C3/901.334
5.0.1	iPhone 3GS	Apple-iPhone2C1/901.405
5.0.1	iPhone 4 GSM	Apple-iPhone3C1/901.405
5.0.1	iPhone 4 CDMA	Apple-iPhone3C3/901.405
5.0.1	iPhone 4S	Apple-iPhone4C1/901.405
5.0.1	iPhone 4S	Apple-iPhone4C1/901.406
5.0.1	iPad	Apple-iPad1C1/901.334
5.0.1	iPad 2 WiFi	Apple-iPad2C1/901.334
5.0.1	iPad 2 GSM	Apple-iPad2C2/901.334
5.0.1	iPad 2 CDMA	Apple-iPad2C3/901.334
5.1	iPhone 3GS	Apple-iPhone2C1/902.176
5.1	iPhone 4 GSM	Apple-iPhone3C1/902.176
5.1	iPhone 4 CDMA	Apple-iPhone3C3/902.176
5.1	iPhone 4S	Apple-iPhone4C1/902.179
5.1	iPad	Apple-iPad1C1/902.176
5.1	iPad 2 WiFi	Apple-iPad2C1/902.176
5.1	iPad 2 GSM	Apple-iPad2C2/902.176
5.1	iPad 2 CDMA	Apple-iPad2C3/902.176
5.1	iPad (3rd generation) WiFi	Apple-iPad3C1/902.176
5.1	iPad (3rd generation) Verizon	Apple-iPad3C2/902.176
5.1	iPad (3rd generation) AT&T / International	Apple-iPad3C3/902.176
5.1.1	iPhone 3GS	Apple-iPhone2C1/902.206
5.1.1	iPhone 4 GSM	Apple-iPhone3C1/902.206
5.1.1	iPhone 4 CDMA	Apple-iPhone3C3/902.206
5.1.1	iPhone 4S	Apple-iPhone4C1/902.206
5.1.1	iPad	Apple-iPad1C1/902.206
5.1.1	iPad 2 WiFi	Apple-iPad2C1/902.206
5.1.1	iPad 2 GSM	Apple-iPad2C2/902.206
5.1.1	iPad 2 CDMA	Apple-iPad2C3/902.206
5.1.1	iPad (3rd generation) WiFi	Apple-iPad3C1/902.206
5.1.1	iPad (3rd generation) Verizon	Apple-iPad3C2/902.206
5.1.1	iPad (3rd generation) AT&T / International	Apple-iPad3C3/902.206
6	iPhone 4 GSM	Apple-iPhone3C1/1001.403
6	iPhone 4 CDMA	Apple-iPhone3C3/1001.403
6	iPhone 4S	Apple-iPhone4C1/1001.403
6	iPhone 5 GSM	Apple-iPhone5C1/1001.405
6	iPhone 5 CDMA	Apple-iPhone5C2/1001.405

If you want know details of Exchange ActiveSync devices syncing in your environment, use this cmdlet:

Run: $AdminSessionADSettings.ViewEntireForest = $true
or it won’t find users in other domains.

Get-Childitem C:\WINDOWS\system32\LogFiles\W3SVC1 | where-object {$_.lastwritetime -gt $DateToCompare} | ForEach { Export-ActiveSyncLog -FileName $_.FullName -OutputPath “C:\ActiveSync_Reporting” -OutputPrefix $_.Name.Replace(“.log”,”_”) -UseGMT:$true}

Get-CASMailbox -ResultSize unlimited -filter {HasActivesyncDevicePartnership -eq $True} | % {Get-ActiveSyncDeviceStatistics -Mailbox $_.identity} | ft Identity, DeviceType, DeviceID, DeviceUserAgent, LastSuccessSync >>”C:\ActiveSync_Reporting\EAS.txt”

Get ActiveSync device details from a list:

gc .\Users.txt | Get-CASMailbox | % {Get-ActiveSyncDeviceStatistics -Mailbox $_.identity} | ft Identity, DeviceType, DeviceID, DeviceUserAgent, LastSuccessSync >>./oo.txt

I have a separate article here:

Exchange 2007 ActiveSync reporting:
https://msexchangeguru.com/2010/05/20/e2k7-activesync-reporting/

Confirm Authentication settings on IIS VDir’s for CAS and MBX roles and recreate IIS Virtual directories

Client Access Role = IIS

CAS servers run completely on IIS service and it is all about virtual directories within. Any OWA, OA or EAS troubleshooting should start with ensuring authentication settings to be properly set on VDir’s in IIS. I would start with the article:

AutoDiscover Troubleshooting- Default authentication for Exchange VDir’s aka Virtual directories on CAS and Mailbox role:
https://msexchangeguru.com/2010/10/05/autodiscover/

Also, you don’t want to make the mistake of not checking if the Application pool for EAS is running or not:

1. Click Start, click Administrative Tools, and then click Internet Information Services (IIS) Manager.
   
2. Double-click to expand the server name, and then double-click to expand the Application Pools folder.
   
3. Right-click MSExchangeSyncAppPool, and then see if it is Started or not
   

If the MSExchangeSyncAppPool is stopped, Exchange ActiveSync will be disabled for that server.

One of the main troubleshooting performed by PSS is to recreate the IIS virtual directories:

Get-ActiveSyncVirtualDirectory 

Remove-ActiveSyncVirtualDirectory -Identity "Contoso.com\Microsoft-Server-ActiveSync" 

New-ActiveSyncVirtualDirectory -WebSiteName "Contoso.com" 

In this article we have focused on the first 4 troubleshooting to be performed for EAS issues. To be continued…

Ratish Nair
MVP Exchange
Team@ MSExchangeGuru

Keywords: Exchange ActiveSync, troubleshooting Exchange ActiveSync, troubleshooting ActiveSync, IIS logs, Exchange 2007, Exchange 2010, ActiveSync not working, Exchange 2010, Http error for ActiveSync, Find all Exchange ActiveSync users.

Posted February 1st, 2012 under Exchange 2007, Exchange 2010, Exchange ActiveSync. RSS 2.0 feed. Leave a response, or trackback.