Exchange 2010 to 2016: Mailbox Move – Undocumented Migration Error
Exchange 2016 is new but migrations are similar so you expect that you will face more or less the same errors or issues which you might have seen in the past.
But this time I got this issue which has no past history and I could not find a solution on the web. Even my MVP distribution list had no idea. Even nothing clearly documents in internal Microsoft solutions system.
We have to run the EXTRA trace with the help of Microsoft Escalation Engineering team. Thank you to the Microsoft Exchange Escalation Engineering team for this help.
Let me explain the issue and how did we fix it.
One of my customer which runs a private cloud and my company Golden Five manage it, raise a request for Exchange 2010 to 2016 migration. We did all good without any issue but 10 users in the whole migration gave us hard time.
-We got the following error.
Unable to open message store. (hr=0x80004005, ec=1292)
Couldn’t switch the mailbox into Sync Source mode.
This could be because of one of the following reasons:
Another administrator is currently moving the mailbox.
The mailbox is locked.
The Microsoft Exchange Mailbox Replication service (MRS) doesn’t have the correct permissions.
Network errors are preventing MRS from cleanly closing its session with the Mailbox server. If this is the case, MRS may continue to encounter this error for up to 2 hours – this duration is controlled by the TCP KeepAlive settings on the Mailbox server.
Wait for the mailbox to be released before attempting to move this mailbox again.
+ CategoryInfo : NotSpecified: (:) [New-MoveRequest], RemoteTransientException
+ FullyQualifiedErrorId : [Server=EXCHANGE,RequestId=eb97598c-7659-4311-a116-2d87cdbc6735,TimeStamp=8/16/2016 4:3 0:14 PM] [FailureCategory=Cmdlet-RemoteTransientException] 2E25D178,Microsoft.Exchange.Management.Migration.MailboxReplication.MoveRequest.NewMoveRequest
-This is the error from exchange management log in the event viewer:
Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: Couldn’t switch the mailbox into Sync Source mode. This could be because of one of the following reasons: Another administrator is currently moving the mailbox. The mailbox is locked. The Microsoft Exchange Mailbox Replication service (MRS) doesn’t have the correct permissions. Network errors are preventing MRS from cleanly closing its session with the Mailbox server. If this is the case, MRS may continue to encounter this error for up to 2 hours – this duration is controlled by the TCP KeepAlive settings on the Mailbox server. Wait for the mailbox to be released before attempting to move this mailbox again. —> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: MapiExceptionMailboxInTransit: Unable to open message store. (hr=0x80004005, ec=1292)
Diagnostic context: Lid: 55847 EMSMDBPOOL.EcPoolSessionDoRpc called [length=132] Lid: 43559 EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=440][latency=5] Lid: 52176 ClientVersion: 15.1.396.33 Lid: 50032 ServerVersion: 14.3.279.2 Lid: 35180 Lid: 23226 — ROP Parse Start — Lid: 27962 ROP: ropLogon [254] Lid: 17082 ROP Error: 0x50C Lid: 26937 Lid: 21921 StoreEc: 0x50C
Lid: 27962 ROP: ropExtendedError [250] Lid: 1494 —- Remote Context Beg —- Lid: 1238 Remote Context Overflow Lid: 63331 Lid: 2439 StoreEc: 0xFFFFF9BF Lid: 3591 StoreEc: 0xFFFFF9BF Lid: 2567 StoreEc: 0xFFFFF9BF Lid: 63331 Lid: 2439 StoreEc: 0xFFFFF9BF Lid: 10804 StoreEc: 0xFFFFF9BF Lid: 54007 Lid: 58103 StoreEc: 0xFFFFF9BF Lid: 44343 Lid: 60727 StoreEc: 0xFFFFF9BF Lid: 60525 Lid: 2523 StoreEc: 0x50C Lid: 55095 Lid: 42807 StoreEc: 0x50C Lid: 51813 Lid: 4071 StoreEc: 0x50C Lid: 17597 Error: 0x0 Lid: 22787 Error: 0x0 Lid: 22787 Error: 0x0 Lid: 22957 Error: 0x0 Lid: 19693 Error: 0x0 Lid: 17917 Error: 0x0 Lid: 27341 Error: 0x0 Lid: 22787 Error: 0x0 Lid: 35383 Lid: 2199 StoreEc: 0x50C Lid: 56415 Lid: 48223 StoreEc: 0x50C Lid: 17097 StoreEc: 0x50C Lid: 8620 StoreEc: 0x50C Lid: 1750 —- Remote Context End —- Lid: 26849 Lid: 21817 ROP Failure: 0x50C Lid: 26297 Lid: 16585 StoreEc: 0x50C Lid: 32441 Lid: 1706 StoreEc: 0x50C Lid: 24761 Lid: 20665 StoreEc: 0x50C Lid: 25785 Lid: 29881 StoreEc: 0x50C — End of inner exception stack trace — at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.<>c__DisplayClass1.<ReconstructAndThrow>b__0() at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation) at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.ReconstructAndThrow(String serverName, VersionInformation serverVersion) at Microsoft.Exchange.MailboxReplicationService.WcfClientWithFaultHandling`2.<>c__DisplayClass1.<CallService>b__0() at Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action serviceCall, String context) at
-But there was none of the three causing this because I was able to move the mailbox from Exchange 2010 to another database in Exchange 2010.
-Microsoft still confirmed that the error code 0x50c is ecMailboxInTransit, which tells you that the move flag on the mailbox was set.
-We decided to run the EXTRA – Microsoft Exchange Troubleshoot Assistant trace on Ex2016 while running the command on Ex2016 server.
-Microsoft provided the EXTRA trace tool and also analyzed the trace.
-Install EXTRA and Run EXTRA
-Click on select a task
-Click on Trace Control and click ok on the popup
-Give the path of the ETL file, Name of the ETL file, size, select “Create a new file when max trace file size is reached”, then update how long you would like to run the trace, then click “Set manual trace tags”
-Select all on top
-Then select all tags for the following Components to trace
MailboxReplicationService
ADProvider
-Leave rest as it is and click on Start tracing
-Now run the migration command and wait for the trace to end.
In EMS run the below mentioned command
New-moverequest id -targetdatabase dbname -archivetargetdatabase archiveDBname
-Once Trace end, send the ETL file to Microsoft.
-Microsoft reviewed the trace and shared the following for every faulty mailbox
This is the mailbox guid we are trying to open and fail on Guid which I see is the archive mailbox msExchArchiveGUID: Guid
-This clarified why I was not getting the error while moving my mailbox from Exchange 2010 to 2010.
-Exchange will move primary and archive mailbox to the same exchange version so even if we didn’t specify the Exchange 2016 archive database, I still got the error because Exchange 2016 was trying to migrate the archive mailbox as well. In Exchange 2010 it was not migrating archive mailbox so there was no error. Fair enough.
-We tried to repair the primary mailbox and got this error.
-We tried to repair the archive mailbox and go this error in the application log.
New-MailboxRepairRequest mailboxemail -CorruptionType ProvisionedFolder,SearchFolder,AggregateCounts,Folderview -Archive
Online integrity check for request 9af648a2-f2b8-4737-a5e6-cd9ae9ec6011 failed with error -1601.
-Now I tried to check the size of the archive mailbox and got the following warning. J
get-mailbox | get-mailboxstatistics -archive -erroraction silentlycontinue | ft displayname,totalitemsize
WARNING: The user hasn’t logged on to mailbox ‘DN’
(‘5715db43-4916-4dfb-a6c6-d55b4b38b9a3’), so there is no data to return. After the user logs on, this warning will no longer appear.
-Verified with the manager and got the update that uses never used the archive mailboxes.
-So deleted archive mailboxes by running the below mentioned command
Disable-Mailbox <Mailbox ID> -Archive
-Now we were able to move the mailboxes.
-After the mailbox move completed, we created new archive mailboxes for those users in Exchange 2016 Archive Database.
This was a discovery of a fix in the history of 20 years of Exchange.
Microsoft MVP | CTO @ Golden Five
Team@MSExchangeGuru
September 16th, 2016 at 2:10 pm
awesome 🙂
September 17th, 2016 at 1:52 am
Great, thanks for sharing.
December 5th, 2016 at 9:44 pm
Thanks, had this same issue on 2016/2010 hybrid environment during migration and had been banging my head against the wall not being able to move 1 last damn mailbox.