MSExchangeGuru.com

Learn Exchange the Guru way !!!

 
«
»

Exchange 2013: Recover a DAG and DAG Node from crash Part 3

In this blog we will focus on bringing DAG node back in the DAG.

 

Check out Part 1 here: https://msexchangeguru.com/2014/02/05/e2013-dr-dag-post-crash1/

Check out Part 2 here: https://msexchangeguru.com/2014/02/05/e2013-dr-dag-post-crash2/

 

Issue 3:

Dag node 2 has crashed and unable to boot the OS. The only option left to do the installation of the OS, recover exchange, join DAG and reseed database.

If you have only 2 DAG nodes then this is a peace of worry and you would like to fix on the day one of the issue. If you have more than 2 DAG nodes with 2 or more Database copies then you can take this issue to the next day.

 

Disaster Recovery Steps for one DAG node recovery:

1. Install OS and apply windows update.


Turn on Automatic updates. We can change this later to manually apply the updates so avoid any update reboot.



Download and installed the required updates. I had required 70 + updates in Feb 2014.

 

2. Reset the computer account the in the Active Directory User and computers


Click yes on this pop up


Now you will see below confirmation


3. Disable Firewall: Click on Public on here.


                Click on Turn windows firewall on or off here


                     Click on “Turn on Windows Firewall”

 

4. Turn off IEESC


Click off on both and click ok.


 

5. Give the old IP and point to the DNS server.

 

6. Give the same server name and join the domain



Type the user name and password then click ok.


You will see the welcome message

 

7. Turn on RDP


 

Click on “Allow remote connections to this computer” then click ok


 

8. Set the time zone


9. Restart the server

10. Copy the exchange 2013 setup files

11. Open the PowerShell with Run as administrator

 

12. Run the following cmdlet to install features.

For Windows 2012:

Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, RSAT-Clustering-Mgmt, RSAT-Clustering-PowerShell, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation

    

For Windows 2008:

Import-Module ServerManager

Add-WindowsFeature Desktop-Experience, NET-Framework, NET-HTTP-Activation, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Web-Server, WAS-Process-Model, Web-Asp-Net, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI

13. Restart the Server

14. Download and install the following softwares :

For Windows 2012:

  1. http://www.microsoft.com/en-us/download/details.aspx?id=34992

  2. http://www.microsoft.com/en-us/download/details.aspx?id=17062

  3. http://www.microsoft.com/en-us/download/details.aspx?id=26604

For Windows 2008:

  1. http://msdn.microsoft.com/en-us/library/5a4x27ek(VS.110).aspx

  2. http://www.microsoft.com/en-us/download/details.aspx?id=34595

  3. Microsoft Unified Communications Managed API 4.0, Core Runtime 64-bit

    http://www.microsoft.com/en-us/download/details.aspx?id=34992

  4. http://www.microsoft.com/en-us/download/details.aspx?id=17062

  5. http://www.microsoft.com/en-us/download/details.aspx?id=26604

  6. http://support.microsoft.com/?kbid=974405&wa=wsignin1.0

  7. http://support.microsoft.com/?kbid=2619234

  8. http://support.microsoft.com/?kbid=2533623

 

15. Restart the server

 

16. Remove the copy of the Database copy by running the following command:

Remove-MailboxDatabasecopy netcom12013vtc2

Confirm when it prompt


This is expected warning as server it down

 

17. Remove the DAG node from the DAG by running the below command:

Remove-DatabaseAvailabilityGroupServer -Identity Netcom -MailboxServer 2013VTC2 –ConfigurationOnly

Confirm when it prompt


 

18. DAG node would be required to remove from failover cluster manager.

Open FCM à Expand Nodes à Select the node and right click à Select more actions à Select Evict.


Click yes on this popup window


 

19. Install the exchange Server setup with Recover switch by running the below command

.setup /m:recoverserver /IacceptExchangeServerLicenseTerms

The important thing is recoverserver switch does not change anything in the active directory.

In the below screen we can see Exchange 2013 is installing 6 components of 2 roles.


 

20. Restart the Server

 

21. Now run the below command to add the Node 2 to the DAG

Add-DatabaseAvailabilityGroupServer identity DAGName -MailboxServer 2013VTC2

You will see this screen on your shell. Install the failover cluster manager from add and remove features to avoid this action from shell.


If you see the below error then just wait for sometime, verifiy failover cluster is installed. Then rerun the same command.


Second time I was able to see my node 2 joined the DAG cluster.


Node 2 has joined the cluster


Node 2 is part of DAG as well


 

22. Run the below command to start seed of the DAG database copy.

Add-MailboxDatabaseCopy Netcom1 -MailboxServer 2013VTC2

You will see the below message while seeding in the Shell. It will take time based on the database size.


After seeding you will see the below message to restart Information store on the Node 2.


 

23. Restart Information store Service.

24. Run the below command the check the status of the seeding.

Get-mailboxdatabasecopystatus **

You will resynchronizing is going on for the DAG database


After sync, you will see Healthy

 

25. You should be able to move the database to the node 2. 

 

We have recovered our node 2 of the DAG.

 

Prabhat Nigam

Microsoft MVP | Exchange Server

Team@MSExchangeGuru

Posted February 5th, 2014 under Clustering, DAG, Database Management, Disaster Recovery, Exchange 2010, Exchange 2013, Powershell. RSS 2.0 feed. Leave a response, or trackback.

18 Responses to “Exchange 2013: Recover a DAG and DAG Node from crash Part 3”

  1. Exchange 2013: Recover a DAG and DAG Node from crash Part 2 « MSExchangeGuru.com Says:
    February 5th, 2014 at 11:05 pm

    […] « Exchange 2013: Recover a DAG and DAG Node from crash Part 1 Exchange 2013: Recover a DAG and DAG Node from crash Part 3 […]

  2. Qhayum Says:
    March 5th, 2014 at 4:05 am

    This is one of the best article.

  3. Pravin Says:
    March 27th, 2014 at 4:29 am

    Very Nicely explain Mr.Prabhat as all can understand. Thanks

  4. movisajdi Says:
    April 21st, 2014 at 1:35 pm

    Hey,
    i Follow your method, its works great until i stop at content index, which shows “FailedAndSuspended”, even databse copies are added but content index is the problem.
    i add ContentSubmitters group method, i delete content index method by stoping the services, but non of them work.

    Seeding of content index catalog for database ‘Managers’ failed. Please verify that the Microsoft Search (Exchange) and the Host Controller service for Exchange services are running and try the operation again. Error: An error occurred while processing a request on server ‘MBX1’. Error: An Exception was received during a FAST operation..

    the passive database content index is showing “FailedAndSuspended” and i am not able to failover.

  5. Prabhat Nigam Says:
    April 21st, 2014 at 2:21 pm

    @Sajdi

    You have reported 2 issues
    1. content indexing
    2. moving active database to passive database when content indexing is failed.

    Here are my recommendations
    1. Content indexing generally takes 15-20 mins to become healthy after the seeding completes. If it does not become health then stop Microsoft Search (Exchange) service then remove the current content indexing folder and start the service. This should fix the content indexing.

    2. Use the shell command switch SkipClientExperienceChecks to skip content indexing check. So the following command should help you.
    Move-ActiveMailboxDatabase DBName -ActivateOnServer PassiveDBServername -SkipClientExperienceChecks

    Let me know if this helps

  6. movisajdi Says:
    April 21st, 2014 at 10:33 pm

    Thanks for your kind help.
    after following you advice i am still getting “Event ID 1009 MSExchangeFastSearch” on passive mailbox server
    how ever i move the database by running the above command.
    thanks

  7. Prabhat Nigam Says:
    April 21st, 2014 at 11:27 pm

    @Sajdi

    I think you have already tried this KB – http://support.microsoft.com/kb/2807668/en-US

    If it has not helped then give a feedback at the end of the KB.

  8. movisajdi Says:
    April 22nd, 2014 at 9:08 am

    Yes,
    i follow that one http://support.microsoft.com/kb/2807668/en-US and this one as well..there is an an other method in this artical..
    http://abdullrhmanfarram.wordpress.com/2013/06/16/event-id-1009-content-index-status-of-the-mailbox-databases-failed/

  9. Prabhat Nigam Says:
    April 22nd, 2014 at 10:10 am

    Did you leave the feedback?

  10. movisajdi Says:
    April 22nd, 2014 at 3:04 pm

    Yes i left, by saying i am using Exchange 2013 sp1, which i upgrade from CU3.
    Regards

  11. Sayed Tahzeeb H Says:
    January 28th, 2015 at 12:15 pm

    Truely a Guru……….

  12. Marco Says:
    January 29th, 2015 at 9:50 am

    Hi all,

    I also have the same problem with ID 1009 and status FailedAndSuspended on Exchange Server SP1.
    I already create group in AD take permission to Administrator and network services.
    Also alredy run Update-MailboxDatabaseCopy and delete folder SINGLE.
    Any suggest?

  13. Prabhat Nigam Says:
    January 29th, 2015 at 5:18 pm

    Restart the following services should fix it or you might need to go for latest update.
    Microsoft Exchange Search
    Microsoft Exchange Search Host Controller

  14. mejor robot aspirador Says:
    March 21st, 2015 at 6:45 am

    Thanks for finally talking about >Exchange 2013: Recover a DAG and DAG Node from crash Part 3 | MSExchangeGuru.com <Loved it!

  15. John Says:
    March 25th, 2015 at 2:42 am

    Hi,

    What if your storage is in tact and only the server goes down ? Can the server be recovered without having to reseed all the databases again ?

  16. Prabhat Nigam Says:
    March 29th, 2015 at 11:19 pm

    Hi John,

    In that case just recover the server and resync the DB. You don’t need to reseed.

  17. Raman Says:
    June 18th, 2015 at 3:49 pm

    Hi Prabhat,

    I have a question regarding Windows 2012 Failover clustering with Exchange 2013 CU7.

    We have Windows 2012 running with Exchange 2013 DAG. 8 nodes 1 witness server.

    There are few instances where one of the nodes lost the quorum due to network issues. When ever that happens cluster service goes in restarting (crashing). I tried to change Cluster service to manual and then start it but, it just keep crashing until I restart the server after that it works fine that node once again gets added into the quorum without any issues.

    My question – Is it normal behavior if node lose the quorum cluster service keep restarting until you restart the server? Or is there any way to bring back that server in the quorum without restart of the server.

    clussvc.exe version 6.2.9200.21268

    Error

    The Cluster Service service terminated unexpectedly. It has done this 15 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.

    Thanks,
    Raman

  18. Prabhat Nigam Says:
    June 18th, 2015 at 9:13 pm

    It looks like restart is doing more than cluster service restart.
    I would suggest to get the server checked by Microsoft.

Leave a Reply

Categories

Archives

MSExchangeGuru.com