MSExchangeGuru.com

Learn Exchange the Guru way !!!

 
«
»

Exchange 2010 DAG errors

Just thought of listing out some recent DAG error I worked and resolved… Happy to help!!!

FSW or the File Share Witness will fail to come online on one node for Exchange 2010 Database Availability Group running Windows Server 2008 R2.

Events:

System Log – Event ID 1564 –

Indicates that the cluster cannot access the file share witness directory. 
File share witness resource ‘File Share Witness (\\EXCHCAS.domain.com\DAG1.domain.com)’ failed to arbitrate for the file share ”. Please ensure that file share ” exists and is accessible by the cluster.

Use PsExec v1.98 to ensure you can connect to the FSW: http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx

I suspected this was an issue with either network itself or Permissions on the FSW

Tried the following steps:

  1. Stopped the FSW share
  2. Renamed that folder
  3. Recreated it:
  4. Open EMC
  5. Go to the properties of the DAG under Organization Configuration- Mailbox
  6. Change the name of the witness directory, removed it and then click Apply
  7. Now a new directory and FSW created

Set-Databaseavailabilitygroup -Witnessdirectory “c:\FSWFolderName” -Witnessserver “FSWServerName” -ID “DAGName”

All set…


Error: 
A server-side database availability group administrative operation failed. Error: The operation failed with message: 
An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=12) 
failed with 0x80070005. Error: Access is denied"' failed. [Server: servername.domain.com] 
An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. 
Error: Cluster API '"AddClusterNode() (MaxPercentage=12) 
failed with 0x80070005. Error: Access is denied"' failed. 
Access is denied 
Warning: 
The network name 'DAG-Exchange2010' is not online. 
Please check that the IP address configuration of the Database Availability Group is correct. 
Warning: 
The operation wasn't successful because an error was encountered. You may find more details in log file 
"C:\ExchangeSetupLogs\DagTasks\dagtask_2011-04-15_14-41-32.171_add-databaseavailabiltygroupserver.log". 
Exchange Management Shell command attempted: 
Add-DatabaseAvailabilityGroupServer -Identity 'DAG-Exchange2010' -MailboxServer 'ServerName' 
Upon further investigation saw that the Server we are trying to add to the DAG 
was not a member of the Exchange Servers or the Exchange Trusted Subsystem. 
Error: Unable to contact the Cluster Service on other 1 members (Exch2) of the database availability group. 
Warning: The operation wasn't successful because an error was encountered. 
You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2011-04-15_14-41-32.171_add-databaseavailabiltygroupserver.log". 
Resolution 
======================== 
We analyzed the Exchange Best Practice Analyzer, the Application and System Logs and the DAG Tasks. 
We saw that the first error is received at this point: 
[2011-05-25T12:19:29] GetRemoteCluster() for the mailbox server failed with exception = 
An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. 
Error: Cluster API '"OpenCluster(Exch2.domain.com) failed with 0x6d9. 
Error: There are no more endpoints available from the endpoint mapper"' failed 
[2011-05-25T12:19:29] Ignoring previous error, as it is acceptable if the cluster does not exist yet. 
[2011-05-25T12:19:29] DumpClusterTopology: Opening remote cluster DAG. 
[2011-05-25T12:19:51] DumpClusterTopology: Failed opening with 
Microsoft.Exchange.Cluster.Replay.AmClusterApiException: 
An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. 
Error: Cluster API '"OpenCluster(DAG1.domain.com) failed with 0x6ba. 
Error: The RPC server is unavailable"' failed.. ---> 
System.ComponentModel.Win32Exception: The RPC server is unavailable 
 --- End of inner exception stack trace --- 
 at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByName(AmServerName serverName) 
 at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor 
(String nameCluster, ITaskOutputHelper output) 
[2011-05-25T12:19:51] Dumping the cluster by connecting to: DAG. 
[2011-05-25T12:19:51] DumpClusterTopology: 
Not dumping, since the cluster could not be contacted. 
[2011-05-25T12:19:51] Opening the cluster on nodes [em1]. 
[2011-05-25T12:19:51] GetRemoteCluster() failed possibly with error 1864554511, 
ex = Microsoft.Exchange.Cluster.Replay.AmClusterApiException: 
An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. 
Error: Cluster API 'OpenByNames(Exch2.domain.com) failed for each server. 
Specific exceptions: 'An Active Manager operation failed. 
Error: An error occurred while attempting a cluster operation. 
Error: Cluster API '"OpenCluster(Exch1.domain.com) failed with 0x6d9. 
Error: There are no more endpoints available from the endpoint mapper"' failed..'.' failed.. ---> 
Microsoft.Exchange.Cluster.Replay.AmClusterApiException: 
An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. 
Error: Cluster API '"OpenCluster(Exch1.domain.com) failed with 0x6d9. 
Error: There are no more endpoints available from the endpoint mapper"' failed.. ---> 
System.ComponentModel.Win32Exception: 
There are no more endpoints available from the endpoint mapper 
 --- End of inner exception stack trace --- 
 at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByName(AmServerName serverName) 
 at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByNames(IEnumerable`1 serverNamesToTry) 
 --- End of inner exception stack trace --- 
 at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByNames(IEnumerable`1 serverNamesToTry) 
 at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.FindClusterForDag() 
[2011-05-25T12:19:51] The operation wasn't successful because an error was encountered. 
You may find more details in log file 
"C:\ExchangeSetupLogs\DagTasks\dagtask_2011-05-25_12-19-29.875_add-databaseavailabiltygroupserver.log". 
[2011-05-25T12:19:51] WriteError! Exception = 
Microsoft.Exchange.Management.Tasks.DagTaskServerCanNotContactClusterException: 
Unable to contact the Cluster service on other 1 members (em1) of the database availability group. 
[2011-05-25T12:19:51] Updated Progress 'Done!' 100%. 
[2011-05-25T12:19:51] COMPLETED 
add-databaseavailabiltygroupserver explicitly called CloseTempLogFile(). 
Resolution 
======== 
Set servers Exch1.domain.com and Exch2.domain.com the Registry Key "DisabledComponents" to 0xFFFFFFFF and we restarted the Servers. 
How to disable IP version 6 (IPv6) or its specific components in Windows 7, 
in Windows Vista, in Windows Server 2008 R2, and in Windows Server 2008: http://support.microsoft.com/kb/929852 
On the mailbox servers Exch1.domain.com and Exch2.domain.com disabled these 3 Features (TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access): 
Disabled the Firewall and then tried to add the Server Exch2.domain.com to the DAG again.

 

All set – Now enabled firewall once again.

Ratish Nair
MVP Exchange
Team@ MSExchangeGuru

Posted June 5th, 2012 under Database Management, Exchange 2010. RSS 2.0 feed. Leave a response, or trackback.

8 Responses to “Exchange 2010 DAG errors”

  1. Prabhat Says:
    June 8th, 2012 at 1:53 pm

    Nice article Ratish.

    If we just uncheck the IPv6 from network connection properties then it causes this issue and this should not be done. It is like roadies everything or nothing. So we should disabled IPv6 completely using ffffffff or keep it checked in network connection properties.

    I propose to keep the firewall off and firewall service running, unless we have configured the firewall to allow the required ports.

  2. xunyang Says:
    June 22nd, 2012 at 6:02 am

    i think this is only a workaround. actually, disable IPV6 and firewall is not the root cause. espically, disable firewall is a bad idea.

  3. Bill Rowland Says:
    July 14th, 2012 at 1:07 pm

    I have not tested, however disabling IPV6 has not been tested in best practices. Most recommend that you do not disable IPV6 as DAG services uses part of the stack to communicate. Have you been able to fully test this?

  4. arunlal Says:
    November 23rd, 2012 at 1:52 am

    Error:
    A server-side database availability group administrative operation failed. Error: The operation failed with message: Windows Failover Clustering timed out while trying to validate server ‘A1’. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server. [Server: WIN-0T78VRONQF8.intel.com]

    A server-side database availability group administrative operation failed. Error: Windows Failover Clustering timed out while trying to validate server ‘A1’. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server.

  5. ZAHOOR Says:
    April 1st, 2013 at 6:50 am

    I am not able to add anoter exchange node to DAG bewlow is the log file last contenets:

    [2013-04-01T10:09:35] InternalValidate() done.
    [2013-04-01T10:09:35] Updated Progress ‘Adding server ‘SAJDDREX01’ to database availability group ‘SAEXDAG1′.’ 6%.
    [2013-04-01T10:09:35] Working
    [2013-04-01T10:09:35] Updated Progress ‘Adding server ‘SAJDDREX01′ to the cluster.’ 8%.
    [2013-04-01T10:09:35] Working
    [2013-04-01T10:35:16] The following log entry comes from a different process that’s running on machine ‘SAJDHQEX01.nbk.com.sa’. BEGIN
    [2013-04-01T10:35:16] [2013-04-01T10:09:35] Opening a local AmCluster handle.
    [2013-04-01T10:09:35] Updated Progress ‘Adding server ‘sajddrex01’ to database availability group ‘SAEXDAG1′.’ 2%.
    [2013-04-01T10:09:35] Working
    [2013-04-01T10:09:35] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = SAJDDREX01, dwStatus = 0x0 )
    [2013-04-01T10:35:16] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseContinue, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 12, szObjectName = SAJDDREX01, dwStatus = 0x800713bb )
    [2013-04-01T10:35:16] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 12, szObjectName = SAJDDREX01, dwStatus = 0x800713bb )
    [2013-04-01T10:35:16] Found a matching exception: Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException: A database availability group administrative operation failed. Error: Windows Failover Clustering timed out while trying to validate server ‘SAJDDREX01’. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server.
    [2013-04-01T10:35:16] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = SAJDDREX01, dwStatus = 0x0 )
    [2013-04-01T10:35:16] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 12, szObjectName = , dwStatus = 0x0 )

    [2013-04-01T10:35:16] The preceding log entry comes from a different process running on computer ‘SAJDHQEX01.x.x’. END
    [2013-04-01T10:35:16] The operation wasn’t successful because an error was encountered. You may find more details in log file “C:\ExchangeSetupLogs\DagTasks\dagtask_2013-04-01_10-09-30.737_add-databaseavailabiltygroupserver.log”.
    [2013-04-01T10:35:16] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: Windows Failover Clustering timed out while trying to validate server ‘SAJDDREX01’. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server. —> Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException: A database availability group administrative operation failed. Error: Windows Failover Clustering timed out while trying to validate server ‘SAJDDREX01’. If this is in a disjoint DNS namespace, the DNS suffixes for all servers in the database availability group must be present on every server.
    at Microsoft.Exchange.Cluster.ClusApi.AmCluster.AddNodeToCluster(AmServerName nodeName, IClusterSetupProgress setupProgress, IntPtr context, Exception& errorException, Boolean throwExceptionOnFailure)
    at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
    — End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.DagTaskValidateNodeTimedOutException) —
    at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
    at Microsoft.Exchange.Cluster.Replay.DagHelper.AddDagClusterNode(AmServerName mailboxServerName, String& verboseLog)
    at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.c__DisplayClass34.b__33()
    at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
    — End of stack trace on server (SAJDHQEX01.x.x) —
    at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
    at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperationDbName(AmServerName serverName, String databaseName, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
    at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunRpcOperation(AmServerName serverName, Nullable`1 dbGuid, Int32 timeoutMs, IHaRpcExceptionWrapper rpcExceptionWrapperInstance, InternalRpcOperation rpcOperation)
    at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunAddNodeToCluster(AmServerName serverName, AmServerName newNode, String& verboseLog)
    at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.JoinNodeToCluster()
    [2013-04-01T10:35:16] Updated Progress ‘Done!’ 100%.
    [2013-04-01T10:35:16] COMPLETED
    add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().

  6. KRISHNA Says:
    May 22nd, 2013 at 4:43 am

    I am getting bellow error as going through above article.
    [PS] C:\Windows\system32>Restore-DatabaseAvailabilityGroup DAG -ActiveDirectorySite Site2

    Confirm
    Are you sure you want to perform this action?
    Restoring Mailbox servers for Active Directory site “Site2″ in database availability group “DAG”.
    [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is “Y”): a
    WARNING: The operation wasn’t successful because an error was encountered. You may find more details in log file
    “C:\ExchangeSetupLogs\DagTasks\dagtask_2013-05-10_07-59-29.367_restore-databaseavailabilitygroup.log”.
    A database availability group administrative operation failed. Error: Unable to form quorum for database availability g
    roup ‘DAG’. Please try the operation again, or run the Restore-DatabaseAvailabilityGroup cmdlet and specify the site wi
    th servers known to be running.

    Still to restart DR site after failover it was working but after Restarting DR site all mail databases are unable to mount, unable to form a quorum, An Active Manager operation failed in DR Site and Cluster service in Disable State cause this DAG network not initialise and showing in EMC.
    Please help for the same.

  7. Amit Says:
    December 23rd, 2013 at 8:23 am

    In my case DHCP Client service was not started. Please make sure to start DHCP service and add again. Hope this will help.

  8. test Says:
    March 24th, 2016 at 8:46 pm

    Nice post. I was checking continuously this weblog and I am inspired!
    Very useful information specifically the ultimate section :
    ) I handle such information a lot. I used to be seeking this
    certain information for a long time. Thanks and good luck.

Leave a Reply

Categories

Archives

MSExchangeGuru.com