When an Exchange Server 2010 Mailbox server that is a member of a Database Availability Group has failed, part of the recovery process is to remove it from DAG membership.
In some scenarios this process may result in an error “A quorum of cluster nodes was not present to form a cluster“. The full error text is below.
[PS] C:\>Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer EX2
Confirm
Are you sure you want to perform this action?
Removing Mailbox server "EX2" from database availability group "dag".
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): y
WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file
"C:ExchangeSetupLogsDagTasksdagtask_2010-11-25_03-48-09.814_remove-databaseavailabiltygroupserver.log".
There was a problem changing the quorum model for database availability group dag. Error: An Active Manager operation f
ailed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"SetClusterQuorumResource() f
ailed with 0x1725. Error: A quorum of cluster nodes was not present to form a cluster"' failed..
+ CategoryInfo : InvalidArgument: (:) [Remove-DatabaseAvailabilityGroupServer], DagTaskProblemChangingQuo
rumException
+ FullyQualifiedErrorId : 80D96894,Microsoft.Exchange.Management.SystemConfigurationTasks.RemoveDatabaseAvailabili
tyGroupServer
To resolve this issue use the -ConfigurationOnly switch instead to remove the failed Mailbox server from the Exchange 2010 DAG. In this example server EX2 is being removed.
[PS] C:\>Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer EX2 -ConfigurationOnly
Next, evict the failed node from the Windows Failover Cluster.
Note: you need to import the Failover Cluster module into your Exchange Management Shell session to perform this task.
[PS] C:\>Import-Module FailoverClusters [PS] C:\>Get-ClusterNode EX2 | Remove-ClusterNode -Force
The failed server has now been removed from the Exchange 2010 DAG and the Windows Failover Cluster.
[adrotate banner=”49″]
Get-Cluster DAG | Remove-Cluster -Force -CleanupAD
You must remove the cluster through the command.
Great article! We did a force quorum on our two Node DAG and created a split brain scenario. We’ve removed secondary node from cluster manager, disabled its NIC, but get error when trying to run the remove commands to remove it from the DAG, but errors saying that there is a still a database copy present.
Any thoughts about how to resolve this?
Error:
[PS] C:\Windows\system32>Set-DatabaseAvailabilityGroup -Identity DAG -WitnessDirectory c:\DAGWitness -WitnessServer Newwitnessserver.domain.local
There was a problem changing the quorum model for database availability group EXGDAG. Error: An error occurred while
attempting a cluster operation. Error: Cluster API failed: “DeleteClusterResource() failed with 0x139f. Error: The
group or resource is not in the correct state to perform the requested operation”
+ CategoryInfo : InvalidArgument: (:) [Set-DatabaseAvailabilityGroup], DagTaskProblemChangingQuorumExcept
ion
+ FullyQualifiedErrorId : [Server=EX01,RequestId=0303ae8a-6df1-47f9-8fd2-1390763b2eb2,TimeStamp=12/19/2017 4:4
4:02 AM] [FailureCategory=Cmdlet-DagTaskProblemChangingQuorumException] 56C85867,Microsoft.Exchange.Management.Sys
temConfigurationTasks.SetDatabaseAvailabilityGroup
+ PSComputerName : EX01.domain.local
The Real Person!
The Real Person!
I Googled your error message “There was a problem changing the quorum model for database availability group”, and found this.
http://markgossa.blogspot.com.au/2016/01/dag-doesnt-match-configured-witness-server.html
Looks like it will fix your problem.
To answer your earlier question, the FSW should be located in the datacenter where you want the databases to be online/active in the event of a WAN failure.
Thanks Paul,
Let me check http://markgossa.blogspot.com.au/2016/01/dag-doesnt-match-configured-witness-server.html
Hi
I checked my quorum resources state, it has already quorum resources as File Share Witness, the above blog talking about when there is no quorum resources present in the DAG.
The above blog speaks about Quorum resources, In my exchange already the Quorum resources showing as File Share Witness
The Real Person!
The Real Person!
If that article hasn’t helped you then I suggest you open a support case with Microsoft so that they can look at your environment directly and provide a solution.
Hi Paul,
Thanks for your time.
Hi Paul,
The issue has been resolved by manually deleted the File share Witness from FCM.
Steps Done:
1. Set No witness (Now witness will be “None”)
2.Check there is any witness shows in FCM. If there is witness under cluster resource, Right click and remove.
3.Set-DatabaseAvailabilityGroup -Identity DAG -WitnessServer Witnessserver.domain.local -WitnessDirectory C:\DAGWitness.
Now My DAG shows
WitnessServer : Witnessserver.domain.local
WitnessDirectory : C:\DAGWitness
AlternateWitnessServer :
AlternateWitnessDirectory :
WitnessShareInUse : Primary (This was showing as Invalidconfiguration)
DxStoreWitnessServers :
No downtime is required.
Set-Databaseavailabilitygroup -Identity DAG -Witnessserver NEWWITNESSSERVER.domain.local -Witnesssdirectory C:\DAGWITNESS
In witness server I have added exchange trusted subsystem to local administrator Group. before run the above cmdlet.
Hi,
I am getting “WitnessShareInuse : InvalidConfiguration” when I get the DAG status. What could be the reason and I tried “Set-DatabaseAvailabilitygroup -Identity -WitnessServer -WitnessDirectory ” to reconfigure the witnessserver. and unable to change.
But there is a difference in witness type between GUI fail over cluster and cmdlet witness type.
In GUI witness type is showing as “file share witness”, In cmdlet quorum type is showing as “nodeandfilesharemajority”
The Real Person!
The Real Person!
What does “unable to change” mean? Do you get an error message?
WARNING: The witness server and directory currently in use by database availability group ‘DAG’ doesn’t match the
configured primary or alternate witness server. This may be due to Active Directory replication latency. If this
condition persists, please use the Set-DatabaseAvailabilityGroup cmdlet to correct the configuration.
WitnessServer : Witnessserver.domain.local
WitnessDirectory : C:\DAGWitness
AlternateWitnessServer :
AlternateWitnessDirectory :
WitnessShareInUse : InvalidConfiguration
DxStoreWitnessServers :
What does “unable to change” mean? (Unable to change the witness server to another Server)
Site : 2 Sites
Servers : PRIMARY 3 and DR 1
DAG Members Total : 4
Witness : Placed in DR (Is this correct?)
DAC : Enabled
The Real Person!
The Real Person!
Why are you unable to change it? What command are you running? What error message do you get?
Get-ClusterNode PRD01
Name State
================================
PRD01 Down
I am getting below error while trying to remove a corrupt node from cluster.
The node is crashed and not able to make it up.
PS C:\> Get-ClusterNode PRD01|Remove-ClusterNode -Force
Remove-ClusterNode : An error occurred evicting node ‘PRD01’.
The cluster request is not valid for this object
At line:1 char:45
+ Get-ClusterNode PRD01|Remove-ClusterNode <<<< -Force
+ CategoryInfo : NotSpecified: (:) [Remove-ClusterNode], ClusterCmdletException
+ FullyQualifiedErrorId : Remove-ClusterNode,Microsoft.FailoverClusters.PowerShell.RemoveClusterNodeCommand
The Real Person!
The Real Person!
No idea. I assume “Get-ClusterNode PRD01” returns a good result?
If you’re stuck on that step I suggest you open a support case with Microsoft for further assistance.
Hi, i have followed the procedure in the previus one article but my cluster dowsn’t go up. if i try to use this procedure i have this error:
[PS] C:\Windows\system32>Get-ClusterNode EXCH-MBX03 | Remove-ClusterNode -Force
Get-ClusterNode : Failed to retrieve the node ‘EXCH-MBX03’ from the cluster ‘DAG01’.
An error occurred opening node ‘EXCH-MBX03’.
The cluster node was not found
At line:1 char:16
+ Get-ClusterNode <<<< EXCH-MBX03 | Remove-ClusterNode -Force
+ CategoryInfo : NotSpecified: (:) [Get-ClusterNode], ClusterCmdletException
+ FullyQualifiedErrorId : Get-ClusterNode,Microsoft.FailoverClusters.PowerShell.GetNodeCommand
How can i do?
Thanks
The Real Person!
The Real Person!
It’s saying that cluster node was not found. Are you sure it’s the right name?
Grate! Worked
Thanks!
Helped me a lot. Had to remove a dead Exchange 2013 server from DAG, and remove-clusternode was the solution
This definitely helped me out in crisis when a server was added to a DAG but the IP still resolved to two servers so there was a conflict. Thanks Paul!!
I am unable to remove a Member from DAG. it shows below error:
An Active Manager operation failed with a transient error. Please retry the operation. Error: A transient error occurred during discovery of the database availability group topology. Error: The fully qualified domain name for node ‘EX2’ could not be found.
will you please help me.
Thanks
The Real Person!
The Real Person!
“The fully qualified domain name for node ‘EX2’ could not be found”
Is your server named EX2?
Great post! Thank you so much! Same situation with Exchnage 2013. It helps me a lot!
This truely sorted me out. Thanks man!
Thanks so much Paul , your always helpful for me .
God blesses you,
imran safi
from afghanistan
I ran these commands on a server that was still in the environment but halfway uninstalled from before. I was unable to uninstall Exchange 2013 from this “orphan” server after running the commands in this article because the server seemed to think that it was still a part of the DAG. MS helped me out with running this command first on one of the servers that is left in the DAG:
cluster node ORPHAN-MBX /forcecleanup
After that I could uninstall Exchange from this “orphan” server, and thus clean up AD properly.
Paul.. great help!
Thanks
Always on point! your articles always work!
Pingback: Unable to Remove Failed Server from DAG in Exchange Server 2010 | Bart Vincke's Blog
Hi Paul and Thanks Again
In my cluster and DAG everything was OK
situation is :
Two Ex 2010 SP1 in the DAG (mailbox01 and Mailbox02) and Two SP3 Added to DAG (mailbox1 and Mailbox2)
Now I want to remove the previous ones
but this error show up again and again while removing mailbox01 with no database copy on it
anyway i used first cmdlet
[PS] C:\>Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer Mailbox0X2 -ConfigurationOnly
and it is removed from DAG but second ones
[PS] C:Windowssystem32>Get-ClusterNetwork Maibox01 | Remove-ClusterNode -Force
Get-ClusterNetwork : Failed to retrieve the network ‘Maibox01’ from the cluster ‘DAG’.
An error occurred opening network ‘Maibox01’.
The cluster network was not found
At line:1 char:19
do not work, give me following error and the server is not removed from windows failover cluster
oh sorry
i had no patience 🙂
just needed some time maybe to clear the configurations from the cluster and then it worked
i think you should mention that some time is needed between running those commands 😉
You have no idea how this helped. I couldn’t remove the member server from DAG because Windows Cluster Service would not start. Thanks!
I have an two exchange 2010 sp2 multi-role servers. In EMC it shows the replication is up to date with no queue built up. However, the DR server shows in the event viewer.
“passive/active copy is not UP according to clustering”
In cluster failover management the node shows as offline and so does the NIC. Both are online and the replication seems to be fine. The primary dag server shows all is well. Can I use the above to remove and readd the recovery exchange server? Do I need to remove all DB copies first or can I just execute the commands above and it will take care of it?
Thanks sir, this was a great help today!
Hi
I am trying to import Import-Module FailoverClusters but msem shell is not getting this command please help
Thanks
The Real Person!
The Real Person!
what is “msem”?
The command is a PowerShell command and should work on a Windows 2008 R2 server that is an Exchange 2010 DAG member.
Hi
msem is microsoft excange management shell, Actually i was Using Pssession from local machine
Thank you so much it works from Dag member.
Great post! It works like a champ…!
Pingback: Exchange Recovery: Failed DAG Member in Exchange Server 2010 - Exchange Server Pro