Database availability groups are reasonably smart and robust systems, but they do need monitoring, or else you might find one day they are not providing the HA that you need them to. One of the useful PowerShell cmdlets for keeping an eye on things is Test-ReplicationHealth. It can be used to help troubleshoot database availability groups by running it locally or against a remote server.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
[PS] C:\>Test-ReplicationHealth Server Check Result Error ------ ----- ------ ----- E15MB2 ClusterService Passed E15MB2 ReplayService Passed E15MB2 ActiveManager Passed E15MB2 TasksRpcListener Passed E15MB2 TcpListener Passed E15MB2 ServerLocatorService Passed E15MB2 DagMembersUp Passed E15MB2 ClusterNetwork Passed E15MB2 QuorumGroup Passed E15MB2 DatabaseRedundancy Passed E15MB2 DatabaseAvailability Passed E15MB2 DBCopySuspended Passed E15MB2 DBCopyFailed Passed E15MB2 DBInitializing Passed E15MB2 DBDisconnected Passed E15MB2 DBLogCopyKeepingUp Passed E15MB2 DBLogReplayKeepingUp Passed |
To test a remote server, use the -Identity parameter.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
[PS] C:\>Test-ReplicationHealth -Identity E15MB2 Server Check Result Error ------ ----- ------ ----- E15MB2 ClusterService Passed E15MB2 ReplayService Passed E15MB2 ActiveManager Passed E15MB2 TasksRpcListener Passed E15MB2 TcpListener Passed E15MB2 ServerLocatorService Passed E15MB2 DagMembersUp Passed E15MB2 ClusterNetwork Passed E15MB2 QuorumGroup Passed E15MB2 DatabaseRedundancy Passed E15MB2 DatabaseAvailability Passed |
The cmdlet also accepts pipeline input, however if you were to simply pipe Get-MailboxServer into it and you have Mailbox servers in the organization that are not DAG members then you risk seeing errors in your results that just get in the way. Instead you can pipe only the members of a database availability group into Test-ReplicationHealth using the following method:
|
1 |
[PS] C:\>Get-DatabaseAvailabilityGroup | select -ExpandProperty:Servers | Test-ReplicationHealth |
There’s a lot of output to look at, so it’s often easier to filter the results to only those that have not passed.
|
1 2 3 4 5 6 7 |
[PS] C:>Get-DatabaseAvailabilityGroup | Select -ExpandProperty:Servers | Test-ReplicationHealth | Where {$_.Result.Valu e -ne "Passed"} Server Check Result Error ------ ----- ------ ----- EX2016SRV2 DatabaseAvailability *FAILED* Failures:... EX2016SRV1 DatabaseAvailability *FAILED* Failures:... |
The error details are usually truncated, so outputting the results as a list will let you see more information.
|
1 2 |
[PS] C:>Get-DatabaseAvailabilityGroup | Select -ExpandProperty:Servers | Test-ReplicationHealth | Where {$_.Result.Val e -ne "Passed"} | Format-List |
By the way, if you’re curious about what each of the tests do, there’s a “CheckDescription” provided for each of the checks that Test-ReplicationHealth performs. Some of them are still a bit vague, but there’s some useful info there that can help point you in the right direction to investigate further.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
[PS] C:>Test-ReplicationHealth | Format-List Check* Check : ClusterService CheckDescription : Checks if the cluster service is healthy. Check : ReplayService CheckDescription : Checks if the Microsoft Exchange Replication service is running. Check : ActiveManager CheckDescription : Checks that Active Manager is running and has a valid role. Check : TasksRpcListener CheckDescription : Checks that the Tasks RPC Listener is running and is responding to remote requests. Check : TcpListener CheckDescription : Checks that the TCP Listener is running and is responding to requests. Check : ServerLocatorService CheckDescription : Checks that the Server Locator Service is running and is responding to requests. Check : DagMembersUp CheckDescription : Verifies that the members of a database availability group are up and running. Check : MonitoringService CheckDescription : Checks that the Monitoring WCF Service is running and is responding to requests. This endpoint is hosted inside the DAG Management service. Check : ClusterNetwork CheckDescription : Checks that the networks are healthy. Check : QuorumGroup CheckDescription : Checks that the quorum and witness for the database availability group is healthy. Check : FileShareQuorum CheckDescription : Verifies that the path used for the file share witness can be reached. Check : DatabaseRedundancy CheckDescription : Verifies that databases have sufficient redundancy. If this check fails, it means that some databases are at risk of losing data. Check : DatabaseAvailability CheckDescription : Verifies that databases have sufficient availability. If this check fails, it means that some databases are at risk of losing service. |
Hi Paul, what are the next steps if you get a failure from Test-ReplicationHealth? We are running Exchange 2010 and despite seeing no ill effects all 12 members of our three DAG’s are now failing with:
The health test of the Microsoft Exchange Server Locator Service on server ‘DAG-MEMBER’ failed.
Error: Server Locator Service call had a communication error.
Searching the for this error it does’t seem to appear in any MS documentation so we’re at a bit of a loss of how to proceed before contacting MS.
Many thanks, Ryan.
According to TechNet that…
“Verifies the Active Manager client/server processes on DAG members and on the Client Access Server that perform lookups in Active Directory and Active Manager to determine where a user’s mailbox database is active.”
I don’t see any info out there on what to do about it. Are there any other event log entries that might be relevant? Particularly events that may be logged when the server restarts?
Thanks for the advice Paul, I tried rebooting a DAG member but couldn’t see anything useful so investigated further…
This issue only came to light because we run Test-ReplicationHealth for each node daily and email an alert on failure. The Server Locator Service error first appeared last month and then disappeared before returning last week. It would always affect all servers or none.
We are in the process of updating from SP2 to SP3 and it seems the Test-ReplicationHealth results vary depending on where you run the cmdlet. If run on SP3 we don’t see ClusterNetwork, QuorumGroup and FileShareQuorum checks but gain the ServerLocatorService failure.
The server running the daily health checks would connect to a different Exchange server each day thus we got differing results. So far we’ve only upgraded our CAS and HT servers so hopefully once we complete the mailbox servers the ServiceLocatorService will report healthy.
Thanks again for initial advice though Paul 🙂
I suppose that would make sense that running from a down level server might throw some odd results for higher SP level servers. Though I’ve never run into that issue myself, because I generally run monitoring scripts from the highest version/SP level in the org.
Hi Paul,
Would you happen to have the firewall ports required for this to work? We have an exchange site behind a firewall and not getting any results when I run this test.
I don’t. Try a wireshark trace?
Hi Paul,
thanks once again for excellent work. Just like the test-exchangehealth script, I would like to have this output to an email that my team could get say twice a day, i.e. at the typical working start time of the day and before we go home for the day.
Any assistance to help generate it will be greatly appreciated.
Sending any PowerShell output as an email is pretty simple really.
http://practical365.com/powershell-how-to-send-email/
Or use my Test-ExchangeServerHealth.ps1 script.
Hi Paul, Thank you for the time you take to write these articles. Not only are they clear, but also help with getting a better understanding of the PowerShell cmdlets.
Nien.