One of the issues that my Get-DAGHealth.ps1 script alerts for is failed content indexes on database copies in a database availability group.
Note: for failed content indexes on servers that are not DAG members refer to this article instead.
Failed content indexes can easily go unnoticed when everything else is working fine however they will eventually begin to cause problems for you, for example by preventing database switchovers.
Fixing a single failed content index is easy, but if there are multiple failed indexes you can speed things up a little by fixing them all with a single PowerShell command.
To demonstrate, here is the Exchange Management Shell command for using Get-MailboxDatabaseCopyStatus to display all database copies that have a content index in a failed state.
[PS] C:\>Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq "Failed"}
Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex
Length Length State
---- ------ --------- ----------- -------------------- ------------
Mailbox Database 1HO-EX2010-MB1 Healthy 0 0 22/02/2013 10:43:26 AM Failed
Mailbox Database 2HO-EX2010-MB1 Healthy 0 0 22/02/2013 10:41:36 AM Failed
So all we need to do is pipe the output of that command into the Update-MailboxDatabaseCopy cmdlet with the -CatalogOnly switch.
[PS] C:\>Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq "Failed"} | Update-MailboxDatabaseCopy -CatalogOnly
Wait for that process to complete, then re-run the first command again. If no results are returned this time then there are no more failed content indexes.
@paul cunningham , May i know , how to fix the service down error for the passive copies while using Get-mailboxdatabasecopystatus (restarted server) multiple times.
Once again TOP Stuff… very helpful!
Thxs Man
You should run thin cmdlet:
Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq “Failed” -and $_.Status -eq “Healthy”} | Update-MailboxDatabaseCopy -CatalogOnly
Because content indexing can be reseeded only for passive nodes.
Dear Paul
i have lost active manager exch1 DAG exchnage 2013 due to power failure but when i start other dag member exch2 server all dag database are disconnected and failed state.While active manager exch1 server database showing service down state but its permanently lost.
So how can i make active manager to exch2 and makes database mounted ? no issue if lost some data i knew copy queue lenghth is not zero.please need advice
If you have just run exchange 2016 CU 19 update and your getting this unhealthy content index count , then I suggest you look at the exchange service Microsoft Exchange Search Host Controller it will now be disabled ( after the update ). Enable and start the service I checked on one of my unpatched servers and it was set to enabled and start automatically and run
Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq “Failed”} | Update-MailboxDatabaseCopy -CatalogOnly
Your reading this and will probably save 3 hours of your life not like me.
Just ran the Exchange 2016 CU19 update for the March 2, 2021 fix. Same issue with the Search Host service disabled. Re-enabling and starting took care of the failed state.
Thanks for the heads up!
Exactly the same here too, thanks Simon for the pointer, totally saved me loads of time. Top man! Nothing about that CU19 upgrade was simple for me from start to finish!
Thanks Simon,
I applied CU23 (KB5001779) to my Exchange 2013 and it disabled “Microsoft Exchange Search Host Controller” and also corrupted IIS.
Regards
I don’t know you or what country you are from but if you ever come to Essex, UK then I owe you a beer!!! 🙂
Saved me a lot of grief
Hi Paul,
We have exchange DAG with two servers and ContentIndexState on both the servers are in “FailedAndSuspended” State.
Can i use the command listed above to repair?
I tried running this on my failed statuses but get this error:
Database ‘DB1\SERVER’ has only one copy. This task is supported only for databases that have more than one copy. It happens for all my DB’s.
Any suggestions?
sorry for the double posting (i submitted a wrong email address)
please note, the correct email is: matanmail@gmail.com
thank you
Hi Paul, hope you can help me out here,
i have a 3 server dag 2013
all db’s are OK except for one passive copy which always stays in Suspend state on a specific serer (“contentindexstate suspended”) no matter what i do
i reseeded, removed and re-created, tried to fix it (update-mailboxdatabasecopy catalog only)…tried them alll
still this DB of all….is in healthy but suspended state….i don’t have any more idease for what can cause this.
please see if you can get some…
thank you
Hi Paul, hope you can help me out here,
i have a 3 server dag 2013
all db’s are OK except for one passive copy which always stays in Suspend state on a specific serer (“contentindexstate suspended”) no matter what i do
i reseeded, removed and re-created, tried to fix it (update-mailboxdatabasecopy catalog only)…tried them alll
still this DB of all….is in healthy but suspended state….i don’t have any more idease for what can cause this.
please see if you can get some…
thank you
Hi Paul
I have a DAG node that still has all its exchange services running but its local database (not replicated ) is dismounted. The replicated db are still healthy. Unfortunately by mistake, two users mailbox were created in the dismounted local database. There is no way to move them out of the dismounted database. How would I fix the Failed content state of a standalone database that prevents it from being mounted? Thanks for your help.
Poul, what command if the mailbox database has a content indexing status of Unknown? I’am confused here. Same command? Any advice is very welcome. Best regards.
Awesome, we have faced the same problem and this article helps to resolve the issue. saved lot of times, Excellent explanation Paul.
Well..after restarting the passive node today, i noticed that ‘content index state’ is appearing ‘Failed’ for all 8 databases. Followed your suggested method. It did not give any error but it did not fix the problem either. Then finally I just restarted the following services: First “Microsoft Exchange Search” then “Microsoft Exchange Search Host Controller” and within 2 minutes or so all the databases started showing the “Healthy” state. No idea what fixed the problem. But it did. Thanks.
Hi Paul,
We had issue with a extreme fast growing Index folder on one of our database servers in a 2 node DAG cluster.
Any idea what triggers a extreme fast growing index? 20 GB in 1 hour..
Kind regards,
Leo
Hi Paul,
We have an Exchange organization of three servers and have a DAG set up for seven databases. Since upgrading to Exchange 2013 from 2010, we have a problem with content indices randomly failing- They seem to go bad about 2-3 per week. I keep fixing them but it seems like its always a different index failing. All servers are running the most current CU, 2 of them are EE and one is SE. The problem began before the two were upgraded from SE to EE. Any ideas?
Incorrectly configured antivirus exclusions is a common cause of index failures.
Hi paul, My OWA search are not working in all mail i have tried all the options to rebuild indexing but not successful it still says failed what shall i do?
i have tried i have stopped the search services and deleted the GUID folder yet indexing still fails. i have one DB of 256GB kindly assist with other ways to solve this issue
Have you configured the recommended file system antivirus exclusions on your Exchange servers?
OWA does not work either…ok will log a call.
Thanks Paul
Thanks for replying..
Just checked as below
Live server – Status is Mounted – ContentIndexState is Healty
Passive server – Status is Healthy – ContentIndexState is Healthy
If you’ve got health indexes, and both Outlook and OWA search are not working, then I suggest opening a support case with Microsoft so they can look at it with you.
Hi Paul – top post as always, could you provide some pointers with a problem we have? We have exchange 2010 in a DAG (active / passive) on exchange 2008 R2. After a reboot of our mailbox servers we often have issues with the Outlook search function stops working for everyone in the organization. Generally it stops working between the hours of 7.30/8.00am and starts working again around 11.15am-12pm without us doing anything!? I have ran various things as below..
After x (sometimes days) amount of time, reboots of servers / switching of passive – active DB’s it starts working on its own!
Moved my Mailbox to a DB that is not in the DAG – Search works all the time
Switched All DB’s to passive mailbox server – no better
Ran this on each DB – New-MailboxRepairRequest -Database “dbx” -CorruptionType AggregateCounts,ProvisionedFolder,SearchFolder,FolderView -DetectOnly – no better
Ran this on both passive and active servers .\Troubleshoot-CI.ps1 -Server name – no better
Ran this on each DB \Troubleshoot-CI.ps1 -database DBx -Action DetectAndResolve – found no errors
Any pointers would be appreciated!
What state are the indexes in when search stops working?
Thanks. Saved my ass..
Paul,
First off, EXCELLENT command! Works like a charm. Unfortunately, I have an issue that I’m hoping you can help me with.
We have Data Centers at different locations around the world and replicating the active index from the active DB, may cause us to have to replicate from Singapore to Chicago, for instance. Because of that, it takes FOREVER to run this command (literally all day) on 10 indexes.
Is there a way to replicate from a healthy passive copy of the index, in the same DC?
Thanks mate!
The Update-MailboxDatabaseCopy cmdlet has a -SourceServer parameter that can be used to specify a server to reseed from. It works for reseeding databases, not sure if it also works for content indexes, but you will soon find out I’m sure.
https://technet.microsoft.com/en-us/library/dd335201(v=exchg.160).aspx
Hi Paul,
I’m running Exchanging 2013 with two mailbox servers in a DAG. Get-MailboxDatabaseCopyStatus * showed both servers in FailedAndSuspended state. I think this issue occurred after I extended disk space on the mailbox servers. I tried Update-MailboxDatabaseCopy -CatalogOnly but that didn’t help I presume as there was no Heatlty copy present.
On the passive node I stopped Exchange Search and Exchange Host Controller service, renamed .single to .single.old and started the service again which created a new folder. But state was still showing FailedAndSuspended, so I tried to do same on the Active node and upon service start status was changed to Crawling on active node and passive was still showing FailedAndSuspended.
It took more than 24 hours and status was changed from Crawling to Failed on Active node. Then again on the passive node I stopped and started Exchange Search and Exchange Host Controller service, after a while status was changed from Failed to Crawling and then to Healthy. Once status was healthy then I ran Hi Paul,
I’m running Exchanging 2013 with two mailbox servers in a DAG. Get-MailboxDatabaseCopyStatus * showed both servers in FailedAndSuspended state. I think this issue occurred after I extended disk space on the mailbox servers. I tried Update-MailboxDatabaseCopy -CatalogOnly but that didn’t help I presume as there was no Heatlty copy present.
On the passive node I stopped Exchange Search and Exchange Host Controller service, renamed .single to .single.old and started the service again which created a new folder. But state was still showing FailedAndSuspended, so I tried to do same on the Active node and upon service start status was changed to Crawling on active node and passive was still showing FailedAndSuspended.
It took more than 24 hours and status was changed from Crawling to Failed on Active node. Then again on the passive node I stopped and started Exchange Search and Exchange Host Controller service, after a while status was changed from Failed to Crawling and then to Healthy. Once status was healthy then I ran MailboxDatabaseCopy -CatalogOnly which made the active node healthy as well.
Now the issue which I’m facing is that search still isn’t working other than recent emails (1 day old emails)
.single.old size is showing as 30 GB while new folder .single is showing only 6 GB. Is there any possible solution I could fix the search issue?
Could you please help me to resolve this.
Regards,
Mobin
Hi Mobin,
im facing the same issue you have, did you manage to solve it ?
thanks,
sorry, wrongly reply to you. additonal this comand works .
i have edited SMTP but not working. please give me any hints
$smtpsettings = @{
To = “40802478@qq.com”
From = “wyh33621999@126.com”
Subject = “$reportemailsubject – $now”
SmtpServer = “smtp.126.com”
}
Hi Paul,
Hope you can help me, we are running a Exchange 2010 DAG group for our exchange archives with 8 databases.
2 of our databases have a Failed index state on both copies if the database. if I remove the catalog and start the search services again, status changes to crawling but after 2 hours status is failed again.
databases are in clean shut down state,
prior to changing back to Failed state the following eventlog entry is showing;
MSSearch service signaled to pause indexing for database . FTERequestStatus = ForcePause, FTERequestStatusChangeReason = ResetNeeded. Indexing on this catalog is being paused. It will be resumed once a resume signal is received.
right after that the status changes to Failed.
How about exchange 2013 mailbox database copy status is keep resynchronizing??
Pingback: Как лечить индекс базы Exchange 2013 — rusmandalay
Thanks for the script!
Worked great as always. Thanks Paul.
I have an issue whereby at random intervals (maybe once a week) one of our servers will get hammered by noderunner.exe using up to 15-20GB RAM and very high CPU usage. I had a call with Microsoft the last time it happened and they said to install CU11 as it had some fixes for high CPU noderunner issues but this hasn’t seemed to help.
The next course of action that they suggested was to rebuild the indexes but they suggested a slightly different process. They said to remove all DB copies from the other DAG members, then stop the search service, delete the CI and once completed add the DB copies back onto the other DAG members.
This seems like a bit of an extreme fix and I was just wondering if in your opinion this is required or whether just running the process that you describe above will work?
Many thanks in advance.
AFAIK there’s no need to remove all DB copies to rebuild an index. Removing just the index files from each DAG member should be sufficient.
Brilliant thank you so much for your help. So can i confirm the process would be as below
– On the active database server stop the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller.”
– Delete the Existing CI catalog folder
– Start the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller” services.
– Once the DB catalog has been rebuilt log onto each of the other servers that has a DB copy and run the following command
At this point do i then
– Run the command Update-MailboxDatabaseCopy -Identity DB_NAMESERVER_NAME -SourceServer ACTIVE_SERVER -DeleteExistingFiles -CatalogOnly
Or
– On the passive database servers stop the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller.”
– Delete the Existing CI catalog folder
– Start the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller” services.
– Run the command Update-MailboxDatabaseCopy -Identity DB_NAMESERVER_NAME -SourceServer SERVER_FROM_ABOVE -CatalogOnly (without the -DeleteExistingFiles switch)
Again thanks in advance for your help.
Thanks You so much, it worked ike a charm, the cause was a failed veeam backup.
Hi. Our database is also getting a passive failed on the secondary but when I get the mailbox database status, it says failed but the context index health is healthy. I even tried to reseed the db and cant seem to have the db replicate. Not sure what to do next. Please help.
If the database copy on the server keeps failing you need to look at the event logs on that server to find out more.
Pingback: Fix Failed Database Content Index for Exchange Server 2013
Is the syntax also compatible in windows 8.I am really confused with powershell syntax.
Yes.
We had an issue in our Exchange 2013 environment where content indexes would seemingly randomly get corrupted when servers were rebooted. It turns out the McAfee clients on the servers had an access protection policy configured to block IRC ports 6666-6688. Many Exchange processes including the search service were using ports in that range which caused a communication issue. Once the policy was removed the indexes were repaired.
Hi, I moved a database to another drive on the same server and accidentally renamed whilst changing the path. Once the move was completed a copy of the database with the new name was created however the original database still remained. When I check the database properties the path is still pointing to the original location. My problem is now I cannot mount the database. I am novice to exchange and any help would be very much appreciated.
Hi Paul,
we are using Exchange 2013(cu5) 3 node DAG, content index got failed in one of the database(passive and DR). I have tried all the recommended solution but none of them saves me.
we have reseeded content index file, created content submitter groups..
still it showing “Failled”, please help me..
Thanks a lot, Paul. You are a life saver.
Paul, in our updated Outlook 2010 we constantly get complaints that our “Search inbox” is not working, and we have to log into our 2 Exchange servers to restart ” Microsoft Exchange Index” and this usually takes time to replicate. any assistance would be greatly appreciated.
We have 2 Exchange servers in our primary site and 2 Exchange servers in our DR site. When I run “Get-MailboxDatabaseCopyStatus” command everything showing up as “Healthy” except “ContentIndexState” on servers in DR site for 1 database showing ups as “Crawling” and it never switches to “Healthy” state. I completely rebuild Content Index directory on the server in primary site and did the same thing on the server in the DR site, but it still showing up as “Crawling.” This state prevents database failover. Is there anything can be done? I wasn’t able to find a way to monitor “Crawling” process, so I’m not sure if it is actually going but slow or just hanging. I tried enabling perfmon to see if there are any activity under “Full Crawl Mode Status” but I get nothing there either. Any help would be greatly appreciated.
Paul, we are have a 2 DAG setup. For some reason the active node got dismounted and passive node is in Failed and Suspended state. I’ve manually removed all transact logs from both databases and copied .edb file from dismounted node to the passive node however I can’t get database mounted 🙁 Can you please help ?? I’ve been working on it all day and ready to pull my hair out
Whoa… not sure I agree with the actions you’ve already taken there.
I suggest you contact Microsoft Support ASAP.
If left without treatment
What will happen ?
Anywhere that you have the management tools installed.
Do I need to run this command in content Indexing failed server or I can rum from any mailbox server ? kindly suggest.
I tried your steps but I get an error on all my DBs:
Database ‘ManagementVEXCH01’ has only one copy. This task is supported only for databases that have more than one copy.
Hello,
Thank you for your advices! I tried your script to fix all the database but nothing happen. It just asks me if i want to apply it to all the database (i say yes for all) and after it does nothing. Do you have an idea?
Thanks.
Paul, you are a legend.
you make us look good!
Thanks.
I have 8 mailbox database copies that show a “FailedAndSuspended” state. I have a total of 36 databases. All but these 8 are healthy. I have tried your procedures and scripts but these 8 still show as FailedAndSuspended”. Is there anything else to try before opening a ticket with Microsoft?
We have to run this script many times per day, just to keep our content indexes healthy. Any tips or ideas to look into as to why they are failing so often?
Worked perfectly. THANK YOU!
Thanks mate, did just what I needed 🙂
Paul,
I enjoy looking through and using the scripts you have put together to better monitor the health of our DAG. I put this script into production today and very much appreciate what you do!
Thanks!
This will seed the catalog from the active database. If your active server is located at a different location for business continuity planning then your WAN will be heavily used.
You can even script it as I’ve done to run every hour or so, and add the failed and suspended status as well:
Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus | where {$_.Status -eq “FailedandSuspended”}| Resume-MailboxDatabaseCopy
Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus | where {$_.Status -eq “Suspended”}| Resume-MailboxDatabaseCopy
Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus | where {$_.Status -eq “Failed”}| Resume-MailboxDatabaseCopy
Hi Paul,
Just curious on what causes these content indexes to fail?
We have a fairly standard exchange 2010 DAG /CAS setup running on 2008r2 VMs
I seem to be fixing content index issues weekly…
HI Paul,
I am using Exchange Server 2013 CU1 installed on Windows Server 2008 Standard R2.
Everything was running fine till yesterday but since morning Every user is facing problem while searching email on Outlook Web App. They are getting “An Error Occurred while searching. Please try Again” Error.
It is also Showing Content index State failed.
What to do? Please help..
I have one database that has Content Index State of “unknown”. I cannot run the CatalogOnly as it fails “Seeding of index catalog for database DB1 has failed. Please verify the Microsoft Search (Exchange) and the Host Controller service for Exchange services are running and try again.”
Yes both services are running and I even tried to restart Microsoft Exchange Search..
I have found the Content Index ‘Unknown’ state while Exchange backup job was running (Symantec Backup Exec). Once backup job completed, Content Index state back to ‘Healthy’.
I am using exchange Server 2013.
Do you need to suspend the database copy first before running update-mailboxdatabasecopy?
Not if you’re just fixing failed content indexes.
Thanks for the tip Paul, this tip along with a couple of others provided by you have really helped us out!
Worked like a charm, thanks for the script Paul! 🙂