One of the issues that my Get-DAGHealth.ps1 script alerts for is failed content indexes on database copies in a database availability group.

Note: for failed content indexes on servers that are not DAG members refer to this article instead.

Failed content indexes can easily go unnoticed when everything else is working fine however they will eventually begin to cause problems for you, for example by preventing database switchovers.

Fixing a single failed content index is easy, but if there are multiple failed indexes you can speed things up a little by fixing them all with a single PowerShell command.

To demonstrate, here is the Exchange Management Shell command for using Get-MailboxDatabaseCopyStatus to display all database copies that have a content index in a failed state.

[PS] C:\>Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq "Failed"}

Name                               Status     CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                   Length     Length                                       State
----                               ------     --------- ----------- --------------------   ------------
Mailbox Database 1HO-EX2010-MB1   Healthy    0         0           22/02/2013 10:43:26 AM Failed
Mailbox Database 2HO-EX2010-MB1   Healthy    0         0           22/02/2013 10:41:36 AM Failed

So all we need to do is pipe the output of that command into the Update-MailboxDatabaseCopy cmdlet with the -CatalogOnly switch.

[PS] C:\>Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq "Failed"} | Update-MailboxDatabaseCopy -CatalogOnly

Wait for that process to complete, then re-run the first command again. If no results are returned this time then there are no more failed content indexes.

About the Author

Paul Cunningham

Paul is a former Microsoft MVP for Office Apps and Services. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server. Paul no longer writes for Practical365.com.

Comments

  1. chris claydon

    Once again TOP Stuff… very helpful!

    Thxs Man

  2. Kr. Donch

    You should run thin cmdlet:
    Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq “Failed” -and $_.Status -eq “Healthy”} | Update-MailboxDatabaseCopy -CatalogOnly

    Because content indexing can be reseeded only for passive nodes.

  3. ali zafar

    Dear Paul
    i have lost active manager exch1 DAG exchnage 2013 due to power failure but when i start other dag member exch2 server all dag database are disconnected and failed state.While active manager exch1 server database showing service down state but its permanently lost.
    So how can i make active manager to exch2 and makes database mounted ? no issue if lost some data i knew copy queue lenghth is not zero.please need advice

  4. Simon Powell

    If you have just run exchange 2016 CU 19 update and your getting this unhealthy content index count , then I suggest you look at the exchange service Microsoft Exchange Search Host Controller it will now be disabled ( after the update ). Enable and start the service I checked on one of my unpatched servers and it was set to enabled and start automatically and run

    Get-MailboxDatabaseCopyStatus * | where {$_.ContentIndexState -eq “Failed”} | Update-MailboxDatabaseCopy -CatalogOnly

    Your reading this and will probably save 3 hours of your life not like me.

    1. Dan E

      Just ran the Exchange 2016 CU19 update for the March 2, 2021 fix. Same issue with the Search Host service disabled. Re-enabling and starting took care of the failed state.

      Thanks for the heads up!

    2. Andrew Anthony

      Exactly the same here too, thanks Simon for the pointer, totally saved me loads of time. Top man! Nothing about that CU19 upgrade was simple for me from start to finish!

    3. Waqas Hashmi

      Thanks Simon,

      I applied CU23 (KB5001779) to my Exchange 2013 and it disabled “Microsoft Exchange Search Host Controller” and also corrupted IIS.

      Regards

    4. Paul Webb

      I don’t know you or what country you are from but if you ever come to Essex, UK then I owe you a beer!!! 🙂

      Saved me a lot of grief

  5. Sheffy

    Hi Paul,

    We have exchange DAG with two servers and ContentIndexState on both the servers are in “FailedAndSuspended” State.

    Can i use the command listed above to repair?

  6. Stephen Wilkins

    I tried running this on my failed statuses but get this error:

    Database ‘DB1\SERVER’ has only one copy. This task is supported only for databases that have more than one copy. It happens for all my DB’s.

    Any suggestions?

  7. Ronen Adler

    sorry for the double posting (i submitted a wrong email address)
    please note, the correct email is: matanmail@gmail.com

    thank you

  8. Ronen Adler

    Hi Paul, hope you can help me out here,
    i have a 3 server dag 2013
    all db’s are OK except for one passive copy which always stays in Suspend state on a specific serer (“contentindexstate suspended”) no matter what i do
    i reseeded, removed and re-created, tried to fix it (update-mailboxdatabasecopy catalog only)…tried them alll
    still this DB of all….is in healthy but suspended state….i don’t have any more idease for what can cause this.
    please see if you can get some…
    thank you

  9. Ronen Adler

    Hi Paul, hope you can help me out here,
    i have a 3 server dag 2013
    all db’s are OK except for one passive copy which always stays in Suspend state on a specific serer (“contentindexstate suspended”) no matter what i do
    i reseeded, removed and re-created, tried to fix it (update-mailboxdatabasecopy catalog only)…tried them alll
    still this DB of all….is in healthy but suspended state….i don’t have any more idease for what can cause this.
    please see if you can get some…
    thank you

  10. Alphonse

    Hi Paul
    I have a DAG node that still has all its exchange services running but its local database (not replicated ) is dismounted. The replicated db are still healthy. Unfortunately by mistake, two users mailbox were created in the dismounted local database. There is no way to move them out of the dismounted database. How would I fix the Failed content state of a standalone database that prevents it from being mounted? Thanks for your help.

  11. Nobby

    Poul, what command if the mailbox database has a content indexing status of Unknown? I’am confused here. Same command? Any advice is very welcome. Best regards.

  12. Kapardhi

    Awesome, we have faced the same problem and this article helps to resolve the issue. saved lot of times, Excellent explanation Paul.

  13. Amit Bhatta

    Well..after restarting the passive node today, i noticed that ‘content index state’ is appearing ‘Failed’ for all 8 databases. Followed your suggested method. It did not give any error but it did not fix the problem either. Then finally I just restarted the following services: First “Microsoft Exchange Search” then “Microsoft Exchange Search Host Controller” and within 2 minutes or so all the databases started showing the “Healthy” state. No idea what fixed the problem. But it did. Thanks.

  14. Leo Hooghart

    Hi Paul,
    We had issue with a extreme fast growing Index folder on one of our database servers in a 2 node DAG cluster.
    Any idea what triggers a extreme fast growing index? 20 GB in 1 hour..

    Kind regards,

    Leo

  15. Dan Spahn

    Hi Paul,
    We have an Exchange organization of three servers and have a DAG set up for seven databases. Since upgrading to Exchange 2013 from 2010, we have a problem with content indices randomly failing- They seem to go bad about 2-3 per week. I keep fixing them but it seems like its always a different index failing. All servers are running the most current CU, 2 of them are EE and one is SE. The problem began before the two were upgraded from SE to EE. Any ideas?

    1. Avatar photo
      Paul Cunningham

      Incorrectly configured antivirus exclusions is a common cause of index failures.

  16. Ezra Omoke

    Hi paul, My OWA search are not working in all mail i have tried all the options to rebuild indexing but not successful it still says failed what shall i do?

    i have tried i have stopped the search services and deleted the GUID folder yet indexing still fails. i have one DB of 256GB kindly assist with other ways to solve this issue

    1. Avatar photo
      Paul Cunningham

      Have you configured the recommended file system antivirus exclusions on your Exchange servers?

  17. leew

    OWA does not work either…ok will log a call.

    Thanks Paul

  18. leew

    Thanks for replying..

    Just checked as below

    Live server – Status is Mounted – ContentIndexState is Healty
    Passive server – Status is Healthy – ContentIndexState is Healthy

    1. Avatar photo
      Paul Cunningham

      If you’ve got health indexes, and both Outlook and OWA search are not working, then I suggest opening a support case with Microsoft so they can look at it with you.

  19. leew

    Hi Paul – top post as always, could you provide some pointers with a problem we have? We have exchange 2010 in a DAG (active / passive) on exchange 2008 R2. After a reboot of our mailbox servers we often have issues with the Outlook search function stops working for everyone in the organization. Generally it stops working between the hours of 7.30/8.00am and starts working again around 11.15am-12pm without us doing anything!? I have ran various things as below..

    After x (sometimes days) amount of time, reboots of servers / switching of passive – active DB’s it starts working on its own!

    Moved my Mailbox to a DB that is not in the DAG – Search works all the time
    Switched All DB’s to passive mailbox server – no better
    Ran this on each DB – New-MailboxRepairRequest -Database “dbx” -CorruptionType AggregateCounts,ProvisionedFolder,SearchFolder,FolderView -DetectOnly – no better
    Ran this on both passive and active servers .\Troubleshoot-CI.ps1 -Server name – no better
    Ran this on each DB \Troubleshoot-CI.ps1 -database DBx -Action DetectAndResolve – found no errors

    Any pointers would be appreciated!

  20. Andreas Engkvist

    Thanks. Saved my ass..

  21. Jim Blunt

    Paul,

    First off, EXCELLENT command! Works like a charm. Unfortunately, I have an issue that I’m hoping you can help me with.

    We have Data Centers at different locations around the world and replicating the active index from the active DB, may cause us to have to replicate from Singapore to Chicago, for instance. Because of that, it takes FOREVER to run this command (literally all day) on 10 indexes.

    Is there a way to replicate from a healthy passive copy of the index, in the same DC?

    Thanks mate!

  22. Mobin

    Hi Paul,

    I’m running Exchanging 2013 with two mailbox servers in a DAG. Get-MailboxDatabaseCopyStatus * showed both servers in FailedAndSuspended state. I think this issue occurred after I extended disk space on the mailbox servers. I tried Update-MailboxDatabaseCopy -CatalogOnly but that didn’t help I presume as there was no Heatlty copy present.

    On the passive node I stopped Exchange Search and Exchange Host Controller service, renamed .single to .single.old and started the service again which created a new folder. But state was still showing FailedAndSuspended, so I tried to do same on the Active node and upon service start status was changed to Crawling on active node and passive was still showing FailedAndSuspended.

    It took more than 24 hours and status was changed from Crawling to Failed on Active node. Then again on the passive node I stopped and started Exchange Search and Exchange Host Controller service, after a while status was changed from Failed to Crawling and then to Healthy. Once status was healthy then I ran Hi Paul,

    I’m running Exchanging 2013 with two mailbox servers in a DAG. Get-MailboxDatabaseCopyStatus * showed both servers in FailedAndSuspended state. I think this issue occurred after I extended disk space on the mailbox servers. I tried Update-MailboxDatabaseCopy -CatalogOnly but that didn’t help I presume as there was no Heatlty copy present.

    On the passive node I stopped Exchange Search and Exchange Host Controller service, renamed .single to .single.old and started the service again which created a new folder. But state was still showing FailedAndSuspended, so I tried to do same on the Active node and upon service start status was changed to Crawling on active node and passive was still showing FailedAndSuspended.

    It took more than 24 hours and status was changed from Crawling to Failed on Active node. Then again on the passive node I stopped and started Exchange Search and Exchange Host Controller service, after a while status was changed from Failed to Crawling and then to Healthy. Once status was healthy then I ran MailboxDatabaseCopy -CatalogOnly which made the active node healthy as well.

    Now the issue which I’m facing is that search still isn’t working other than recent emails (1 day old emails)

    .single.old size is showing as 30 GB while new folder .single is showing only 6 GB. Is there any possible solution I could fix the search issue?

    Could you please help me to resolve this.

    Regards,

    Mobin

    1. Hisham A

      Hi Mobin,

      im facing the same issue you have, did you manage to solve it ?
      thanks,

  23. Royal Wang

    sorry, wrongly reply to you. additonal this comand works .

  24. Royal Wang

    i have edited SMTP but not working. please give me any hints

    $smtpsettings = @{
    To = “40802478@qq.com”
    From = “wyh33621999@126.com”
    Subject = “$reportemailsubject – $now”
    SmtpServer = “smtp.126.com”
    }

  25. Richard

    Hi Paul,

    Hope you can help me, we are running a Exchange 2010 DAG group for our exchange archives with 8 databases.

    2 of our databases have a Failed index state on both copies if the database. if I remove the catalog and start the search services again, status changes to crawling but after 2 hours status is failed again.

    databases are in clean shut down state,

    prior to changing back to Failed state the following eventlog entry is showing;

    MSSearch service signaled to pause indexing for database . FTERequestStatus = ForcePause, FTERequestStatusChangeReason = ResetNeeded. Indexing on this catalog is being paused. It will be resumed once a resume signal is received.

    right after that the status changes to Failed.

  26. Fajar Pambudi

    How about exchange 2013 mailbox database copy status is keep resynchronizing??

  27. Weili

    Thanks for the script!

  28. Dan Burchfield

    Worked great as always. Thanks Paul.

  29. Matt Steele

    I have an issue whereby at random intervals (maybe once a week) one of our servers will get hammered by noderunner.exe using up to 15-20GB RAM and very high CPU usage. I had a call with Microsoft the last time it happened and they said to install CU11 as it had some fixes for high CPU noderunner issues but this hasn’t seemed to help.

    The next course of action that they suggested was to rebuild the indexes but they suggested a slightly different process. They said to remove all DB copies from the other DAG members, then stop the search service, delete the CI and once completed add the DB copies back onto the other DAG members.

    This seems like a bit of an extreme fix and I was just wondering if in your opinion this is required or whether just running the process that you describe above will work?

    Many thanks in advance.

    1. Avatar photo
      Paul Cunningham

      AFAIK there’s no need to remove all DB copies to rebuild an index. Removing just the index files from each DAG member should be sufficient.

      1. Matt Steele

        Brilliant thank you so much for your help. So can i confirm the process would be as below

        – On the active database server stop the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller.”
        – Delete the Existing CI catalog folder
        – Start the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller” services.
        – Once the DB catalog has been rebuilt log onto each of the other servers that has a DB copy and run the following command

        At this point do i then

        – Run the command Update-MailboxDatabaseCopy -Identity DB_NAMESERVER_NAME -SourceServer ACTIVE_SERVER -DeleteExistingFiles -CatalogOnly

        Or

        – On the passive database servers stop the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller.”
        – Delete the Existing CI catalog folder
        – Start the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller” services.
        – Run the command Update-MailboxDatabaseCopy -Identity DB_NAMESERVER_NAME -SourceServer SERVER_FROM_ABOVE -CatalogOnly (without the -DeleteExistingFiles switch)

        Again thanks in advance for your help.

  30. Bugnon Laurent

    Thanks You so much, it worked ike a charm, the cause was a failed veeam backup.

  31. Ryan Favila

    Hi. Our database is also getting a passive failed on the secondary but when I get the mailbox database status, it says failed but the context index health is healthy. I even tried to reseed the db and cant seem to have the db replicate. Not sure what to do next. Please help.

    1. Avatar photo
      Paul Cunningham

      If the database copy on the server keeps failing you need to look at the event logs on that server to find out more.

  32. PeterMyagkov

    Is the syntax also compatible in windows 8.I am really confused with powershell syntax.

  33. Thomas Perry

    We had an issue in our Exchange 2013 environment where content indexes would seemingly randomly get corrupted when servers were rebooted. It turns out the McAfee clients on the servers had an access protection policy configured to block IRC ports 6666-6688. Many Exchange processes including the search service were using ports in that range which caused a communication issue. Once the policy was removed the indexes were repaired.

  34. John Loi

    Hi, I moved a database to another drive on the same server and accidentally renamed whilst changing the path. Once the move was completed a copy of the database with the new name was created however the original database still remained. When I check the database properties the path is still pointing to the original location. My problem is now I cannot mount the database. I am novice to exchange and any help would be very much appreciated.

  35. Dineshbabu

    Hi Paul,
    we are using Exchange 2013(cu5) 3 node DAG, content index got failed in one of the database(passive and DR). I have tried all the recommended solution but none of them saves me.
    we have reseeded content index file, created content submitter groups..
    still it showing “Failled”, please help me..

  36. John Sam George

    Thanks a lot, Paul. You are a life saver.

  37. Benito 34

    Paul, in our updated Outlook 2010 we constantly get complaints that our “Search inbox” is not working, and we have to log into our 2 Exchange servers to restart ” Microsoft Exchange Index” and this usually takes time to replicate. any assistance would be greatly appreciated.

  38. Igor P

    We have 2 Exchange servers in our primary site and 2 Exchange servers in our DR site. When I run “Get-MailboxDatabaseCopyStatus” command everything showing up as “Healthy” except “ContentIndexState” on servers in DR site for 1 database showing ups as “Crawling” and it never switches to “Healthy” state. I completely rebuild Content Index directory on the server in primary site and did the same thing on the server in the DR site, but it still showing up as “Crawling.” This state prevents database failover. Is there anything can be done? I wasn’t able to find a way to monitor “Crawling” process, so I’m not sure if it is actually going but slow or just hanging. I tried enabling perfmon to see if there are any activity under “Full Crawl Mode Status” but I get nothing there either. Any help would be greatly appreciated.

  39. alex

    Paul, we are have a 2 DAG setup. For some reason the active node got dismounted and passive node is in Failed and Suspended state. I’ve manually removed all transact logs from both databases and copied .edb file from dismounted node to the passive node however I can’t get database mounted 🙁 Can you please help ?? I’ve been working on it all day and ready to pull my hair out

    1. Avatar photo
      Paul Cunningham

      Whoa… not sure I agree with the actions you’ve already taken there.

      I suggest you contact Microsoft Support ASAP.

  40. Josh

    If left without treatment
    What will happen ?

  41. Anil

    Do I need to run this command in content Indexing failed server or I can rum from any mailbox server ? kindly suggest.

  42. Chris Darrow

    I tried your steps but I get an error on all my DBs:

    Database ‘ManagementVEXCH01’ has only one copy. This task is supported only for databases that have more than one copy.

  43. Fabian

    Hello,

    Thank you for your advices! I tried your script to fix all the database but nothing happen. It just asks me if i want to apply it to all the database (i say yes for all) and after it does nothing. Do you have an idea?

    Thanks.

  44. Phillip

    Paul, you are a legend.

    you make us look good!

    Thanks.

  45. Barry Sherman

    I have 8 mailbox database copies that show a “FailedAndSuspended” state. I have a total of 36 databases. All but these 8 are healthy. I have tried your procedures and scripts but these 8 still show as FailedAndSuspended”. Is there anything else to try before opening a ticket with Microsoft?

  46. Daniel B

    We have to run this script many times per day, just to keep our content indexes healthy. Any tips or ideas to look into as to why they are failing so often?

  47. Steve Graham

    Worked perfectly. THANK YOU!

  48. Leigh

    Thanks mate, did just what I needed 🙂

  49. Ken

    Paul,

    I enjoy looking through and using the scripts you have put together to better monitor the health of our DAG. I put this script into production today and very much appreciate what you do!

    Thanks!

  50. Toan

    This will seed the catalog from the active database. If your active server is located at a different location for business continuity planning then your WAN will be heavily used.

  51. Edwin

    You can even script it as I’ve done to run every hour or so, and add the failed and suspended status as well:

    Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus | where {$_.Status -eq “FailedandSuspended”}| Resume-MailboxDatabaseCopy
    Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus | where {$_.Status -eq “Suspended”}| Resume-MailboxDatabaseCopy
    Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus | where {$_.Status -eq “Failed”}| Resume-MailboxDatabaseCopy

  52. Deni

    Hi Paul,

    Just curious on what causes these content indexes to fail?
    We have a fairly standard exchange 2010 DAG /CAS setup running on 2008r2 VMs

    I seem to be fixing content index issues weekly…

  53. Manoj

    HI Paul,

    I am using Exchange Server 2013 CU1 installed on Windows Server 2008 Standard R2.

    Everything was running fine till yesterday but since morning Every user is facing problem while searching email on Outlook Web App. They are getting “An Error Occurred while searching. Please try Again” Error.
    It is also Showing Content index State failed.

    What to do? Please help..

  54. David

    I have one database that has Content Index State of “unknown”. I cannot run the CatalogOnly as it fails “Seeding of index catalog for database DB1 has failed. Please verify the Microsoft Search (Exchange) and the Host Controller service for Exchange services are running and try again.”

    Yes both services are running and I even tried to restart Microsoft Exchange Search..

    1. Nazeef

      I have found the Content Index ‘Unknown’ state while Exchange backup job was running (Symantec Backup Exec). Once backup job completed, Content Index state back to ‘Healthy’.
      I am using exchange Server 2013.

  55. Thomas

    Do you need to suspend the database copy first before running update-mailboxdatabasecopy?

  56. SimonF

    Thanks for the tip Paul, this tip along with a couple of others provided by you have really helped us out!

  57. Keith K.

    Worked like a charm, thanks for the script Paul! 🙂

Leave a Reply