Home » Exchange Server » PowerShell Tip: Fix All Failed Exchange Database Content Indexes

PowerShell Tip: Fix All Failed Exchange Database Content Indexes

One of the issues that my Get-DAGHealth.ps1 script alerts for is failed content indexes on database copies in a database availability group.

Note: for failed content indexes on servers that are not DAG members refer to this article instead.

Failed content indexes can easily go unnoticed when everything else is working fine however they will eventually begin to cause problems for you, for example by preventing database switchovers.

Fixing a single failed content index is easy, but if there are multiple failed indexes you can speed things up a little by fixing them all with a single PowerShell command.

To demonstrate, here is the Exchange Management Shell command for using Get-MailboxDatabaseCopyStatus to display all database copies that have a content index in a failed state.

So all we need to do is pipe the output of that command into the Update-MailboxDatabaseCopy cmdlet with the -CatalogOnly switch.

Wait for that process to complete, then re-run the first command again. If no results are returned this time then there are no more failed content indexes.

Paul is a Microsoft MVP for Office Servers and Services. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server. Paul is a co-author of Office 365 for IT Pros and several other books, and is also a Pluralsight author.
Category: Exchange Server

45 comments

  1. David says:

    I have one database that has Content Index State of “unknown”. I cannot run the CatalogOnly as it fails “Seeding of index catalog for database DB1 has failed. Please verify the Microsoft Search (Exchange) and the Host Controller service for Exchange services are running and try again.”

    Yes both services are running and I even tried to restart Microsoft Exchange Search..

    • Nazeef says:

      I have found the Content Index ‘Unknown’ state while Exchange backup job was running (Symantec Backup Exec). Once backup job completed, Content Index state back to ‘Healthy’.
      I am using exchange Server 2013.

  2. Manoj says:

    HI Paul,

    I am using Exchange Server 2013 CU1 installed on Windows Server 2008 Standard R2.

    Everything was running fine till yesterday but since morning Every user is facing problem while searching email on Outlook Web App. They are getting “An Error Occurred while searching. Please try Again” Error.
    It is also Showing Content index State failed.

    What to do? Please help..

  3. Deni says:

    Hi Paul,

    Just curious on what causes these content indexes to fail?
    We have a fairly standard exchange 2010 DAG /CAS setup running on 2008r2 VMs

    I seem to be fixing content index issues weekly…

  4. Edwin says:

    You can even script it as I’ve done to run every hour or so, and add the failed and suspended status as well:

    Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus | where {$_.Status -eq “FailedandSuspended”}| Resume-MailboxDatabaseCopy
    Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus | where {$_.Status -eq “Suspended”}| Resume-MailboxDatabaseCopy
    Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus | where {$_.Status -eq “Failed”}| Resume-MailboxDatabaseCopy

  5. Toan says:

    This will seed the catalog from the active database. If your active server is located at a different location for business continuity planning then your WAN will be heavily used.

  6. Ken says:

    Paul,

    I enjoy looking through and using the scripts you have put together to better monitor the health of our DAG. I put this script into production today and very much appreciate what you do!

    Thanks!

  7. Daniel B says:

    We have to run this script many times per day, just to keep our content indexes healthy. Any tips or ideas to look into as to why they are failing so often?

  8. Barry Sherman says:

    I have 8 mailbox database copies that show a “FailedAndSuspended” state. I have a total of 36 databases. All but these 8 are healthy. I have tried your procedures and scripts but these 8 still show as FailedAndSuspended”. Is there anything else to try before opening a ticket with Microsoft?

  9. Fabian says:

    Hello,

    Thank you for your advices! I tried your script to fix all the database but nothing happen. It just asks me if i want to apply it to all the database (i say yes for all) and after it does nothing. Do you have an idea?

    Thanks.

  10. Chris Darrow says:

    I tried your steps but I get an error on all my DBs:

    Database ‘ManagementVEXCH01’ has only one copy. This task is supported only for databases that have more than one copy.

  11. alex says:

    Paul, we are have a 2 DAG setup. For some reason the active node got dismounted and passive node is in Failed and Suspended state. I’ve manually removed all transact logs from both databases and copied .edb file from dismounted node to the passive node however I can’t get database mounted 🙁 Can you please help ?? I’ve been working on it all day and ready to pull my hair out

  12. Igor P says:

    We have 2 Exchange servers in our primary site and 2 Exchange servers in our DR site. When I run “Get-MailboxDatabaseCopyStatus” command everything showing up as “Healthy” except “ContentIndexState” on servers in DR site for 1 database showing ups as “Crawling” and it never switches to “Healthy” state. I completely rebuild Content Index directory on the server in primary site and did the same thing on the server in the DR site, but it still showing up as “Crawling.” This state prevents database failover. Is there anything can be done? I wasn’t able to find a way to monitor “Crawling” process, so I’m not sure if it is actually going but slow or just hanging. I tried enabling perfmon to see if there are any activity under “Full Crawl Mode Status” but I get nothing there either. Any help would be greatly appreciated.

  13. Benito 34 says:

    Paul, in our updated Outlook 2010 we constantly get complaints that our “Search inbox” is not working, and we have to log into our 2 Exchange servers to restart ” Microsoft Exchange Index” and this usually takes time to replicate. any assistance would be greatly appreciated.

  14. John Loi says:

    Hi, I moved a database to another drive on the same server and accidentally renamed whilst changing the path. Once the move was completed a copy of the database with the new name was created however the original database still remained. When I check the database properties the path is still pointing to the original location. My problem is now I cannot mount the database. I am novice to exchange and any help would be very much appreciated.

  15. Thomas Perry says:

    We had an issue in our Exchange 2013 environment where content indexes would seemingly randomly get corrupted when servers were rebooted. It turns out the McAfee clients on the servers had an access protection policy configured to block IRC ports 6666-6688. Many Exchange processes including the search service were using ports in that range which caused a communication issue. Once the policy was removed the indexes were repaired.

  16. Ryan Favila says:

    Hi. Our database is also getting a passive failed on the secondary but when I get the mailbox database status, it says failed but the context index health is healthy. I even tried to reseed the db and cant seem to have the db replicate. Not sure what to do next. Please help.

  17. Matt Steele says:

    I have an issue whereby at random intervals (maybe once a week) one of our servers will get hammered by noderunner.exe using up to 15-20GB RAM and very high CPU usage. I had a call with Microsoft the last time it happened and they said to install CU11 as it had some fixes for high CPU noderunner issues but this hasn’t seemed to help.

    The next course of action that they suggested was to rebuild the indexes but they suggested a slightly different process. They said to remove all DB copies from the other DAG members, then stop the search service, delete the CI and once completed add the DB copies back onto the other DAG members.

    This seems like a bit of an extreme fix and I was just wondering if in your opinion this is required or whether just running the process that you describe above will work?

    Many thanks in advance.

      • Matt Steele says:

        Brilliant thank you so much for your help. So can i confirm the process would be as below

        – On the active database server stop the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller.”
        – Delete the Existing CI catalog folder
        – Start the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller” services.
        – Once the DB catalog has been rebuilt log onto each of the other servers that has a DB copy and run the following command

        At this point do i then

        – Run the command Update-MailboxDatabaseCopy -Identity DB_NAMESERVER_NAME -SourceServer ACTIVE_SERVER -DeleteExistingFiles -CatalogOnly

        Or

        – On the passive database servers stop the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller.”
        – Delete the Existing CI catalog folder
        – Start the “Microsoft Exchange Search service” as well as the “Microsoft Exchange Host Controller” services.
        – Run the command Update-MailboxDatabaseCopy -Identity DB_NAMESERVER_NAME -SourceServer SERVER_FROM_ABOVE -CatalogOnly (without the -DeleteExistingFiles switch)

        Again thanks in advance for your help.

  18. Richard says:

    Hi Paul,

    Hope you can help me, we are running a Exchange 2010 DAG group for our exchange archives with 8 databases.

    2 of our databases have a Failed index state on both copies if the database. if I remove the catalog and start the search services again, status changes to crawling but after 2 hours status is failed again.

    databases are in clean shut down state,

    prior to changing back to Failed state the following eventlog entry is showing;

    MSSearch service signaled to pause indexing for database . FTERequestStatus = ForcePause, FTERequestStatusChangeReason = ResetNeeded. Indexing on this catalog is being paused. It will be resumed once a resume signal is received.

    right after that the status changes to Failed.

  19. Royal Wang says:

    i have edited SMTP but not working. please give me any hints

    $smtpsettings = @{
    To = “40802478@qq.com”
    From = “wyh33621999@126.com”
    Subject = “$reportemailsubject – $now”
    SmtpServer = “smtp.126.com”
    }

  20. Mobin says:

    Hi Paul,

    I’m running Exchanging 2013 with two mailbox servers in a DAG. Get-MailboxDatabaseCopyStatus * showed both servers in FailedAndSuspended state. I think this issue occurred after I extended disk space on the mailbox servers. I tried Update-MailboxDatabaseCopy -CatalogOnly but that didn’t help I presume as there was no Heatlty copy present.

    On the passive node I stopped Exchange Search and Exchange Host Controller service, renamed .single to .single.old and started the service again which created a new folder. But state was still showing FailedAndSuspended, so I tried to do same on the Active node and upon service start status was changed to Crawling on active node and passive was still showing FailedAndSuspended.

    It took more than 24 hours and status was changed from Crawling to Failed on Active node. Then again on the passive node I stopped and started Exchange Search and Exchange Host Controller service, after a while status was changed from Failed to Crawling and then to Healthy. Once status was healthy then I ran Hi Paul,

    I’m running Exchanging 2013 with two mailbox servers in a DAG. Get-MailboxDatabaseCopyStatus * showed both servers in FailedAndSuspended state. I think this issue occurred after I extended disk space on the mailbox servers. I tried Update-MailboxDatabaseCopy -CatalogOnly but that didn’t help I presume as there was no Heatlty copy present.

    On the passive node I stopped Exchange Search and Exchange Host Controller service, renamed .single to .single.old and started the service again which created a new folder. But state was still showing FailedAndSuspended, so I tried to do same on the Active node and upon service start status was changed to Crawling on active node and passive was still showing FailedAndSuspended.

    It took more than 24 hours and status was changed from Crawling to Failed on Active node. Then again on the passive node I stopped and started Exchange Search and Exchange Host Controller service, after a while status was changed from Failed to Crawling and then to Healthy. Once status was healthy then I ran MailboxDatabaseCopy -CatalogOnly which made the active node healthy as well.

    Now the issue which I’m facing is that search still isn’t working other than recent emails (1 day old emails)

    .single.old size is showing as 30 GB while new folder .single is showing only 6 GB. Is there any possible solution I could fix the search issue?

    Could you please help me to resolve this.

    Regards,

    Mobin

Leave a Reply

Your email address will not be published. Required fields are marked *