In the comments of my blog post about repairing failed content indexes, Tipza asks:
How do you monitor the status of this rebuild?
To answer the question, here's an excerpt from the Exchange Server Troubleshooting Companion.
When the content index for a database has become corrupt, it will need to be rebuilt, or reseeded from another database copy in the DAG. For now, let's look at the process for a non-DAG Mailbox server, and demonstrate the different procedure for DAGs later in this chapter.
This process involves removing the existing content index files, which will trigger Exchange Search to re-index that database. The re-indexing process can cause a high load on the Exchange server, which may impact performance for the server. So you should carefully consider the timing of any content index rebuilds, and how it might impact your end users. The content index files are located in the same path as the database EDB file, in a sub-folder named with a GUID.
Before the corrupt index files can be removed, the Exchange Search services must be stopped. While these services are stopped, searches in OWA will not be able to be performed by end users, and all of the database content indexes on the server will be reported as “Failed” by Get-MailboxDatabaseCopyStatus.
|
1 |
PS C:\> Invoke-Command -ComputerName EX2013SRV1 {Stop-Service MSExchangeFastSearch; Stop-Service HostControllerService} |
Next, delete the GUID-named folder that contains the content index files. If the folder will not delete due to files in use, then it's likely that either:
- You haven't stopped the correct search services
- Another process, such as file-level anti-virus software, has a lock on the folder (and may be the cause of the index corrupting to begin with)
After deleting the files, start the search services again.
|
1 |
PS C:\> Invoke-Command -ComputerName EX2013SRV1 {Start-Service MSExchangeFastSearch; Start-Service HostControllerService} |
After a delay while Exchange Search evaluates the situation, the database will be re-indexed. The content index will have a state of “Crawling” while this is occurring.
|
1 2 3 4 5 6 7 8 |
[PS] C:\>Get-MailboxDatabaseCopyStatus -Server EX2013SRV1 | ft -auto Name Status CopyQueueLength ReplayQueueLength LastInspectedLogTime ContentIndexState ---- ------ --------------- ----------------- -------------------- ----------------- DB01EX2013SRV1 Mounted 0 0 Crawling DB02EX2013SRV1 Mounted 0 0 Healthy DB03EX2013SRV1 Mounted 0 0 Healthy DB04EX2013SRV1 Mounted 0 0 Healthy |
You can monitor the progress of the database crawl by watching the MSExchange Search IndexesCrawler: Mailboxes Remaining counter in Performance Monitor for that database instance.
Paul is a Microsoft MVP for Office Apps and Services and a Pluralsight author. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server.
Once I had to run this rebuilding process for some bigger databases in a customers Exchange environment. The rebuilding took about 1-1.5 days per database.
If you don’t need to fully rebuild the index, you can update it from a non-corrupted database copy by:
Update-MailboxDatabaseCopy -Identity DATABASE-NAMEDATABASE-NAME -CatalogOnly
That’s way faster. Sadly I could use that command only for a few databases, because in most of the cases all database copies had damaged search indexes.
I also wonder why Microsoft removed the “ResetSearchIndex.ps1” which still existed in Exchange 2010.
I migrated my Database and Log folder to an external drive because of my C: is too small, and now my index is failed. I’ve done the steps above twice and it failed both times. Everything is working as far as I know except we can’t search for anything (which is kind of a big deal apparently). Am I missing something?
Is there a way to show when last index rebuild has occured ?
You could search the admin audit log for use of the cmdlet that starts a rebuild.
Before I go dropping $35 on your (I’m sure excellent) troubleshooting ebook Paul, is the following scenario covered in it? Our Exchange infrastructure has gotten itself into a pickle: two server DAG, both servers up and happy, but the production mail database (DB1) stopped replicating for a while, so I tried manually reseeding it, and that’s when I noticed that something was wrong. We now have the primary server (EX1) serving ProdDB-1 as “healthy”, but with failed content indexing, and the secondary (EX2) refuses to reseed at all until the content indexes on EX1 are healthy. When I check the CI “health” on EX1, Get-MailboxDatabaseCopyStatus reports “Catalog is dismounted externally for database” (the catalog folder was removed by ResetSearchIndex.ps1 – and the indexes haven’t been recreated at all)
The book is written to teach people how things work, and where to look when they break, so that they become better problem solvers. There are some specific tips and solutions in there, but I don’t think your exact case is covered. Sounds to me like you’ve already done some troubleshooting and that a support case with Microsoft would be a good next step for you.
Paul, could you post the procedure for deleting the corrupt index files and rebuilding them for DAG members?
How do you get that database index to show up in the performance monitor? I open performance monitor and it show s last, average minimum – no idea what it is trying to tell me but it doesn’t look like yours