In the comments of my blog post about repairing failed content indexes, Tipza asks:
How do you monitor the status of this rebuild?
To answer the question, here's an excerpt from the Exchange Server Troubleshooting Companion.
When the content index for a database has become corrupt, it will need to be rebuilt, or reseeded from another database copy in the DAG. For now, let's look at the process for a non-DAG Mailbox server, and demonstrate the different procedure for DAGs later in this chapter.
This process involves removing the existing content index files, which will trigger Exchange Search to re-index that database. The re-indexing process can cause a high load on the Exchange server, which may impact performance for the server. So you should carefully consider the timing of any content index rebuilds, and how it might impact your end users. The content index files are located in the same path as the database EDB file, in a sub-folder named with a GUID.
Before the corrupt index files can be removed, the Exchange Search services must be stopped. While these services are stopped, searches in OWA will not be able to be performed by end users, and all of the database content indexes on the server will be reported as “Failed” by Get-MailboxDatabaseCopyStatus.
|
1 |
PS C:\> Invoke-Command -ComputerName EX2013SRV1 {Stop-Service MSExchangeFastSearch; Stop-Service HostControllerService} |
Next, delete the GUID-named folder that contains the content index files. If the folder will not delete due to files in use, then it's likely that either:
- You haven't stopped the correct search services
- Another process, such as file-level anti-virus software, has a lock on the folder (and may be the cause of the index corrupting to begin with)
After deleting the files, start the search services again.
|
1 |
PS C:\> Invoke-Command -ComputerName EX2013SRV1 {Start-Service MSExchangeFastSearch; Start-Service HostControllerService} |
After a delay while Exchange Search evaluates the situation, the database will be re-indexed. The content index will have a state of “Crawling” while this is occurring.
|
1 2 3 4 5 6 7 8 |
[PS] C:\>Get-MailboxDatabaseCopyStatus -Server EX2013SRV1 | ft -auto Name Status CopyQueueLength ReplayQueueLength LastInspectedLogTime ContentIndexState ---- ------ --------------- ----------------- -------------------- ----------------- DB01EX2013SRV1 Mounted 0 0 Crawling DB02EX2013SRV1 Mounted 0 0 Healthy DB03EX2013SRV1 Mounted 0 0 Healthy DB04EX2013SRV1 Mounted 0 0 Healthy |
You can monitor the progress of the database crawl by watching the MSExchange Search IndexesCrawler: Mailboxes Remaining counter in Performance Monitor for that database instance.
Paul is a Microsoft MVP for Office Apps and Services and a Pluralsight author. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server.
Once I had to run this rebuilding process for some bigger databases in a customers Exchange environment. The rebuilding took about 1-1.5 days per database.
If you don’t need to fully rebuild the index, you can update it from a non-corrupted database copy by:
Update-MailboxDatabaseCopy -Identity DATABASE-NAMEDATABASE-NAME -CatalogOnly
That’s way faster. Sadly I could use that command only for a few databases, because in most of the cases all database copies had damaged search indexes.
I also wonder why Microsoft removed the “ResetSearchIndex.ps1” which still existed in Exchange 2010.
I migrated my Database and Log folder to an external drive because of my C: is too small, and now my index is failed. I’ve done the steps above twice and it failed both times. Everything is working as far as I know except we can’t search for anything (which is kind of a big deal apparently). Am I missing something?
Is there a way to show when last index rebuild has occured ?
You could search the admin audit log for use of the cmdlet that starts a rebuild.
Before I go dropping $35 on your (I’m sure excellent) troubleshooting ebook Paul, is the following scenario covered in it? Our Exchange infrastructure has gotten itself into a pickle: two server DAG, both servers up and happy, but the production mail database (DB1) stopped replicating for a while, so I tried manually reseeding it, and that’s when I noticed that something was wrong. We now have the primary server (EX1) serving ProdDB-1 as “healthy”, but with failed content indexing, and the secondary (EX2) refuses to reseed at all until the content indexes on EX1 are healthy. When I check the CI “health” on EX1, Get-MailboxDatabaseCopyStatus reports “Catalog is dismounted externally for database” (the catalog folder was removed by ResetSearchIndex.ps1 – and the indexes haven’t been recreated at all)
The book is written to teach people how things work, and where to look when they break, so that they become better problem solvers. There are some specific tips and solutions in there, but I don’t think your exact case is covered. Sounds to me like you’ve already done some troubleshooting and that a support case with Microsoft would be a good next step for you.
Paul, could you post the procedure for deleting the corrupt index files and rebuilding them for DAG members?
How do you get that database index to show up in the performance monitor? I open performance monitor and it show s last, average minimum – no idea what it is trying to tell me but it doesn’t look like yours
By default it’s showing CPU % counter. You can uncheck that one.
You need to right click on performance counter in tree on the left, and select properties.
Then on the DATA tab you need to add Counters, scroll down in the list of counters until you see “MSEXchange Search Indexes”, under that category you select “Crawler: Mailboxes remaining”. Below that in instances, you select all the databases individually instead of total.
That’s it.
You can still adjust the scale of the graph in properties, of switch between different graph styles.
Hi Paul/everyone,
single on-premises server, needed to rebuild Content Index on 2 of my databases and renamed/removed the GUID folders of the databases while opening performance monitor to follow progress:
Countdown in Performance monitor on first database finished, then for another 30 minutes or so afterwards ContentIndexStatus (when checking Get-MailboxDatabaseCopyStatus) was “Crawling” before it went to “Healthy”.
Took another 7 hours before countdown in Performance Monitor on second database finished. However, status on this particular database is still “Crawling”, and it’s now like 16 hours since countdown in Performance monitor was done. Is this normal, or is something stuck? As far as I know there is no other way than what I’ve been doing to track progress?
Second database still “Crawling”. Been like 36 hours since Performance Monitor stopped counting down. I’ve read the content index rebuild can take a long time, and I wouldn’t worry if PM was still counting down. But this long with same status, when PM seems to think the process is finished? Can’t be normal?
not sure how active this blog is, but will leave another brief update in case anyone feel like replying: So it’s been 1 week, and the last database still has status crawling. So something must for sure be stuck.
I did all this several times and it never recreates the deleted folder (stop.start services, reboot whatever) and the state is Healthy. OWA search still fails.