On your Exchange 2016 databases you may encounter a situation in which the content index fails. A failed content index will impact Outlook on the web (OWA) users trying to search their mailbox contents, and can also cause database switchover and failover problems for you.
Failed content indexes can be seen in the output of Get-MailboxDatabaseCopyStatus.
[PS] C:\scripts\>Get-MailboxDatabaseCopyStatus * | sort name | Select name,status,contentindexstate Name Status ContentIndexState ---- ------ ----------------- DB05\EX2016SRV1 Mounted Healthy DB05\EX2016SRV2 Healthy FailedAndSuspended DB06\EX2016SRV1 Healthy Healthy DB06\EX2016SRV2 Mounted Healthy DB07\EX2016SRV1 Healthy Healthy DB07\EX2016SRV2 Mounted Healthy DB08\EX2016SRV1 Healthy Healthy DB08\EX2016SRV2 Mounted Healthy
For an Exchange 2016 server that is not a member of a database availability group the failed content index can be rebuilt using the following procedure. Note that rebuilding the content index may cause high CPU utilization on the server. If you have any concerns about performance impact you should perform this work at an off-peak time for your customer.
First, stop the search services on the server.
[PS] C:\>Stop-Service MSExchangeFastSearch [PS] C:\>Stop-Service HostControllerService
Next, open Windows Explorer and go to the folder that contains the files for that database. If you aren’t sure where that is located you can query the database attributes.
[PS] C:\>Get-MailboxDatabase DB05 | select EdbFilePath EdbFilePath ----------- E:DB05DB05.edb
In that folder (E:DB05 in this example) there will be a sub-folder with a GUID for the name, which contains the content index files. Delete that GUID folder.
Start the search services on the server again.
[PS] C:\>Start-Service MSExchangeFastSearch [PS] C:\>Start-Service HostControllerService
The content index will be crawling and rebuilding, which will take some time depending on the size of the database.
For an Exchange 2016 server that is a member of a database availability group the content index can be fixed by reseeding it from a healthy copy.
[PS] C:\>Update-MailboxDatabaseCopy DB05\EX2016SRV2 -CatalogOnly -BeginSeed
When the content index has finished rebuilding or reseeding you should see all healthy results again.
[PS] C:\>Get-MailboxDatabaseCopyStatus * | sort name | Select name,status,contentindexstate Name Status ContentIndexState ---- ------ ----------------- DB05\EX2016SRV1 Mounted Healthy DB05\EX2016SRV2 Healthy Healthy DB06\EX2016SRV1 Healthy Healthy DB06\EX2016SRV2 Mounted Healthy DB07\EX2016SRV1 Healthy Healthy DB07\EX2016SRV2 Mounted Healthy DB08\EX2016SRV1 Healthy Healthy DB08\EX2016SRV2 Mounted Healthy
Cybersecurity Risk Management for Active Directory
Discover how to prevent and recover from AD attacks through these Cybersecurity Risk Management Solutions.
Very grateful for you detailing this – working a charm reindexing now, after Exchange 2016 CU19 must have screwed this up on the quiet. Much thanks 🙂
Awesome you da’ man. This happened after application of CU19 to Exchange 2016 for Zero day exploit.
Micros*** blah.
Yes. This happened to us too.
I got same issue of ContentIndex “failed” after upgrading to Exchange 2016 CU19.
Checked the below services, one of them was disabled, after enabling/starting everything is fine.
 Microsoft Search (Exchange)
 Host Controller service for Exchange services
Thanks Waseem,
We applied the Exchange 2016 CU19 on March 8 2021 and you were right!
Our Host Controller service was disabled and after enabling and starting the service the search functionality started working.
Such was the case with me. I found the service disabled as well after upgrading from CU17 to CU19. Re-enabling the service resolved the issue with ContextIndexing.
We ran into this same problem last night after upgrading to Exchange 2016 CU19.
Thanks!!!
Bro I have Also Exchange server 2016 with CU19 but this issue mostly when replication between database and database copies on a different servers. when I fix my replication issues between servers in my environment after that no copy fails on any servers.
Muito obrigado, fiz o processo como mencionado, e para minha alegria, após reiniciar os serviços, a pesquisa estava funcionando. TOP
Hello,
After migrating from EX2010 to EX2016, we have similar search issues
I could see all ‘ContentIndexState’ of 2016 servers is Unknown.
could you update us how to proceed to fix the issue.
Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex
Length Length State
—- —— ——— ———– ——————– ————
Mailbox Database 2047897560\XXXX Mounted 0 0 Unknown
Mailbox Database Primary\XXXX Mounted 0 0 Unknown
Cheers.
Paul,
Great article just what i was looking for this morning.
I have now foxed the issue.
Found in the event viewer that there was an error with space, eventhough I still have heaps of space, it said it has passed the 70% threshold. So all I did was to relocate some files out of the drive to have extra space in the exchange drive, then I followed the above steps again, and after several hours, this time it worked.
Thanks a lot.
Hi Paul,
Thanks for the note really helpful
Hello,
I have this problem on Exchange 2013.
I have followed this steps every day for a week now.
It goes to Healthy. (search still won’t work in outlook or owa)
Then the next few hours is back to FailedAndSuspended.
Any idea what could be causing it?
I have only one Database no DAG
Cheers.
Running a test-exchangesearch from EMS reports time outs/errors and Outlook searches are inconsistent. Exchange shows the content index as healthy for all DAG servers containing the database copy.
It’s been advised to rebuild the content index from scratch. In a case where Exchange lists the content index as healthy and you have a DAG, what should be the correct steps to re-build the content index from scratch? I don’t want it attempting to copy over an index from the other DAG servers since they all have same ‘healthy’ but unsearchable indexes. I haven’t found anything yet but my google-fu is a bit lacking since I’ve working other issues related to this restore.
In my mind, I think you would just stop the MSExchangeFastSearch/HostControllerService(s) on all 3 members, disable the services on the two non-active members, remove the index on the active server and then let it re-build on service restart. When complete, repeat the index delete on the remaining two DAG server, enable/restart services and let them complete.
My reasoning for disabling services on the other two DAG server is to not have them attempt to auto-start/build while trying to get a clean index made on the active server.
Has anyone had to do this? Have I missed an obvious article for such a process?
Anyone familiar with this error?
ContentIndexErrorMessage : Could not find registry value of Index Status for database {2134923498 9083 8-02384-0 8-0823940}
(standalone on premise)
thanx in advance
Hey Paul,
Just want to say thanks! Great resource.
Hi Paul,
We were updating our Exchange 2016 CU9 servers today in our dags one at a time. After a few were done we noticed a lag in our environment. When checking we see our Passive Databases reseeding. Would be nice to know why.
If you’re having regular issues with the index failing, check to see if there is enough hard drive space on the drive that the database/index is on for a copy/expansion.
We encountered a problem where we had to rebuild the index daily. Everyday we rebuilt the index based on your recommendations above and the next morning it failed.
Upon further investigation we found in the event log an event 1006 followed by an immediate 1009. This appeared to happen every night around the same time.
The daily recurrence was a backup of the database. We assume the software that we used must’ve stopped the database, made a temporary copy and restarted the database. Finally we came to the understanding that there was not enough space to complete this process and the end result was the failure of the index. There was no indication on the backup software or in the event log of the failure.
We added drive space and the issue went away.
@Jim L says
What backup solution you’re using?
Hey Jim,
After an extensive search online, your description of the problem is the closest to mine.
Mine also fails every day and I do have a daily backup solution. (shadowprotect)
Did you just add drive space to the backup unit? or to the local drive?
I do have plenty of space available in both, not sure if there is a threshold there.
Cheers.
Hi Paul,
I want to express that you are greatly appreciated!
in my scenario (3 databases), I ran a series of commands to try to get things to work (below after command that fixed my situation).
This is what fixed it for me:
Update-MailboxDatabaseCopy -identity DB01\MBX01 -CatalogOnly
note: when I used this command there was no change:
Update-MailboxDatabaseCopy -identity DB01\MBX01 -SourceServer MBX02 -CatalogOnly
But the command without -SourceServer gave a warning:
The mailbox database copy ‘DB01\MBX01’ has failed to update from server . Do you want to clean up that update request now? Seeding cannot be
requested for the same database copy until the failed request has been cleaned up by the server, which should automatically happen within 15 minutes.
After clicking through the warning, 2 of the 3 databases were Healthy immediately, and the DB01 had to complete the seed.
Here are the commands I tried before my solution:
Set-ServerComponentState MBX01 -component “HighAvailability” -requester Maintenance -State Active
Set-ServerComponentState MBX01 –Component ServerWideOffline –State Active –Requester Maintenance
Set-MailboxServer MBX01 –DatabaseCopyAutoActivationPolicy Unrestricted
Set-MailboxServer MBX01 –DatabaseCopyActivationDisabledAndMoveNow $false
Resume-ClusterNode –Name MBX01
Set-ServerComponentState MBX01 –Component HubTransport –State Active –Requester Maintenance
Hi Paul,
I’m showing Database01 and Database02 in the same folder and two GUID-named folders there as well. Only Database01’s Content index state is showing as FailedAndSuspended. So how do I know which folder to delete?
Thank you in advance 🙂
NM. May be the long way around, but I was able to dig around in my AD Microsoft Exchange System Objects until I found the right property under Attribute Editor. Booya!
Hello Paul,
I have two servers in a DAG but it is giving error while creating a mailbox database copy and the error is related with content Index. But ran a command with -Skip ClientExperienceChecks and it worked. Now, I am not able to either switchover the database nor move the mailbox to this database.
Tried with this below command but still the that content indexing is in failed and suspended state.
Update-MailboxDatabaseCopy DB05EX2016SRV2 -CatalogOnly -BeginSeed
Hi,
This solution works but i must do it every time I have to switch.
Under Exchange Server 2016 CU8 in DAG (Labs), I use the recommandations of :
https://support.microsoft.com/en-us/help/2807668/content-index-status-of-all-or-most-of-the-mailbox-databases-in-the-en
It works for me.
Regards.
Some of my database copies are with ContentIndexState: Disable, although are configures as True.
Can the IndexSearch “fail” due a low ram memory configuration?
My DAG hosts (2) are in a virtual environment and I want to know if increment Ram will be usefull.
Hi Paul,
When you run Update-MailboxDatabaseCopy, this command mus be run with the MSExchangeFastSearch and HostControllerService stop?
Regards
My indexes are showing healthy
I have a server with 3 databases DB 1 and DB3 work fine DB2 when I run a test-exchangesarch -idenetiy name gives me a time out of thread the other 2 work fine. I repaired the index on those servers but still has same results. any suggestions?
Thanks Paul. This worked great on my Exchange 2016 server and fixed the searching issue!
Hi Paul,
again, I had to repair this because New-MailboxImportRequest could not import anymore.
greetz bpoelen,
the netherlands
Oh Boy, Paul,
you saved my Day, I was in a endless for days loop to resolve this problem.
I was still a bit afraid to delete a folder in the exchange folder.
But is is working now, amazing
thanks,thanks, thanks
Thank you for your efforts on this Exchange blog.
My Content index state = failed.
My services list does not have those two present.
Why might they be missing (hence my real problem)?
I am pursuing other steps to identify this.
I always learn something from you and the commentators.
try run “Stop-Service …” from powershell.
not exchange mgmt shell
I have tired all the steps mentioned here https://www.practical365.com/exchange-server/exchange-2016-failed-content-index/ but still am getting same error Content index state: FailedAndSuspended.
Please help
Thank You! Great instructions and worked perfectly!
I agree. 🙂 Thank you Paul. Very Helpful.
Paul
Thanks as always for the great Exchange related content. A lot of us out here rely on your insight.
I have a situation that maybe you can offer some guidance on. I have a test environment setup testing Exchange 2016. I have three mailbox servers that are in a DAG. The servers currently have 16GBs of ram and a 40GB hard drive where the databases are stored (test mailboxes only, but no real content). The DAG has its own replication 10G network.
It seems like every time I add a new database copy, the content index comes up as FailedAndSuspended and never reaches a healthy state. I have followed several different posts that say to use the Update-MailboxDatabaseCopy, but nothing seems to be helping. I also added an AD group called ContentSubmitters based on an article I read.
Any thoughts on why the content indexes never get healthy or take a long time to reach a healthy state? (example: I added a test database around 6:40PM along with two copies, it has been 2.5 hours and the content indexes for the copies are still in a state of FailedAndSuspended.)
Thanks
Kyle
The Real Person!
The Real Person!
Make sure you’ve configured antivirus exclusions.
Also try simplifying your networking by removing the replication network.
After reseeding the Lag copy database we are seeing the content indexing state as Autosuspend and replay queue length is also too high (in thousands). Replay Lag time and truncation is set to 7 Days. Is this normal behaviour? Please advice as we reseeded one database again but still it is showing Autosuspended and the replay queue length is high. We are using Exchange 2016. Safety net hold time is set to 1 day.
its expected on lagged copies..I confirmed it
Are you running RU3? If so, the recommendation is: Reinstall your cluster with RU2.
Unfortunately not joking – we’ve been there.
im getting failed for content now on ALL . since i renamed the old guid.
Name Status ContentIndexState
—- —— —————–
PublicFolderDatabaseSRV-EXCHANGE Mounted Failed
QCDBEXCH01SRV-EXCHANGE Mounted Failed
QCDBEXCH02SRV-EXCHANGE Mounted Failed
QCDBEXCH03SRV-EXCHANGE Mounted Failed
QCDBEXCH04SRV-EXCHANGE Mounted Failed
where before it was
Name Status ContentIndexState
—- —— —————–
PublicFolderDatabaseSRV-EXCHANGE Mounted FailedAndSuspended
QCDBEXCH01SRV-EXCHANGE Mounted Healthy
QCDBEXCH02SRV-EXCHANGE Mounted Healthy
QCDBEXCH03SRV-EXCHANGE Mounted Healthy
QCDBEXCH04SRV-EXCHANGE Mounted Healthy
any help ?
Hi Paul,
Thanks for the great post !
BTW after migrating from EX2010 to EX2016, we have similar search issues for some huge mailboxes. Numeric searches are not properly working (M/B Size 25 GB)
I could see all ‘ContentIndexState’ of 2016 servers as Healthy.
Please advise.
Hello Paul,
Thank you for the excellent posts on Exchange! I followed it and did not run into any obvious errors, but still have a problem. I hope you can help me out.
Last week our Exchange 2016 database got corrupted when power was cut to the server. I have restored the database but forgot the index folder.
Users are complaining that the search option in is not working in Outlook and OWA.
With the “Get-MailboxDatabase | select EdbFilePath” I located the index folder and deleted it. When I started the search services the folder is not recreated and I am still not able to search for e-mails.
When I run the “Get-MailboxDatabaseCopyStatus” command the output states that the ContentIndexState is healthy.
I hope that you can point me in the right direction.
I have the same issue,
But i can see that the utilization of the command line takes a lot of time based on how big the database is. So i keep this running overnight to see if this resolves it.
Other tip , create new DB and move the databases to that new database , as the index will recreate itself on the new DB
Thank you for your great resources!
I have seen, that after following your guide and restarting the services Get-MailboxDatabaseCopyStatus will still show “FailedAndSuspended” for a while (you will see the noderunner.exe task(s) working at full speed).
After a while it will change to “Crawling” and after that hopefully to “Healthy”, but I’m not there, yet.
In our environment after installing CU1 for Exchange 2016 crashed serach service.
We couldn’t find any way to resolve this issue (we try index repair, etc.)
Yesterday we installed CU2 for Exchange 2016 and do repairing content index as described above. Now it’s working fine.
is the command Update-MailboxDatabaseCopy DB05EX2016SRV2 -CatalogOnly -BeginSeed only for Exchange 2016 or it also works with 2013?
The Real Person!
The Real Person!
If you’re ever wondering whether a parameter is available for a cmdlet, or you want to find out more about a parameter, just check the TechNet page for that cmdlet. That’s what I do.
No. No. No. No.