At some point in your career as a hard-working IT professional you’re likely to encounter a scenario in which an Exchange 2016 server has completely failed. For example, you might have a physical server with a dead hardware component preventing it from booting, or a virtual machine with a corrupt operating system volume that is preventing it from starting up.
There are many ways out of these types of situations, and I won’t dictate that one particular solution is the best one in all cases. You’ll have a variety of factors to take into account in your particular scenario, for example perhaps the hardware component that failed can be quickly replaced and the server brought online, negating the need for any other software-driven recovery process to take place.
But let’s assume for the sake of this article that your situation involves a failed server that will not boot, and you either have replacement server hardware that you can use or are planning to spin up another virtual machine to run Exchange 2016.
In either case a recovery installation of Exchange Server 2016 may be the appropriate course of action for you to take. At a high level the process goes like this:
- You install a new Windows Server instance with the same characteristics as the failed server (the same server name, Windows Server version, drive letters, and performance/capacity)
- You perform a recovery install of Exchange 2016 by running setup with the /mode:recoverserver switch
- You re-apply any custom configurations that were not automatically re-applied by the recovery install
- You restore the Exchange databases if those volumes were also lost in the server failure.
Let’s take a look at an example of a recovery installation for a standalone Exchange Server 2016 Mailbox server. If your server is a DAG member, refer to my article on how to recover a failed database availability group member instead.
Preparing the Server for Exchange Server 2016 Recovery
To begin you’ll need to prepare the server that you’ll be recovering Exchange Server 2016 on. Whether that is the same server or a different server depends on your specific circumstances, but either way you’ll need to:
- Install the same Windows Server operating system and service pack level on the server
- Configure your storage volumes to use the same drive letters as the previous server
- Join the server to the domain (note that you will first need to reset the computer account that already exists in Active Directory)
- Install the Exchange Server 2016 pre-requisites
Performing a Recovery Install of Exchange Server 2016
On the server that you’ve prepared copy the Exchange Server 2016 setup files to a location where they’ll be accessible on the server. You should use the same setup files for the Exchange Server 2016 cumulative update that was previously installed on the server, do not try to use this recovery process to upgrade or downgrade the server.
Open an elevated command prompt and change to the folder containing the Exchange 2016 setup files. Run the following command.
C:\> setup /mode:recoverserver /IAcceptExchangeServerLicenseTerms
Restoring Custom Exchange Server 2016 Configurations
There is often confusion around what counts as a “custom configuration” in this scenario. Exchange Server 2016 stores most of its configuration in Active Directory, such as virtual directory URL settings, transport settings, the names and locations of databases, and so on.
However anything that is machine-specific or stored locally on the server such as IIS settings, SSL certificates, modified config files, or registry keys, will not be restored by the recovery install. You should always document and automate the post-install configuration of your Exchange servers so that any such customizations can be reapplied in this type of scenario.
On the most basic of Exchange 2016 deployments you’re likely to have to at least reinstall the SSL certificate for the server. If you have multiple Exchange 2016 or 2013 servers that have the same SSL certificate installed then you can simply export the certificate from one of those servers, and then import it to the recovered server. Note that the steps for this are the same for Exchange 2016 as they are for Exchange 2013. After reinstalling the SSL certificate you can then also enable it for Exchange services.
Restoring Exchange Server 2016 Mailbox Databases
If the volumes that contain your mailbox databases and transaction logs were not lost in the server failure, and are configured with the same drive letters or mount points as before, then you will likely find that the databases are able to successfully mount and continue operation without data loss.
If the databases and log files are available, but will not mount, you may need to perform a soft recovery of the database first using ESEUtil.
If the databases and log files were lost in the server failure, then you will need to restore them from your last backup.
Summary
In this article I’ve provided an overview of performing a recovery installation of a failed Exchange Server 2016 server. As you can see the process requires some knowledge of the server you’re recovering, such as drive letters and any custom configurations, that should always be captured in documentation or scripts in readiness for this type of disaster recovery scenario. When all of the required information and data is available a recovery of the failed Exchange 2016 server is achievable.
Hi paul,
I have a standalone serveur exchange 2016 cu19, when i tried to update to cu20 it crash.
can i made a Setup /m:RecoverServer straight to cu20.
Hi Paul,
Sorry for being late here.
You said to use the same CU version when doing a recovery. However ours was a 2016 CU 10 server(when dead), however all the other DAG servers are 2016 CU14 now. And we are in a process of upgrading all the servers to CU15 already. Hence wants to recover this one with 2016 CU15 as a fresh server and then add it to DAG.
Do you suggest anything different or if there are any recommendations with it.
Would really appreciate your help.
Don’t know If I should create a new Thread but cant see where to do that.
This all stems back to a corrupt database which happened 3 months ago and would not mount and failed to allow all the usual Get-Mailbox etc etc commands. We left it there for a while but Backups also kept failing. We had managed to transfer all the users mailboxes to new databases and the Exchange server has continues to run fine.
I tried to remove the Corrupt database but had no luck so in a fit of pique I deleted it. Big Mistake
The exchange server continued to run but certain features didn’t work such as creating a new outlook profile/account hence we couldn’t move users to a new PC.
So far we have restored from backups and the then incremental that only had the database’s in.
VERITAS BUExec moaned and didn’t complete the restore until but only after it had been running for 8 hours!! (why it couldn’t tell us a bit earlier I don’t know!!)
Anyway after researching the subject a lot I am firmly of the opinion that the/a system mailbox was on the corrupt database but I this Database was not mounted I assumed it didn’t matter.
incidentally a CU14 update also failed.
So my question is do I just need to locate the missing system mailboxes in ADSIedit and delete them or will /prepareAD recreate them.
OR is there such a thing as a repair install?
I will be applying Cumulative Update 14 for Exchange Server 2016 n a stand alone server this weekend in preparation for an office 35 migration project. I have done all he ground work and I believe I am well prepared for it. It’s a stand alone server. I wish it was in a DAG configuration, would have made it more easier to stagger the CU update between DAG members. I want to say that your articles have been very helpful over the years. Thank you and keep on sharing your knowledge.
get-tip : The term ‘get-tip’ is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\Program Files\Microsoft\Exchange Server\V15\bin\RemoteExchange.ps1:264 char:1
+ get-tip
+ ~~~~~~~
+ CategoryInfo : ObjectNotFound: (get-tip:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
Connect-ExchangeServer : The term ‘Connect-ExchangeServer’ is not recognized as the name of a cmdlet, function, script
file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct
and try again.
At line:1 char:76
+ … ge Server\V15\bin\RemoteExchange.ps1′; Connect-ExchangeServer -auto – …
+ ~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (Connect-ExchangeServer:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
Dear Paul
Thank you for articles you provide in site.
I need to recover an exchange server 2016 CU4 installation.
But now download links in MS site has been disabled.
Now, how can I do it?
I really need to same source files.
Hi
Thank you for your brilliant web site
in our organization we had a problem in Additional DC
we removed it from Active Directory site and services and run a new
after that exchange server services won’t start
and we have lots of error in event
Hi Paul,
Can we recover an exchange server with different Operating System version?
For example the failed Exchange Server is: Exchange 2010 SP3 with OS Win Server 2008 R2.
Can I setup Setup/Mode:RecoverServer in Windows Server 2012?
Hi Paul
i came here from another post from you(https://www.practical365.com/exchange-server/exchange-server-error-an-incomplete-installation-was-detected/)
im decomisioning 2013 server and the uninstaller stops at 6% (step 1 of 13) i think when is stopping some service IIS related.
Now i cant use the unistaller again, so i decide to install again. But i have the error:
Prerequisite Analysis FAILED
A Setup failure previously occurred while installing the PreFileCopy role. Either run Setup again for just this rol
e, or remove the role using Control Panel.
For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.InstallWatermar
k.aspx
any idea?
Hi Paul, as you recommend me in another article to do a recovery for a lost server, this server I need to restore it just to do a decommission properly, mi question is:
¿Do I need to have the Database of the lost server in order to do the decommission process?
I can tell the database was also installed in the C drive in the lost server.
The Real Person!
The Real Person!
No, you won’t need it. The server should just initialize a new database file in the previous location.
am trying to uninstall exchange 2016 but am getting the errors
An Incomplete Installation Was Detected
Hi Paul,
You said to use the same CU version when doing a recovery. I am on 2016 CU5 and don’t have the installation media that was used to do the initial installation. I’ve searched all over for CU5 and everything is pointing to microsoft website where CU5 is no longer available. Do you have any secret repository where I can still get CU5, or in worst case can I use the latest cumulative update installer?
The Real Person!
The Real Person!
I do not have a secret repository. I keep them all stored on a NAS for just this type of occasion, but it’s not possible for me to share them.
I always recommend customers and consultants do the same and maintain their own stash of CUs. If you have an outside consultant you’ve ever used for Exchange work, you could ask them if they can drop off a copy for you. Otherwise you can open a support case with Microsoft and I believe they can provide the retired CUs for this type of situation.
Hi Paul
We had a mail system failure. It was taking awhile to recover the system so we recreated by disabling and then enabling again some of the key mailboxes we needed. What will happen when we bring the recovered exchange server back online?
The Real Person!
The Real Person!
I assume what you mean is you’ve enabled the mailboxes on another server/database, so they were empty but could keep working to send/receive email.
When you recovery your failed database you’ll need to restore the data for those mailboxes into their new locations. Take a look at “dial tone recovery” for the steps.
Recovery fails with the following error “An exception ocurred while setting shared config DC”.
Thank you for your articles Paul, they are very helpful!!
When doing a recovery is it possible to restore everything back to the original install rather than keep the mailbox databases, virtual directory URLS, and anything else stored in Active Directory?
The Real Person!
The Real Person!
No, it’s going to use the existing config for that server name. Not sure exactly what you’re trying to achieve though?
Hi Paul,
Some sevices in the Exchange 2016 cannot start, or are start pending. Does this require a reinstall of the Exchange 2016?
Cheers
The Real Person!
The Real Person!
Did this just start happening on its own, or after you’d done some updates/maintenance, or something else that might provide some context to the situation?
Paul,
thanks for reply. This happened after a power off situation and the server got off by itself. The UPS for the server turned off after it ran out of charge because the generator fail to start in the morning at 4:30am.
That was the context. Any help pls.
Thanks in advance.
Cheers.
For an example , We have eight Exchange VMsin DAG and applying Exchange CU/patches on all at same time. We got a problem with all serves after applying the patches.
What is the best recovery methods?
Snapshot is not recommended so we did not take the snapshot.
The Real Person!
The Real Person!
You updated 8 DAG members at the same time? That’s not a very sensible approach. If you’ve got that many DAG members you should be able to update them one at a time, allowing enough time in between each update to ensure that no issues or bugs appear.
You haven’t described your problem, but the only way to recover a DAG member is to uninstall and reinstall it, or do a recovery install (https://www.practical365.com/recovering-a-failed-exchange-2016-database-availability-group-member/).
You should probably contact Microsoft Support though, to find out the best way to deal with whatever problems you’re seeing.
How do we recover the VM servers in a DAG/standalone if something goes wrong after applying SP and patches? If snapshot is not supported.
The Real Person!
The Real Person!
If it’s a standalone server then the article above explains what to do.
If it’s a DAG member then there’s some additional steps outlined here: https://www.practical365.com/recovering-a-failed-exchange-2016-database-availability-group-member/
I can’t get past it – any guidance is much appreciated…
————————
Performing Microsoft Exchange Server Prerequisite Check
Configuring Prerequisites COMPLETED
Prerequisite Analysis FAILED
The language pack bundle could not be found or is corrupt.
For more information, visit: http://technet.microsoft.com/library(EXCHG.160)/ms.
exch.setupreadiness.LangPackBundleCheck.aspx
500 error ecp when logging In after restore.
Fix an error 500 after a reinstall
This is due to the an unmounted database.
### Get Mailbox name
Get-MailboxDatabase -IncludePreExchange2013
### Get the status of the
Get-MailboxDatabase -Identity “Mailbox Database” -Status | Format-List
### Mount the database:
Mount-Database -Identity “Mailbox Database”
After doing a recover install of Exchange Server 2016, I am unable to login to the Admin Center and Outlook email clients are unable to connect to the server. I am receiving an “HTTP 500 Internal Server Error” when I attempt to sign-in the Admin Center. There is no database copy to restore to bc this was also lost.
How can I correct this?
Fix an error 500 after a reinstall
This is due to the an unmounted database.
### Get Mailbox name
Get-MailboxDatabase -IncludePreExchange2013
### Get the status of the
Get-MailboxDatabase -Identity “Mailbox Database” -Status | Format-List
### Mount the database:
Mount-Database -Identity “Mailbox Database”
What if you want to recreate a database because everything was lost…including the last backup?
amazing article, just need to add a point here,
if your exchange server wasn a part of a DAG and you are trying to recover it using a new hardware, you may wanna try using this command
Setup /m:RecoverServer /IAcceptExchangeServerLicenseTerms /InstallWindowsComponents
this takes care of the pre-reqs 🙂
thanks paul, you are the best 🙂