This article will demonstrate the step by step process for installing cumulative updates for Exchange Server 2016.
The steps for installing cumulative updates on Exchange 2016 are:
- Prepare by downloading update files, checking backups, and reviewing known issues
- Update mailbox servers in the internet-facing sites
- Update mailbox servers in any remaining internal sites (if any)
- Update Edge Transport servers (if any)
- Perform health checks and rebalancing of servers
Preparation
Before you install any cumulative updates on your Exchange 2016 servers, you should first:
- Download the cumulative update from Microsoft. Do not download from any third party websites. You can download the latest cumulative update and upgrade an Exchange 2016 to the latest version in one update. You do not need to install all of the cumulative updates released between your current version and the latest version.
- Verify that you have confirmed, working backups of your Active Directory.
- Verify that you have confirmed, working backups of your Exchange servers and databases.
- Verify that you have documented any customizations to your Exchange server that will need to be re-applied, such as custom OWA login pages, web.config changes, registry changes, or third party add-ons. Generally speaking you do not need to re-apply standard Exchange configurations that are set via the Exchange Admin Center or Exchange management shell (e.g. changing default message size limits).
- Verify that your Exchange SSL certificates have not expired.
- Check the Exchange Supportability Matrix and verify that you are maintaining the .NET Framework on your servers to remain compatible with Exchange.
Known Issues
Comprehensive lists of known issues with cumulative update installation generally do not exist, however to improve your awareness of issues experienced by other customers, you should read the comments on the Exchange team blog entry for the relevant cumulative update, and check the TechNet forums for other reported issues.
You should also be aware of the following issues:
- Slow Installation of Exchange Server Updates on Windows Server 2012 R2
- Always use a CMD prompt to run Exchange setup, not PowerShell
- New DAG activation preference behaviour in Exchange 2016 CU2 (this is not a problem, but you need to be aware of it)
Order of Installation of Exchange 2016 Updates
Cumulative updates for Exchange 2016 should be installed in the internet-facing site first, before installing in other sites in the organization.
- Mailbox servers are updated first
- Edge Transport servers can be updated last
For load-balanced servers and Exchange 2016 DAG members, there will be a period of time during which all servers are not at the same version. This is expected, and supported, but you should plan to continue upgrading servers so that they are all updated within a reasonable period of time. You can balance that recommendation with the need for caution, e.g. waiting for issues to arise on the first upgraded server before deploying to the other servers. As a rule of thumb, aim for “days or weeks” rather than “months” between server upgrades, depending on the size of your environment.
Deploying Exchange 2016 Cumulative Updates
The process for installation is as follows:
- Perform the Active Directory schema changes and updates. This is performed once for the entire Active Directory environment. You do not need to repeat this for each server being upgraded.
- Upgrade servers. For each server in turn:
- Place the server into maintenance mode.
- Install the update.
- Perform testing.
- Take the server out of maintenance mode.
- Perform post-installation tasks:
- Rebalance database availability groups.
- Restore customizations.
- Perform a health check of the environment.
Active Directory Schema Changes and Updates
Most cumulative updates will include Active Directory schema changes, as well as other updates such as changed to RBAC roles. In some cases, the existence of changes will depend on which previous CU you’re upgrading from. So as a general rule you should plan for AD schema changes and updates to occur.
The AD preparation tasks can be run in advance of your server upgrades, or they can be allowed to run automatically as part of the first server upgrade process. In either case, Enterprise Admins and Schema Admins rights will be required. And if you’re running the update from an Exchange server, the RSAT-ADDS feature must be installed.
Before applying the schema update follow the steps provided by Michael B Smith to retrieve the existing Exchange schema version, so that you can compare it before and after the AD preparation steps have been completed to verify that the schema update was applied.
- Run setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms (requires Enterprise Admins and Schema Admins permissions, and must be performed in the same AD Site as the Schema Master on a server with the RSAT-ADDS-Tools feature installed – the Schema Master itself would meet these requirements)
- Run setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms
- Run setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms in each domain in your forest that contains Exchange servers or mailboxes. If you have a single domain, the previous step has already done this for you.
When the Active Directory changes have been applied, on each server run the upgrade.
Upgrading Exchange 2016 Servers
For Exchange 2016 Mailbox and Edge Transport servers, whether they are standalone, load-balanced, or part of a DAG, use the following procedure.
Set the HubTransport component to “Draining”, and redirect any messages currently in the queue to another server. If you’re running a single Exchange server, you can skip the redirect command.
[PS] C:\>Set-ServerComponentState EX2016SRV1 –Component HubTransport –State Draining –Requester Maintenance [PS] C:\>Redirect-Message -Server EX2016SRV1 -Target EX2016SRV2.exchangeserverpro.net
If the server is a DAG member, run the following commands. If your server is not a DAG member, skip to the command for setting ServerWideOffline.
[PS] C:\>Suspend-ClusterNode –Name EX2016SRV1 Name ID State ---- -- ----- EX2016SRV1 1 Paused
Disable database copy auto-activation. This command will also move any active database copies to other DAG members, assuming there are other healthy DAG members available. This is not instantaneous, it can take several minutes for the moves to occur. We’ll check on it shortly anyway.
[PS] C:\>Set-MailboxServer EX2016SRV1 –DatabaseCopyActivationDisabledAndMoveNow $true
Make a note of the database copy auto-activation policy on the server, so you can set it back to this value at the end of maintenance.
[PS] C:\>Get-MailboxServer EX2016SRV1 | Select DatabaseCopyAutoActivationPolicy DatabaseCopyAutoActivationPolicy : Unrestricted
If the policy is not already set to “Blocked”, run the following command to set it.
[PS] C:\>Set-MailboxServer EX2016SRV1 –DatabaseCopyAutoActivationPolicy Blocked
Check for any database copies that are still mounted on the server. This command should return no results. If any database copies are still active on the server, and there are other DAG members that host copies of the database, perform a manual switchover.
[PS] C:\>Get-MailboxDatabaseCopyStatus -Server EX2016SRV1 | Where {$_.Status -eq "Mounted"}
Place the server into maintenance mode.
[PS] C:\>Set-ServerComponentState EX2016SRV1 –Component ServerWideOffline –State InActive –Requester Maintenance
For servers that are in a load-balanced pool:
- Verify that the load balancer health checks have taken the server out of the pool or marked it as offline/inactive.
- If the load balancer does not automatically do this, manually mark the server as offline/inactive.
For servers that are in a DNS round robin group, remove the DNS record for this server’s IP address.
Before you run Exchange setup to install the cumulative update:
- Perform a restart of the server to clear any pending reboot status that will stop Exchange setup from running.
- Verify that the PowerShell execution policy is set to Unrestricted as per KB981474.
After the restart, launch an elevated CMD prompt, and run the following command from the folder where the Exchange setup files are located:
X:> setup /m:upgrade /IAcceptExchangeServerLicenseTerms
After the cumulative update has installed, restart the server. When the server has been restarted, perform a basic health check of the server:
- Review event logs for new or excessive errors and warnings
- Check that auto-start services on the server have started
You can now remove the server from maintenance mode. Note, if the server is not a DAG member, then only the first and last commands are necessary. If the server is a DAG member, use the database copy auto-activation policy value that was set on the server prior to being placed into maintenance mode (the default is “Unrestricted”).
[PS] C:\>Set-ServerComponentState EX2016SRV1 –Component ServerWideOffline –State Active –Requester Maintenance [PS] C:\>Resume-ClusterNode –Name EX2016SRV1 Name ID State ---- -- ----- EX2016SRV1 1 Up [PS] C:\>Set-MailboxServer EX2016SRV1 –DatabaseCopyAutoActivationPolicy Unrestricted [PS] C:\>Set-MailboxServer EX2016SRV1 –DatabaseCopyActivationDisabledAndMoveNow $false [PS] C:\>Set-ServerComponentState EX2016SRV1 –Component HubTransport –State Active –Requester Maintenance
Post-Installation Tasks
After deploying an Exchange 2016 cumulative update there are some post-installation tasks that you should perform.
Rebalance Database Availability Groups
Throughout the update process the database copies in your DAG will have been moved between DAG members, possibly multiple times. If you want to return your active database copies to their most preferred DAG member (aka “rebalancing the DAG”), use the PowerShell script supplied by Microsoft.
[PS] C:\>cd $exscripts [PS] C:\Program Files\Microsoft\Exchange Server\V15\scripts\>.\RedistributeActiveDatabases.ps1 -DagName EX2016DAG01 -BalanceDbsByActivationPreference
Restore Customizations
After you have completed updating your servers you will need to re-apply any customizations that you had documented during the preparation steps above.
Perform a Health Check of Servers
Here are some suggestions for health checking your Exchange servers after applying updates.
- Check the cluster nodes are all up – verify that you have not left any DAG members suspended in the cluster by running the Get-ClusterNode cmdlet on one of the DAG members.
- Test service health – use the Test-ServiceHealth cmdlet to verify that all required services are running on each server.
- Test MAPI connectivity to every database – use the Test-MAPIConnectivity cmdlet to verify that all databases are mounted and accessible.
- Check the database copy status for DAGs – use the Get-MailboxDatabaseCopyStatus cmdlet to verify that all database copies, copy/replay queues, and content indexes are healthy.
- Test replication health for DAGs – use the Test-ReplicationHealth cmdlet on each DAG member to verify replication health is good.
- Check the database activation policy for each Mailbox server – verify that each Mailbox server that is in a DAG has the correct database activation policy for your environment.
- Check server component status – use Get-ServerComponent to verify that you have not left any servers in maintenance mode.
- Run Exchange Analyzer to check for best practices compliance.
The Test-ExchangeServerHealth.ps1 script can perform some of the steps above for you. You should also consider running one or more tests from ExRCA.com to verify client connectivity and inbound mail flow are working.
Comments for this blog post are now closed; please contact team@practical365.com for any additional questions and comments, thank you.
Hello I need to upgrade cumulative update 19 for exchange server 2016 ,currently we are using cumulative update 17 for exchange server 2016 .please let me know what are prerequisites for the same. is This update contains AD Schema changes, so please let me know how to plan this
Hi Paul,
We’ve come across a rather bizarre issue after what appeared to be a successful upgrade 2016 CU17. Both OWA and ECP are unable to load – ECP displays the error;
[Ecp] An internal server error occurred. The unhandled exception was: System.MissingMethodException: Method not found: ‘Void Microsoft.Exchange.Security.Authentication.Utility.DeleteFbaAuthCookies(System.Web.HttpRequest, System.Web.HttpResponse)’.
at Microsoft.Exchange.HttpProxy.FbaModule.RedirectToFbaLogon(HttpApplication httpApplication, LogonReason reason)
at Microsoft.Exchange.HttpProxy.FbaModule.OnEndRequestInternal(HttpApplication httpApplication)
at Microsoft.Exchange.HttpProxy.ProxyModule.c__DisplayClass20_0.b__0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
OWA displays the generic “something went wrong” – all other services are fine.
Have tried – DependentAssemblyGenerator.exe as well as updatecas.ps1 both to no avail.
Any help appreciated.
I am having the same (or very similar issue). OWA and ECP both initiate a second prompt for authentication. We cannot seem to figure this one out. Were you able to remedy your issue?
Hi, did you find a solution? I upgraded one server and got a similar problem, but the other server which is still on CU14 is OK.
Thanks
Hi,
We are experiencing this exact problem as well.
Did you find any solution?
Thanks!
thanks for the article…
Hi Paul,
We have an On-Premise Exchange Server 2016 environment which includes an Edge server. Both Mailbox Servers and the Edge are at CU5. We are looking to migrate to On-Premise Exchange Server 2019 and understand we need to update to CU16 to successfully move the mailboxes.
Do we need to update the Edge server to CU16 along with Mailbox Servers?
Because we have limited space on the Edge we may not be able to update it to CU16 and are considering simply installing an Edge Exchange Server 2019 and using it with our Exchange 2016 Mailbox servers until we can upgrade them to Exchange Server 2019. Is this a plausible option.
Please advise.
Thank you
Hello All
My customer has Exchange 2016 CU5 and along the way someone has updated the server to .Net Framework 4.7.0. What can I do about updating .Net Framework and getting Exchange to CU14 supported by .net 4.7.2 or CU 16 supported by 4.8?
Thank you
Paul,
We performed a Cumulative Update 15 days ago and I still have a database stuck in HealthyAndUpgrading status with 88 mailboxes left in ContentIndexMailboxesLeftToCrawl
We have 6 databases in the DAG and the other 5 are fine. At first I thought it would just take a few days but there’s no way it should take this long. Something is wrong.
What can I do to correct this?
Hi Paul,
Currently I have Exchange Server 2016 CU5 (Build Number 15.1.845.34) running on Windows Server 2012 R2
Current AD is running Windows Server 2008 R2
I need to install Exchange Server 2016 CU15
Do we need to run /PrepareDomain or /PrepareAD
1. Can I go directly from Exchange Server 2016 CU5 to CU15?
2. If not, then how should I go? for example: from Exchange Server 2016 CU5 to 10 and then go to CU15?
3. How many steps Do I need to go from Exchange Server 2016 CU5 to CU15?
4. Could you please kindly let me know Exchange Server Upgrade 2016 CU5 to CU15 Step-by-Step
Thank you.
Babu – I’m curious if you have completed this yet? I’m in the same position, except the domain is 2012 R2. If you have not already, you need to upgrade AD, because 2008 R2 was end of support in January. Also, make sure you complete DFSR migration if you haven’t already.
Hello ,
Great Blog , I’ve found this how to works perfect .
I have a question , We have a few database dag , would you recommend we set the “Activation preference number:” to 1 well before we start, for all database to the server we are not upgrading . that way we can do controlled failover for testing . Plus when place it in maintenance mode , we would have to do a Server Switch ,
Sry for the Double post , My earlier post was there .
HI Paul,
thanks so much for all your blog. they are GREAT!
I have a 2 member dag . I just about to do my first 2016 CU upgrade . Our plan is to do one at a time (waiting a few days for burn in and testing ) , then do the other.
So would you recommend we set database “Activation preference number:” on all dag database member to the server we not going are not upgrading . Making that the Primary . This way we test 1 database in a control setting. Allowing the other to stay where they are . Minimise auto moves .
Do you have to do these steps on the passive boxes in the DAG? I need to patch my two passive DR ones and felt like I should be able to just patch through.
Hi Paul,
Great article. A quick question. I hope you will shed light on it.
We are going to upgrade to Exchange CU14 from CU12.
Do we have to run Step 1 and 2
Step 1-
E:\Setup.exe /IAcceptExchangeServerLicenseTerms /PrepareSchema
step 2
E:\Setup.exe /IAcceptExchangeServerLicenseTerms /PrepareAD /OrganizationName:”Contoso Corporation”
Just ran through the same CU update 12 -> 15. No need to PrepareSchema but yes to prepareAD.
See the following.
https://docs.microsoft.com/en-us/exchange/plan-and-deploy/active-directory/ad-schema-changes?view=exchserver-2016
Great article Paul, well written.
Hi Paul,
As always your directions are impeccable.
Trying to run the upgrade, I get the following error message.
Installing product C:\Exchange CU14\exchangeserver.msi failed. Fatal error during installation. Error code is 1603. Last error reported by the MSI package is ‘The Installer has insufficient privileges to modify this file: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\mapi\emsmdb\global.asax
Anyone have any ideas how to get past this error?
Use the elevated CMD instead of powershell.
Do we need AD prep commands if updating from CU12 to CU14 ?
Thanks.
Did you get the answer yet? I’m to upgrade from CU 7 to 14 by tomorrow.If you got any answer please contact me :my Email :eng.nor@gmail.com
Yes it is required. see the technet articale. before CU13 ad prep is required.
https://support.microsoft.com/en-ae/help/4522150/cumulative-update-15-for-exchange-server-2016
you are upgrading from Cumulative Update 13 for Exchange Server 2016 or a later cumulative update for Exchange Server 2016 to Cumulative Update 15 for Exchange Server 2016, then there’s NO NEED to run the /PrepareAD or /PrepareDomain. No additional actions (prepareAD, prepareDomain, or assigning permissions) are required.
Do we have to run /PrepareAD /PrepareDomain if we upgrade from CU12 to CU14?
Yes, you still have to. Check this.
https://support.microsoft.com/en-us/help/4556414/cumulative-update-17-for-exchange-server-2016
We have on premises server and hybrid servers as well. So while upgrading shall we First upgrade hybrid servers and then on premises. Some articles saying IST Hybrid servers and then on-prem servers.
Currently both on-prem and hybrid are at Exchange 2016 CU10.
We have on premises server and hybrid servers as well. So while upgrading shall we just upgrade hybrid and then on premises. Some articles saying IST Hybrid servers and then on-prem servers.
Currently both on-prem and hybrid are at Exchange 2016 CU10.
Hi Paul,
Great article and thanks for the informative content. A quick a question:
After upgrading the first server and mounting databases on the new CU level, can these databases then be mounted on the other server at the lower CU level?
Or is it once the DB’s are mounted on the newer CU they can only be mounted on server with the same CU or greater?
Regards,
David
I have read some more documentation and it seems the database schema is not updated automatically. Once the all members of the DAG are updated to the same CU level, the update-databaseschema command can be issued:
https://docs.microsoft.com/en-us/powershell/module/exchange/mailbox-databases-and-servers/update-databaseschema?view=exchange-ps
Going well in co-existence but don’t upgrade everything at the same time and read everything carefully. I will never learn Exchange till Paul does everything. Keep up the good work and I better learn programming to script the whole thing, not sleeping enough is bad for your health, best regards
I successfully completed an Exchange 2016 Mailbox server CU update from RTM directly to CU11 by following these instructions. There were a few Outlook client issues afterwards, fixed by recreating profiles. There are a few App Event log errors to resolve. But a day after the update, all is still functioning. Thanks Paul.
Jim
The problem is solved by reinstalling. During the analysis of the problem, I came to the conclusion that the installation in cmd did not finish correctly, although there were no errors, but there were too few complete messages. I do not understand why MS can not write a message in case of incorrect completion.
Typo in the previous message. In IIS the certificate is established correctly.
Hi.
Aftef update CU 10, there was the following error:
Connecting to remote server failed with the following error message : The WinRM client cann
ot process the request. It cannot determine the content type of the HTTP response from the destination comput
er. The content type is absent or invalid. For more information, see the about_Remote_Troubleshooting Help topic.
In ISS сертифакт it is established. Who can help? Thanks.
Paul
Great article Just upgraded my Exchange 2016 DAG from CU9 to CU10 on NODE 2
Just add this before you start
Install Visual C++ Redistributable Packages for Visual Studio 2013 64 bit
Add this
Close EMS before running
setup /m:upgrade /IAcceptExchangeServerLicenseTerms
Now need to upgrade NODE 1 but this process helped a lot.
Thank you
Hi Paul,
Is is possible to use different CU version of Exchange Servers 2013 in the single Exchange organization. They are not in the DAG.
Does any cause arise?
Thank you.
Hi Paul,
Thank you for the quick reply.
I already tried the /m:upgrade initially and I got the message “The parameter ‘roles” is not valid for current operation ‘Upgrade’.” This is why I’m using the Install mode on the EdgeTransport (2nd) server.
Yes, I am aware that CU4 is very old. The plan is to upgrade to CU4 then finally to CU9. Going forward, I was thinking of having the current release being 2-3 months behind the latest CU available.
Kind regards,
Noel
When using /m:upgrade you do not need to specify the roles. It just upgrades the roles that are already installed. If you’re ever unsure about the cmd line parameters they are documented on TechNet.
Hi Paul,
The CU 9 has been successfully applied to both servers.
Thank you very much for your advice.
Kind regards,
Noel
Hi Paul,
I followed this article to upgrade the main exchange server 2016 RTM to CU4. But I’m having issue in upgrading the Edge Transport server to CU4.
I tried to run the unattended setup.exe as follows
setup.exe /mode:Install /role:EdgeTransport /IAcceptExchangeServerLicenseTerms
In the setup log, it shows the following snippets of msgs
….
RuntimeAssembly was started with the following command: ‘/mode:install /role:EdgeTransport /IAcceptExchangeServerLicenseTerms /sourcedir:D:\ExchangeServer2016-x64-cu4’
…
[ERROR] Setup encountered a problem while validating the state of Active Directory: ADAM is installed on this machine; only the Microsoft Exchange Edge Transport server role may be installed…
…
‘GatewayRole’ is installed on the server object.
The installation mode is set to: ‘Install’.
…
…
Applying default role selection state
…
InstallModeDataHandler has 8 DataHandlers
RootDataHandler has 1 DataHandlers
Mailbox role: Transport service
…
[ERROR] The Edge Transport role cannot be installed with other roles.
[ERROR] Transport service cannot be installed without Mailbox service.
…
… Setup did not complete!
Could you please advise if I’m updating the EdgeTransport server incorrectly?
Thank you.
Noel
You’re using the wrong command line switches for an upgrade.. It’s not /m:install to perform an upgrade. Use /m:upgrade instead.
Also CU4 is way out of date now. The latest release is CU10, and only CU9 and CU10 are currently supported.
Hi Paul,
Thanks for a great article (again). We are in the middle of migrating from 2010 to 2016. I was wondering if I should the steps you’ve outlined above, or perform a server switchover to perform the monthly updates to the OS on the 2016 servers.
Best regards,
Duncan
Should you perform the same maintenance steps for installing OS updates? Yes. The same maintenance steps apply to pretty much any maintenance/update/planned outage scenario.
how to Perform the Active Directory schema changes and updates ?
Hello,
What is the exact procedure to patch exchange server 2016 ?
Do i need to update AD schema as well ?
Thanks in advance !
Read the article above.
Hi Paul,
Thank you very much for kind reply!
Can you please let me know how to update the schema while performing CU update?
I know the procedure of schema update while doing Exchange server installation.
Best regards,
Anup
Setup.exe /PrepareAD
Exchange 2016 has its own StartDagServerMaintenance.ps1 and StopDagServerMaintenance.ps1 scripts in *ExchangeInstallDir*\scripts – are they not written explicitly for Exchange 2016, or are those just old versions included?
They aren’t suitable for Exchange 2016. AFAIK they are just the Exchange 2010 scripts left over.
According to the following technet article, the scripts are still valid. You just need to run other commands along with them.
https://technet.microsoft.com/en-us/library/dd298065(v=exchg.160).aspx
Scroll down to the section titled Performing maintenance on DAG members.
I tested the process outlined in the technet article and there are a couple problems but it does seem to work.
The scripts are called by the wrong name in several locations. The scripts correct names are startdagservermaintenance.ps1 and stopdagservermaintenance.ps1.
They also tell you to run:
.\StartDagMaintenance.ps1 -serverName -MoveComment Maintenance
But left out the -pauseClusterNode parameter. The command should look like:
.\StartDagMaintenance.ps1 -serverName -MoveComment Maintenance -pauseClusterNode
Correction, the command should look like:
.\StartDagServerMaintenance.ps1 -serverName -MoveComment Maintenance -pauseClusterNode
Hey Paul, I am hoping you can point me in the right direction with an issue. I recently took over a Server 2012R2 Exchange 2016 environment. I am trying to get it updated to CU8 and 4.7.1 .net
It is currently on CU2 with .net 4.6.1. My plan was to take it to CU4, update the .net to 4.6.2 and then update to CU8 and finally update .net to 4.7.1. My first problem is CU4 is no longer available to download.
How should I proceed? Update to CU8 and then update .net to 4.7.1? Will this break anything to do it, in that order? This is production environment with standalone single server.
Thanks for all your continued assistance.
See my reply to the comment above yours.
Snap! Looks like you already answered that same question above.
Thanks again Paul!
Hi Guys,
I have an exchange 2016 with CU2 (I know I am way out on updates) and my question is , can I install .Net 4.7.1 now and upgrade straight to CU8?
I know the CU updates allow for upgrade from any version, just not sure how the .Net installation will affect CU2 if I upgrade the .Net 4.7.1 now.
N.B I have a standalone server with all roles and databases on one server. No personalized configurations.
There’s a note on the supportability matrix about situations like yours.
https://technet.microsoft.com/en-us/library/ff728623(v=exchg.150).aspx
“When upgrading Exchange from an unsupported CU to the current CU and no intermediate CUs are available, you should upgrade to the latest version of .NET that’s supported by Exchange first and then immediately upgrade to the current CU. This method doesn’t replace the need to keep your Exchange servers up to date and on the latest, supported, CU.
Microsoft makes no claim that an upgrade failure will not occur using this method, which may result in the need to contact Microsoft Support Services.”
Thanks for the reply, I have done the update this past Saturday. Did the .net 4.7.1 and then CU8.
I did have only one issue whereby OWA did not work. The link and fix i used was from:
https://social.technet.microsoft.com/Forums/lync/en-US/6d67e5ef-555e-41a5-8de9-2a56cf95363a/missing-the-microsoft-exchange-server-auth-certificate?forum=exchangesvradmin
Hello Paul,
I have a 2 node Exchange 2016 DAG setup, currently running CU4
Due to quite a lot of know bugs – and to be fully up2date – I want to update my Exchange 2016 DAG to CU8 as soon as possible.
What would you recommend I do ?
– Taking 2 CU´s at a time (I.e. from CU 4 to CU 6 – and then from CU 6 to CU 8 ) as I think is Microsofts best (and tested)
practise ?
or
– Should I update directly from CU4 to CU8 ?
Please advise here 🙂
Thanks
BR
Peter, Hostnordic
Microsoft supports updating from any CU to the latest.
Microsoft only tests the N-2 upgrade scenarios though.
I’m not aware of any specific issues with your scenario. Given you’ve got a DAG i would probably just go straight to the latest one.
Thanks for the great article Paul.
I have a 4 node DAG exchange 2016 setup behind a hardware load balancer. All running CU2. I upgraded one of the server to CU4 then CU5. All seemed to go well until I added the server back into the load balancer. Random users started getting kicked out of Outlook and being prompted for user name and pw. They cant get back in until I remove the updated server from the load balancer. After doing a lot of digging in many different log files I think I found something. In the IIS Exchange Back End logs it looks like the only traffic between the updated server and the rest is the Healthmailbox. On all other three servers I seem users being redirected to other servers, but not on the updated server. I’m just not sure where to go from here. Any suggestions??
Thanks so very much!!
small type-O:
You wrote Exchange 2013 instead of 2016 at Perform a Health check of Servers.
Thanks a lot Paul.Much Appreciated.
Hello Paul,
Thanks for the wonderful article. I’m facing an issue when trying to run the upgrade setup from Exchange Powershell I received the following error message and setup fails
Please help
Performing Microsoft Exchange Server Prerequisite Check
Configuring Prerequisites COMPLETED
Prerequisite Analysis FAILED
Setup can’t continue with the upgrade because the powershell (3564) has open files. Close the process, and then restart
Setup.
Exchange setup isn’t run from PowerShell. Use a CMD prompt. You will need to close down any PowerShell sessions that have Exchange files locked as well (e.g. close all Exchange Management Shell windows).
Paul, I am updating from EXCH 2016 RTM (15.1.225.42) to EXCH 2016 RU6. We are in O365 now and only keep this server around to allow for the receive connector to send email anonymously (UPSs, Storage devices). Is there anything that we need to be concerned with in 0365?
BTW, great article. I’m never let down when reading your posted content.
No. You’re out of date so you definitely need to update (MS wants you at latest or N-1 for hybrid deployments).
Hi Paul,
When referencing “If the server is a DAG member, run the following commands. If your server is not a DAG member, skip to the command for setting ServerWideOffline.”, I do not see the command for setting ServerWideOffline. Could you provide that for me?
Thank you.
It’s there. Do a CTRL-F and search for the word and you’ll find it.
Hi Paul,
Sure is, totally missed it. Sorry about that.
Lodctr exited with error code ‘2001’.” is probably caused by the presence of mdbperf.ini in C:\Program Files\Microsoft\Exchange Server\V15\Bin\perf\AMD64
Remove it and the CU installation will work again. See here more:
https://social.technet.microsoft.com/Forums/de-DE/98a831e8-505b-4c57-b6f4-c064cbb8c6ac/performance-counter-names-and-help-text-failed-to-unload-lodctr-exited-with-error-code-2001?forum=Exch2016GD
Hi Paul,
Im experiencing a weird issue. Its really not giving me a more descriptive note with regards to the problem. It only says “Performance counter names and help text failed to unload. Lodctr exited with error code ‘2001’.” ive been stuck in language installing ever since. i tried doing the lodctr /r and other things from google but with no luck.
Hello all,
I had two Exchange 2016 servers in my infrastructure and I started to upgrade them to Exchange 2016 CU3.
One of the servers has been updated successfully, but the second one ran into a problem at ‘Client Access Front End service’ stage.
You can see the CMD output below.
F:>Setup /m:upgrade /IAcceptExchangeServerLicenseTerms
Welcome to Microsoft Exchange Server 2016 Cumulative Update 3 Unattended Setup
Copying Files…
File copy complete. Setup will now collect additional information needed for installation.
Languages
Management tools
Mailbox role: Transport service
Mailbox role: Client Access service
Mailbox role: Unified Messaging service
Mailbox role: Mailbox service
Mailbox role: Front End Transport service
Mailbox role: Client Access Front End service
Performing Microsoft Exchange Server Prerequisite Check
Configuring Prerequisites COMPLETED
Prerequisite Analysis COMPLETED
Configuring Microsoft Exchange Server
Language Files COMPLETED
Restoring Services COMPLETED
Language Configuration COMPLETED
Exchange Management Tools COMPLETED
Mailbox role: Transport service COMPLETED
Mailbox role: Client Access service COMPLETED
Mailbox role: Unified Messaging service COMPLETED
Mailbox role: Mailbox service COMPLETED
Mailbox role: Front End Transport service COMPLETED
Mailbox role: Client Access Front End service FAILED
The following error was generated when “$error.Clear();
.
“$RoleInstallPathScriptsUpdate-AppPoolManagedFrameworkVersion.ps1″ -AppPoolName:”MSExchangeServicesAppPool”
-Version:”v4.0″;
get-WebServicesVirtualDirectory -server $RoleFqdnOrName | set-WebServicesVirtualDirectory
-windowsAuthentication:$true -WSSecurityAuthentication:$true -OAuthAuthentication:$true
” was run:
“System.Runtime.InteropServices.COMException (0x800700B7): Filename: \?C:Program FilesMicrosoftExchange
ServerV15FrontEndHttpProxyEWSweb.config
Line number: 8
Error: Cannot add duplicate collection entry of type
‘add’ with unique key attribute ‘key’ set to ‘HttpProxy.ProtocolType’
at
Microsoft.Web.Administration.Interop.IAppHostAdminManager.GetAdminSection(String bstrSectionName, String bstrPath)
at
Microsoft.Web.Administration.Configuration.GetSectionInternal(ConfigurationSection section, String sectionPath, String
locationPath)
at
Microsoft.Exchange.Management.SystemConfigurationTasks.ExchangeServiceVDirHelper.EwsAutodiscMWA.EnableOrDisableCafeEndpo
int(Configuration
configuration, String endpointName, Boolean enableEndpoint)
at
Microsoft.Exchange.Management.SystemConfigurationTasks.ExchangeServiceVDirHelper.EwsAutodiscMWA.OnSetManageWCFEndpoints(
Task
task, EndpointProtocol protocol, Boolean enableWSSecurity, ExchangeVirtualDirectory adVirtualDirectory)
at
Microsoft.Exchange.Management.SystemConfigurationTasks.SetWebServicesVirtualDirectory.InternalProcessRecord()
at
Microsoft.Exchange.Configuration.Tasks.Task.b__b()
at
Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean
terminatePipelineIfFailed)
at Microsoft.Exchange.Configuration.Tasks.Task.ProcessTaskStage(TaskStage taskStage,
Action initFunc, Action mainFunc, Action completeFunc)
at Microsoft.Exchange.Configuration.Tasks.Task.ProcessRecord()
at System.Management.Automation.CommandProcessor.ProcessRecord()”.
The Exchange Server setup operation didn’t complete. More details can be found in ExchangeSetup.log located in the
:ExchangeSetupLogs folder.
F:>
At this stage I cannot do anything with the server, and /owa is not working anymore. It seems that I have a corrupt installation.
Do you have any ideas how can I solve the error and ran the installation task again?
Regards,
Mircea
Hi Paul – Great information as always. I have a question about removing the server from maintenance mode. What is the difference between Set-ServerComponentState EX2016SRV1 –Component ServerWideOffline –State Active –Requester Maintenance and the subsequent same command but for HubTransport? Doesthe ServerWideOffline also take care of the HubTransport?
No, ServerWideOffline is a separate component state to HubTransport, so they both need to be set individually.
Hi,
very useful info. Do you recommend performing this on live servers? I usually do regular Widows updates during scheduled downtime (i.e the wekend), will this procedure cause issues to end users or does it still need downtime? (for a system with multiple CAS servers and a DAG with 3+ copies of each database).
Yes. What’s the point of high availability if you still need to patch late at night and on weekends?
Great thanks, will try it this week!
Hi Paul,
Great job with your documentation of the install process, thank you!
I’ve installed 2016 RTM a while ago and accidentally installed .Net 4.6. I don’t recall if it was before or after installing Exchange. So if you’ve already installed .Net 4.6, can you still install CU2? I see that it’s recommended to install Exchange 1st then .Net but that’s not the my current situation.
If not, how would you recommend proceeding? Thank you.
If you mean 4.6.1, I believe the guidance is to uninstall it (using the correct steps) and only reinstall it after you’ve upgraded Exchange to a supported CU, and make sure you apply the appropriate hotfixes as well.
More info:
https://www.practical365.com/net-framework-4-6-1-and-exchange-compatibility/
Yes, 4.6.1 thanks Paul.
Will the removal of 4.6.1 cause any problems with the current install of Exchange if I don’t have time to install CU2 right after the uninstall of 4.6.1?
No. It is recommended if you’ve inadvertently installed 4.6.1 to remove it following the guidance. You don’t need to immediately install or reinstall Exchange afterwards.
Hey Paul, running into an issue during setup and followed your instructions to a “T.”
Here is the error that’s getting thrown. It seems to be a permissions issue but I can’t figure out why. I am running from an elevated cmd.exe and I have all the appropriate permissions on the Exchange server.
Write-ExchangeSetupLog -Info (“An exception ocurred while configuring
Search Foundation PowerShell Snapin. Exception: ” + $_.Exception.Message);
}
}
” was run: “System.Exception: Failure cleaning up SearchFoundation Data
folder. – C:Program FilesMicrosoftExchange
ServerV15BinSearchCeresHostControllerData – Exception calling “Delete”
with “2” argument(s): “Access to the path
‘Microsoft.ClientResourceView.FlowService.dll’ is denied.”
at
Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception,
ErrorCategory category, Object target, Boolean reThrow, String helpUrl)
at
Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception,
ErrorCategory category, Object target)
at
Microsoft.Exchange.Management.Deployment.WriteExchangeSetupLog.InternalProcessRe
cord()
at Microsoft.Exchange.Configuration.Tasks.Task.b__b()
at
Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String
funcName, Action func, Boolean terminatePipelineIfFailed)”.
Any help would be greatly appreciated.
Locked file perhaps by a service still running, or antivirus has it locked.
Hi Paul. Great Arcticle. I have a problem with the unattenend upgrade – First it shows CU1 but it is definetly CU2 – When i start the GUI installation i see CU2… very strange…
But the setup hang @ Restoring Services COMPLEDTED and nothing happen anymore. Do you know what can be the Problem ?
Thanks for help
Running from powershell.exe or cmd.exe? You should be using cmd.exe.
Can the exchange 2016 scripts: StartDagServerMaintenance.ps1 and StopDagServerMaintenance.ps1 be used for this ?
If not, why not ?
No. The scripts don’t do all the steps that are recommended for DAG member maintenance.
Thanks Paul, do you think that the Microsoft exchange team will change the scripts accordingly. I guess these scripts are there for a reason namely to do maintenance on a Dag member server.
No, I don’t anticipate them putting any development time into those scripts. They were originally introduced in Exchange 2010, and were recommended for use with 2010. But that had a different maintenance process than 2013/2016. The scripts were changed, but like I said they don’t include all the recommended steps for performing maintenance (same is true for both 2013 and 2016).
Hello Paul,
We run Exchange 2010, but are moving to 2 new Edge servers in a different DMZ. First off I setup 2 Exchange 2016 (Edge Servers) in the new DMZ, but am not sure if I have to PrepareAD internally to successfully introduce them. Because they are the latest version (and internally its still all 2010) is it necessary to prep AD and the schema first?
Thanks for this awesome post.
Do you have or know a maintenance script?
Michael Van Horenbeeck has one for Exchange 2013 that he published (do a Google search to find it) but I don’t know if it works the same for Exchange 2016. I suspect so.
At first glance, the answer is “no”:
—
WARNING: The specified Exchange Server is not an Exchange 2013 server!
WARNING: Aborting script…
—
I’m sure MVH will update this at some point.
If he hasn’t updated it by now …. 🙂
A keen community member might patch it and send him the updated code.
Hi Paul, you’re the best! <3
Here is my function to manage maintenance on 2016 DAG node:
https://github.com/tivrobo/exchange-scripts/blob/master/SetDagServerMaintenance.ps1
Greetings Paul ,
I have come across a weird issue while upgrading My exchange 2013 Mail Box server from CU 5 to CU 11 and would dearly need your help. We have 4 2013 servers running in our environment 2 CAS and 2 MBX servers. Have successfully upgraded the first 3 servers to CU 11 without any issues and this is the last server in the organization. Find below the Details of the issue.
1. The server is a Mailbox server participating in a DAG. The other mailbox server is already upgraded to CU 11 without any issues.
2. Have tried installing CU 11 upgrade through GUI, intial screen appears and disappears immediately. Exchange setup logs do not have any conclusive logs.
3. Tried to install through powershell (setup /m…..), the setup failed but with some conclusive error messages. please find below a snippet of the log files.
[04/20/2016 06:11:31.0475] [0] The following roles have been unpacked: BridgeheadRole ClientAccessRole MailboxRole UnifiedMessagingRole AdminToolsRole
[04/20/2016 06:11:31.0475] [0] The following datacenter roles are unpacked:
[04/20/2016 06:11:31.0490] [0] The following roles are installed: BridgeheadRole ClientAccessRole MailboxRole UnifiedMessagingRole AdminToolsRole
[04/20/2016 06:11:31.0537] [0] [ERROR] Exception has been thrown by the target of an invocation.
[04/20/2016 06:11:31.0553] [0] [ERROR] Requested value ‘15.0.913.22’ was not found.
[04/20/2016 06:11:31.0553] [0] CurrentResult SetupLauncherHelper.loadassembly:444: 1
[04/20/2016 06:11:31.0553] [0] The Exchange Server setup operation didn’t complete. More details can be found in ExchangeSetup.log located in the :ExchangeSetupLogs folder.
[04/20/2016 06:11:31.0553] [0] CurrentResult main.run:235: 1
[04/20/2016 06:11:31.0553] [0] CurrentResult setupbase.maincore:396: 1
[04/20/2016 06:11:31.0568] [0] End of Setup
The things which i derived from the setup logs
1. It is clear that the steup is failing when the setup tried to find the adminversion ID of the CU5 installer15.0.913.22.
2. Now, when i see the exchange setup logs of the servers where the installation was successful, it says 15.0.913.22 was present.
3. Would dearly want to know whether you have faced this issue , or a solution would be nice. even now the setup fails with the same error ” [ERROR] Requested value ‘15.0.913.22’ was not found.”
Thanks in Advance.
eric
Hi Paul,
Would appreciate a reply.
Hi Paul,
Thanks for this VERY useful post (thus, thanks for your entire blog, and the knowledge you share with us)
Some Technet articles or blog posts (like this one: https://technet.microsoft.com/en-us/library/dd298065(v=exchg.150).aspx#Pm ) mention that the transport services should be restarted to take into account their new state (e.g. draining, active, …)
From your experience, is this necessary or not?
Regards
In my experience it has not been necessary, but it’s been coming up in conversation lately so I assume Microsoft is making that recommendation for good reasons.
Hi,
To be able to start setup.exe /prepareschema ou prepareAD from E2016 source files, you do require to use Windows 2012+ server OS. So if it is done from your Schema master AD server, it has to have that OS (Which might not be the case)
Regards.
HI Paul, thank you so much for your blog. I’m a new Exchange Server Administrator and your tutorials are invaluable to me. I have 2 questions:
1) If I run the schema update with the setup.exe /PrepareSchema or the setup.exe /PrepareAD switch and is not required by the CU, will I break anything in by doing this?
2) At the beginning of the Article you said “Verify that you have documented any customizations to your Exchange server that will need to be re-applied”. If I inherited the Exchange environment, how will I know what has been done? or how can I check this in more detail?
Thank you in advance.
1) No
2) Look for things that make your Exchange installation different from a “vanilla” installation, such as customized OWA login pages, integration with third party products that are installed on the Exchange server, that sort of thing.
Hi Paul,
Thank you very much for the information and your article!
I’m currently plan to migrate from Ex2010 to 2016 and have installed Ex2016 RTM with 2x ExNodes (DNS-RoundRobin) and 1x DAG, mainly with your instructions. Again, THX for this great Job! 🙂
At the past, I’m using for my Ex2010 enviroment PS-Script ‘StartDagServerMaintenance.ps1’ for maintenance start and ‘StopDagServerMaintenance.ps1’ for maintenance stop.
At some other posts I can found for Ex2013 Server Update to use also post scripts ‘UpdateCas.ps1’ and ‘UpdateConfigFiles.ps1’ scripts.
… “After each installation of a cumulative update for Exchange 2013, remember to execute both the UpdateCas.ps1 and UpdateConfigFiles.ps1 Windows PowerShell scripts.”
Here some questions:
1. Should the script ‘StartDagServerMaintenance.ps1’ no longer be used?
2. If yes, in which order and combination with your post information?
3. Should the both ‘UpdateCas’ scripts are also executed after the CU update?
4. How is the order at a Exchange coexistence (Ex2010/Ex2013/Ex2016) to expand the Scheme/Forrest/Domain and updating each ExNode?
Excuse my bad English and greetings.
1. Correct, the 2010 maintenance scripts should not be used for 2013/2016.
2. n/a
3. I’ve never found it necessary for 2013, and only once have needed to run UpdateCas.ps1 for a 2010 server, many, many years ago.
4. Schema/AD update is performed once only. Update the highest version of Exchange first.
Hi Paul,
THX for your reply.
Was not really clear for me, but I have now understood. Updated yesterday Exchange 2016 CU1 with no problem, thanks to your guidance.