Update March 2021: This post relates to issues found in March 2020. A new, more serious set of security issues have been found in March 2021. See Tony Redmond’s post on the topic for more information and then join our live panel discussion on protecting Exchange Server from #HAFNIUM attacks on March 12.
If you haven’t already done so, this week you should be applying patches to your Exchange Servers. A reasonably easy to exploit vulnerability has been disclosed by Microsoft as CVE-2020-0688.
Sigi and I talked about this in this week’s episode of The Practical 365 podcast, but this is important enough to write about separately in case you missed the show.
The TL:DR version of this is – anyone with the username and password for a mailbox on your Exchange Server and access to the /ecp directory can execute code as the SYSTEM account on the server itself.
The proof-of-concept exploits show this can be achieved by sending it an easily crafted URL using freely obtained open source tools.
This vulnerability is due to Microsoft using the same set of cryptographic keys on every Exchange Server installation. The keys being stored in plain text in a web.config file on every server.
The bad news is that although it has taken nearly ten years for this to be noticed, now it has, would be attackers are acting quickly. Security researchers are already seeing scans for public-facing Exchange servers to obtain build version information. If you publish OWA and ECP externally the next step by these malicious people will be to attempt to gain a set of working credentials to access Exchange. There are already tools available to use social networks like LinkedIn to harvest email addresses and of course tools to automate login attempts.
Make no mistake – if you publish Exchange externally and don’t patch, it’s only a matter of time until this is exploited.
Even if you don’t publish Exchange externally, you should consider this a threat. Not all staff follow the rules and it’s possible an insider within your organization may use this opportunity to take advantage of the rights an Exchange computer account has. The amount of damage someone could do easily shouldn’t be understated.
Find out more: Working with the new Exchange cmdlets
Simon Zuckerbraun, of the Zero Day Initiative, has produced a comprehensive write-up on their blog that provides a vast amount of detail about the vulnerability including a walk-through on how to exploit it. It makes for worrying reading – all an attacker needs is a compiled version of ysoserial.net, the validation key from either his article – or any Exchange Server– and then by crafting a URL GET request to /ecp/default.aspx they can do as they wish. If that hasn’t worried you yet, read the article now.
Getting patches for Exchange
Patches are available for all support versions of Exchange Server – from 2010 through to 2019 and all are affected.
The update comes in the form of an Update Rollup, UR30, for Exchange 2010, and security updates for the latest Cumulative Updates for Exchange 2013, 2016 and 2019. If you aren’t running the latest CUs for Exchange 2013 or higher, then you will need to get current as well as subsequently applying the patch.
You can find links to the patches on the Microsoft Security Response Center page for CVE-2020-0688.
Does applying updatecas.ps1 deprecate the KB5000871 exchange security update?
We installed Exchange CU 23 yesterday. A few notes from our experiences:
1. Make sure you install .Net 4.8 prior to the install
2. As others have mentioned, the install needs to be run as an admin
In our case, the CU failed updating the mailbox database service. Luckily, our server is still running but we are getting odd errors related IIS crashing and have to periodically reboot the server. We have a call in to Microsoft but it has been a full 24 hours at this point and we have not gotten a response.
mine crashed on a system account with insufficient creds (no oabgen).. it broke search.. I fixed the system account, reran, and it straightened out.. so if you haven’t approached it that way it might be an option.
We installed our’s last week as well, issues encountered is that the OWA/ECP broke in our environment.
We followed instructions on KB article but did not mention of the .NET 4.8 being installed first before the security update.
I’m wondering if that’s causing our OWA issue in our case.
We followed the KB suggested by Microsoft about OWA/ECP being broken after CU security update but issue still persists:
https://docs.microsoft.com/en-us/exchange/troubleshoot/client-connectivity/owa-stops-working-after-update
I’ll install 4.8 and re-apply the CU23 security update tonight I’ll see if this resolves our issues.
Were you able to hear a word from Microsoft, I’m curious to know how they responded to your issue.
Also, I noticed that there’s a new release for the Security Update but only applied to 2016/2019 Exchange servers. Wondering if Microsoft discovered bugs on the initial release.
I’d suggest dropping into Tony’s post on the topic here – remember this post is about the security issues from a year ago (2020!). There are new releases today, to cover older CUs and make things less painful for those updating.
https://practical365.com/blog/microsoft-issues-critical-security-updates-for-exchange-server/
Hi Steve,
a Customer have an atypical configuration with two Exchange 2013 with Cu23 installed. One with Client Access role, the second with Mailbox role.
I wonder if i should installed KB4536988 on both servers like any other Cu (Mailbox first, CAS after) or simply on CAS.
Thank you in advance for your reply
we did cas first since its the first box they’d hit to crack at
Ah great… we are still on exchange 2007…
I like your blog and getting spun up on Exchange as this will be a full-time job for me. Appreciate the feedback from everyone. I’ll save this blog as a favorite.
Hi guys
After logging with Microsoft, the issue was this:
“https://support.microsoft.com/en-us/help/3032024/outlook-web-app-and-ecp-redirect-to-the-fba-page-in-exchange-server-20”
Interesting to note: the public cert worked fine before the patch, and then the restriction of no support for “CSP=Microsoft Storage Key Provider” kicked in and the cert did not comply.
Also, the certificate CSP conversion process (explained in the Tech Article) failed on the existing cert, so a re-issue (with the Provider = Microsoft RSA SChannel Cryptographic Provider in the new request) was requested from Digicert, installed with services, and OWA & ECP started working correctly.
Hope this helps someone with this issue…
Hey guys,
I just installed on Exchange 2013 CU23 and everything went fine with the installation itself, but I noticed that i could not search in Outlook. I checked the Content IndexState of my databases and they were all failed.
It looks like it set the Microsoft Exchange Search Host Controller to Disabled. I changed this to Automatic and started it and restarted the Microsoft Exchange Search and everything was showing Healthy in about 5 minutes.
We use Zerto for DR replication and I performed a test failover to prior to the patch and it was set to automatic confirming that after the patch it was Disabled.
I can’t speak for other Exchange versions, but this is something for people to keep an eye on while applying this patch
I ran into this same issue. The security patch must be ran from an elevated CMD. I didn’t see that on the fine print initially and it led to a corrupted OWA and ECP virtual directory.
It still installed without showing an error but we started getting calls about users getting a 400 error when trying to access via OWA.
Hi, did you have a resolution to this issue?
this.. plus you cant right click, run as admin, on the cu install.. you have to invoke it from the command line of an elevated cmd window
Hi Steve, I have just applied Exchange 2016 CU15 yesterday. DO i need to still install this patch or it would be inbuilt as this Patch is available since Feb.
I am a bit afraid hearing about ECP/OWA break issues.
Please suggest.
Virsingh,
This security patch is NOT built into any CU for any version of Exchange. It is a separate installation that can only be installed once the Exchange system is upgraded to the latest CU.
As above – you need to install the security update *in addition* to the CU.
If your environment doesn’t expose exchange urls externally then locking down the internal ECP virtual directories with the IIS IP Address and Domain restrictions feature is likely a decent temporary option until patching can be done. Can lock down ECP to subnets or IP’s only your admin staff have access to.
Hello and yes, that is what I did. We do not publish OWA externally at all, in fact, all of our desktops use OWA internally instead of using a full Outlook client. It is just a cleaner, leaner solution that works for us. Hence, the IP based restriction for the ECP default web site (in IIS) worked very well. Set it to 127.0.0.1 as the only address that can access the ECP, so you must be physically at the server to have access, etc. Thanks for the suggestion!
Great write up Steve. Couple of things to add. Make sure you run this patch from an elevated command prompt else it may not install all files.
There has also been some people having issues with it on Exchange 2013 where it breaks OWA/ECP. The fix is a bit two fold, I have a link for all the details but didnt want to share it without asking first.
We personally haven’t had any issues with any of our customers but worth others being aware.
I’m interested in the post-patch OWA/ECP fix if you’d care to share?
I managed to fix OWA with updatecas.ps1 script, ocated in Exchange scripts folder.
Sorry, cant paste link, reported as spam???
Hi Chris/Paul
I have a client who has just run this update Exch 2013, CU23 (but NOT from an elevated prompt …!), and now neither ECP or OWA are functional.
The Login page for both appears, but after entering password, the screen simply keeps requesting a password.
I have tried a number of tasks to resolve, but none have been successful.
I will attempt to run the patch again this evening (with elevated prompt !), but would be eager to try any other fixes in the meantime…
I’m also interested in the possible post-patch fixes.
Ralph
Ralph,
Did that work? I installed CU23 and the patch. A couple weeks later OWA/ECP are broke. Waiting on a call from MS but, they are taking forever.
Hi Chris,
Could you please the solution/fix of ecp/owa break? I installed the cu23 with this security patch and things are not good after that.
My server’s running 2013 sp1 (CU4),if i install CU23, my server works well? Please help me to update CU23.
Even though every Cu is a new exchange install, the prerequisites are different, for example, your .net version would need to be updated etc. I would advise you to read the prerequisite first before jumping to the latest CU especially since you are so far out on updates.
I have updated my Exchange 2016 Cu14 with the Cu14 patch however my ecp does not work any longer, OWA works,I get error HTTP Error 500.19 – Internal Server Error only when trying ecp.
I’ve not personally seen the same thing with this patch, but would check:
– You’ve rebooted the server
– When you ran the patch you did so from an elevated command prompt – I don’t know if it’s safe to run the patch again if you didn’t do so from an elevated command prompt but would personally consider doing so.
If either of those don’t help – then until someone else reproduces it and can solve it then a first point of call would be to raise a SR with Microsoft, as poking around is likely to make it worse.
This is a known / published issue with exchange updates, when not run from an elevetad prompt
Update to my own comment, I noticed that the ecp web.config file was blank, restored it from backup, did an iisreset and ecp back up and running!
Thank you! That might well help someone else reading this
I would assume exchange online users are safe
There is no more recent CU for 2013 than CU23, which is from July 2019. So apparently there is no patch for 2013.
Hi Jeff,
The link to the security update for Exchange 2013 is broken in the Microsoft article.
The KB (https://support.microsoft.com/en-us/help/4536988/description-of-the-security-update-for-microsoft-exchange-server-2013) does link to it though.
It is a patch – a security update to apply *to* CU23, rather than a new CU. The release date of the CU is Feb 2020.
https://www.microsoft.com/en-us/download/details.aspx?id=100909
Steve
Does applying updatecas.ps1 deprecate the KB5000871 exchange security update?
There is, if you have the CU23 installed, all you need is this patch. I just finished installing CU23 and this patch after without any issues.
Hi Steve
What happens if you still running Exchange 2010 SP3 on Server 2008 R2? I believe RU30 for Exchange 2010 SP3 is not available for Server 2008 R2, unless I am mistaken.
We are currently in processing of deploying Exchange 2016 within our organization.
Appreciate any advice.
No problem Exch 2010 SP3 and Server 2008 R2
I alredy do it on a client.
I agree with Rafael, RU30 should apply like any existing update rollup for Exchange Server 2010 SP3.
Steve
Exchange 2010 stops at 2008 R2 so it will be available.
https://docs.microsoft.com/en-us/previous-versions/exchange-server/exchange-140/aa996719(v=exchg.140)?redirectedfrom=MSDN
Practical365 is my go to resource reading for Exchange, so wanted to share my “scary CU stories to tell in the dark”.
An environment I was brought on to work last week were still on 2016 CU4 (version December 13, 2016). I needed to get them up to CU15, then apply the related patch that is only compatible on the newest and one version back per Microsoft’s SOP.
It was…a long weekend, but overall came out of the tunnel successful. Going that far in CU’s is always scary, so many changes, improvements, items that can fall over – but wanted to take this latest exploit as serious as it really is for the clients health. Plenty examples are now published of how to leverage it, there are strong hints of scanning afoot of bad folks attempting to locate exploitable company servers.
Best advice for other Exchange admins: if you have a history of holding on to a CU, choose today to correct that situation, don’t be an easy target.
Well done on what was likely a tough weekend. Moving forward that far, plus applying security updates is not a small undertaking.
My immediate actions with my clients that I am mid-project with has been to mitigate against the risk urgently before getting patches installed. For those mid-migration patching is difficult as well, especially long migration jobs that can’t just be halted.
I agree – keep up to date and it makes not only applying the patches easier but also makes sure there is less fear of applying CUs and patches.
Hello, would you mind fleshing out a bit more the phrase “mitigate against the risk urgently” since, from the context of your comments, I assume this means these clients are mid-project and you cannot immediately update versions / apply patches, etc. Thanks!
If you can’t update or patch your environment right away, you can stop the ECP application pool in IIS as a temporary security measure.
If it’s not possible to update your platform, you can manually change the Machine Key for your ECP applications in IIS. Although this is not a supported solutions, it seems to work.. but you should really update. 🙂
Yes, I would agree that that should be fine on the face of things, though Microsoft aren’t suggesting it as a workaround. The issue often with disabling ECP is it is the options page in OWA, so it can have unintended effects. However as a temporary measure, doing as above, or stopping publishing that specific path externally (no solution for insider threats, but..) can work until the patch window.