Wired Calls Exchange Server a Security Liability
In March 2021, during the middle of the Hafnium attack against Exchange servers, I said that the attack gave impetus for on-premises customers to move to the cloud. Eighteen months later, things have not improved. If anything, they might be even worse despite Microsoft’s best efforts to mitigate attack vectors and close off holes in the venerable Exchange code base.
Recent day-zero vulnerabilities underline the point. But you know things have reached a critical point when Wired magazine calls Exchange server a security liability in a well-argued article by Andy Greenberg, summarized with “it’s time to say goodbye to on-premise(s) Exchange.”
Greenberg points to a stream of vulnerabilities (like ProxyRelay, described by researcher Orange Tsai), ongoing hacking campaigns, and the difficulty organizations often experience when they try to patch Exchange servers. These are facts. Exchange Server is a huge target for attackers because it was such a popular on-premises email server. Although the number of servers is gradually reducing, there’s still a lot of servers out there for attackers to pursue.
On Demand Migration
Migrate all your workloads and Active Directory with one comprehensive Office 365 tenant-to-tenant migration solution.
Five Reasons Why Exchange Server is Vulnerable
Customers might have some questions about why Exchange Server is so vulnerable. I think it boils down to five factors:
The age of the code base. No version of Exchange Server was designed to deal with the kind of threat-filled environment now prevalent on the internet. The web components in the current release (Exchange 2019) use an architecture laid down fifteen or more years ago (arguably for Exchange 2003 but definitely for Exchange 2007). The dependency on IIS and the difficulty in configuring web virtual directories to make sure that OWA is secure reflects thinking that wouldn’t happen today.
The reluctance of the installed base to move to new server versions. Even if Microsoft closes holes and improves the security of Exchange Server, their efforts are worth precisely zero if customers don’t upgrade their servers. Exchange has always been a slow application in terms of moving forward, largely because of the requirement for new hardware.
For example, Exchange 2019 mail servers have a recommended memory of 128 GB to allow Exchange to cache “hot mailbox data” and improve performance. Many organizations run Exchange on virtual machines hosted on VMware or Hyper-V, introducing another complication in the upgrade process.
The fragility of the upgrade/patch process: It takes too long to apply security updates or the regular quarterly cumulative updates. Servers must be taken offline, and everything must be right (including Windows patches) before an update will install. Even then, odd things can happen to prevent an update from completing. These things work out in the end, but update difficulties have caused too many grey hairs for Exchange administrators, as I know to my cost.
The detachment of the Exchange engineering team: There was a time when the Exchange engineering team was very connected to its customers. I don’t believe this is the case any longer for two reasons. First, the most experienced engineers have moved to new positions within Microsoft in roles that concentrate on the Microsoft 365 substrate or Exchange Online. These folks knew the product inside out. They also had great connections with customers and MVPs and a genuine appreciation of operational challenges. The current engineering team responsible for Exchange Server just doesn’t possess this background. They are talented people, but their connectivity with the real world is not the same as it was.
The lack of connection with their base is compounded by the demise of the in-person Ignite conference. Traditional Ignite conferences allowed engineers to interact with customers in a very personal manner. People brought technical issues to the conference to debate with engineers. It was a perfect two-way learning conduit that hasn’t happened since 2019. The recent “hybrid Ignite” is not the same. The rumor is that Microsoft will try and bring back an in-person Exchange Conference (MEC) in 2023. That might help, but I suspect that the content will focus on Exchange Online because that’s where Microsoft’s future lies.
Microsoft’s use of Exchange Online: Microsoft eats its own dog food, but the current dog food is cloud-flavored. If Microsoft used Exchange Server as its mail server, we might see more aggressive action to harden the server, close holes as they emerge, and introduce new software solutions to improve updates and patches. For instance, Microsoft might have been as assiduous in removing basic authentication from Exchange Server as they have been for Exchange Online.
Microsoft generates $100-plus billion annually from cloud services, and that’s where its focus will remain. The number of resources Microsoft assigns to on-premises server engineering will only decrease over time.
Exchange Online
Exchange Online is very different from Exchange Server, and the gap widens all the time, not least because of the central role Exchange Online plays for the Microsoft 365 substrate. Although Exchange Online spans over 200,000 physical mailbox servers, the risk of compromise is much lower than for any on-premises environment because of the security resources Microsoft dedicates to protecting its cloud infrastructure. Quite simply, few other companies could afford to erect and manage the same kind of defenses.
Time to Take the Migration Pain
I don’t know how many Exchange servers remain operational in on-premises organizations. The FBI found tens of thousands of servers to patch last year. Given that Office 365 has more than 345 million paid seats (almost all of whom use Exchange Online), the bulk of the migration from on-premises Exchange is over.
Some organizations that remain absolutely need to run on-premises (military servers are the classic example, including those on submarines). Many of those organizations have the security smarts to be able to defend their infrastructures. Others don’t, and that fact is obvious because of the ongoing level of attacker interest in exploiting Exchange flaws. Putting the special cases to one side, any regular commercial implementation of Exchange Server must ask if things have become so bad that they should migrate ASAP, even if they follow Paul Robichaux’s security principles for Exchange Server.
Different circumstances affecting companies will influence the decision, but at this point, it just makes sense to remove themselves from the target list and migrate. Migrations are painful and costly, but a compromised Exchange server is so much worse (as people discover on an all-too-frequent basis).
Microsoft Platform Migration Planning and Consolidation
Simplify migration planning, overcome migration challenges, and finish projects faster while minimizing the costs, risks and disruptions to users.
Managing Ex on prem for 15+ GW before that for years with minimal issues. There are more problems and outages in the cloud than there was on prem. This is just my optics, but I feel I have my hands in my pocket telling 50k users…Sorry cloud issue nothing I can do.
Hi Tony, thanks for sharing. When you say the bulk of the migration is over, do you have a sense of how many users or customers are still out there on prem?
Who knows? Microsoft isn’t saying and we never had very accurate data about the number of Exchange on-premises mailboxes to work with. If pushed into a corner, I’d say maybe 10% of the original base, which could be about 40 million accounts. But this is guessing.
Does your opinion consider organizations with on-prem exchange servers only supporting mail relay functionality?? In an ideal world, all of our on-premises devices support modern authentication, but that isn’t always the case.
Organizations that only use on-premises servers for mail relay are fine to do that. I am more concerned about mailboxes used by humans. However, it is absolutely critical to keep servers updated and patched, even if they’re only used for mail relay. There are many known vulnerabilities that can affect servers that run outdated software and those servers represent a real risk. So, run Exchange 2016 or Exchange 2019 on your mail relay servers and make sure that you apply cumulative and security updates as Microsoft releases updates and all will be well.
Nice to read so far. From my point of view you could turn your phrase just the other way around:
“ Although Exchange Online spans over 200,000 physical mailbox servers, the risk of compromise is much lower than for any on-premises environment because of the security resources Microsoft dedicates to protecting its cloud infrastructure.”
When they are compromised then everybody is really fried. Not so much comments about the fact that MS relies also on the security of 3rd parties, just to name Solarwind. It was clear that the hackers who came through that door had access to the sources of Windows and and a few month after we have a zero day like Haffnium. I don’t want to put too much into it but is your recommendation really to put all economical eggs in just one basket?
Your argument is mostly about the complexity’s of maintenance and migration. I agree on that and I know what I’m talking because I work closely with Exchange since 5.5 and developed a lot of bad hacks against every version in order to solve issues or crashes of my clients.
You can’t sell it as a self healing worry free product. But tell everybody go to the cloud?
I’m not convinced…
Actually no. Exchange Online is split up across datacenter regions and forests so any compromise won’t penetrate everywhere. Also, compromises tend to rely on access to privileged accounts and there are very few of these, all of which have time-limited access to perform actions. There are other security measures in place that I won’t go into here. In the on-premises world, there are still thousands of unpatched and vulnerable servers in operation (you can’t expect the FBI to patch them all as they did for U.S. servers after Hafnium). That doesn’t happen in the cloud because all servers run the latest software. In comparing the two environments, I see a cloud system operated by the development group and protected by thousands of security professionals where holes like basic authentication are steadily being closed off. Against that, I see on-premises environments that don’t have the same protection and expertise (exceptions do prove the rule, but there are few) where known holes exist. So I conclude that the cloud option is best.
Happy to debate the point at length if you want to attend the TEC Europe Tour event in Frankfurt on April 21. https://www.quest.com/event/tec-talk-five-things-microsoft-365-security-administrators-should-do-in-2023/
I don’t think on-premise will go 100% away because of the government. We have 3 of them and we’re just one org.
Some government institutions are definitely among the last hold-outs of on-premises Exchange Server. I just hope that they secure those servers!
With the recent breach on Rackspace Infrastructure, this is a final nail in the coffin for Exchange On Premise. An infrastructure as robust as Rackspace which offered Gold Standard Hosted Exchange for Small Businesses has seen a massive breach. And these small businesses have either moved to Microsoft 365 or chose to move to Google Workspace (which is way worse from enterprise scale perspective). This comes when Microsoft 365 Education was banned in Germany and school service providers were planning to provide hosted exchange on German Data Centers.
I don’t think Microsoft 365 Education has been banned in Germany. An opinion has been expressed by some state-level authorities and the German Federal Data Protection Authority (See https://www.computerweekly.com/news/252527842/Microsoft-365-banned-in-German-schools-over-privacy-concerns) that Microsoft 365 does not comply with some of the GDPR requirements. However, that’s not a ban, and there’s been considerable pushback against the opinion. And it’s always ‘on-premises’ and never on-premise’ (just to be picky).
Support must need to know atleast why the mailbox’s and it’s spam filters were acting strangely.., it’s a basic., they simply saying tell to your recipient to add our address into trusted sender., and They don’t even know about SCL which was mentioned in the documentation. we can able to say 1 or 2 clients/recipients to add my email address in your contact list., and it’s hard to say 1000+ Clients., and for that we need to call everyone to add my email address in your trusted address and it takes atleast 1week or maybe more then a week., and we don’t even know all of them will answer or not., I hope you’ll understand. And please, I request you to degrade/update the Outlook Mail Servers and its Exchange Internal Headers. I Hope you’ll soon solve these issues.
Thanking You
As no one here works for Microsoft, we can’t help. You need to work the issue with Microsoft support.
After the recent Exchange Update, Mails were going to Junk Folder, Even shortText Mails/Personal Mails., Or even if am gonna send mails to Clients/Recipients, They were receiving Emails to Junk Folder, within Organization Outlook/ No in Organization Outlook. Filters were working strangely.. And also with Recent Updates while using SMTP Client Submission LocalHost Submitting IP/ Public IP has been removed and showing only Microsoft IP’s. Even Customer Support (CS) don’t know issues properly. Exchange Server Year 2019 15.02.0529.005 far better then latest Exchange Server.
You’d be better off complaining to Microsoft Support about issues like this than expressing them in comments here. Formally documented issues are the only ones that Microsoft will action, and that means reporting the problem to support.
Hey! Nicely written , Intresting to read
Thanks & Regards
V.saibabu
Hi, Thank you for sharing this beautiful blog….
Hi Tony,
Thanks for the article. I personally believe that there is still a strong argument for on prem Exchange as a Hybrid server, for EXO management in synced AD scenarios (still the majority of enterprise) and simplified mail relay for multifunction devices (scan to email) and application SMTP.
Otherwise you are faced with creating scripts for every scenario such as hiding an object from the GAL, mail aliases, etc. And RBAC becomes that much more of a nightmare when delegating specific rights to service desks etc.
Fact is, to simplify BAU management, less experienced generalist in-house engineers need a GUI.
Just don’t publish endpoints externally, or use Azure AD App proxy for conditional access layers of protection, and a lot of risk is mitigated.
I suspect you might be an experienced Exchange admin, so you’re probably very comfortable with the maintenance operations needed to keep the server healthy. I wasn’t really addressing you in the article. I’m really after the folks who have on-premises servers and struggle to manage them. Maybe it’s too little time. Maybe too little knowledge. Whatever it is, they struggle. And we see this every time an attack emerges. Those servers should be decomissioned and their mailboxes moved to the cloud. Pronto.
The folks who know what they are doing can make their own minds up. But I think you’d find that the cloud is a pretty good place to be in most circumstances…
In a world of fake sysadmins, texts like these can pass as revelatory, but it’s just a long exchange online sales pitch.
Man, on prem exchange is fine. Don’t expose it to the Internet in any way (outer perimeter for email is on linux) and for owa you have a reverse proxy. That exchange is untouchable. And Exchnage is easy to maintain and patch. It’s a joke. And you are in full control. But IT WILL COST YOU.
There’s no sales pitch here. As a matter of record, I have documented the development of Exchange Server since 1995 over a series of books (like Microsoft Press Exchange 2010 Inside Out) and have strong and enduring relationships with many people who have worked on the product. It pains me to make such a recommendation, but it’s the right thing to do at this point. It is possible to run Exchange Server in a secure manner. Unfortunately, many instances exist when the server is badly maintained or not maintained at all. We see this evidence in the success of attackers in finding vulnerable servers to compromise. By all means stay on-premises, but do so in the full knowledge that you’ll need to work hard to maintain security against ongoing threat.
I think you both have valid points. Those that want/can invest in the dollars to maintain on-prem Exchange, for whatever reason, should be able to. Though Microsoft actually needs to reconnect with these organizations to understand the use case better, including dealing with low bandwidth deployments, like mentioned in the article.
Those companies, that don’t have the money, or want, to invest in the resources to keep on-prem Exchange updated and secure, should absolutely migrate to the cloud. I would lean toward Tony being right that the majority of on-prem Exchange installs should probably be migrated to Exchange on-line now.
Dude preaching to Tony Redmond about Exchange is like skater preaching to Tony Hawk about skateboarding.
One of the primary reasons we still have on-premises Exchange servers is for mass mailings, since Exchange Online won’t work for that. All of our mailboxes are in the cloud. What’s your recommendation for us to be able to send out mailings to our entire customer base if we remove onprem mail servers?
Mass mailings is an appliance-type activity rather than the norm of hosting user mailboxes. You could keep an Exchange Server for mass mailings. As long as it’s not exposed to the internet (no OWA, EAS, etc.), it should do just fine.
Use SendGrid or MailChimp for mass mailings