What We Can Learn from the Rackspace Breach

In early December 2022, Rackspace posted this notice to their customers:

On Friday, Dec 2, 2022, we became aware of an issue impacting our Hosted Exchange environment. We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact. After further analysis, we have determined that this is a security incident. 

This notice might have surprised you for several reasons. One is that you might not have known that any companies still offer hosted Exchange services. Another is that you might not know Rackspace, which has grown over two decades to a US$3 billion/year publicly-traded hosting company. A third is to discover that attackers can compromise such a large enterprise. Yet, surprising though it is, all these things are true, and now we’re dealing with the aftermath.

What Happened?

It can be difficult to unravel security incidents like this, but, to their credit, Rackspace did pretty much everything right: they went public with the incident, hired a very well-known security firm (CrowdStrike) to help them clean up, and then published a postmortem discussing what happened. Here’s what we know based on that postmortem and on comments from CrowdStrike.

CrowdStrike’s analysis shows that these attackers (part of a ransomware group known as Play) used a previously-unknown zero-day exploit to penetrate the Rackspace-hosted Exchange environment. This exploit is related to the ProxyNotShell exploit, which Microsoft first acknowledged in September 2022. As an initial mitigation, Microsoft recommended adding a rewrite rule for IIS to drop connection requests attempting to use the exploit. In the November 8, 2022, Exchange security update (KB5019758), Microsoft patched the underlying vulnerabilities. Note my use of the plural! There were two problems addressed in that patch: CVE-2022-41040 is the ability to force a server-side request against the Autodiscover endpoint, and CVE-2022-41082 is a remote PowerShell vulnerability. ProxyNotShell attacks required the attacker to succeed in exploiting both vulnerabilities. Adding the rewrite rule block the 41040 exploit and stopped an attack. Or so it seemed.

When they released the patch, Microsoft updated their guidance to say the following:

We recommend that customers protect their organizations by applying the updates immediately to affected systems. The options described in the Mitigations section are no longer recommended. For more information, review the Exchange Team blog.

However, in a development that will probably not surprise very many of you given the low uptake of previous patches… Rackspace didn’t install the patch. The CrowdStrike analysis appears to indicate that Rackspace didn’t patch because Microsoft didn’t explicitly say that CVE-2022-41082 was remotely exploitable. That may have led them to believe that the mitigation rule, originally recommended in September, was good enough to obviate the need to apply the November patch. This turned out to be a really bad decision because attackers could also exploit the remote PowerShell issue (CVE-2022-41082) by attacking OWA with a forged server-side request. That’s what the Play attackers did.

Because Rackspace didn’t patch, they got torched. File that fact away for later.

The Impact on Customers

The incident turned into a big mess for all concerned. Rackspace’s stock price took a major beating, the formerly supportive local news media turned on them, and they were pilloried by customers. The impact for most customers, according to Rackspace’s postmortem,  was that they had zero access to their email from about December 2 until sometime in late December, depending on when each customer’s data was restored. Along the way, Rackspace helped some of its customers to move their mailboxes to Microsoft 365—but the results overall are pretty grim. “More than half of impacted customers have some or all of their data available,” said Rackspace, but that data is limited to email that arrived before December 2.

Alongside the outage, a total of 27 customers (out of roughly 30,000 hosted Exchange customers) had PST files containing mail data stolen by the attackers. The reason why the attackers went after PST files is unclear, but it’s likely that it’s because PST files are portable and easy to access with any Outlook client. The infamous Sony Pictures email hack (2014) also resulted in attackers copying a large number of PST files containing confidential information that later found its way into public view. PSTs are insecure. Plain and simple.

Rackspace hasn’t said publicly which specific customers were affected, whether they fall into any specific categories or verticals, or anything at all about that aspect of the attack. This is probably smart, considering the number of lawsuits currently being aimed their way.

Moving forward

The longer-term impact of this incident is interesting. First, Rackspace spent on the order of about $30 million to clean up. That’s real money, even for a $3 billion company. Second, they are exiting the hosted Exchange business. That leaves Intermedia and a few smaller companies as the last remaining hosted Exchange providers in the US, with a handful of midsize hosters remaining in the rest of the world. It’s hard to see how this fiasco is good news for them, though, since the whole mess underlines Microsoft’s point that their investments in Exchange Online can’t be matched by organizations hosting their own environments.

Third, Rackspace is offering to pay Microsoft to migrate customers to Microsoft 365, a not-insignificant cost to them. That may not be enough to convince customers to stay with them, but it means that line of business and its associated revenue is gone.  They may recoup some revenue from the sale of Microsoft 365 licenses, but once customers are in Microsoft 365, it’s trivial for them to change to another managed service provider, and I expect that many of them will.

Lessons learned?

Rackspace obviously made the strategic decision to avoid ever repeating this problem by getting rid of hosted Exchange. What can the rest of us learn? Well, we can start with a lesson that far too many people seem to have still not internalized: patch your Exchange servers. Maybe shouting it a few more times will result in some incremental security improvements. The bigger lesson is probably also one of strategy: if you have resisted moving to Exchange Online for business or operational reasons, you should be prepared with a strategy to ensure that your users are protected against ransomware and other threats, and that you have a viable way to recover data if that protective strategy fails. The cost of doing so is high, but the cost of not doing it is worse—and the comparison of those costs may well lead you to reconsider whether the cloud is the right place for your email after all.