Storm-0558: Understanding How Microsoft Failed to Protect Itself

Penetration of Corporate User Account Led to Data Loss and Exploitation

You’re undoubtedly familiar with the so-called Storm-0558 attacks from July 2023. If not a quick recap: these attacks (widely attributed as the work of the Chinese government) compromised a number of high-value Exchange Online mailboxes, including the US Secretary of Commerce and the US Ambassador to China. Given the sensitivity of the mailboxes, it’s likely that they were hosted in GCC-High or DOD tenants.

The episode was a huge embarrassment for Microsoft and has led to their decision to move some Purview Audit Premium features back to the standard tier, as well as to an investigation by the US Department of Homeland Security’s Cyber Safety Review Board.

In Microsoft’s initial disclosure of the breach, they made some fairly vague statements about how the threat actor had been able to breach the service by obtaining a key, and on September 6, 2023, they released a more detailed post-mortem on how the attacker got the key. It’s sobering, not to mention alarming, and you should understand the nature of the failure(s) to better understand the inherent risk in using a cloud service like Microsoft 365.

Microsoft Goes Zero for Five

Aviation accident investigators talk about “Swiss cheese” as a metaphor for understanding how accidents happen—you only get a bad result if enough holes in the cheese slices line up. Microsoft’s postmortem contains the phrase “(this issue has been corrected)” a total of five times. That’s a pretty good way to frame the discussion: There were five separate holes in the cheese that the attackers chained together to mount the attack.

Let’s take them in sequence. Note that Microsoft said they’ve fixed all these issues, but they didn’t provide any real details on the nature or scope of the fixes.

1. In April 2021, a system used to generate signing keys for the consumer Microsoft services crashed and generated a Windows crash dump. These dumps should never include cryptographic keys, but in this case, an unspecified bug (“race condition”) allowed the key material to be copied from RAM to the crash dump file.
2. The system that’s supposed to scan crash dumps for sensitive data didn’t detect the presence of the key data in the dump file, again because of an unspecified bug.
3. As part of normal debugging, the crash dump was moved to Microsoft’s corporate network that’s connected to the Internet. The corporate network is where Microsoft personnel, including engineers, work. The credential-scanning tools on the debugging network didn’t detect the key data there either. Microsoft didn’t say that this was because of a bug, probably because their credential scanner is looking for actual credentials and not just key material.
4. Microsoft updated a set of internal libraries used for validating cryptographic signatures. These libraries were meant to be shared between their consumer and enterprise services, but because they did a bad job of the updates, the libraries didn’t properly enforce the intended separation of scope between the consumer and enterprise services. I’ll score this as another bug.
5. Because Microsoft did a bad job of documenting the changes (the ones they did a bad job of implementing), “developers in the mail system” relied on the (non-working) libraries to catch scope violations of the type the attackers used.

It’s not surprising or uncommon that there are bugs in the software we use, and bugs apparently account for four of the five failures above. The fifth is the developers’ assumption that a library did what its documentation actually said… well, every developer has been in that situation before. However, to an experienced security practitioner, there are a few things about the above list that immediately scream for attention.

Unanswered Questions in Microsoft’s Response

First, how did the attacker get the crash dump in the first place? Buried in Microsoft’s description is this line (emphasis added): “the Storm-0558 actor was able to successfully compromise a Microsoft engineer’s corporate account.” Re-read that. The attacker compromised a Microsoft employee’s account—presumably one that was protected with MFA and conditional access policies—and that compromise gave them access to the crash dump. The dump was generated in April 2021.  Microsoft’s recap says that attacks started on May 15, 2023, but they also cite attack activity patterns starting in April 2023 so it’s hard to tell when the attacks truly began.

That’s a two-year period in which Microsoft’s vaunted internal security controls apparently failed to notice the Chinese government rooting around in their network.

Second, how did the attackers know to find the crash dump? Microsoft said that they didn’t have enough log data to definitely prove that the attacker stole and exfiltrated the crash dump, because they don’t retain logs for long enough. If they have any evidence indicating what else the attacker did, they aren’t saying so, but I am betting that because of the time span of the attack, they do not have that evidence.

Third, what else did the threat actor have access to during that time period? One of the reasons that advanced persistent threats (APTs) are so hard to deal with is that they establish persistence—and the longer they’re in a network, the more difficult it is to ensure that you’ve removed all of their points of presence and persistence.

Practical Actions to Stay Safe

The fatalists among us might be tempted to shrug and say “Well, if Microsoft gets hacked, what chance do I have?” That attitude ignores the fact that you’re probably not going to be targeted by the Chinese government and that there are some practical steps you can take to reduce your risk.

First, are you appropriately applying MFA and conditional access policies? Are the correct policies being applied, consistently, to the correct set of users? Can you demonstrate that?

Second, how’s your logging? Some of the organizations attacked by Storm-0558 didn’t notice because they didn’t have the MailItemsAccessed event enabled because that costs money. Mailbox auditing is available for all Exchange Online customers, but ingestion of mailbox audit data into the unified audit log is only for Office 365 E3 and above. The MailItemsAccessed event used to be available for Office 365 E5 and above, but now (deployment to tenants starts in September 2023) it’s available for Office 365 E3.

If you haven’t enabled MailItemsAccessed for logging, or aren’t sure if you do, you should enable it ASAP. That event is the key indicator that an application, authorized or not, has accessed items in an Exchange mailbox. The original attacks were detected when the target organization (a Federal Executive Civilian Branch agency) noticed an unusual application ID had generated MailItemsAccessed events in their audit logs. If you have other sensitive data items, users, or network resources, you should ensure that you’re logging the right things for them, but also that you have the appropriate detection and review mechanisms in place to catch exceptions.

Analysis of audit data can be done through systems like Splunk or Sentinel. Microsoft recently changed the way that the Search-UnifiedAuditLog cmdlet works, so make sure that the data flowing from the unified audit log into your data repository is complete and accurate. You could also use the technique explained in this article to monitor events added to the unified audit log for inconsistencies. Whatever tools you use to capture and analyze audit data, make sure that events are reviewed.

Third, how easy would it be for an attacker to move laterally into (or out of) any protected networks you maintain? The idea of having an isolated production network is well accepted in the industry, but as soon as you start allowing users or data to move from that isolated network to other less-isolated networks, you’ve created a potential vulnerability of which you should be both mindful and cautious. You should verify that you have appropriate security controls, adequately enforced, along with logging and alerting to monitor movement between these networks.

More Attacks Coming

We haven’t seen the last of nation-state-sponsored attacks, nor have we seen the last of the impact of this particular attack. While we wait to see what embarrassing revelations come next, spend some time investigating the three items I mentioned above to help protect yourself in the future.