Block Old Exchange Servers From Sending Email to Exchange Online
Updated March 28
Today, Microsoft announced that their tolerance for obsolete and unpatched on-premises Exchange servers is reaching an end, at least in terms of communicating with Exchange Online. Instead of allowing the old servers to send email to Exchange Online, Microsoft says that they will start to protect their service by first throttling and then rejecting inbound messages from on-premises servers if the operators don’t keep their servers updated.
The experience of the past few years beginning with the Hafnium attack of March 2021 influenced Microsoft’s plan. Despite the widespread havoc wreaked by the Hafnium attackers, on-premises servers remain unpatched and exposed. The FBI might have closed the gaps for U.S.-based servers, but that still leaves servers in other countries for attackers to target.
Microsoft doesn’t know (or isn’t saying) exactly how many on-premises Exchange servers are in use. They know about the servers that connect to Microsoft as part of an Exchange hybrid organization and have some idea about other servers from analyzing inbound SMTP traffic. One thing is for sure: far too many of the on-premises servers run obsolete and vulnerable versions of Exchange Server. The set of vulnerable servers spans Exchange 2007, Exchange 2010, and Exchange 2013 (which becomes unsupported on April 11, 2023). Eventually, the set could include Exchange 2016 and Exchange 2019 servers that are not being updated.
TEC Talk: What to Do About Exchange On-Premises After Microsoft Starts to Block Messages
Hear what Tony Redmond has to say about what might happen if your org is using older on-premises Exchange servers.
Unpatched, Weak, and Vulnerable On-Premises Servers
Because Microsoft wrote the software running on these servers at a time when few could imagine the current threat landscape, these versions of Exchange server are achingly vulnerable . The problem is that a swelling bag of known attack techniques are available to anyone from script kiddies to nation state actors, all ready to deploy to inflict harm against old servers.
Even with the best will in the world, it’s not possible to upgrade these Exchange servers to cope with the known threats. Even companies with lots of experience running on-premises servers run into issues keeping systems updated. The three end-of-life versions of Exchange Server are out of support and don’t receive security or cumulative updates. As each month goes by, the software becomes more liable to become a target for potential compromise. This statement is not hyperbole. Its truth is obvious to anyone who’s followed the history of recent attacks against Exchange Server like the two day-zero vulnerabilities discovered in September 2022.
I have long said that the best course of action for most on-premises customers is to move to Exchange Online and Microsoft 365. Business and other requirements mandate that some need to remain on-premises. What’s happening now is that Microsoft is saying that if these customers want to use on-premises Exchange to communicate with Exchange Online, they must run a recent version of Exchange Server and commit to keeping the software current with security updates and cumulative updates. In this context, “current” means that servers run the latest available or the previous updates.
Message Restrictions Imposed by Transport
To make their position clear and attempt to convince those running old servers to upgrade to a supported release, Microsoft will introduce a change to the Exchange Online transport service to check the SMTP headers of inbound messages arriving in through inbound connectors configured with the ConnectorType set to be OnPremises. This limits the set of servers Microsoft will monitor and potentially block to those in hybrid organizations that handle email traffic to Exchange Online.
If a message comes from Exchange Server, the Exchange Online transport service checks the version number in the header to know what version the transmitting Exchange server runs. For example, Exchange Online might see this header in an inbound message:
Received: from <domain>.com (xx.xx.x.xx) by HOST.mail.protection.outlook.com (xx.xxx.xxx.xx) with Microsoft SMTP Server (TLS) id 14.3.496.0 via Frontend Transport; Mon, 27 Mar 2023 13:20:21 +0000
There’s nothing strange or secret here. SMTP servers send details about themselves in the initial handshake before the servers agree to communicate further. Version 14.3.496.0 means that the sending server runs Exchange 2010 SP3 with roll-up update 30. Microsoft released this version on February 11, 2020, so it’s over three years old. More importantly, Exchange 2010 is out of support and has received no updates since March 2, 2021. In passing, let me remind everyone that Exchange 2013 reaches the end-of-support milestone on April 11, 2023.
Equipped with knowledge about the sending server, Exchange Online can decide to accept or drop the message. Microsoft knows that it can’t declare outright war against old servers (it did, after all, sell the software running on those servers), so a more subtle and nuanced approach is necessary.
Report – Throttle – Block
Microsoft says that they will preview the report identifying affected servers in April. May 23 is when customers using obsolete Exchange 2007 servers will see the generally available version of the report. On June 26, ninety days after the announcement, Microsoft will begin to throttle the processing of messages sent from Exchange 2007 servers over an inbound connector type of OnPremises. Traffic that arrives in from obsolete servers via another route will be unaffected – for now.
Exchange Online implements the throttle by delaying acceptance of inbound messages for onward delivery to mailboxes or to the internet for external recipients. The throttling will be progressive, starting with five-minutes of throttling per hour and escalating gradually over 30 days to 20 minutes of throttling. If its owner does nothing to upgrade the server, Exchange Online will begin to block messages. Figure 1 shows the scheme of enforcement used by Exchange Online. As you can see, enforcement splits into three 30-day chunks for reporting, throttling, and blocking. On July 26, 2023, Exchange Online will block inbound traffic from obsolete Exchange 2007 servers.
When Exchange Online throttles inbound messages, it issues a temporary SMTP error to cause the messages to queue on the source servers. It’s as if Exchange Online was temporarily unavailable. When the throttle condition ceases, Exchange Online accepts inbound messages again and the queue clears. At the early stages, where throttling lasts for five or ten minutes, users might not be aware of delays. Server administrators might notice queues growing but attribute this to network glitches. However, as throttling increases to twenty or thirty minutes per hour, the pain will increase to noticeable levels.
What’s for sure is that people will realize that their messages are not flowing as smoothly as before and that administrators will notice message queues accumulating. Well, they should notice that queues grow, but given that some of these servers are sadly neglected, that might not be true.
The Throttling-Blocking Cycle
Before throttling commences, Microsoft will post notifications in the Microsoft 365 message center to make administrators of hybrid organizations aware that their organization contains obsolete servers. Microsoft is still working out final details of the notification cycle, but it should go something like this:
- Now: Microsoft is analyzing message traffic to detect outdated servers.
- Once Microsoft detects that a server is in scope, they add the server to a new mail flow report that Microsoft is releasing to the Exchange admin center in Exchange Online. The new report provides details of throttling and blocking, along with information about what happens next if the server operators take no action.
- Steady-State: Microsoft decides to block a set of outdated servers. The first set is traffic from Exchange 2007 servers that comes into Exchange Online using an inbound OnPremises type of connector (in other words, hybrid servers connected to the internet). Microsoft posts a message center notification to these tenants where these servers exist to say that Microsoft will put throttling into effect in 60 days.
- Throttling starts and goes for 30 days, getting progressively longer every 10 days.
- After 30 days of throttling, blocking begins and increases over the next 10 days, while throttling decreases commensurately, until Exchange Online completely blocks traffic from the server.
The details of how and when tenants receive notifications might change. What won’t change is the need for administrators to act. Of course, the problem again is the lack of management in some tenants. People need to read message center notifications and then react by looking for outdated servers, (the Health Checker script is helpful here) understanding the role the servers play in the organization, and arranging for an upgrade (which might involve some hardware replacement). Not everyone proactively responds to message center notifications.
After administrators have a chance to absorb the hint implicit in throttling, Microsoft will increase the pressure to upgrade servers by starting to reject some messages. Exchange Online will block inbound messages permanently and senders will receive an NDR with a 5.7.230 code to say that their email isn’t getting through. Hopefully, the senders will ask their server administrators what’s going on and ask them to fix the problem. At that point, the proverbial penny might drop for the on-premises server administrators as they see a bunch of failure errors in their message tracking logs.
Upgrade or Stop Sending Email
If the server operators ignore the hint, Microsoft will gradually increase the pressure by throttling and then blocking messages until Exchange Online reaches the point of refusing to accept any messages from the old server. The operators now have a simple choice: either upgrade their servers to a supported version or lose the capability of sending email to Exchange Online. Given the volume of email handled by Exchange Online, choosing not to upgrade could be described as incurring a considerable handicap (or even shooting yourself in the foot).
Microsoft plans to start with the relatively small set of Exchange 2007 servers that send email to Exchange Online, but the pressure will soon come on those running Exchange 2010 and Exchange 2013 servers. Expect the same rinse and repeat formula of throttling and rejection to occur to build the pressure to upgrade to a supported version, including when Exchange 2016 and Exchange 2019 servers run unsupported software. Microsoft’s move against old servers running unsupported software is a hint to apply security and cumulative updates to Exchange 2016 and Exchange 2019 servers.
Let’s be clear about Microsoft’s intention: they want to force/cajole/persuade people who run on-premises Exchange servers to use maintainable software. Those who decide to follow Microsoft’s guidance will continue to enjoy full interoperability with Exchange Online. Those who don’t will have their communications blocked. It’s as simple as that.
It’s obvious that Microsoft could take the same route to deal with other vulnerable servers if they identify that attackers use a certain type of server to inject malware into Exchange Online. For instance, if someone’s running an old version of a Postfix server on Linux and attackers exploit a flaw to compromise Exchange Online mailboxes, it’s easy to imagine that Microsoft might step in and block email from the problematic servers to protect the integrity of Exchange Online.
Microsoft Platform Migration Planning and Consolidation
Minimize the risk, time, cost, and complexity associated with your next Exchange Online Domain Transfer
We live in an imperfect world and Microsoft must accept that some customers who operate a hybrid Exchange organization include old Exchange servers in their inventory. The old servers probably don’t host user mailboxes and instead take care of activities like sending commercial emails (marketing communications). Applying throttling or a block to the email sent by these servers could disrupt legitimate customer operations.
Microsoft still wants to convince customers to upgrade the servers. To allow time for this to happen, customers can request a 90-day “enforcement pause” per calendar year. The idea here is that many companies have periods during a year when they do not perform upgrades or that some customers simply need some extra time before they can attend to the outdated servers. In these situations, customers can ask Microsoft to pause throttling/blocking for a single block of 90 days or divided into multiple blocks over the year amounting to 90 days. The mail flow report in the EAC will include an option to request a pause.
After the pause period lapses, the blocks will pick up again at the same point (for instance, if a 15-minute throttling delay applied when the pause started, that’s what it will be when the pause finishes) if the outdated servers remain operational.
The Right Thing to Do
I’ve no doubt that some will consider Microsoft’s actions to be unwarranted and high-handed. It’s a valid perspective, but against that, Microsoft has the right to protect a service used by hundreds of millions of people. Whatever you think, if your organization uses Exchange Online and has some obsolete on-premises Exchange servers, you don’t get to vote. The direction is clear: update those servers or face loss of communication with Exchange Online. And while you’re at it, remember to keep your Outlook clients updated too.