Block Old Exchange Servers From Sending Email to Exchange Online
Updated March 28
Today, Microsoft announced that their tolerance for obsolete and unpatched on-premises Exchange servers is reaching an end, at least in terms of communicating with Exchange Online. Instead of allowing the old servers to send email to Exchange Online, Microsoft says that they will start to protect their service by first throttling and then rejecting inbound messages from on-premises servers if operators don’t keep their servers updated.
The experience of the past few years beginning with the Hafnium attack of March 2021 influenced Microsoft’s plan. Despite the widespread havoc wreaked by the Hafnium attackers, on-premises servers remain unpatched and exposed. The FBI might have closed the gaps for U.S.-based servers, but that still leaves servers in other countries for attackers to target.
Microsoft doesn’t know (or isn’t saying) exactly how many on-premises Exchange servers are in use. They know about the servers that connect to Microsoft as part of an Exchange hybrid organization and have some idea about other servers from analyzing inbound SMTP traffic. One thing is for sure: far too many of the on-premises servers run obsolete and vulnerable versions of Microsoft Exchange Server. The set of vulnerable servers spans Exchange 2007, Exchange 2010, and Exchange 2013 (which becomes unsupported on April 11, 2023). Eventually, the set could include Exchange 2016 and Exchange 2019 servers that are not being updated.
TEC Talk: What to Do About Exchange On-Premises After Microsoft Starts to Block Messages
Hear what Tony Redmond has to say about what might happen if your org is using older on-premises Exchange servers.
Unpatched, Weak, and Vulnerable On-Premises Servers
Because Microsoft wrote the software running on these servers at a time when few could imagine the current threat landscape, these versions of Microsoft Exchange server are achingly vulnerable . The problem is that a swelling bag of known attack techniques are available to anyone from script kiddies to nation state actors, all ready to deploy to inflict harm against old servers.
Even with the best will in the world, it’s not possible to upgrade these Exchange servers to cope with the known threats. Even companies with lots of experience running on-premises servers run into issues keeping systems updated. The three end-of-life versions of Microsoft Exchange Server are out of support and don’t receive security or cumulative updates. As each month goes by, the software becomes more liable to become a target for potential compromise. This statement is not hyperbole. Its truth is obvious to anyone who’s followed the history of recent attacks against Exchange Server like the two day-zero vulnerabilities discovered in September 2022.
I have long said that the best course of action for most on-premises customers is to move to Exchange Online and Microsoft 365. Business and other requirements mandate that some need to remain on-premises. What’s happening now is that Microsoft is saying that if these customers want to use on-premises Exchange to communicate with Exchange Online, they must run a recent version of Exchange Server and commit to keeping the software current with security updates and cumulative updates. In this context, “current” means that servers run the latest available or the previous updates.
Message Restrictions Imposed by Transport
To make their position clear and attempt to convince those running old servers to upgrade to a supported release, Microsoft will introduce a change to the Exchange Online transport service to check the SMTP headers of inbound messages arriving in through inbound connectors configured with the ConnectorType set to be OnPremises. This limits the set of servers Microsoft will monitor and potentially block to those in hybrid organizations that handle email traffic to Exchange Online.
If a message comes from Exchange Server, the Exchange Online transport service checks the version number in the header to know what version the transmitting Exchange server runs. For example, Exchange Online might see this header in an inbound message:
Received: from <domain>.com (xx.xx.x.xx) by HOST.mail.protection.outlook.com (xx.xxx.xxx.xx) with Microsoft SMTP Server (TLS) id 14.3.496.0 via Frontend Transport; Mon, 27 Mar 2023 13:20:21 +0000
There’s nothing strange or secret here. SMTP servers send details about themselves in the initial handshake before the servers agree to communicate further. Version 14.3.496.0 means that the sending server runs Exchange 2010 SP3 with roll-up update 30. Microsoft released this version on February 11, 2020, so it’s over three years old. More importantly, Exchange 2010 is out of support and has received no updates since March 2, 2021. In passing, let me remind everyone that Exchange 2013 reaches the end-of-support milestone on April 11, 2023.
Equipped with knowledge about the sending server, Exchange Online can decide to accept or drop the message. Microsoft knows that it can’t declare outright war against old servers (it did, after all, sell the software running on those servers), so a more subtle and nuanced approach is necessary.
Report – Throttle – Block
Microsoft says that they will preview the report identifying affected servers in April. May 23 is when customers using obsolete Exchange 2007 servers will see the generally available version of the report. On June 26, ninety days after the announcement, Microsoft will begin to throttle the processing of messages sent from Exchange 2007 servers over an inbound connector type of OnPremises. Traffic that arrives in from obsolete servers via another route will be unaffected – for now.
Exchange Online implements the throttle by delaying acceptance of inbound messages for onward delivery to mailboxes or to the internet for external recipients. The throttling will be progressive, starting with five-minutes of throttling per hour and escalating gradually over 30 days to 20 minutes of throttling. If its owner does nothing to upgrade the server, Exchange Online will begin to block messages. Figure 1 shows the scheme of enforcement used by Exchange Online. As you can see, enforcement splits into three 30-day chunks for reporting, throttling, and blocking. On July 26, 2023, Exchange Online will block inbound traffic from obsolete Exchange 2007 servers.
When Exchange Online throttles inbound messages, it issues a temporary SMTP error to cause the messages to queue on the source servers. It’s as if Exchange Online was temporarily unavailable. When the throttle condition ceases, Exchange Online accepts inbound messages again and the queue clears. At the early stages, where throttling lasts for five or ten minutes, users might not be aware of delays. Server administrators might notice queues growing but attribute this to network glitches. However, as throttling increases to twenty or thirty minutes per hour, the pain will increase to noticeable levels.
What’s for sure is that people will realize that their messages are not flowing as smoothly as before and that administrators will notice message queues accumulating. Well, they should notice that queues grow, but given that some of these servers are sadly neglected, that might not be true.
The Throttling-Blocking Cycle
Before throttling commences, Microsoft will post notifications in the Microsoft 365 message center to make administrators of hybrid organizations aware that their organization contains obsolete servers. Microsoft is still working out final details of the notification cycle, but it should go something like this:
- Now: Microsoft is analyzing message traffic to detect outdated servers.
- Once Microsoft detects that a server is in scope, they add the server to a new mail flow report that Microsoft is releasing to the Exchange admin center in Exchange Online. The new report provides details of throttling and blocking, along with information about what happens next if the server operators take no action.
- Steady-State: Microsoft decides to block a set of outdated servers. The first set is traffic from Exchange 2007 servers that comes into Exchange Online using an inbound OnPremises type of connector (in other words, hybrid servers connected to the internet). Microsoft posts a message center notification to these tenants where these servers exist to say that Microsoft will put throttling into effect in 60 days.
- Throttling starts and goes for 30 days, getting progressively longer every 10 days.
- After 30 days of throttling, blocking begins and increases over the next 10 days, while throttling decreases commensurately, until Exchange Online completely blocks traffic from the server.
The details of how and when tenants receive notifications might change. What won’t change is the need for administrators to act. Of course, the problem again is the lack of management in some tenants. People need to read message center notifications and then react by looking for outdated servers, (the Health Checker script is helpful here) understanding the role the servers play in the organization, and arranging for an upgrade (which might involve some hardware replacement). Not everyone proactively responds to message center notifications.
After administrators have a chance to absorb the hint implicit in throttling, Microsoft will increase the pressure to upgrade servers by starting to reject some messages. Exchange Online will block inbound messages permanently and senders will receive an NDR with a 5.7.230 code to say that their email isn’t getting through. Hopefully, the senders will ask their server administrators what’s going on and ask them to fix the problem. At that point, the proverbial penny might drop for the on-premises server administrators as they see a bunch of failure errors in their message tracking logs.
Upgrade or Stop Sending Email
If the server operators ignore the hint, Microsoft will gradually increase the pressure by throttling and then blocking messages until Exchange Online reaches the point of refusing to accept any messages from the old server. The operators now have a simple choice: either upgrade their servers to a supported version or lose the capability of sending email to Exchange Online. Given the volume of email handled by Exchange Online, choosing not to upgrade could be described as incurring a considerable handicap (or even shooting yourself in the foot).
Microsoft plans to start with the relatively small set of Exchange 2007 servers that send email to Exchange Online, but the pressure will soon come on those running Exchange 2010 and Exchange 2013 servers. Expect the same rinse and repeat formula of throttling and rejection to occur to build the pressure to upgrade to a supported version, including when Exchange 2016 and Exchange 2019 servers run unsupported software. Microsoft’s move against old servers running unsupported software is a hint to apply security and cumulative updates to Exchange 2016 and Exchange 2019 servers.
Let’s be clear about Microsoft’s intention: they want to force/cajole/persuade people who run on-premises Exchange servers to use maintainable software. Those who decide to follow Microsoft’s guidance will continue to enjoy full interoperability with Exchange Online. Those who don’t will have their communications blocked. It’s as simple as that.
It’s obvious that Microsoft could take the same route to deal with other vulnerable servers if they identify that attackers use a certain type of server to inject malware into Exchange Online. For instance, if someone’s running an old version of a Postfix server on Linux and attackers exploit a flaw to compromise Exchange Online mailboxes, it’s easy to imagine that Microsoft might step in and block email from the problematic servers to protect the integrity of Exchange Online.
Microsoft Platform Migration Planning and Consolidation
Minimize the risk, time, cost, and complexity associated with your next Exchange Online Domain Transfer
The Exception
We live in an imperfect world and Microsoft must accept that some customers who operate a hybrid Exchange organization include old Exchange servers in their inventory. The old servers probably don’t host user mailboxes and instead take care of activities like sending commercial emails (marketing communications). Applying throttling or a block to the email sent by these servers could disrupt legitimate customer operations.
Microsoft still wants to convince customers to upgrade the servers. To allow time for this to happen, customers can request a 90-day “enforcement pause” per calendar year. The idea here is that many companies have periods during a year when they do not perform upgrades or that some customers simply need some extra time before they can attend to the outdated servers. In these situations, customers can ask Microsoft to pause throttling/blocking for a single block of 90 days or divided into multiple blocks over the year amounting to 90 days. The mail flow report in the EAC will include an option to request a pause.
After the pause period lapses, the blocks will pick up again at the same point (for instance, if a 15-minute throttling delay applied when the pause started, that’s what it will be when the pause finishes) if the outdated servers remain operational.
The Right Thing to Do
I’ve no doubt that some will consider Microsoft’s actions to be unwarranted and high-handed. It’s a valid perspective, but against that, Microsoft has the right to protect a service used by hundreds of millions of people. Whatever you think, if your organization uses Exchange Online and has some obsolete on-premises Exchange servers, you don’t get to vote. The direction is clear: update those servers or face loss of communication with Exchange Online. And while you’re at it, remember to keep your Outlook clients updated too.
Would be a interesting feature. With two very big caveats. 1) they will NOT notify admins if you run into this throtteling/blocking. And 2) there is a 6 month old bug with the detection of the on-prem version and they still block e-mails even when you’re up-to-date with a wrongly detected version.
Just google “Remove message header in Exchange Server” and EXO won’t know the version.
And what do you think will happen then? Removing X-headers https://practical365.com/remove-internal-exchange-server-names-ip-addresses-message-headers/ is not a way to prevent Exchange Online recognizing the server version at the far side of a connector.
Hi Tony,
We migrated from E2013 hybrid to E2019 hybrid on April 6 2024. The E2013 server has been shut down since that date but not yet technically uninstalled from the on-premise environment. The M365 outdated Exchange server report still only lists the E2013 server. I’ve had a (useless) ticket open with M365 support since April 6. Our CIO is nervous to try a knee-jerk uninstall of E2013 until support says that’s the fix but I’d like to amputate that old server ASAP.
I don’t understand how the message headers could reference an offline E2013 server. Do you think if we uninstall E2013 from the environment that’ll cure our need to continue pausing the throttling?
If the Exchange 2013 server has been shut down since April 6, I think you’ve had enough time to discover if any problems are lurking in the environment and should be safe to remove it. Did you run the get-onpremserverreportinfo cmdlet to update the information that the Exchange server report depends on?
MS requested I run the get-onpremserverreportinfo cmdlet to show the output (it only showed the E2013 server), but they didn’t give me any directions on how to ” update the information” their report depends upon.
My impression is that the cmdlet updates the information shown in the report. This might take a little time as it’s likely that the data is cached.
I ran it for MS on Apr 18, and again today (Apr 22) with no obvious change. I found a redditor had the same issue a few months ago but a solution was not posted. I guess I’ll have to wait for MS or uninstall E2013 (whichever comes first), I appreciate the quick feedback.
I found this comment from MS here, see comment dated Feb 1 2024 10:53PM from Kevin.
https://techcommunity.microsoft.com/t5/exchange-team-blog/how-to-pause-throttling-and-blocking-of-out-of-date-on-premises/ba-p/4007169#comment-on-this
Kevin’s a senior guy in the transport section of the Exchange development team. I’d trust what he says.
Is there any kind of online test that you can send and email to or copy your headers to that would determine if the block would affect our enterprise? We are on O365 but still have hybrid turned on simply as a way to capture and redirect bounced emails to a catch-all account. We also send thru a couple of smart devices, Barracuda for spam and Code-Two for signatures. It would be good to make sure that we know we’re safe from this, as we just came up from Ex2k10sp3, but it has been decommissioned.
Why not simply run the Get-ExchangeServer cmdlet to check the servers in your on-premises environment and the software versions they run? You know that Microsoft will block old servers that host the connectors to send email to Exchange Online, so simply make sure that these servers run an up-to-date and supported version and you’re fine.
Tony,
Just so I get this right, Microsoft will be inspecting the entire SMTP header and all “Received: from xxxxxxx” statements for the id: field for anything less the the current supported range of servers, and not just the connecting SMTP server?
Microsoft will inspect the x-headers for inbound email sent across an inbound connector configured as an on-premises connectors to check the version number of the sending sender. That’s it. It doesn’t matter if the originating server is version 2010, 2013, or 2016. The problem arises and blocking will ensue if the server that handles the message traffic to Exchange Online is 2007.
Coming at a time where we are looking to move back on-Orem. Microsoft is a joke, every patch they release wreaks havoc on our company, Teams is a pig, Windows just keeps getting worse and worse for the enterprise, and their support gets worse and worse each year.
No one forces you to use Teams, just like no one forces you to use Microsoft 365. It’s up to organizations to decide what platform and software best serves their purpose. Good luck with your move back on-premises. I’m sticking with the cloud.
Does inbound onpremise connectors mean that this change is aimed at exchange online customers with hybrid environments and not random incoming SMTP traffic? Seems like a slightly overkill move to start generally blocking legitimate and otherwise clean traffic to your own customers because a third party senders email has been routed through a software you don’t like. Bit of a dangerous precedent one might think.
That’s Microsoft’s opening gambit. The first target is obsolete Exchange 2007 servers that handle email traffic on behalf of hybrid organizations with the aim of blocking malicious traffic that might penetrate the on-premises organization before being relayed to Exchange Online. The scope of the exercise will steadily increase to include Exchange 2010 and 2013 servers, and then Exchange 2016 and 2019 servers. This might seem like overkill, but if it stops Exchange Online becoming contaminated because someone runs an on-premises server with a known vulnerability, it’s worthwhile.
So only hybrid customers for now? Not being an exchange online customer I don’t have much recourse except for spoofing mail headers, and I don’t think we want thst to become best practise.
It is objectively and irrefutably not “worthwhile” blocking incoming third party SMTP traffic based solely on the software versions of MTAs in the chain. That is just lazy spam filtering that is going to cause problems for everyone. Today it’s Exchange onprem, tomorrow it’s your competitors. For the sake of the children, obviously.
Only hybrid customers for now.
SMTP is not really involved here. It happens to be the communications protocol and the initial test used by Exchange Online is based on headers inserted by Exchange on-premises. However, the transport service is capable of disassembling messages from A to Z, so you can imagine that some more sophisticated analysis will be used when traffic from non-hybrid environments are assessed.
My experience with MS products tells me that they will eventually apply the same rules to all incoming mail. After all its the same situation you are running an out of date and unsupported/insecure mail system.
To make sense of the next 3 Replies, please go to the bottom one and work your way up …..
But the problem is that customers who have purchased from the Business family of Microsoft 365 and are running an on prem Exchange 2013 server for management, are simply abandoned, i.e. to continue to run in a Hybrid environment, they must run a supported version of Exchange which means they will need to purchase a brand new Exchange Server license purely for management.
So my first question is this – Is my understanding correct? If my understanding is not correct, what is the current state of the Hybrid Key entitlement?
On the other hand, if my understanding is correct, I would highly encourage Microsoft to extend the Hybrid Key entitlement to all Exchange Online customers so as to provide a uniform roadmap that all customers can leverage to better secure their environments and meet the requirements that Microsoft is imposing, justified or not.
Please note that starting with April 2022, HCW does provide a ‘hybrid key’ for Exchange Server 2019 also. Please see the following post for more information: https://techcommunity.microsoft.com/t5/exchange-team-blog/released-2022-h1-cumulative-updates-for-exchange-server/ba-p/3285026
Additionally, Microsoft has an entitlement for organizations that purchase Office 365 / Microsoft 365 from their Enterprise family of Office 365 / Microsoft 365 product line that will license an on premise Exchange Server that does not host any mailboxes (i.e. an Exchange Server used purely for management) via a Hybrid Key –
https://www.microsoft.com/en-us/microsoft-365/exchange/microsoft-exchange-licensing-faq-email-for-business
Based on very significant feedback from the customer base, Microsoft extended the Hybrid Key entitlement to Exchange 2019 (originally it was limited to only Exchange 2016 and I think Tony Redmond was a part of a pretty large and nasty forum “riot” on this topic)
https://learn.microsoft.com/en-us/answers/questions/938611/exchange-hybrid-2019-license-requirements
I’m not sure I was ever a party to a “large and nasty forum riot”… but however…
I think that there are additional concerns that need to be addressed. The most pressing of which is that it is Microsoft’s recommendation that organizations who have a Hybrid Identity (Windows Server AD connected to Azure AD) and use Exchange Online must continue to run an Exchange on prem server to manage Exchange attributes –
https://learn.microsoft.com/en-us/exchange/decommission-on-premises-exchange
As far as I know this recommendation is still in place. Is Microsoft working to eliminate this requirement?
No longer a requirement. See https://learn.microsoft.com/en-us/exchange/manage-hybrid-exchange-recipients-with-management-tools
Microsoft created the software that allowed the issue, then created a policy of forced upgrade with associated costs, then stops supporting their product, then complains that the issue is there.. stay toxic Microsoft.
How about a good Linux based email messaging platform. When will Ms block them?
How about creating a product that is continually developed without continually charging.. opps.. that would be bad business!
Hold on. We’re talking about Exchange 2007 servers right now. Microsoft designed the architecture for these servers in the 2003-2006 period. At that time, no one knew what the threat horizon would be like in 2023. The fact is that these are very old servers that are no longer being maintained and patched. Four separate waves of major releases (2010, 2013, 2016, and 2019) have happened since Exchange 2007 was king of the walk. Do you expect that Microsoft would maintain software that was so old when the work needed to bulletproof Exchange 2007 essentially means rewriting large parts of the code from scratch? Isn’t it simpler to install Exchange 2016 or 2019?
I’m unaware of any Linux-based email server from the 2006 era that runs in production today without substantial levels of upgrade and update that is invulnerable to current threats. Maybe you know of such a beast?
Look, the fact is that people who run these old servers are vulnerable. They need to do something to fix that problem. The FBI had to reach out and fix some U.S. based servers after Hafnium situation, but the FBI isn’t going to do it for every old server especially those outside the U.S. If you run old software, it’s your responsibility to make sure that the software is fit for purpose. Exchange 2007 no longer falls into that category.
I use Cisco ESA as a SmartHost to send email to internet. Will be MS identify my emails as a Exchange Server or other MTA?
If an interim host leaves the SMTP mail headers created by Exchange Server intact, they will be processed by Exchange Online for the purpose of this exercise. If not, they won’t.