At Microsoft Ignite 2018 this year Jeff Kizner, Microsoft’s Principal PM Manager in the M365 and Exchange Online Engineering team ran a groundbreaking session on existing and new Exchange Hybrid features for a more efficient and streamlined migration to Office 365. In his session, he discusses the challenges Messaging Admin face when moving to Exchange Online. Such as ongoing communications with Network Admin and security implications. In this blog, I am going to elaborate on these issues further and inform how you can overcome them with the new and enhanced Hybrid Exchange features.
The Hybrid Challenge
The first workload in Office 365 is mostly Exchange because other Office 365 applications like Microsoft Teams or Planner rely on the mailbox already being migrated to Exchange Online for a full functionality. For example, Microsoft Teams voicemail or configure connectors in Teams will only work if the mailbox is migrated to Exchange Online. Another reason is that the users don’t have to learn or use new tools because Outlook and OWA will not be changed after the move. This means you want to migrate your mailboxes as easy and fast as possible to Exchange Online.
For some of you it’s going quickly and for others, you might be having issues to migrate to Exchange Online – it’s necessary, but it’s hard. There are a lot of dependencies first, like configuring Active Directory synchronization, choosing the right sign-in method for your organization, and importantly communicating with your security department why Microsoft needs published endpoint from your Exchange on-premises organization to Office 365 (and back).
What we currently have is two separate organizations managed separately. The admin must go over to the on-premises Exchange Admin Center or Exchange Management Shell for one task, like add a new proxy address to an Exchange Online mailbox , then your admin must go to Exchange Online Exchange Admin Center or Exchange Online PowerShell for another task, like add mailbox permissions for a migrated mailbox, and so on. The idea of hybrid isn’t the reality because currently there are two separate organizations managed separately. The vision of hybrid should be an idea of a virtual organization that brings all your users and data under one view. Therefore, everything should be seamless and transparent for the users. Microsoft is working very hard on their vision of a single virtual organization, however, there is still a long way to go.
Administration and Configuration
The current problem is the administration and configuration between both organizations, Exchange on-premises, and Exchange Online.
To adjust your Exchange Online organization with your on-premises organization, there were a lot of things that you must have done manually prior to migrating your users. Customizations of policies like retention policy, OWA mailbox policy, mobile device mailbox policy, etc. must be created and configured in Exchange Online as well. The management of those policies will be always in both of your organizations if you have mailboxes both on-premises and in Exchange Online.
Organization Configuration Transfer
OCT v1 was released in June 2018, it allows you to do a one time copy of Organization Configuration objects to Exchange Online. The subset of policies and objects are:
- Retention Policy
- Retention Policy Tag
- OWA Mailbox Policy
- Mobile Device Mailbox Policy
- Active Sync Mailbox Policy
But those one-time transfer is limited to New-* actions only.
For example, if you had an OWA mailbox policy on-premises that didn’t exist in Exchange Online, HCW will do a comparison between both organizations. The comparison is done by the name attribute of the policy, which means if there is no OWA mailbox policy matching the name attribute, HCW will read all the configured properties on-premises, saves it in the memory and creates a new one (New-* actions) with the same properties, like you had on-premise. If HCW saw a match based on the name attribute, it will skip the OWA mailbox policy completely.
Now in October 2018, OCT v2 is available which allows you to transfer more Org configurations to Exchange Online:
- All OCT v1 objects
- Active Sync Device Access Rule
- Active Sync Organization Settings
- Address List
- DLP Policy
- Malware Filter Policy
- Organization Config
- Policy Tip Config
As part of v2, if a policy object with the same name attribute already exists both in Exchange on-premises and Exchange Online, you can now choose to either overwrite the values of the objects in Exchange Online or keep them as is (Set-* actions). And, just in case you overwrite the cloud settings and didn’t mean to, Microsoft provided a rollback script to undo those changes. Keep in mind that there is not a synchronization engine built in here, it is still a one-time transfer. Every time you run the HCW it’s doing a transfer, which you can run that as many times as you want.
OCT supports Exchange Server 2019, 2016, 2013 and 2010. OCT requires the latest cumulative update or update roll-up available for the version of Exchange you have installed in the on-premises organization. But if that’s too much of a burden, the immediate previous release is also supported.
Hybrid Setup and Onboarding
Microsoft talked to a lot of customers and wanted to figure out why they couldn’t go from hybrid to Office 365. A prevailing problem was the security implications, especially for financial institutes, because most security departments said they can’t allow any inbound traffic to the Exchange on-premises environment, even if it is locked down to the Exchange Online IP addresses. Another example is organizations that outsource their IT resources and can’t make any changes in an appropriate time frame to run the Hybrid Configuration Wizard, etc.
Microsoft has started to think about how they can fix this problem with a few simple principles:
- Keep it simple: It must be integrated with the experiences that you as an Exchange admin had control over.
- Start simple, prove the model, create a platform: Just like OCT, prove what we were doing was possible. But ensure you create a platform that you could continue to extend on.
- Don’t change Exchange authentication, authorization, or throttling: Work within the framework that Exchange has today. We wanted to make as few changes as possible to Exchange.
- Most importantly: we wanted to fix or eliminate the most common problems we saw engaging with all the customers.
- No customer DNS changes
- No certificate changes
- No firewall or network changes
- Protect on-premises systems
The Exchange Modern Hybrid will install an agent, built on the same technology as Azure Application Proxy, and publish your Exchange on-premises environment to Exchange Online without requiring any of the changes customers have struggled with.
All you need are outbound connections from that agent to the Internet on port 443 for the data moves and port 80 for the CRL checks. What that does is the agent talks to the hybrid proxy service which runs in Azure with a tenant specific endpoint (or a unique host name URL for your environment). This endpoint has nothing to do to identify your environment because it will not be discoverable and this random GUID will be assigned unique to your tenant. The only place you can find this URL is the on-premises logs, in the migration endpoint in your tenant, and the Organization Relationship in your tenant – it is not discoverable anywhere.
For security reasons, the agent has an IP white list and that’s how connections are controlled. It is locked down to the Exchange Online IP addresses.
V1 of the Hybrid Agent will support the core scenarios of mailbox moves and free/busy for your hybrid deployment and is in private preview now.
There is no change for the free/busy request form your on-premises organization to Exchange Online. This means you still must allow outbound connections to Exchange Online. For the connection from Exchange Online to your on-premises organization, the TargetSharingEpr parameter will be overwritten with the unique URL where the Azure App Proxy runs for your tenant.
The same URL change is used for the migrations where the remote server is the unique URL from the Azure App Proxy. Migrations are always a call from the cloud to on-prem and there is only one inbound direction that goes through the Azure App Proxy.
Hybrid Agent – Setup Details
The Hybrid Agent is built in to the Hybrid Configuration Wizard to make the installation as simple as possible.
- The Hybrid Agent Installer MSI file will be downloaded in the background to always receive the latest information.
- Then the installer will be kicked off and will do some basic installation tasks, like copy the bits to the machine you are installing on, register some PerfMon counters, and register a Windows Service (Microsoft Hybrid Agent).
- Then Azure will be contacted to register the agent. What happens here is that a unique identifier will be created for your tenant. A certificate request (CSR) will be created and sent up to Azure along with your admin auth token to validate your tenant. During this process there will be some other properties stamped on the certificate to make sure this certificate can only be used for your specific environment.
- The certificate will be handed back to you and installed on the box. It is valid for 180 days; the private key is marked as non-exportable and the certificate will be rolled out again automatically 30 days before it expires. For the configuration the backend URL will be stored, and this is where the agent itself on-premises talks to Exchange on-premises. Is it a hostname, a VIP? Microsoft stores that to match the inbound request to where it must hit on your Exchange on-prem servers.
- Then the frontend URLs will be registered within Azure and validated for Exchange. The validation is super simple and only Test-MigrationServerAvailability will be executed to pass it to the Azure App Proxy endpoint.
- To complete the configuration, the Organization Relationship will be configured with the TargetSharingEpr endpoint and set the migration endpoint.
The Hybrid Agent v1 will be contain the following functionalities:
The Hybrid Agent will be automatically updated, this means if Microsoft extends the agent with new features, you will get them automatically.
As a recommendation, put the agent(s) on your Exchange servers because it only talks 443 outbound to Exchange Online. Talk to your security department and if they want to put the agent(s) in the DMZ, you can put them also in the DMZ. But this requires maybe an additional server to be installed – keep it simple.
The Send-As Problem
The Send-As problem can’t be fixed with Directory Synchronization, as Microsoft did this for delegate permissions.
The Hybrid Agent can solve this problem, which was shown in a demo during the session. For example, you can add Send-As permissions on-premises for a cloud mailbox and remove those permissions from the cloud again. This is a very early stage and not rolled out to the productive Exchange Online environment now, which means there are no further details available. This is what Microsoft wants to talk about during the next year – how can they improve the goal to be a single virtual organization, what features are coming, what features are missing, etc.
- Exchange Online Hybrid setup has never been easier (still in private preview)
- Your networking and security teams can bother other people now
I’m glad to see that improvements will come to the Exchange Hybrid Configuration Wizard to make the move to the cloud easier. But not only mailbox moves, especially the administration and configuration will be a lot simpler than it was before, because the goal to a single virtual organization is now one step closer.
If you’re considering the move to Office 365, you might like the excellent white paper: Email Migration to Office 365.
Does this replace any currently configured on-prem hybrid solutions? We have two virtual win 2012 r2 std exchange 2013 CU20 servers with active MS Azure AD connect sync server. Also, when will this be out of private preview?
No, you can keep your existing hybrid infrastructure but you can also switch to modern. It’s was already this week:
I have installed the agent and have a migration batch in Synced status. It is stated that the Hybrid agent does not support SMTP mail flow, so will a Completed mailbox in Office 365 be able to email a local user? Will Office 365 send to the local user over the internet using a DNS lookup of the local server public MX record?
Mail flow between on-premises and Exchange Online occurs via Hybrid connectors. If you run the agent, there will be two connectors with the hybrid routing domain tenantname.mail.onmicrosoft.com which is stamped as the targetaddrese for migrated users in your local AD.
How is autodiscover supposed to be configured in this scenario (using Hybrid Agent)?
For on-prem to O365 like usual (classic hybrid).
But for O365 to on-prem…? (org relationship)
And for external clients…? (external DNS)
In other words, how should the external autodiscover DNS record and the online TargetAutodiscoverEpr attribute be configured?
The on-prem URLs (server.domain.com) are not externally available/published in this scenario.
Shouldn’t TargetAutodiscoverEpr point to the Hybrid Agent endpoint, just like TargetSharingEpr? Or simply to https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity ?
Have a look at the screenshot of the EWS example above: the Autodiscover connect from O365 to on-prem will still be available via your published URL (/WSSecurity) in the OrgRel & IOC configuration. Only EWS and MRS calls are routed through the hybrid agent with the new unique tenant ID.
Thanks for the article, do you know if teams will be able to do f/b lookups to on-prem using hybrid agent? Also will auto discover for on-prem need to still be published? Finally what about outlook mobile is this supported for FB lookups to on-prem using hybrid agent?
I’m not a Teams expert, but the Teams client gathers the availability data from Graph and (probably) at the backend it’s an EWS call to your on-prem infrastructure. If this works with classic hybrid for now, it should also work with the modern hybrid agent. Same for Outlook mobile – all EWS free/busy request stays as it is.
V1 of the hybrid agents works only for mailbox migrations and free/busy calls. Autodiscover must be still published, yes. But you can expect further improvements in v2.
Pingback: Choosing between Minimal and Full Exchange Hybrid
It seems like my comment got lost.
One question however. For me the end goal in migrating to cloud is get rid of the on-premise servers. However to my knowledge their is now way to decommission the servers once you have had a hybrid set up. ref, https://docs.microsoft.com/en-us/exchange/decommission-on-premises-exchange
“When directory synchronization is enabled for a tenant and a user is synchronized from on-premises, most of the attributes cannot be managed from Exchange Online and must be managed from on-premises. This is not due to the hybrid configuration, but it occurs because of directory synchronization”
Are you aware of any work around on this “issue”? Cause you will never get a clone only environment as long as you need to keep you on-premises servers.
Your are right – that’s the goal of many companies. Unfortunately, there is currently no “workaround” or something like that. Microsoft is working very hard to make this possible, e.g. through AAD Connect or other possible solutions.
The only solution is to keep the (free) last Exchange Server or disable directory synchronization (O365 only environment) for now. Maybe there will be some news at Ignite 2019.
Thanks for your reply, really apricate it.
Maybe this is MS way in pushing organizations the become pure cloud only organizations 🙂
I’ll keep following the news, and hopefully this will be available in the near future.
Hi Rkast, thank you – glad you like the post. Yes, the Hybrid Agent will use a Microsoft managed certificate. This means you don’t have to use a third-party certificate and also it is not required to publish Autodiscover and EWS externally. You can use your corp certificate without affecting the hybrid connectivity because every tenant gets a unique URL from the Hybrid Agent.
One thing I’ve been wondering about is that organization moving from hybrid set up to online, still are unable (post the migration ) to decommission their on-premises exchange servers.
“Customers with a hybrid configuration often find after a period of time that all of their mailboxes have been moved to Exchange Online. At this point, they may decide to remove the Exchange servers from on-premises. However, they discover that they can no longer manage their cloud mailboxes.
When directory synchronization is enabled for a tenant and a user is synchronized from on-premises, most of the attributes cannot be managed from Exchange Online and must be managed from on-premises. This is not due to the hybrid configuration, but it occurs because of directory synchronization. In addition, even if you have directory synchronization in place without running the Hybrid Configuration Wizard, you still cannot manage most of the recipient tasks from the cloud.”
Are you aware of any work on this “issue”? Cause one of the end goals (for me at least) is to get rid of them on-premises servers.
Hi, first of al great write up. Second does using the agent mean we can use an self signed/corp pki certificate for autodiscover and ews services ( cause with azure app proxy when creating an app and leave the url on msappproxy.net it uses Microsoft/azure cert and you dont need to bring your own)