During an upgrade of Exchange Server 2013 to a new cumulative update some customers may experience an issue where setup fails, leaving the server in a non-operational state.
The error thrown by setup is:
Mailbox role: Transport service FAILED
The following error was generated when “$error.Clear();
$connectors = Get-ReceiveConnector -Server $RoleFqdnOrName;
foreach($connector in $connectors) { if($connector.MaxLocalHopCount -gt 1) { Set-ReceiveConnector -Identity $connector.Identity -MaxLocalHopCount 5 }};
” was run: “Microsoft.Exchange.Management.SystemConfiguration Tasks.ReceiveConnectorRoleConflictException: The values that you specified for the Bindings and RemoteIPRanges parameters conflict with the settings on Receive connector “EX2013SRV2Test”. Receive connectors assigned to different Transport roles on a single server must listen on unique local IP address & port bindings.
This error is caused by a receive connector conflict. When the error occurs setup stops, but unfortunately it has already removed the previous build of Exchange in readiness to install the newer build. So the server is left in a completely non-operational state.
In this article I’ll explain:
- Why this error occurs in the first place
- How to fix it so that you can run setup again to repair your failed server
- How to check for this problem before you try to run a cumulative update
Why This Error Occurs in the First Place
The reason that this error occurs is that the server has two (or more) receive connectors bound to different transport services that are trying to listen on the same IP address and port number. My fellow Exchange MVP Andrew Higginbotham has a very detailed explanation of this here.
Every Exchange 2013 or Exchange 2016 server is configured with a set of default receive connectors that follow a standard naming convention that includes the server name.
Three default receive connectors are homed on the Front End Transport service:
- Default Frontend SERVERNAME – this connector listens on TCP port 25 to accept SMTP connections, and acts as the entry point for email into the Exchange organization. It is pre-configured to perform that role, and there are few scenarios in which modifying this connector is appropriate. As a general rule I recommend you do not disable, modify, or remove it. Think of this receive connector as the connector of last resort.
- Client Frontend SERVERNAME – this connector requires client authentication, accepts secure SMTP connections (with TLS applied), and listens on port TCP 587.
- Outbound Proxy Frontend SERVERNAME – this connector listens on TCP port 717 and is only used when a send connector is configured to proxy outbound email through the front end servers or services. This is an optional configuration, and not very common.
Two default receive connectors are homed on the Transport service:
- Default SERVERNAME – this connector accepts connections from other Mailbox and Edge Transport servers using TCP port 2525. Clients do not connect directly to this connector, and there are no reasons for you to ever disable, modify, or remove this connector.
- Client Proxy SERVERNAME – this connector accepts connections from Front End Transport services on the same server, or on other servers, using TCP port 465. Again, clients do not connect directly to this connector, and you should not disable, modify, or remove it.
You should not modify or remove the default connectors. Any other connectors on the server are likely to have been manually created, and that’s where the problem can occur. The short version is that when creating the receive connector the wrong transport role was chosen.
“Hub Transport” is the wrong choice here. Unfortunately it is also the default choice, and a lot of Exchange admins experienced in previous versions of Exchange would recognise “Hub Transport” as the role where a receive connector would have been created. However, there is no “Hub Transport” role in Exchange Server 2013, nor is there a service called “Hub Transport”. You could argue that “Hub Transport” shouldn’t even be an option in that particular dialog.
Fortunately the most recent builds of Exchange Server 2013 (SP1 and above, from what I’m told) contain logic to prevent an administrator from choosing the wrong option and causing the connector conflict.
Of course, that doesn’t help customers who have perhaps deployed the RTM version of Exchange Server 2013 first, and then later tried to upgrade to the latest CU. This is more common than you might think it would be, due to some customers acting with the incorrect assumption that the RTM build needs to be deployed first before any service packs or cumulative updates are applied.
Those customers, if they happen to have created conflicting receive connectors, are going to have a bad day when they upgrade. So let’s look at how to fix it.
How to Fix it so You Can Run Setup Again to Repair Your Failed Server
For organizations with multiple Exchange 2013 servers the fix is simpler, because you can still access the Exchange Admin Center on one of the operational servers. In mail flow -> receive connectors you can select the non-operational server and see the list of connectors on that server. In this example the connector named “Test” is highlighted, and you can see that it is bound to the “HubTransport” role.
That same connector name was in the error that was thrown by setup earlier, which is how you know which connector(s) to look for.
The values that you specified for the Bindings and RemoteIPRanges parameters conflict with the settings on Receive connector “EX2013SRV2Test”. Receive connectors assigned to different Transport roles on a single server must listen on unique local IP address & port bindings.
To change the role for the connector open the Exchange Management shell and run Set-ReceiveConnector as follows.
[PS] C:\>Set-ReceiveConnector EX2013SRV2Test -TransportRole FrontendTransport
For single server organizations it’s a little tricker. With no operational Exchange 2013 servers the management tools can’t be used. Instead we need to dive into ADSIEdit.msc to fix the connector. Launch ADSIEdit.msc and connect to the “well known Naming Context” of Configuration.
The receive connector objects can be found in Configuration -> Services -> Microsoft Exchange -> Your Organization Name -> Administrative Groups -> Exchange Administrative Group (FYDIBOHF23SPDLT) -> Servers -> Server Name -> Protocols -> SMTP Receive Connectors.
The attribute we’re interested in is the msExchSmtpReceiveRole. In the example below are two receive connectors. The one on the left has the role attribute set to 16384, which is Frontend Transport. The one on the right is set to 32, which is Hub Transport.
Modify the value from 32 to 16384 on your receive connector and then apply the change.
After taking either of the corrective actions above re-run the cumulative update and setup should proceed past the failed step and complete successfully.
How to Check for this Problem Before Running an Update
As I mentioned earlier there is logic in current builds of Exchange Server 2013 that should prevent this issue from occurring in future, but for those of you reading this who are planning to upgrade from a very early version of Exchange 2013 you might be concerned and want to avoid the problem entirely. There are two ways to do this.
The first is to simply open your Exchange Admin Center and look at the receive connectors on your servers. If you see custom connectors that you’ve created and that are bound to the Hub Transport role then you can pro-actively apply the Set-ReceiveConnector fix demonstrated above and you should be fine from there.
If you’re still unsure then you can simply run the same PowerShell commands that setup is executing when it throws the error. I’ve adjusted them slightly here for readability and added the -WhatIf switch so that Set-ReceiveConnector doesn’t make any actual changes to your configuration.
$connectors = Get-ReceiveConnector;
foreach($connector in $connectors)
{
if($connector.MaxLocalHopCount -gt 1)
{
Set-ReceiveConnector -Identity $connector.Identity -MaxLocalHopCount 5 -WhatIf
}
}
Pasting those lines into the Exchange Management Shell and pressing enter will check all of the receive connectors in your org for the conflict.
Read the errors carefully though, because it will throw one for your mis-configured connector but also for other default connectors that are not mis-configured. Obviously you should only apply the fix to your custom connector and not modify any of the default connectors on the server.
Summary
As you can see this is an unfortunate error condition that has quite a severe impact on Exchange 2013 servers during an upgrade. In particular, single server organizations are left with no operational servers at all after this error occurs. Hopefully anyone in that situation will be able to quickly find the solutions demonstrated in this article and get their server up and running again.
I get this error while running the hybrid configuration wizard. Disabling the connector as suggested in other documentation did not help. I am out of options
We have 3 Exchange 2013 servers. Can i install the CU23 on just one server and then the other 2 on a later date?
Thank You. You save may (Sun)-Day.
Hi Paul,
I have same issue recently for install the CU17 on the Exchange 2016. we have 4 Exchange server and do we need to change on the all exchange servers or just one by one when we want to install the CU
You Made and Rescue my Day (And my ass 🙂 )
Greetings from Germany
Great work paul, you save me once again.
I have a 2013 RTM with all (both) roles installed on one server (VM).
There are a couple Receive Connector Hub Transport rules, Default, and Client Proxy installed. I ran the test script and it passed. Can I keep these, or should I change to FrontEnd as the article recommends?
Believe it or not I just completed a 2013 RTM installation and went straight to CU21. Single server installation. Your article just saved my day. Thx again :O)
I would love to know if you did any special pre-install work or had any issues?
Hi Mr. Paul ,
Thanks for the Great Tip, Too Bad mine problem wasn’t resolved with what you gave us.
Mine Error is at the
MailBox role: Mailbox Service.
Error:
” was run: “Microsoft.Exchange.Data.DataValidationException: Database is mandatory on UserMailbox.
at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)
at Microsoft.Exchange.Configuration.Tasks.Task.WriteError(Exception exception, ErrorCategory category, Object target, Boolean reThrow)
at Microsoft.Exchange.Configuration.Tasks.DataAccessTask`1.Validate(TDataObject dataObject)
at Microsoft.Exchange.Configuration.Tasks.SetTaskBase`1.InternalValidate()
at Microsoft.Exchange.Configuration.Tasks.SetRecipientObjectTask`3.InternalValidate()
at Microsoft.Exchange.Management.Common.SetMailEnabledRecipientObjectTask`3.InternalValidate()
at Microsoft.Exchange.Management.RecipientTasks.SetUserBase`2.InternalValidate()
at Microsoft.Exchange.Management.RecipientTasks.SetMailboxBase`2.InternalValidate()
at Microsoft.Exchange.Management.RecipientTasks.SetMailbox.InternalValidate()
at Microsoft.Exchange.Configuration.Tasks.Task.b__b()
at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)”.
This is part of what the Error sais.
Hope you can help me to with error.
Thx
Thanks, you saved my life!
This is a fantastic article – it saved my bacon after an abortive upgrade from Exchange 2013 RTM to CU20. Thank you very much, Paul!
Thank you!!! You saved our upgrade night.
as checked below two connectors are showing with hub transport
•Default SERVERNAME
•Client Proxy SERVERNAME
i think in article these were mentioned, default, so do i need to change this ? only these are two with Hub Transport
The Real Person!
The Real Person!
Don’t modify the connectors that Exchange creates itself.
Hello Paul,
I have Hybrid envrionment of Exchange 2013 with Office365. there are few recieve connectors are showing pointing to hub transport on all 4 servers
So how can i check if those are default connectors or created while going to hybrid environment. I dont want to take a risk.. Please suggest
and changing from hub transport to front-end transport, will make any difference ? or it is very much fine to change ? any impact
Looking forward for your response..
and thank you so much to u. i read this article before upgrade..
The Real Person!
The Real Person!
I don’t understand your question. Are you asking me what the names are for the receive connectors that Exchange creates for itself automatically during setup? Those connector names are listed in my article above.
what do you think, is this issue can come from upgrading from cu13 to cu17 ? in sp1
and changing from hub transport to front-end transport, will make any difference ?
The Real Person!
The Real Person!
It could happen in any upgrade scenario, so you should check it before you upgrade.
as i see recieve connectors are in type hyb transport ..
so can i change all to front-end transport, ? will it impact anything in current envrionment
we are in hybrid environment, with O365
OK, this is like the third time one of your articles has saved my ass. You need to put a “buy me a beer” button on your profile.
The Real Person!
The Real Person!
When people buy my ebooks or watch my Pluralsight courses I get money that I can use to buy beer, so yeah 🙂
We just got Pluralsight. I will try to find you on there. Thanks.
You good sir, saved my day.
THANKS!
Thank you!!
The ADSIEdit fix for a single server worked like a charm!
FYI, before I started the CU17 install I did run your script but it did not return any errors, but then it did error out during the install.
Either way, thanks! You saved me many hours!
Thank you!! ADSIEdit worked perfectly. Life-Saver article! Saved me from rolling back to backup.
The Real Person!
The Real Person!
If you mean rolling back using a VM snapshot, that isn’t supported and would likely have caused more damage to your environment.
Hi Paul,
Does this only apply to multi-role servers?
Thanks
Dan
The Real Person!
The Real Person!
Well, you should only be deploying multi-role servers anyway 🙂
For Exchange 2013 it applies to multi-role servers. For Exchange 2016 there’s no separation of CAS and MBX anyway.
It’s probably possible to mess it up with separate CAS/MBX servers as well but since I never deploy Exchange that way I couldn’t go into more specifics.
You saved my ass on July the 4th man, thanks so much.
Um, if I change the receive connector to front end, mail stop flowing all together. So the hub transport is doing something, if I change it back (In ADSIEDIT) mails working again?
The Real Person!
The Real Person!
Which connector are you changing?
This is a great article! I did exactly what you had mentioned for a test environment, trying to update from a RTM build to CU6 with a custom receive connector set to HubTransport instead of FrontendTransport. Great article!
hi Paul,
i cant find :
Configuration -> Services -> Microsoft Exchange -> Your Organization Name -> Administrative Groups -> Exchange Administrative Group (FYDIBOHF23SPDLT) -> Servers -> Server Name -> Protocols -> SMTP Receive Connectors.
The Real Person!
The Real Person!
Which bit do you get stuck on?
Great job on the article.. Going to watch for this one in the future.
Many thanks !
Mush appreciated Paul, you’re the MAN
Hi Paul,
I think there is a correction needed at this article,
Wrong One: Configuration -> Services -> Microsoft Exchange -> Your Organization Name -> Administrative Groups -> Exchange Administrative Group (FYDIBOHF23SPDLT) -> Servers -> Server Name -> Protocols -> SMTP Receive Connectors.
Correct one: Configuration -> Services -> Microsoft Exchange -> Your Organization Name -> Administrative Groups -> Exchange Administrative Group (FYDIBOHF23SPDLT) -> Servers -> Server Name -> Protocols -> SMTP Receive Connectors -> Default Exchange 2013 –> properties
I have experienced the same issue in my lab and corrected it some time back.
The Real Person!
The Real Person!
No, you should never modify that connector.
hi Paul,
but I can’t locate the attribute “msExchSmtpReceiveRole” under the properties of “SMTP Receive Connectors” , I can see it only under the properties “Default/Default FrontEnd /ClientProxy/ClientFrontEnd/OutboundProxyFrontEnd” after expanding “SMTP Receive Connectors”
i’m still confused.
The Real Person!
The Real Person!
The servers have several receive connectors that are configured by Exchange setup. They are listed in the article above. Those should never be touched.
HI,
I have problem with my Exchange server 2013 Sp1 ,the stucked emails to the draft folder 3 week after installation. please help me.
thank you.
F. R
Paul, you are the friggin’ man! Wow. CU13 just takes a dump and leaves you cold and alone in the wilderness to fend for yourself. I’m a single site, and running RTM for over two years! (Hey, it was 99.9% functional…so, stop your laughing!)
Your article and fix here is very concise and a lifesaver. Many thanks from the Old Mule, in Louisville, KY. I’m indebted to you for your knowledge and sharing it here.
Thanks Paul! I got this error with CU12, just saved my weekend!
Thank you, I have read many articles you have written in the past also. You are a real Systems Admin. One that shares his details and thus is not all about me type person. I am currently on a 200k 2010 to 2013 migration with one of the biggest companies everyone knows. I have great praise for your time and thoughts. I love my complex job and friends that share their vast experience’s. Although I have worked at MS twice in my career their resources and services tickets fail me time and time again and the TechNet articles are slow to change to keep up. By that time me and many of the worlds engineers are stuck in “crises mode” and are forced to find a solution while the customers suffer for lack of better words “they loose money”.
Thank you for all u do !
Charles Giles
I am so glad I found this before I run the CU’s!
I have a DAG which was set up before I started and now have the task of getting them up to date. Each server in the DAG has Outbound, Deafult and Client pointing to FrontendTransport and then a Client and Default pointing to HubTransport which are left over from the 2010 migration. They both cover all IPs like their FrontEnd counter part but are set to port 2525 and 465 with the Frontend connectors using 25 and 587.
My intial thought is to just delete the HunTransport connectors as we have the Frontend Transports connectors. Am I right in thinking that as they point to Hub Transport and this is no longer a service then they can’t be working anyway?
Many Thanks,
Natalie
The Real Person!
The Real Person!
No … don’t modify or delete any of the connectors that Exchange setup created. This article only applies to custom connectors that have been incorrectly configured by an administrator when they were created. I’ve updated the post to clarify.
Great, thank you! So I shouldn’t get failed update leaving them as they are and pointing to the HubTransport?
Just read your amendment, thank you very much.
YOU made it 8-D
Thanks a lot Paul. It’s awesome to learn from you!
Greetings from Germany
patrick
Adding my heaps of praise and thanks to the thread, thanks Paul! Saved my bacon for sure! Had a hard time finding anything related to the error until I limited my search and found this. As an above comment said, it was like a needle in a haystack! Very well written article, perfect step-by-step cleanup/fix. Thanks!
Thank you Paul, you rescued me from my heart attack! I would have never been able to figure this out. (Single Exchange server, I am managing an installation done previously by someone else)
Cheers,
Hi Paul,
We currently have Exchange 2013 SP1 CU8, trying to install CU11.
It says we need to update the Schema/AD domain, which I did not expect.
Is this behavior normal or different thing, I cant see anyone told about this here in forum.
The Real Person!
The Real Person!
It is normal behaviour. You can expect every Exchange cumulative update to include a schema update.
Great comment. It helped me big time. thks a lot. I got this error while upgrading to CU10.
Whew… Thanks to you and google my heart has stopped trying to pound out of my chest. I could have restored to last backup, but I never like having to tell the boss that. Thanks!
Fantastisk! Big Thanks indeed !
Life = Saved.
Thank you.
Needless to say. Thanks again Paul.
I have a question.
I am trying to setup Exchange 2013 to receive email from a Spam Server but I get the following error message “the values that you specified for the Bindings and RemoteIPRanges parameters conflict with the settings on Receive connector.” How am I to allow incoming email from this server?
The Real Person!
The Real Person!
The default receive connectors on an Exchange 2013 server are already configured and ready to accept email for internal recipients.
Other than that, any new connector you create should be bound to Frontend Transport not Hub Transport.
you saved me 🙂
thanks !!!!!
Wow. This is a needle in a haystack problem and your multiple solutions (either with 2+ Exchange servers or with 1) are fantastic. Thank you Thank you Thank you.
Worked like a charm! Thanks Paul!
I faced this issue while upgrading from Exch2013-SP1 to CU9 on DAG server with two nodes ( Active/Passive ) starting with the passive node.
I got the error ( Receive Connector Conflict )
I had to delete the Customized Receive Connector with Hub Transport on Both DAG Nodes and restart some Exchange Services.
Finally It’s working and resuming the upgrade process
You Saved My Time .
Thankssssss Paul
If one has a CU8 already installed (and upgrade from CU7), can it be assumed that this will not be an issue when installing CU9? Thanks!
The Real Person!
The Real Person!
Why rely on assumption when it only takes a few seconds to check?
Fair point – to be safe I checked – the script that you provided returned no errors at all.
Great article, came in handy. I’ll note that SP1 does not have sufficient logic to prevent this, I just updated SP1 -> CU9 and a disabled HubTransport connector caused this.
Same case – single exchange- you use ADSI tool and change it in the AD. Works like a charm !!
Hi,
Great post, but what to do if you only have a single exchange 2013 server with this error?
The Real Person!
The Real Person!
Use ADSIEdit as described in the article.
Hi,
been using adexplorer from sysinterals. ADSIEdit doesnt seem to show all the information. While with adexplorer everything shows up as described in the article.
After that the setup failed again, due to not able to find the grammars directory (C:Program FilesMicrosoftExchange ServerV15UnifiedMessaginggrammars) . I have created this folder manually and setup completed after a new run.
You rule !!!
What about receive connectors on a mailbox server that is not co-located with the ClientAccess role? Do these need to be set to FrontEndTransport as well?
Hi Paul,
Nice article, thanks for taking the time to write it up! Interested to hear why you’re choosing to change Test to a FrontendTransport connector, rather than changing the bindings to port 2525?
I thought that the default FrontendTransport connector already allowed access from anyone on any source IP address anyway, so wouldn’t you normally create additional HubTransport receive connectors with the different Auth Mechanism and Permission Groups as needed for a smaller set of IPs?
Thanks in advance
Twan
The Real Person!
The Real Person!
No, absolutely not.
Great article, this has caught me out in the past. I restoring from backups to get up and running again. I’m nervous about trying again as I don’t have a test environment to try it in beforehand 🙁
The Real Person!
The Real Person!
Using ADSIEdit to fix the role for the connector, or deleting the offending connector entirely, seems to be a reliable solution from all reports I’ve received.