Home » Exchange Server » Troubleshooting Email Delivery with Exchange Server Protocol Logging

Troubleshooting Email Delivery with Exchange Server Protocol Logging

This week I tweeted a tip that can help you troubleshoot email delivery for your Exchange servers.

In this article I’m going to expand on that topic and explain why protocol logging is useful, and how you can enable it in your own environment.

What is Exchange Server Protocol Logging?

Protocol logs capture the SMTP communications that occur between servers. The information that is written to the protocol log files looks very similar to what you see when you are using Telnet to make an SMTP connection.

Example of a protocol logging log file

This information is invaluable in troubleshooting scenarios, because it captures events that occur during message delivery that may not appear in other logs on the server.

For example, many administrators are used to looking in message tracking logs when they troubleshoot email delivery. But message tracking logs only record events for messages once they are in the transport pipeline. If a message is never sent/received because the SMTP connection itself is rejected, the message tracking log will show no useful troubleshooting information.

There are two parts to the configuration of protocol logging in Exchange Server, and they are basically the same across Exchange 2007, Exchange 2010, and Exchange 2013.

Configuring Protocol Logging on Transport Servers

The first is the per-server settings, configured on Hub Transport and Edge Transport servers for Exchange 2007/2010, or either Client Access or Mailbox servers in Exchange 2013.

Protocol log paths can be set in the Exchange Management Console in the properties of the server.

However, most of the other useful settings can only be configured in the Exchange Management Shell.

You can use Get-TransportServer to view existing settings.

Note: I’ve left out the HTTP protocol log settings from the above output since they are not relevant to this article.

You’ll notice that send and receive logs have their own settings, allowing you to keep those log files in separate paths and use different retention periods to suit the specific server’s role.

The default settings on a server are fine for low volumes of email traffic. The 30 days retention is useful in theory but the max directory size of 250mb will mean that high volumes of email traffic will probably result in far less than 30 days of log files retained (250mb would cover a few hours of traffic on one of my servers at the moment).

So think of the max directory size as a safety net for your log retention to make sure that logs don’t fill up the disk on the server. My experience these days is that Transport servers tend to have plenty of free disk space so I like to set the max log directory to something generous like 4GB for both send and receive.

If you have any concerns about storing logs on the C: drive, either for performance or free space reasons, you can use the protocol log path settings to move the log directory to a different disk.

Configuring Protocol Logging on Connectors

The second part of the configuration is the per-connector settings. Even though protocol logging is enabled by default on Transport servers, it is not enabled by default on any send or receive connectors.

One approach is to enable protocol logging on every connector in your organization. However some administrators will prefer to enable it only as needed.

I recommend that you enable protocol logging on any of the following:

  • Send/receive connectors on any servers involved in internet email flow
  • Receive connectors on any servers that act as SMTP relays for internal devices and applications

Those recommendations are based on my experience that most SMTP troubleshooting cases are for situations where email is not delivering to/from the internet, or from internal devices and applications that relay through your Exchange servers.

I’ll give you two real world examples of those situations:

  1. Outbound emails to specific domains are queuing on your Edge Transport servers. Protocol logging is how you can discover what reason the recipient’s email server is giving for rejecting the connection from your server.
  2. You’ve added an IP address of an internal server to your relay connector. Protocol logging is how you can ensure that the correct receive connector on your Hub Transport server is processing the connections from that internal server.

You can see all of your receive connector protocol logging settings using the Get-ReceiveConnector cmdlet.

The setting is also visible in the properties of a receive connector.

Protocol logging on a receive connector

Send connectors also have their own protocol logging level, visible when you run Get-SendConnector (you’ll notice there is no “server” for a send connector).

Again, you will also find the setting in the properties of the send connector.

Protocol logging level on a send connector

Using Protocol Logging Data

Once you have protocol logging enabled there are a few ways you can put it to good use.

The first is in specific troubleshooting cases. Protocol logs are quite readable so all you need to do is open them in Notepad or a similar text editor and look at the data.

However, finding what you’re looking for can be a challenge if you’ve got a lot of log files.

Lots of logs files…

You can often narrow down the file you’re looking for because they are named according to date. But if there are multiple files for a given day you can search for strings in the files.

You can use PowerShell to search for string matches in the log files. In this example Get-Childitem returns the list of log files in the current directory, and I’m piping those into Select-String to look for “microsoft.com”.

The lines with the pattern matches as well as the file name itself (eg RECV20120806-1.LOG) are displayed in the results.

You also get the remote IP address (eg 114.42.130.106) returned, so you can perform a further search to see the entire SMTP conversation that occurred with that host.

Those results can be a bit ugly though. Sometimes its easier to output them to a file for reading.

Protocol logs were useful when I was troubleshooting the case of the Hub Transport server load imbalance.

Protocol logs can also be used for general reporting and monitoring of your servers. I’ve published some tutorials that show how to use Log Parser to extract useful insights from protocol logs.

Summary

As you can see, protocol logging is quite useful. I do recommend you check your protocol log configurations and enable them on the most important send and receive connectors so that when problems happen you have this valuable log data available to assist you with troubleshooting.

Paul is a Microsoft MVP for Office Servers and Services. He works as a consultant, writer, and trainer specializing in Office 365 and Exchange Server. Paul is a co-author of Office 365 for IT Pros and several other books, and is also a Pluralsight author.
Category: Exchange Server

59 comments

  1. Leo says:

    Hi Paul,

    You are producing high quality content and great article layout. Can I suggestion to add a ‘print’ icon whereby you can print the article as printable version which removes all the ads, comments etc.

    A printable version whereby a clean article……

    Keep up the good work mate.

    cheers

  2. Dinesh says:

    Paul,

    Article is amazing very depth knowledge of the troubleshooting for mail flow.

    Thank you for sharing such a deep dive

  3. Joseph says:

    Very Good article Paul with lots of good details. This one is going to really help admins as Troubleshooting issues with mail flow is always an issue and most of the time we cannot identify the cause of the issue when it happens in an Org.
    Thanks for sharing with the forum.
    Cheers!

  4. Azad Kumar says:

    Hello Paul,

    Its an amazing experience to read this great article.

    Very precisely, in depth and well work done!

    Many Thanks!

    -Azad

  5. Richard says:

    Excellent article !
    But one question, when you’ve got a proxy in order to filter spams in the architecture, Exchange saw only one IP Address (the anti-spam gateway address).
    If a Saas application must send a mail with relay. How exchange choose the right connector ?

  6. William Gonzales says:

    Paul,

    You’ve been a consistent knowledgeable resource for me throughout the years as I carved myself a career as an Exchange Engineer. So I would like to thank you for all of the nuggets of wisdom you have shared. I have a question for you. In your article “Don’t leave pop/imap protocol logging enabled” you referenced this article and you mentioned the following:

    “As a side note, I surveyed a few of my fellow MVPs on the topic of protocol logging and opinions were split about 50/50 on whether it should be left on all the time or not. You can certainly make up your own mind about that”

    I’m curious as to a few reasons why it should not be left “enabled”. Of course if disk space is a concern, there’s circular logging. But what I tend to hear is that it creates additional “stress” on the hub transport server, and words like “overload” are mentioned.

    I’ve transitioned into a new position and I previously maintained a small environment with 3 sites and 8 servers. Today I’m working at a company that has quadruple the servers and is the largest environment I’ve ever supported. Recently I found myself in need of the protocol logging and found that they weren’t enabled at all. I find them very useful. Would enabling on a very large environment really cause such a noticeable strain on the environment?

    • There’s no stress or strain. Exchange is already writing a ton of log files to disk. My opinion is that a server would already need to be overloaded for protocol logging to contribute to the issue.

      SMTP protocol log settings have max age and max directory size thresholds you can set to ensure that logs don’t grow forever and consume all available disk space.

      POP/IMAP protocol logging does not have those thresholds and carries the risk of consuming all available disk space.

      That is why I don’t recommend leaving POP/IMAP logging enabled (unless you implement your own cleanup scripts), whereas I do recommend SMTP logging be left on for all the benefits it delivers for troubleshooting scenarios.

  7. Jose Byron Gonzalez says:

    Just wanted to thank you for another great piece, where you manage to teach an arcane subject and make it almost enjoyable to be unraveling the inner working of Exchange. As the previous comment mentioned, you are a consistently valuable resource and I wanted to express my appreciation for that.

  8. Steve says:

    If you’re unfamiliar with “Get-ChildItem” make sure that you drill down to the actual SMTP protocol log file path in PowerShell before you enter the commands.
    Example:
    [PS]C:ProgramFilesMicrosoftExchangeServerV14TransportRolesLogsProtocolLogSmtpSend>Get-ChildItem | Select-String -Pattern “example.com.au”

  9. Hany Kamal says:

    Please help me to understand the following lines that are captured from SmtpLog files:

    Date: 30/06/2015 02:09:32.498
    Session-ID: 08D271F4045EF862
    Local: None
    Remote: 41.128.142.5 port 25

    02:09:32.498 [attempting to connect]
    02:09:32.873 220 **************************************************************************************
    02:09:32.873 EHLO mail.gsfmo.gov.sa
    02:09:33.060 250-bmail.linkdatacenter.net [85.194.65.196], this server offers 7 extensions
    02:09:33.060 250-AUTH LOGIN
    02:09:33.060 250-SIZE 21504000
    02:09:33.060 250A

    Notes:
    1 – each time we are trying to send to a domain esmartsoft.com.eg we received this error.
    2 – I used a program called “LogView.exe” to enhance the output of the log.

    Please I need your advice ASAP

  10. Hany Kamal says:

    As well as I want to tell you one more thing that could clarify my situation.
    The outgoing mail server (SMTP) for domain esmartsoft.com,eg was wmail.link.net and at this step the gsfmo domain was able to send all mails to esmartsoft.com.eg domain but once it’s changed by the ISP to be linkmail.hosting.link.net the gsfmo failed to send any mails after that changes.

  11. KERR says:

    Hi, thanks for the info, how would you find the log location for connectors in Exchange 2013? I’ve enabled the verbose protocol logging, and confirmed with your powershell command.

    Cheers

    • KERR says:

      EDIT, found them.

      Receive connector protocol log files for the Transport service on Mailbox servers %ExchangeInstallPath%TransportRolesLogsHubProtocolLogSmtpReceive

      Receive connector protocol log files for the Mailbox Transport service on Mailbox servers %ExchangeInstallPath%TransportRolesLogsMailboxProtocolLogSmtpReceive

      Receive connector protocol log files for the Front End Transport service on Client Access servers %ExchangeInstallPath%TransportRolesLogsFrontEndProtocolLogSmtpReceive

      Send connector protocol log files for the Transport service on Mailbox servers %ExchangeInstallPath%TransportRolesLogsHubProtocolLogSmtpSend

      Send connector protocol log files for the Mailbox Transport service on Mailbox servers %ExchangeInstallPath%TransportRolesLogsMailboxProtocolLogSmtpSend

      Send connector protocol log files for the Front End Transport service on Client Access servers %ExchangeInstallPath%TransportRolesLogsFrontEndProtocolLogSmtpSend

  12. Loreman says:

    Hi Paul!
    What about Exchange 2016?
    After few hours of investigation :), i notice that paths for logs is changed.
    e.g. if you create a custom receive connector for Anonymous relay and set logging to verbose, you’ll not be able to find logs in the expected path.
    Logs will be wrote in the default path:
    C:Program FilesMicrosoftExchange ServerV15TransportRolesLogsFrontEndProtocolLogSmtpReceive

    I don’t know why, but hope this help 😉
    Loreman

  13. mar says:

    Hi Paul,

    I’ve had encountered an issue regarding smtp forwarding. We have a large email volumes each day so we decided to try out a 3rd party email archiver. by enabling smtp connector, we have integrated our exchange 2010 to archiver, and everything was running okay, then one day the archiving server was shutdown. We have congestion in the mail system, so we decided to remove the smtp connector. Can you help us to determine if exchange have some type of feature that can detect if the archive/backupserver which Exchange supposed to send a copy of email is down or up. Then what action can an exchange server can do to immediately stop the smtp connector without human intervention.

    thanks!

  14. Gurwinkle says:

    Is it quite similar to the telnet basically we are looking for error to eliminate the problem.

    We can check the MX record by set type=mx and then the domain name.

    Then we can check the rules of the firewall if rule is blocking the port no 25.

    Then checking SMTP , we can increase the maximum directory size of protocol logging or we can check the error in the protocol logging.But that error can be accomplished by the echo telnet command.

    Is I am right and then we can fix the issue as it persists or we can check the performance of the exchange by using backpressure

  15. Gurwinkle Singh says:

    If that looks healthy then there is some other issue which we can check with the message tracking if it is outlook or exchange server or spam issues.Junk is marked or blacklisted while in the pipeline or user is blocked in pipeline or some kind of a hybrid issue is there.If outbound connectors are configured properly or some transport rule is blocking the email.On premises we can repair the database if something is also wrong with dirsync.

  16. Rossan says:

    Hi,
    I have my Exchange server 2010 which was working fine. Just few days back it was observed that incoming emails were irregular as some were not coming through. I have checked all firewall issues were intact. What might be the problem? Please assist!

Leave a Reply

Your email address will not be published. Required fields are marked *