A common load balancer configuration for Exchange Server scenarios involves using source NAT.
Although some load balancing terminology differs from vendor to vendor, for the context of this article “source NAT” will refer to a configuration where the source IP address of a connection is changed from the client IP address to one of the IP addresses of the load balancer.
In this scenario the client connects to the load balancer Virtual IP address (VIP), the load balancer connects to the Exchange server, and then the Exchange server replies to the load balancer, which sends the reply back to the client.
Although there may be nothing technically wrong with this from a functionality point of view (ie, everything works for your clients without any issues), it may become a problem for you in any support situation where you need to look at IIS logs on the Exchange server. The reason this may be a problem is that the IIS logs will show a “client IP” of the load balancer, not the originating client computer or device.
For example, here the client IP of 10.1.1.9 is logged for some OAB download connections from an Outlook client, which is actually the IP address of the load balancer.
Assuming you can’t change your load balancer topology or configuration to resolve this, there are options with IIS itself that can resolve the problem instead. Two of these options are:
- Using the Advanced Logging module for IIS from Microsoft
- Using the F5XForwardedFor ISAPI filter from F5
As a side note, both of these rely on the X-Forwarded-For header being sent by the load balancer to the Exchange servers. Refer to your load balancer documentation for how to configure this option. For example, on F5 load balancers you can enable it on the HTTP profile associated with a virtual service. And on my Kemp VLM I can change the L7 configuration from default X-ClientSide to X-Forwarded-For instead.
Option 1 – Advanced Logging Module
The Advanced Logging module can be installed and configured on your Client Access servers and enables you to configure a log definition that includes the X-Forwarded-For IP address details.
This is relatively easy to do, however it means an additional set of IIS logs is being generated on your server that you’ll need to manage. It also means you’ll need to look in two sets of logs whenever you’re investigating something and need to know the client IP address. In addition, my servers were not logging the client username correctly, which is apparently a common issue. I tried this fix, but it seemed to break Advanced Logging entirely. Your mileage may vary.
Installation and configuration instructions here.
Option 2 – F5XForwaredFor ISAPI Filter
The other option I looked at is an ISAPI filter from F5. This filter looks for the X-Forwarded-For header and, if found, replaces the client IP address with the X-Forwarded-For IP address instead. Effectively this means the IIS logs contain the correct client IP address instead of the load balancer’s IP address.
This is a better approach than the Advanced Logging module from the point of view that it doesn’t require a separate set of log files to be managed. However, although it works fine in my testing (it even worked with non-F5 load balancers) I am not sure whether any support is available from F5 for the filter, though they do provide the source code.
Download and installation instructions here.
Summary
If you are running a load balancer for your Exchange Server environment and you are doing source NAT then this is an issue that you should look into sooner rather than later. It is no good to wait for a support scenario where you need to know client IP addresses and then discover that the data simply isn’t available. Instead I recommend you get ahead of the issue and implement a solution that will give you the right data when you need it.
Pingback: Exchange 2013/2016: Switching from Zen Load Balancer to HAProxy – A random blog from a sysadmin
I have used ARR Helper to achieve the above. The F5 module didn’t work on the box I tried it on (IIS 8.5).
Hi Paul, one of the clearest articles on the topic once again. Thanks. Question: is there any way to make sure that the correct client IP also shows up in message tracking? Because there as well the problem is that the load balancer IP is being shown in the “OriginalClientIP” field.
The Real Person!
The Real Person!
No solution for the message tracking logs AFAIK, sorry.
I have used ARR Helper to achieve the above. The F5 module didn’t work on the box I tried it on (IIS 8.5).
The ARR Helper has also fixed the message tracking logs! It seems the message tracking logs are also using the client IP from the IIS logs.
The Real Person!
The Real Person!
That is interesting to know.
I am recently started working on IIS 8.5 (windows 2012 r2 ) .unable capture client IP in IIS logs instated of client IP it’s getting NetScaler IP .(two proxy ) and cs user name also not getting and I did X FORWORD FOR but it giving proxy IP’s kindly show the solution
Hey there!
We are using Web Application Proxy on our organization and we a trying to log the original client IP, tried the F5 sollution and the advanced logging, but nothing seems to work.
Seems that X-FORWARDED-FOR is not being sent over by WAP.
Any help would be greatly appreciated!
Seems that WAP in server 2012 R2 does not include X-FORWARDER-FOR at all. MS anounced that it would be included in WAP 2016 though!
Hello,
I’ve used the advanced logging with IIS and it works great, we can see the IP but as on your screenshot the CS-USERNAME is equal to –
Have you found a way to get the username in the advanced log ?
The Real Person!
The Real Person!
Nope. Like I wrote in the article, I tried a potential fix for it but it seemed to break it entirely.
Here the solution to fix the missing Username:
http://support.microsoft.com/kb/2529915
Have you tried using the F5 ISAPI Filter on a Windows Server 2012 server running IIS 8? I am contemplating upgrading our CAS servers to 2012R2 just to I can utilize the built in advanced logging in IIS 8.5.
The Real Person!
The Real Person!
No I haven’t yet. Andreas above has indicated it may not be required. But I haven’t looked into that yet.
I confirmed that the HTTP Filter module written by the same author works great with Server 2012 with IIS 8. Here is the link to his HTTP version which was updated to replace the ISAPI Filter: https://devcentral.f5.com/articles/x-forwarded-for-http-module-for-iis7-source-included#comments
I got this working very well but had to use the F5 http module from this related site: https://devcentral.f5.com/articles/x-forwarded-for-http-module-for-iis7-source-included#.U3z3g_ldWQc
Is there any way to get the client’s IP inserted into SMTP headers from outbound Exchange 2010 email? I realize that the X-Forwarded-For is an HTTP header and not related to SMTP. It would be very handy if outbound email would report the actual client IP in our SMTP headers. Currently the value is from the load balancer nat.
Thanks so much for your wonderful scripts and insights to Exchange.
Windows Server 2012 R2 includes this functionality out of the box (IIS 8.5).
Against which IIS version have you tested both options listed above – IIS 7.0 I guess? As both solutions are a few years old I wonder if they would work against IIS 8.0 (W2012) as well? Unfortunately did not have time to test yet.
The Real Person!
The Real Person!
IIS 7. Haven’t looked at 2012/R2 scenario yet.
Paul,
For those running 2012R2 IIS have some new advanced logging features,
http://www.iis.net/learn/get-started/whats-new-in-iis-85/enhanced-logging-for-iis85
The above link has a quick run through for adding X-FORWARDED-FOR to your logs mapping to a originating IP field. I do not know if it generates a new log file or not but from the reading it looks like it will not!
Once you add the new custom field for X-FORWARDED-FOR and apply the setting the website’s application pool is recycled and IIS creates a new log file with “_x” appendix. The original client IP is included as the last field there.
Note: for IIS to include this Custom header in request headers you need to enable it first
– Select server root in IIS
– Application Request Routing Cache > Server Proxy Settings > Preserve client IP
– Value should be: X-Forwarded-For
I currently have the same issue. I configured the custom logging on the Exchange servers, now I get an extra column in my logs, but it is not being filled (it only shows a hyphen).
This solution: “Application Request Routing Cache > Server Proxy Settings > Preserve client IP”
This is not something that would be configured directly on the Exchange servers, right? If I am correct, this would be used on an IIS server ‘in front of’ the Exchange servers, as a load balancing/reverse proxy solution? It would then be placed as an alternative for the Juniper PulseSecure appliance we are currently using? Is that correct?
Sorry, I forgot some information. I also enabled adding the X-Forwarded-For information in the header on the Juniper PulseSecure (but it is not showing anything in the logs).
Great
thanks
didn’t see anybody cover this “issue”
Great article, Paul – I’ve been wondering about this for a little while. Can you tell me, are you aware of any performance hit that comes with IIS advanced logging? I suspect it’ll have some impact.