This article is an excerpt from the Exchange Server Troubleshooting Companion.
Email spam continues to be a huge problem for organizations these days, and it usually falls on the Exchange administrator to do something about it. Aside from the usual anti-spam measures we can put in place to protect our own servers from spam, we also need to consider how to prevent spammers from spoofing (imitating) the domain names for our own organization. After all, it can be very embarrassing or cause serious brand damage to have spam and malware that uses your domain name.
To detect spoofed email many receiving servers, particularly those operated by large email providers such as Microsoft, Yahoo, Google, and AOL, will perform a check of the Sender Policy Framework (SPF) record for the sender’s domain when a sending server is attempting to send an email message.
SPF records allow a domain owner to specify which mail servers are permitted to send email for that domain name. When the sending server issues its “MAIL FROM” command in the SMTP conversation, the receiving server will look up the SPF record in the domain name of the MAIL FROM email address to see if there is a match for the source IP address of the SMTP connection.
If you were reading about SPF records on the internet you may find advice from some websites that it is better to have no SPF record than it is to have an incorrect SPF record. There’s some truth to that, but also some risks. Some mail hosts will reject mail if there is no SPF record for the domain. It tends to be few hosts that do that, but because they are very large mail hosts the impact can be quite noticeable. Ultimately, it is best to have a correctly configured SPF record in DNS for your domain.
An SPF record is simply a TXT record with a certain syntax. The syntax is made up of two parts; mechanisms, and modifiers. Modifiers are optional and are not commonly used except for special circumstances. During management and troubleshooting of transport you’ll most often be dealing with SPF records containing only mechanisms.
The mechanisms for an SPF record define the sets of hosts that can send email from the domain. Mechanisms can be defined by:
- all – matches any host, and is placed at the end of the SPF record as a “catch all” for any senders that did not match other mechanisms listed ahead of it.
- ip4 – matches a single IPv4 address or IPv4 network range.
- ip6 – matches a single IPv6 address of IPv6 network range.
- a – matches a host name or domain name. The IP addresses that the name resolves to in DNS are matched against the sender’s IP address. This mechanism is useful for matching against a web server IP address based on the domain name.
- mx – matches against the MX records for the domain. This mechanism is useful when the outbound mail is handled by the same servers as the MX records resolve to for inbound mail.
- ptr – reverse DNS queries are used to match the sender IP address to the host names that it resolves to. This mechanism is generally not recommended due to the DNS load it causes.
- exists – simply checks that the domain exists in DNS.
- include – matches the sender IP against the SPF record another domain. This is commonly used when your outbound email is routing via a cloud service such as Exchange Online Protection.
Mechanisms are used in combination with a qualifier that tells the server what to do when a match is found. The qualifiers are:
- “+” for pass (this is the default if no qualifier is explicitly provided)
- “–” for fail (email from unauthorized hosts should be rejected)
- “~” for SoftFail (may result in email being accepted but marked as “likely spam”)
- “?” for Neutral (regardless of the result the email should be accepted)
An example of a mechanism paired with a qualifier is “-all” at the end of an SPF record, which means “Fail/reject email from any sender who did not match an earlier mechanism in the SPF record.”
If this all seems very complicated to you, don’t worry, it starts out that way for everyone who has to deal with SPF records. Fortunately, there are many tools available to help you construct and validate your SPF records. For example, Microsoft provides the Sender ID Framework SPF Record Wizard, which has an awkwardly long name but is nonetheless very useful. Update: Microsoft’s tool has been down for a few months, so you can use the SPF Wizard instead if you need to.
After entering your domain name the wizard will step you through a series of questions to determine the most likely SPF record that you will need. In this example I answered the questions as follows:
- Domain’s inbound servers may send mail (in other words, the servers listed as MX records also handle outbound email)
- An additional domain name whose A record is a valid outbound email server (a common example of this is an externally hosted website that uses its own SMTP service to send notifications and other emails)
- This domain sends mail only from the IP addresses identified above (in other words, anything else trying to send email from my domain name should be considered unauthorized)
The resulting SPF record looks like this.
By adding that string as a TXT record in the public DNS zone for the domain name I will have prevented unauthorized email servers from spoofing my domain name. At least, they won’t be able to do it when sending to any receiving server that checks SPF records. Anyone who is not checking SPF records can still receive the spoofed email, but may reject it for other reasons such as spam content or malware.
Apart from tools to generate your own SPF record, many email services will provide you with the exact strings to add to your SPF record. When you add a domain name to Office 365 Microsoft advises you of the SPF record they suggest, which is appropriate for organizations sending their outbound email using Exchange Online Protection. Similarly, email marketing services and SMTP hosting services will also have documented solutions to adjust your SPF record so that you can successfully use their services without your email being rejected.
After you have your SPF record in place you should validate it. And in fact, you should repeat this validation test any time you suspect an external organization may be rejecting your email because of your SPF record. MXToolbox has an SPF record validator that takes a domain name and IP address as input and lets you know what the result will be if that IP address sends email for your domain.
Aside from the result for that specific IP address, the MXToolbox SPF record lookup tool will also validate the general health of your SPF record for problems such as excessive DNS lookups or syntax problems.
Despite the importance of SPF records for internet email delivery, your internal mail flow between Exchange servers in your own organization is not dependent on SPF records. The Exchange servers in your organization already understand that other Exchange servers in the same organization are authoritative for your domains.
Pingback: Paul: Do You Really Need an SPF Record? – Seo On Exchange
Paul, thank you for such good articles and explanations.
I would like to ask if we should disable in exchange2013 with:
Set-SenderIDConfig -Enabled $false
when we have an external email security gateway which does the spf check for us.
Is that safe?
thank you
hi Paul, I have a concern for envelope sender and “from address”
Can exchange server catch it if envelope sender is different from “from address”?
Most spammer, phishing email, the “from address ” using friendly email addresses ,looks like customer,boss,etc
and the envelope sender , spammer can send email via SPF pass email service provider.
some users may get trapped in this kind of emails.
So i want to indicate user about possible spam if envelope sender vs “from address” are different.
I check external email gateway, and exchange server 2013 does not have this feature.
Do you have any idea?
The Real Person!
The Real Person!
I don’t think Exchange has this capability. You should implement an email security product or service to filter the mail and look for those kinds of signals. Most antispam products like Exchange Online Protection should catch it.
its worth pointing out microsoft products DO NOT check use spf
they use an incompatible (and abandoned) protocol sender-id
all users of SPF records must (to be received correctly by systems running exchange/sender-id) have a null sender-id policy also (or their correct spf policy will be mis-used as a sender-id policy
the sender-id policy is thus
“spf2.0/pra ?all”
meaning if checking from: header (sender-id) return neutral
and their normal spf
“v=spf1 ip4:xx.xx.xx.xx ip4:xx.xx.xx.xx ip4:xx.xx.xx.xx ip4:xx.xx.xx.xx -all”
also note when testing a new spf policy the terminator should only ever be ?all (aka ignore failures)
http://www.alandoherty.net/info/mailservers/spf/
Hi Paul, do you recommend DMARC? And what’s your opinion on it, i.e. pro’s and con’s. Would appreciate your thoughts.
Thanks. Lee
The Real Person!
The Real Person!
I recommend looking at it. I haven’t spent enough time with it to give a thorough pros/cons view of it.
Pingback: Do You Really Need an SPF Record?
Hi Paul
What is the recommended way to setup PTR records if I have 2 X Exchange servers in a DAG?
The public IP’s are set in the SPF record already and hard fail has been setup.
However each exchange server has its own public IP for example:
Server 1 1.2.3.4
Server 2 1.2.3.5
mail.domain.com resolves to 1.2.3.4 and reverse lookups work ok
Scenario is a client sends from Server 2 1.2.3.5 and gets a bounce back
Client host rejected: cannot find your hostname, [1.2.3.5]
This is because the PRT is not setup and the destination server is running a PRT check and 1.2.3.5 does not resove back to mail.domain.com.
So the answer is to ask my ISP to setup PTR records? Does this sound correct?
The Real Person!
The Real Person!
I’m going to assume your send connector has both servers as a source transport server.
In that case, you need the same DNS, PTR, SPF requirements for both servers’ public IP addresses.
Hi Paul,
I see that website is not working for creating SPF records do you have a alternative website to use?
Thanks
Nice article Paul.
We are using Westnet email relay filter service (relay-filter.westnet.com.au) and all incoming emails will go through them first. If I want to add SPF record to DNS zone, do I need to mention them in data field?
The Real Person!
The Real Person!
SPF records are for any sending hosts for your domains, in other words, outbound email. So if you’re sending out via Westnet’s servers as a smart host, then yes they should be included in your SPF record (they probably have some specific guidance to help you with that, most providers do).
Hello Paul,
the “Sender ID Framework SPF Record Wizard” in your post contains a dead Link – could you be so kind and provide the correct URL ?
Thank you very much
Alex
Vienna – Austria
Hi Alex,
The link was live at the time of publishing, unfortunately Microsoft have moved / removed the SPF Record Wizard.
I’ve been creating SPF records for clients over the last few months and, between creating the first SPF record and the second, the Wizard returned an HTTP 500 error.
Just to add…
If you use a Smarthost for outbound email, your experience may vary. For instance, some providers maintain a list of their Mail Server IP addresses, making things simple.
For instance, the UK ISP Zen. To configure the rest of my SPF record I just had to add ‘include:_spf.zen.co.uk’. Whereas, for the UK ISP Demon, I had to add their IP ranges in CIDR format.
Great article Paul,
One area that may need clarification is: “When the sending server issues its “MAIL FROM” command in the SMTP conversation, the receiving server will look up the SPF record in the domain name of the “From” address to see if there is a match for the source IP address of the SMTP connection.”
The SPF lookup isn’t done on the ‘From’ address, it is done on the domain in the return-path field. If you do not specify the return-path specifically, it is automatically added to the header with the value from the “From” field.
The difference in this is important to outline how spammers bypass SPF lookups, as Marc has pointed out.
Keep up the great work.
The Real Person!
The Real Person!
Updated for clarity. MAIL FROM sets the return-path which is also used as the From if no other From is specified later in the SMTP conversation.
Paul are you speaking about the X-MAIL-FROM or the From: header?
One battle we are facing is that the email headers X-MAIL-FROM, Return-Path, X-Sender, envelope-from all contain the outside address (spammer@gmail.com) or whatever crazy mail domain is made available. The issue I’ve been seeing is that it’s only From: field that has been changed. Sometimes the message is made to look like a customer’s address as well as sometimes it’s an address of an employee made to look like it came from that employee asking for information. It’s been more a targeted / spear-fishing thing and not just a spam bot that received a bunch of email addresses from someone’s address book.
I don’t suppose SPF’s would help combat this type of situation if it’s not going to match anything in the From: field header vs. Return-Path, X-MAIL-FROM, X-Sender.
It’s important to note, that many Spoofers are putting the spoofed address in the header only, and they use a different address on the envelope that they know have no SPF. This way the SPF test is neutral, but the receiver sees the spoofed address. I do use SPF but it doesn’t mean that your users will not received spoofed email even from domain that have SPFs. You can’t do much for your own domain from this, but to protect your users from receiving spoofed emails, you can increase the score of an address mismatch, I believes that most gateway score it very low. Not sure how you could deal with this on Office 365 or Exchange Online Protection.
The Real Person!
The Real Person!
There’s two things here that can help. The first is that a lack of SPF record will cause some mail hosts to outright reject the mail. Some email security products/services have this as an option you can enable as well. If you’re willing to be that aggressive and reject some legit mail from senders with email admins who haven’t implemented SPF, then it is a good solution 🙂
The second is that some products (as you say) will score any email that has those characteristics (a different return-path than from address) differently in some way, such as “bulk mail” or “likely spam”. That, combined with other message characteristics (poor IP reputation, content scanning) can often block most spam that tries that technique.
As always, nothing is foolproof and a layered approach is necessary to achieve the highest rate of detection with a low rate of false positives.
Nice article, how would this apply to exchange servers using smart hosts for outbound mail?
Thanks,
Hi Mike,
You would only add the public IPs of the outbound mail servers to your SPF record, and not the internal Exchange servers. If you add the ‘mx’ value to your SPF record like Paul did in the example, it should cover your smart hosts (if they are both inbound and outbound of course).
Paul, I’m dealing with a lot of clients right now who have invalid PTR records and are being rejected from Comcast and a few other providers – do you have an article on this? Is there any other way to record the PTR record besides contacting the IP range owner (arin)?
The Real Person!
The Real Person!
Nope, contact the network providers to register a PTR record.
One of your best articles yet. Clear and concise and doesn’t get lost in the trees. Nice job. Thanks!
Hi Paul,
Any idea why i get error ” One or more addresses has invalid syntax” on Microsoft Sender ID Framework SPF Record Wizard , if I enter two ip’s (one per line) on the “Enter any additional IP addresses (or ranges of addresses) you wish to add to your SPF record (one address or address range per line)” ?