Email verification using SPF - Sender Policy Framework

Email Authentication

Within the SMTP protocol, it's easy to forge most of the data in an email message. Therefore, methods are being devised to verify authenticity. One method is electronic signature, but obtaining and working with certificates is complex (placing high demands on the end user). So, methods that work on the server side are being addressed.

Generally, the term Email Authentication is used for methods of verifying the origin of email messages. The goal is to prevent the possibility of forging the identity of a message (Email Spoofing), thus Phishing and potentially SPAM. It verifies that the email comes from the address listed as the sender, at least at the domain level. To some extent, it can be said that the mail servers (Message Transfer Agents - MTA) involved in sending and transmitting the email are checked.

Because an email message typically contains multiple sender addresses, it's important which address is shown to the user and which is checked. We described the issue of addresses in the SMTP protocol in the article SMTP protocol and e-mail addresses.

For verifying the origin of messages, techniques such as SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail), or the extension DMARC (Domain-based Message Authentication, Reporting and Conformance) are predominantly used today. These methods are defined in Internet RFC standards (where we find the most detailed information). The SPF and DKIM techniques complement each other rather than replace each other. Each verifies something slightly different.

It should be noted that all these techniques only work between organizations that use them. Configuration must be done for sending messages. And checks must be performed when receiving messages (if any of the techniques were used).

SPF - Sender Policy Framework

One of the widely used methods is Sender Policy Framework (SPF) described in RFC 7208 (proposed standard). Terms such as SPF Verification or SPF Authentication are used. Its use is very simple, but it only works for parties that use it. We create DNS records for our domains and enable SPF protection on receipt (our mail gateway must support SPF).

SPF verifies the domain used in the Envelope Sender address (it's recommended to check both MAIL FROM and HELO), that the email message was sent by a correct (authorized) mail server for this domain. It can be said that SPF binds the sender's domain with IP addresses of servers that can send email. Verification can take place already when receiving the first SMTP commands (at the same time when checking the existence of the recipient's address), the entire message doesn't need to be received.

SPF Principle

SPF allows the domain owner to specify which mail servers are allowed (authorized) to send emails for a given domain. The setting is done using a text (TXT) DNS record with a specific format.

• When a mail server receives a message, it looks at the domain in the Envelope Sender, taking the domain from the HELO and MAIL FROM commands (some implementations use only MAIL FROM).
• For this domain, an SPF record (SPF Record) is looked up in DNS.
• If the record exists, the IP address of the mail server from which the message came is verified (Checking Authorization) according to it.
• Evaluation is defined by the check_host() function and possible results of the SPF test (Results of Evaluation) are given.
• The RFC also describes recommended behavior for various results (Result Handling), but we typically determine the real behavior ourselves.

SPF Check Results (Results of Evaluation)

Below are possible results of the SPF verification evaluation. A brief description and recommended behavior (manipulation) with the message according to the result.

• None - no SPF record exists, accept the message
• Neutral ? - the record specifies that validity cannot be determined, accept the message
• Pass + - the server is authorized to send the message, accept the message
• Fail - - the server is not authorized to send the message, reject the message
• Softfail ~ - the server is probably not authorized to send the message (but we don't want to reject), accept and mark the message
• Temperror - a temporary error occurred during SPF verification (usually DNS)
• Permerror - a permanent error occurred (invalid SPF record)

SPF Record in DNS

The SPF record format, and other requirements for DNS records, are specified in RFC 7208. A more readable description can be found in the article SPF Record Syntax. There are also various tools on the internet for checking SPF records SPF Record Check - Lookup SPF Records, or generating them SPF Record Generator.

The SPF record must be published as a DNS TXT (type 16) Resource Record (RR) directly on the given domain. Its format begins with specifying that it's an SPF record v=spf1. This is followed by a list of directives separated by spaces, which are evaluated in the given order. It may also contain modifiers.

A directive consists of a qualifier and a mechanism. The qualifier determines the behavior (evaluation) for the given mechanism, which is simply the sending address. If the address matches, the evaluation is determined according to the qualifier. The qualifier doesn't have to be written, then the default is Pass. The options are

• + Pass
• - Fail
• ~ Softfail
• ? Neutral

Mechanisms are for example ip4:, where the compared IP address is specified, a:, where an A record in DNS is specified for comparison. A special mechanism is include:, which uses the SPF record of the specified domain. And all, which always matches (specifies all addresses), is typically used at the end of the record.

A simple SPF record, for FQDN company.com (on this domain the TXT record is created), might look like this:

"v=spf1 ip4:1.2.3.4 mx -all"

If mail comes from IP address 1.2.3.4 or an IP address that is assigned to the MX record, then the result is Pass. For all other cases, it's Fail.

SPF HELO

Typically, during SPF check, the domain in MAIL FROM is verified (i.e., the domain part of the sender's address - Envelope Sender). For example, for the address smith@company.com, the domain is company.com. RFC 7208 recommends that the domain in the HELO/EHLO command should also be verified. In HELO/EHLO, it's recommended to specify the server name (hostname - FQDN), which is for example mail.company.com (I've seen an email address used as well).

I thought that only the domain would be taken from the name (although it's not described anywhere), but apparently not. Various testing services for SPF checking verify not only MAIL FROM (mfrom), but also HELO (helo), and return that the SPF record for the domain mail.company.com was not found. So I tried adding a TXT DNS record for the subdomain as well, and suddenly the tests come out as Pass. Also, the Cisco Email Security gateway allows optionally enabling the HELO Test when checking SPF/SIDF. It then adds a second (third, if we also have PRA) item Received-SPF to the message headers, where the result of the HELO domain test is.

I tried searching for many well-known domains, but I didn't find anyone who had an SPF record for the HELO domain. So it's probably not used in practice.

SPF Problem

There's one problem with SPF technology, and that's with forwarding emails to another email address. If standard forwarding is used, as offered by a common email client under the Forward function, there's no problem. In that case, a new message is created where the sender is the given user (it's visible that I forwarded the message). The behavior is the same as with reply, only the recipient's address is not filled in and (something like) FW: is inserted into the subject.

The problem is caused by an operation that's referred to as email redirection (Redirect). Redirection cannot be done manually in common email clients (such as MS Outlook), but we can create an automated rule (Rule) that supports this function. During redirection, the message doesn't change and looks as if it came from the original sender. If the recipient replies, they reply to the original sender and not to the one who performed the redirection. The problem is that the message is sent by the mail server of the one performing the redirection, and it uses the address of the original sender. So it's practically a forgery of the sender. And if the original sender and the forwarder are on different domains, the recipient's server, when using SPF, detects the message as forged.

Practical impact (which I used to encounter very often, but the situation may be improving) is described by the following example. I send an email (me@company.com) to a recipient in another company (recipient@somewhere.com). This person has a rule set to forward all messages to their address recipient@seznam.com (and in the worst case, not to save a copy). Seznam receives an email where the sender is from the domain company.com, but it was sent by the mail server of the domain somewhere.com. For our domain company.com, we have an SPF record where only our mail server is authorized. Seznam checks SPF and considers this a spoofed email, so it rejects it. Interestingly, the error does not reach the person who set up the forwarding. But the NDR is sent to us (it is the only address in the header).

Below is an example of the error that is listed in the NDR when the message is rejected. It shows that although I wrote to an address in the domain somewhere.com, the response is from the server seznam.com for a different address.

recipient@seznam.com
mx1.seznam.com #<mx1.seznam.com #5.7.1 smtp; 550 5.7.1 Sender Policy Framework of `company.com' domain denied your IP address.> #SMTP#

In most cases, when we create an automatic rule for forwarding incoming messages, we can choose one of three options. Either classic forwarding (which we should prefer) or forwarding as an attachment, the last option is redirecting (leave the message unchanged).

Note: DKIM, which we will describe later, does not have a problem with redirection because it works on a completely different principle.

Sender ID, Sender ID Framework (SIDF)

Sender ID, described in RFC 4406, is based on SPF and is quite similar to it, although it has certain extensions. It is probably not used much today. Microsoft implemented Sender ID as the Sender ID Framework (SIDF). It allows checking the domain from the Envelope Sender, keyword mfrom. But also the Purported Responsible Address (PRA), which is an address determined by various rules, also considering the Header Sender.

The text DNS record for SPF starts with v=spf1. For Sender ID it is some variant like spf2.0/mfrom, spf2.0/pra,mfrom, etc.

Practical implementation of SPF

Enabling SPF

If we want recipients to be able to perform SPF checks for our domain, we just need to create a DNS record for the domain as described above.

For example, for sending from addresses like storm@company.com, we create a TXT record on the domain company.com. The record value can be:

"v=spf1 ip4:10.20.30.40 -all"

Alternatively, we can create the same record for the HELO name of the server, for example mail.company.com.

To view the record we can simply use a command line on Windows.

nslookup

> set type=txt
> company.com
Server:  ns.company.com
Address:  192.168.0.10

Non-authoritative answer:
company.com     text =
    "v=spf1 ip4:10.20.30.40 -all"

> mail.company.com
mail.company.com     text =
    "v=spf1 ip4:10.20.30.40 -all"

SPF verification

We need a mail gateway (server) that supports SPF. Enabling verification is usually simple. We need to decide how to handle messages that fail the check (e.g., mark them in the subject). An example of a gateway with SPF support is Cisco Email Security. I briefly described enabling SPF verification in the articleCisco Email Security - AntiSpam solution configuration.

SPF is also directly supported by Exchange Server, but we need to install AntiSpam agents for the Transport service Enable antispam functionality on Mailbox servers.

Online SPF check

On the internet, we can find many tools that will find the SPF DNS record for a given domain and verify its syntax. They may also compare the given sending IP address to see if it matches the record. One example is SPF Record Check - Lookup SPF Records. Other services work by sending a message to their email address and returning a report.