Del via

Sending mail with invalid From: addresses to Office 365

  • Artikel

One of the changes to go into Office 365 in the past year is an antispam rule that rejects on messages with an invalid From: address. When this occurs, the message is rejected with:

550 5.7.512 Access denied, message must be RFC 5322 section 3.6.2 compliant and include a valid From address

If you look up RFC 5322 section 3.6, it says that each message must have one and only one From: address:

    +----------------+--------+------------+----------------------------+
   | Field          | Min    | Max number | Notes                      |
   |                | number |            |                            |
   +----------------+--------+------------+----------------------------+
   | from           | 1      | 1          | See sender and 3.6.2       |
   +----------------+--------+------------+----------------------------+

The structure of a From address is then described in section 3.6.2.

For many years, Exchange server allowed senders and recipients to send messages with malformatted From: addresses. That is, something like this was permitted:

From: <blah>

From: “Example Sender”

Even though this is against the RFC 5322 (published in 2008), and RFC 2822 (published 2001) before it, there are still lots of mail servers that send malformatted email in this way. However, if you try to send to other services, it doesn’t work. For example, sending a message that way to Hotmail/Outlook.com results in the message bouncing; sending it to Gmail similarly results in the message bouncing. Indeed, Gmail even forces you to put angle brackets around the email address in the SMTP MAIL FROM. For example, the first line below is rejected by Gmail, the second is accepted:

MAIL FROM: not@acceptable.com

MAIL FROM: <okay@acceptable.com>

Exchange accepts them both. So does Office 365.

Exchange has more relaxed enforcements because in the corporate environment, many applications are running on older or buggy platforms but send wanted email; or, people frequently write scripts to transmit email but do not configure them to send RFC-compliant mail. Large services like Gmail and Outlook.com are more picky about protecting their users, but in a corporate environment that sends messages privately, it is not as strictly enforced if it’s just you sending to yourself.

Given all that, late in 2015, we started seeing massive outbound spam attacks from malicious spammers who signed up for the service. They would send spam with an empty MAIL FROM and an empty From: address:

MAIL FROM: <>
From: <>

We measured the proportion of spam using this pattern; 98-99% of it was being marked as spam (and thus delivered out of our high risk delivery pool), and its total volume was well into the millions per day.

This had numerous drawbacks:

  1. The amount of spam being generated was taking up bandwidth from legitimate email
  2. We were still relaying junk to the Internet and the double null-sender was making it difficult to track down the spammers
  3. The misclassified spam was high enough that it was impacting the quality of our low risk outbound delivery pools. This means that customers were impacted because spammers would get our IPs listed on IP blocklists, affecting our entire customer base

Combining the fact that RFC 2822 was published in 2001 and specified the proper format of an email address, and that there was so much outbound spam, and the workaround was for script-owners of system-generated email to fix their scripts (rather than having to continually chase down spammers), Office 365 decided to crack down on these types of messages:

If you send email to Office 365 with a null SMTP MAIL FROM <>, then the From: address must contain <email@address.TopLevelDomain> including the angle brackets.

From time to time, we get senders telling us that we are mistakenly blocking the following message with the previously mentioned error response:

MAIL FROM: <>
From: sender@contoso.com

It is not a mistake, we require a From: address to have angle brackets if the SMTP MAIL FROM is <>. Just as different email services have different requirements – Gmail requires angle brackets around the SMTP MAIL FROM, Hotmail requires a valid From: address always – Office 365 requires that email addresses get formatted in a certain way if the MAIL FROM is <>.

Because Office 365 deals with legacy mail servers sending traffic to the service, there are certain RFC requirements that the service is not in a position to enforce without potentially causing customer disruption. At the same time, we are trying to keep the null sender rule simple; it is too confusing to have a complicated if-then-elseIf-elseIf-else logic tree for sending with a null sender [1]. And Office 365 is still much more relaxed than almost any other mail platform even with this restriction in place.

This is the balance that has been struck in our attempts to support legacy mail servers while running a cloud-based service, yet keeping spammers off the network.

[1] There are lots of other different ways that spammers try to shove email through the system using visual display tricks that are rejected by some recipients, but allowed by others. Yet a complicated AND-OR-NOT would be too difficult to explain to anyone who asked what the logic is, and it wouldn’t be long before even the engineering team couldn’t maintain it. Simplicity is our goal here, and we achieved it.

For example, when someone says their email is getting rejected, it’s a simple explanation to say “Add angle brackets around the From: address.”

Comments

  • Anonymous
    August 29, 2016
    Can I find out if any of the emails I receive will be (is?) affected by this change?
    • Anonymous
      August 29, 2016
      Just send a test message to your account that's protected by Office 365. It will bounce if it's affected (this change went out 8 months ago).
  • Anonymous
    August 31, 2016
    If database journaling is enabled on Onprem exchange server as per https://technet.microsoft.com/en-in/library/bb123817(v=exchg.141).aspx and if the Journal Report Mailbox is on o365 then it breaks Journaling.
  • Anonymous
    September 27, 2016
    The problem with this is that angle brackets around the From: address is not an RFC specification, so basically O365 is rejecting a lot of valid mail as well.The idea about subscribing and building a product to RFC specifications is so heterogeneous mail systems can talk to each other, or least it used to be that way.Dealing with spam is one thing, rejecting valid mail is quite another.
    • Anonymous
      September 27, 2016
      Hi, Alan,I understand what you're saying but I explain the rationale for this behavior in the second part of the blog post.
      • Anonymous
        October 06, 2016
        Hi Terry, Good to hear from you.I noticed that we had been receiving a lot of mail with RFC2822 addresses which did not have a smiley emoji in the friendly phrase part, can you imagine?!!!So anyway we decided to go ahead and reject those type of mail because they do not commit to our idea of what a fun address should look like & we are primarily concerned with having fun when mailing.Hopefully that is ok and doesn't cause any problems! I was going to make a Request For Comment but...
        • Anonymous
          October 06, 2016
          Per RFC3464 (Delivery Status Notifications): The envelope sender address of the DSN SHOULD be chosen to ensure that no delivery status reports will be issued in response to the DSN itself, and MUST be chosen so that DSNs will not generate mail loops. Whenever an SMTP transaction is used to send a DSN, the MAIL FROM command MUST use a NULL return address, i.e., "MAIL FROM:".Nowhere in RFC 5322 does it insist that angle brackets be used in the From header. Only when a display/friendly name is being used are they required. By applying this ad hoc implementation of RFC 5322, you are denying your own users of their right to know when their sent messages have not been delivered.
          • Anonymous
            October 08, 2016
            @Tony: The messages are rejected in SMTP, not silently dropped, so the sender knows whether or not the message was delivered.
          • Anonymous
            October 10, 2016
            @Terry: You are missing the point. I am trying to send a non-delivery report to your user. You are rejecting it back to my server. Your user has no idea that his original mail never reached the intended recipient.
            • Anonymous
              October 10, 2016
              @Tony:I understand the point you are making - the filtering mechanism we've implemented affects your particular scenario. However, we have dozens of different scenarios we need to protect against, or allow. As I say in the blog post, we are lenient in some areas but that means we have to be more restrictive in others. Other mail services have their own requirements which are stricter than the RFCs say. For example, there are mail filters that reject on failed DKIM signatures even though RFC 6376 says not to. In a case like that, a service can either make adjustments to allow email through, or not make the change and live with the false positives.In this case, we've decided to live with the false positives because a bouncing mail server can make the change on their side easier than we can make the change on ours; it's not a simple matter of saying "Now we will relax this requirement" because there are other spam implications for allowing it. By contrast, adding angle brackets on a bounce message is a small change on the sender's side with few side effects.
          • Anonymous
            October 11, 2016
            @Terry:Your customers, your call. If I were a business user on O365, I would be very concerned about potential business loss, due to my unawareness of deliver failures. But you know better, I suppose.
        • Anonymous
          October 25, 2016
          We're dealing this very issue right now as we front-end our Office 365 with a 3rd party filtering service. As it stands the dsn's generated from this service are no longer getting back to our Office 365 clients, so they have no way of knowing about the failures. Really dislike Microsoft's current approach to this. They should provide some type of exception to allow for this scenario through the use of something like a mail flow connector. As always, you break things and ask other work around the problems you create. Nice.
          • Anonymous
            October 26, 2016
            Connectors still allow customers to relay malformatted email out the service, potentially degrading the IP reputation of Office 365 and affecting the mail flow of our entire customer base. The way connectors work makes it difficult to differentiate between inbound and outbound email.