Terry Zink: Security Talk

The Terry Zink Security Talk blog comes to an end

tzink — Wed, 20 Jun 2018 11:48:58 GMT

Please note: The Terry Zink Security Talk blog is being deprecated in March 2019 in order to focus our attention (and yours) to the vast amounts of information we already have on the support.office.com and the docs.microsoft.com sites.

Some of you may have noticed that the amount that I post on this blog has lessened over the years, and especially this year. This is not for lack of ideas, but simply lack of time to write good, in-depth articles. However, irrespective of that, this is my last blog post on this blog. After spending 14 years at Microsoft (which includes my time with Frontbridge) in the antispam business - fighting the spam, malware, and phishing, and talking about reputation and everything related to email - I’ve decided change is in the air and I’ve made the difficult decision to move on from Microsoft to the next chapter in my professional life. I have thoroughly enjoyed my time at Microsoft, having been given the opportunity to work on world class projects and features. It is satisfying to see so many of the things we’ve worked on together getting used by millions of people around the world. Yet the key piece in that sentence is the word “together” because beyond working on great software, the most rewarding piece is working with so many great, hard-working, and talented people. There really is no better reward than that. I know we all get heads-down with developing the things we need to get done, but when I look around and see the things that everyone else (other than me) has gotten done, it makes me proud to say “Hey, we’ve all done something great here!” I have also enjoyed writing this blog. It has given me the time to practice my writing skills, helped me organize my thoughts out-loud, given me a log to refer back to, and made it possible for me to refer back to features I worked on years ago. If any of you reading this found it valuable, I am pleased I could be of service. You can still find me at various places around the web:

On Twitter
On LinkedIn
On Facebook (I know Facebook sometimes takes flack, but I really enjoy seeing the personal stories people have like posting vacation pics, or life milestones; this is something that I would never see on Twitter or LinkedIn. It helps get us out of a professional context and instead into a friendship context which I hope transcends work… and politics)
On Quora

I haven't decided whether or not I will start another blog. Maybe I will, maybe I won't. It depends on where I end up, and those other platforms have a bigger audience anyway. I thought it would be appropriate to share the oath we here in the antispam team who have been here a little longer had to swear by when we first joined. It’s quite techno-geeky, but they make us do it in a secret ceremony. The part in bold is my own addition: Spam fighters, and now my watch begins, It shall not end until my metaphorical death (that is, when I leave the company). I shall send no unwanted email, hold no typo-squatted domains, populate no spamtraps. I shall wear no black hats and win no glory. I shall learn how to send email via Telnet and Powershell. I am rod that reels in the lures of phish. I am the spreader of the net. I am the shield that guards the realms of email servers. I pledge my honor to the Spam Fight, for this night and all the nights to come. And now my watch is over. Thank-you for reading this blog.

The unauthenticated sender '?' comes to Outlook

tzink — Wed, 20 Jun 2018 11:20:57 GMT

Update: This blog post is being deprecated and information is being moved to support.office.com: Identify suspicious messages in Outlook.com and Outlook on the web

About a year ago, in Office 365, we released the feature that - similar to Gmail - Outlook Web Access stamps a '?' in the sender photo when the message is not authenticated with either SPF or DKIM. Well, I was pleasantly surprised to see this morning it's also in Outlook (for Windows, not sure about other clients): This is great to see, we're now managed to unify the behavior between both Outlook and Outlook Web Access, at least in Office 365. Woot!

11/1/2018 - Clarification: this feature is actually still in Preview.

Chasing the (very) long tail of unauthenticated domains

tzink — Fri, 08 Jun 2018 11:36:04 GMT

One of the requests that frequently crosses my desk (computer screen) is a vulnerability claim that a certain domain that is owned by Microsoft is prone to spoofing because it does not have email authentication records - neither SPF, DKIM, nor DMARC. Because this can be used to spoof, it is a vulnerability. Microsoft Corporation owns many domains, from microsoft.com to skype.com to xbox.com to live.com. Some of these domains have strong auth records (DMARC reject or quarantine) and some have nothing published at all (e.g., microsoftstore.com). We used to go and chase down the most commonly spoofed domains and published SPF, DKIM, and DMARC records for them; monitor the domains; and then move to a DMARC quarantine/reject record. That would lock down the domains. However, that is no longer our strategy because it doesn't scale.

Within Office 365 Advanced Threat Protection [1], we stopped relying upon SPF and DMARC records exclusively as a means to stop spoofing. For our ATP customers, they have antispoof protection available to them, see http://aka.ms/LearnAboutSpoofing. We also added some additional protection in the event the From domain has "microsoft" anywhere in it to ensure it gets marked as spam/phish (I call this "substring spoofing", or sometimes "brand impersonation" which by itself has multiple categories which includes putting the brand name into the Display Name, rather than the domain). .
The simple addition of a hard fail to the SPF record to any of Microsoft's domains does not solve the spoofing problem for all variants of Microsoft properties – outlook.com, Hotmail.com, Hotmail.ca, msn.fr, etc., so we’d have to protect those as well. That is, we could protect one domain, but unless we went and protected everything we own, another claim could come up that yet another domain isn't protected. The locking down of all these domains is time consuming. .
Furthermore, and most importantly due to the way attacks are sent these days, it doesn’t solve the spoofing problem of variants that Microsoft does not own like msn.com.com, my-msn.com, etc. Those are also damaging to our brand and can be used to spoof, but since we don't own them, we can't update those DNS records. We'd have to buy them, or figure out their current owner and shut them down. But we can't stop any and all variants, it's a nearly infinite long tail, especially as Email Address Internationalization ramps up.

Our approach to solving the problem of spoofing is to on-board customers to ATP/E5 [2] where they can get antispoofing protection regardless of the sending domain; if/when we expand Antispoofing to the rest of the customer base, those variants of the Microsoft brand will also be covered. We are also looking to expand the way we protect Top Brands like MSN by checking to see if the domain name contains a Top Brand, or a variant of it, and using other heuristics to detect it as a spoof. In the diagram below, the green part represents authenticated email (SPF, DKIM, DMARC), the blue represents unauthenticated email that is not marked as spam, and the red is everything else. Thus, we won't need to publish any auth records because we are flipping the model on its head - rather than relying upon a domain to publish auth records and defaulting back to assuming something is good, we instead default back to assuming something is bad. We do this because we try super hard to implicitly authenticate a message. Despite our best efforts, this can still result in false positives; but all false positives can be resolved by either publishing auth records by the domain owner, or our customer creating their own local overrides. Publishing an SPF hard fail has its drawbacks – especially when email is forwarded. As I say above, we are looking to move protection away from the SPF/MailFrom domain to the From domain, and then use better heuristics built upon the principles of DMARC rely upon that instead. We think the industry is better served by treating unauthenticated email as suspicious regardless of whether or not it publishes auth records, and then using signals to rescue it - such as authentication or sender reputation - in the event the message isn’t malicious. That’s how we are planning to scale up protection of our brands; we still publish auth records opportunistically, but not necessarily for all of our domains [3]. And the ones that do have SPF records may not necessarily have SPF Hard Fail. Instead, we encourage others to add their own intelligence on top of what is published to repudiate the ones that don't auth. That's what we do. It's the only way to make this scale.

[1] And eventually the rest of the Office 365 customer base, and eventually our consumer email platform [2] Same as [1] [3] We will do this to improve delivery, such as a new domain is being spun up and wants better delivery because we are treating unauthenticated email as spam/phish/spoof. Adding SPF, DKIM, and DMARC will help in self-identifying that the domain is authorized to send email. It's kind of (?) like a Spamhaus PBL for domains

If your MX record doesn't point to Office 365, how do you disable spam filtering in Office 365?

tzink — Tue, 05 Jun 2018 16:55:03 GMT

One of the questions that has come up recently, especially as a lot of customers migrate over from an existing spam filtering solution to Office 365, is how to force Office 365 to rely upon the spam/non-spam verdict of the service that's sitting in front of Exchange Online and not double-filter. Office 365 already has a way to do this via connectors if the email server in front is an on-premise Exchange server. It does this by using TLS certs to "promote" the pre-existing properties of a message stamped by the Exchange server so they are re-used by Office 365. But what if your server in front is not Exchange? Then what? There are a few ways to do this, but the key thing is that there is no simple way to disable spam filtering in Office 365. The option for "Do not filter spam" does not exist at an organizational level. Instead, you have to do a few tricks depending on the configuration.

You want the service in front of Office 365 to get rid of spam, such as sending it to a spam quarantine. . In this case, the spam is taken care of somewhere else, and all email going to Office 365 is non-spam (according to the upstream filter). Therefore, you can create a set of IP Allow List entries corresponding to the relay IPs into the service. This will set SCL -1 and send all email into your users' inbox, bypassing spam scanning, and stamp IPV:CAL and SFV:SKN in the X-Forefront-Antispam-Report header.Alternatively (even preferably) you can create an Exchange Transport Rule (ETR) for those connecting IPs that (a) sets the SCL to -1, and (b) sets an x-header: .
X-Relay-IP-for-service: Allow email from
. ... as it is already filtered for spam. . This way, when someone in your organization gets a message that is spam, and you decide to escalate to Office 365 for missing spam even though we didn't filter it because you said not to, we'll be able to quickly look at the headers and see that's why the message was delivered to the user's inbox..

You want to use Office 365 to manage both spam (in the Junk Email folder) and non-spam. . In this case, you want non-spam to end up in users' inboxes, and spam to go to the Junk Email folder the way it would for something like an Outlook.com or Gmail account.This requires you to configure the upstream mail service to configure two headers: one that is stamped when the service marks a message a spam, and a second one that is stamped when the service marks a message as non-spam. For example, Office 365 uses the X-Forefront-Antispam-Report header to stamp SCL -1 or 1 for non-spam, and SCL 5 or 6 when it is spam, and SCL 9 when it is high confidence spam. . Then, in the Office 365 platform, you would write an ETR that sets SCL -1 when it sees the stamp for non-spam, and SCL 6 (or 9, depending on what you want the action to be) when the message has the spam stamp. . You can extend this for safe and blocked senders, safe and blocked domains, etc. Whatever stamp the upfront service stamps in each case should be consumed downstream.

There is another method such as disabling junk mail filtering on a per-mailbox basis which will force messages to go through filtering but still land in the inbox (this is what I do on my own personal email, although my MX points to Office 365). However, this is an advanced scenario and causes customers a lot of trouble. Putting additional services in front of Office 365 is what we call "complex routing"; there are several scenarios for complex routing, and the MX pointing to another service is the simplest case. Finally, if you do use complex routing like this, there are some things to be aware of:

You lose a big chunk of the benefits of pointing your MX to Office 365 as the first hop in the path. Make sure you read the first article in the Related Posts section at the bottom of this article .
Outlook Web Access (OWA) has some client-side integration that lights up when hooked together with Office 365, and some of those features depend on Office 365 being the first hop in the path (e.g., showing a '?' in the sender photo). When these behaviors come to Outlook desktop client, it will have the same requirement. That's another feature loss for complex routing. .
Additional services in front of Office 365 results in additional layers of management complexity .
Advanced Threat Protection works much better when Office 365 is the first hop in the path, we usually recommend putting other services behind us, and then looping the email back in via a connector. You'll have to use ETRs again and they are more complex, but that way you get the full protection .
If you get missed spam or false positives because of this configuration, it is challenging for us to do much with them because parts of our filter have been disabled

Related posts

A way to (sort of) approximate DMARC aggregate reports in Office 365

tzink — Mon, 21 May 2018 15:52:18 GMT

One of the most common questions people ask me is "How do you get Office 365 to send out DMARC aggregate and forensic reports?" This is followed by "When is Office 365 going to send out DMARC aggregate and forensic reports?" Office 365 doesn't send out DMARC reports, nor is it on our public roadmap. However, there's a hacky way you can kind of approximate aggregate reports. This is how I would do it. Many people don't know about this, but there's basic reporting for spoofed messages flowing into your organization. This is so that you can see who is sending you spoofed messages and create allow/block lists as required (that is, for a message that doesn't authenticate and is marked as spam by our service because of it, you can allow that domain-pair [1] to go through regular filtering; conversely, if Office 365 thinks a domain-pair is legitimate, you can override it and say it's not). Anyhow, the Get-PhishFilterPolicy gives you an overview of everything sending into the service that doesn't authenticate, so you can actually extract this information and send it off to a DMARC reporting address. You can do this via the following: 1. Connect to Exchange Online using Powershell 2. Run the following script, updating this value for your own domain:

$domain = "<yourDomain>"
# Get today's date
$date = get-Date -Format "yyyy-MM-dd"
# Create a variable for exporting the results
$filename = "AuthResults-" + $date + ".csv"
# Get the required data and export it to the filename as a CSV file
$spoof = Get-PhishFilterPolicy -SpoofAllowBlockList -Detailed | Select-Object SpoofedUser,Sender,NumberOfMessages,DomainPairsInCategory,LastSeen,SpoofType
$spoof | where { $_.SpoofType -eq "Internal" } | Export-Csv $filename# Lookup the DMARC rua address from DNS
$dmarcDomain = "_dmarc." + $domain
$dmarc = Resolve-DnsName -Type TXT -Name $dmarcDomain# Convert the Object to a string and strip off the irrelevant parts
$dmarcString = $dmarc.Strings
$dmarcString = $dmarcString -replace ".*rua=mailto:",""
$dmarcString = $dmarcString -replace ";.*",""

# Store this result as the recipient to whom you will send the "DMARC" aggregate report
$to = $dmarcString

# Lookup the DMARC rua domain's MX record
$dmarcRuaDomain = $dmarcString -replace ".*\@","" | Out-String
$dmarcRuaDomain = $dmarcRuaDomain -replace "\s",""
$mxRua = Resolve-DnsName -type MX -Name $dmarcRuaDomain
$smtpServer = $mxRua[0].Name | Out-String
$smtpServer -replace "\s",""

# Construct the variables for sending an email
$from = "DMARC report for " + $domain + " "
$subject = "DMARC report for " + $domain + " on " + $date
$body = gc $filename

# Strip off the localpart in the email address. You may need to adjust this regex if you have localparts that contain other characters
$regex = "[a-z0-9\-\.\+]+@"
$body = $body -replace $regex,""
$body = [string]$body

# Send the message with the Get-PhishFilterPolicy's contents
Send-MailMessage -From $from -To $to -Subject $subject -Body $body -smtpserver $smtpServer -port 25 -usessl

What this does is get all the spoofed email sending as you into your organization, and then send a copy to the address in the DMARC's DNS record. There are some things to be aware of: a) Whoever you send this DMARC "aggregate" report to needs to understand that you are not sending a DMARC aggregate report which is in xml format. Instead, it is the Office 365 phish filter policy for antispoofing reporting and the fields do *not* map 1:1 for a regular DMARC aggregate report. The report has to be parsed and the necessary data extrapolated. And this is the best I can do at the moment. Please don't come back to me saying "But this is not a DMARC report! When are we getting DMARC reports?" b) This script sends your organization's data to a third party outside of the Office 365 compliance boundary. It includes what we classify as End User Identifiable Information because it contains an email address in clear text. When you run this script, be aware of data that you are sending. The script above strips off the localpart of the email address, although there may be other email addresses it doesn't match. So, beware. c) When you run this script, make sure you run it from a machine with an IP address that is in your organization's SPF record. If it isn't, you will need to do some customization. d) I've omitted several fields from the Get-PhishFilterPolicy when sending the "aggregate" report, but IMHO they aren't required e) This cmdlet gets all of your data, whereas you might want to only pull data every 24 hours. If it were me, I would automate it and add the following line:

$yesterday = (Get-Date).AddHours(-24)

Then replace the corresponding line above with:

$spoof | where { $_.SpoofType -eq "Internal" } | where { $_.LastSeen -gt $yesterday } | Export-Csv $filename

f) I'm not a Powershell expert, there are many of you out there who are much better than me and can better optimize the script above. This script is verbose so you can see what's going on. So, this script isn't perfect, and it excludes some key information. But it does give some useful data.

[1] A domain-pair is a combination of the From: + the sending infrastructure. If the From domain is internal to your organization (one of your org's accepted domains), the SpoofedUser is the exact email address; if the From domain is external, then the SpoofedUser is only the From: domain. The sending infrastructure (Sender) is the organizational domain of the PTR record of the sending IP. If the sending IP has no PTR record, then the Sender is the /24 range of the sending IP, e.g., 1.2.3.4 would be 1.2.3.0/24.

How to get images to load in Outlook.com, Office 365, and Outlook email clients

tzink — Fri, 23 Feb 2018 12:12:10 GMT

People sometimes ask me "How do I, as a sender into Office 365, get images to load by default? Every time I send, the images are blocked." I've decided to finally answer that question so I don't need to keep typing my response. 1. Images in Outlook.com load by default if you're a good sender, but as a user you can modify this behavior By default, in Outlook.com, if you're a good sender (determined by a bunch of back-end algorithms) and the message makes it to the inbox, images will be displayed. Here's an example from Groupon, I did nothing to ensure that these images are loaded:

However, if I wanted to, I could flip a setting under Options (gear icon in top right) > Mail > Junk email > Filters and reporting. As you can see, the default behavior is to Show attachments, pictures, and links for senders with good reputation. If I flipped it to the other setting, all images would be suppressed for senders not in my Safe senders list.

(In the image above, I have removed some other sections that are not relevant to this discussion). 2. In Office 365, images do not load by default even if you're a good sender By contrast, in Office 365, if you're viewing your email in Outlook Web Access (OWA), also known as Outlook on the Web, images do not load by default. As a user, you can choose to either (a) show the images within the message for the current session (if you navigate away from the message and come back later, images will be suppressed); or, (b) you can choose to always show content from that sender at which point it adds to your Safe senders list. You can also manually populate your Safe senders list. For example: To populate your Safe senders list manually (and thereby bypass spam filtering as well), select Options (gear icon in top right > Mail > Accounts > Block or allow, and then add to your Safe Senders list. You can also Trust email from your contacts. If you do, it will skip spam filtering and I think load images as well.

You may be wondering "Okay, so how do I get all legitimate senders to load images by default, the way I can in Outlook.com?" The answer is - You can't. There is no equivalent option in Office 365.

That's one of the differences between the Consumer and Enterprise versions of the web interface for checking your email. BTW, images in the Junk folder don't load at all, neither in Consumer nor in Enterprise. 3. In Outlook desktop client, images also do not load by default even if you're a good sender but there is a little more configurability Outlook desktop client, when working together with Office 365 (I'm not sure how it behaves when used standalone) behaves similar to OWA for Office 365. However, there are some key differences. Just like OWA, external senders do not load images by default unless they are in your Safe senders list. However, unlike OWA: a) If you choose to show images on that particular session, your selection will be remembered if you navigate away and then come back b) There is no one-click option to add the sender to your safe senders list from the infobar. Instead, you can right-click on a message and add them to your Safe senders list, or you can do it manually by right clicking and selecting Junk Email Options > Safe senders.

When it comes to downloading and showing images, these settings are controlled via the Outlook Trust Center. You can access this in the newest version of Outlook via File > Options > Trust Center > Trust Center settings > Automatic Download. Outlook actually suppresses all images for downloading, but by default enables the option to download for safe senders, and the trusted zones (published by your organization or perhaps manually populated, I don't recommend playing around with the Trusted Zone).

So you see that Outlook lets you suppress images from everyone except your Safe senders list, whereas OWA for Office 365 shows you images from your Safe senders list without giving you a way to change it. However, just like OWA for Office 365, there's no way for a sender to get images to load by default.

If you're a sender sending email to Office 365, you may be thinking "Are you telling me that I can't have images load by default in the absence of being on a user's Safe sender list?" Correct. You have to be on the user's Safe sender list. "But what if I'm on a good sender's list? Like the green shield list?" That doesn't matter. "How can I get myself added to all your users' safe senders list so that images will show in Office 365 users' mailboxes?" You can't, that is not a service we provide.

That's a quick overview of how loading images works in our various email interfaces. I know that I haven't included Outlook mobile, nor the native Windows mail clients. Perhaps one day I will add those to this list. But for now, this is how images load in the most used email clients that I regularly interact with.

If you use Office 365 but your MX record doesn't point to Office, you may want to close down your security settings

tzink — Thu, 28 Dec 2017 16:44:27 GMT

Even though it's not a recommend configuration for our customers (in terms of spam filtering), some customers of Office 365 route their email through a competing spam filtering service in the cloud, or through an on-prem server. That is, the mail flow looks like this: I've written previously about the problems this can cause, see Hooking up additional spam filters in front of or behind Office 365. However, if you must do it, you may want to ensure that you force all email to go through the 3rd party server. If you hook up a 3rd party server, email can be delivered to your organization in Office 365 through that server or by connecting directly to EOP (Exchange Online Protection). This is not good because our spam filters don't understand that email was sent directly to the service and not through a gateway; if your MX record does not point to Office 365 or EOP, some spam filtering checks are suppressed automatically to avoid false positives. Therefore, to get the fullest protection possible, I recommend relying upon the 3rd party service, and then maybe or maybe not doing double-filtering in EOP (accepting the fact that there will be false positives and false negatives). But, don't just rely on EOP. So, to force email through your on-prem server, you will need to install a TLS cert in your on-prem server, and then always ensure that it's used when connecting from the server to Office 365. Then, create a partner connecter using a seldom used attribute which isn’t exposed via the mainstream UX but only through cmdlet. It’s called AssociatedAcceptedDomains. To do this with TLS-cert-based connectors using cmdlets: New-InboundConnector –Name "OnlyAcceptEmailFrom" -ConnectorType Partner -SenderDomains * -RestrictDomainsToCertificate $true -TlsSenderCertificateName -AssociatedAcceptedDomains For more information on TLS connectors and how to set up a partner connector, see Set up connectors for secure mail flow with a partner organization. You may have to tweak this a bit to get it right, so you may want to experiment with some smaller domains before enabling it for every domain in your organization. What this does is reject messages that don't come over the TLS cert; so long as your on-prem server is correctly configured, any email that tries to connect directly to Office 365/EOP should be rejected. You would want to use this when you are connecting through an on-premise mail server and you can control the certificate. However, if you are connecting through a shared service and cannot specify the TLS-cert, then this would probably not be appropriate without some modifications to the connector. Hopefully, you find this useful.

When creating support tickets about spam, be sure to include message headers

tzink — Thu, 30 Nov 2017 08:00:10 GMT

When users get spam and phishing messages in the inbox, we ask users to submit them back to us, using the instructions here: Submit spam, non-spam, and phishing scam messages to Microsoft for analysis. I explain why this is important in Why does spam and phishing get through Office 365? And what can be done about it? But sometimes, we get customer support tickets, or random questions, from customers or other people within Microsoft. Unfortunately, many times these submissions are not that useful. The following are examples that don't help us:

Forwarding the original spam message (i.e., hitting "Forward" in your email client of the spam message to someone else who asks for a sample)
Screenshots of a spam message
Partial headers, and not the full set, in a text file of a message
Partial headers, and not the full set, from the message header analyzer
A verbal description with no example (e.g., "I'm having trouble delivering to Microsoft" or "I'm getting lots of spam messages")
Message traces in a csv or Excel file without the originals attached
Screenshots showing deliverability going down over time

None of these are helpful. Occasionally for the first one we can do a message trace and find a candidate message, and then go back to the logs and figure out what happened. That works, but it is time-consuming and doesn't scale beyond one or two messages. All of those can help, but without the original message headers, it limits our ability to investigate. So, for everyone who has ever asked for help with deliverability to Microsoft (or too much deliverability in the way of spam), here's a quick primer. Getting message headers in Outlook Getting message headers in Outlook Web Access (OWA, also referred to as Outlook on the web) BTW, this also works in Outlook.com's web interface. Attaching messages using drag-and-drop in Outlook (or attach item) sometimes drops important headers Sometimes people will do the correct thing and forward a message as an attachment. That is, in Outlook, people will compose a new message and then drag-and-drop the spam message as an attachment. Or, they'll use the "Attach item" feature. This is supposed to preserve the entire contents of the message, but sometimes it doesn't. Sometimes, for reasons I don't understand, when dragging-and-dropping a message from the Outlook list view into a new message as an attachment, a bunch of headers get stripped from the original message. This appears to be some header firewall issue between certain versions of Outlook and Exchange. You'll never notice when it happens, it's done secretly with no notification to the user. So, to be safe, you may want to copy/paste the headers of a message into the message body. That's what I do, and it's what some versions of our spam reporting plug-in used to do - to avoid this very problem. Basic troubleshooting Sometimes, you can troubleshoot simple missed spam, or false positive, problems yourself. The key header to look for is the X-Forefront-Antispam-Report header, which is documented at Antispam message headers. If it's a user configuration, you can either adjust it yourself or contact an administrator. Example 1 – Missed spam caused by SCL -1 and reason SKN X-Forefront-Antispam-Report: CIP:104.47.32.118;IPV:NLI;CTRY:;EFV:NLI;SFV:SKN;SCL:-1;SFS:(These are spam rules);DIR:INB;SFP:;SRVR:CO2PR0801MB677;H:NAM01-SN1-obe.outbound.protection.outlook.com;FPR:;SPF:Pass;PTR:mail-sn1nam01on0118.outbound.protection.outlook.com;A:1;MX:1;LANG:en; Example 2 – False positive with SCL 6 and Reason SKB X-Forefront-Antispam-Report: CIP:104.47.32.118;IPV:NLI;CTRY:;EFV:NLI;SFV:SKB;SCL:6;SFS:(These are spam rules);DIR:INB;SFP:;SRVR:CO2PR0801MB677;H:NAM01-SN1-obe.outbound.protection.outlook.com;FPR:;SPF:Pass;PTR:mail-sn1nam01on0118.outbound.protection.outlook.com;A:1;MX:1;LANG:en; Example 3 – Missed spam SCL -1 and reason IPV:CAL X-Forefront-Antispam-Report: CIP:104.47.32.118;IPV:CAL;CTRY:;EFV:NLI;SFV:SKN;SCL:-1;SFS:(These are spam rules);DIR:INB;SFP:;SRVR:CO2PR0801MB677;H:NAM01-SN1-obe.outbound.protection.outlook.com;FPR:;SPF:Pass;PTR:mail-sn1nam01on0118.outbound.protection.outlook.com;A:1;MX:1;LANG:en; Example 4 - Bulk email To determine if a message is marked as spam because it's bulk email, you need to look at the X-Microsoft-Antispam header. This will contain the BCL value. In Office 365, the default value for BCL blocking is 7 (meaning 7-9 are blocked by default). Some customers will lower the BCL which will be more aggressive in terms of bulk email filtering, but this is usually associated with more false positives (see How to securely add a sender to an allow list in Office 365 for tips on how to avoid this). X-Forefront-Antispam-Report: ...SFV:SPM; [Message is marked as spam, there is no special code] X-Microsoft-Antispam: UriScan:;BCL:7;PCL:0;RULEID:(SpamRules);SRVR:BN6PR1101MB2113; X-CustomSpam: Bulk mail [This is stamped when a message is spam due to bulk email filtering and BCL value is non-default] Conclusion This should help provide both you and us with the necessary information to troubleshoot spam problems when you experience them in Office 365. It lets you see if there's some configuration problems on your side, and if there isn't, then getting us data in a usable format reduces friction on side to quickly fix the problem so you can get on with your day.

How to securely add a sender to an allow list in Office 365

tzink — Wed, 29 Nov 2017 12:37:34 GMT

Background We sometimes see users creating allow rules, either through Exchange Transport Rules (ETRs), or Domain Allows, or Safe Senders, when they want to receive email from senders. However, they frequently do this insecurely; spammers then spoof the sending domain (or even the full email address) which skips all spam filtering and the message lands in a user's inbox. This causes significant user-dissatisfaction. Or, on the flip side, organizations sometimes want to receive email from a particular sender (either their own domain, or an external one) and messages go to Junk because it's (legitimately) flagged as spam by our filters. This can occur a couple of ways:

1. The sender is spoofing (usually your own domain) and being marked as spam/phish This occurs when you have some service, such as an on-premise mail server or a cloud service like AWS or Azure, send automated reports to yourself. These frequently get marked as spam and have the red fraud detection tip on them. 2. The sender is being flagged as spam because of bulk email filtering It's common to send bulk email, such as marketing or transactional messages, through a bulk email service like MailChimp, SalesForce, SendGrid, SparkPost, SailThru, etc. These bulk email services often have the option for customers to send email through dedicated IPs, but they more frequently have customers send through shared IP pools. These bulk IPs are assigned a Bulk Confidence Level (BCL); because they come from shared IPs, some IPs have a high (more spammy) BCL and therefore get marked as spam. The net result is that the bulk email service you subscribe to can't send email to yourself (for more details, see Bulk Complaint Level values.)

This article describes how to fix this. The wrong way to allow a sender domain One of the most common methods to allow a sending domain is through ETRs. However, the most insecure way to do it is to create one that looks at the sending domain, and if it matches, sets SCL -1: This is the least secure ETR you could possibly write. What happens is the email hits the Transport Rule engine, and then does a string match on the domain in the From: address. If it matches, it skips spam filtering and delivers to the Inbox. This is insecure because it's trivially easy for a spammer or phisher to put anything they want into the From: address. So while you think you're allowing email from the sender, you're actually allowing email from anyone who puts that particular domain in the From: address. Because of the volume of spear-phishing attacks that is so prevalent today, it's fairly easy to guess whose sending you email legitimately that you might want to allow. Common examples are members of an organization's senior leadership team like the CEO, CFO, etc. Other guessable examples are companies you do business with. A large energy company probably talks to other large energy companies and its suppliers. A large (or medium, or small) technology company probably talks to other large (or medium, or small) technology companies. You maybe even have your own domain allowed so you get all sorts of external reports. These are all easily guessable by spammers and phishers. The secure way to allow a domain to send you email The best way to create an ETR is to do it securely. You should allow a domain + something else:

Use the sending domain plus authentication status. You can put the sending domain as one condition, and then the Authentication-Results header contains "dmarc=pass" or "dmarc=bestguesspass". This ensures that the sending domain has been authenticated and is not spoofing. The advantage of doing it this way is if you want to add more domains, you can simply add to the list of domains
Or, if the sending domain does not authenticate, use the sending domain plus a sending IP (or IP range). However, be restrictive in the amount of IPs you add, we recommend going no wider than a /24. If you go wider than that, you may be allowing too many sending infrastructures that is not under control of the sender
Optionally, add a condition that the message originates from outside the organization (this is implicit, but it's fine to add it as a condition to account for on-premise servers that may not be configured correctly)

The rule actions should be the following:

Set the SCL to -1 (Bypass spam filtering)
Add an x-header that says what the rule does. In the example below, I've added a simple header "X-ETR: Bypass spam filtering for authenticated sender contoso.com". If you have more than one domain in this rule, you can change the header text as appropriate.When a message skips filtering due to an ETR, it stamps SFV:SKN in the X-Forefront-Antispam-Report header (if it's on an IP Allow list, it also stamps IPV:CAL). However, there's no way to see if the ETR was secure or not without manually unwinding everything and viewing the ETR syntax. That's why it is important to add the additional header so you can figure what's going on in the event that you ever get a spam message that skipped filtering.

You should only include domains that are under the control of the sending organization, that is, don't add free consumer accounts like @gmail.com, @outlook.com, etc. If you want to add a free consumer account, add the entire email address. What about Domain Allows, and Sender Allows? There's another way to bypass filtering, and that's by adding a Domain Allow, or Sender Allow (full email address). The problem with Domain Allows as currently implemented by Office 365 is that they are simple string matches, similar to insecure ETRs. You should only use them when you want to allow a particular sender that is not easily guessable, e.g., noreply-alerts@contoso.com (although even that is somewhat guessable). When a message skips filtering due to a Domain Allow or Sender Allow, it stamps SFV:SKA in the X-Forefront-Antispam-Report header. You shouldn't use these to add your own domain, or when the sending domain has no authentication, or when the sending domain is a free consumer email service. This is because the first two are easily spoofable, and the third one doesn't have enough sign-up roadblocks to prevent phishing. The good news is that we are fixing the problem of authentication and spoofing with Domain Allows - if a message fails authentication, we will not respect the Domain or Sender Allow and instead send it through regular filtering. Until then, probably the best advice is to avoid this feature to allow sending domains unless you're really, really confident a spammer would not spoof it. What about Safe Senders? Safe senders are user-actions that your users can take to skip filtering from senders that are ending up in Junk (or they do it proactively to prevent the filter from ever doing it). Rather than get into the full details, please see How to set up safe senders and blocked senders in Office 365. It is difficult to prevent your users from using Safe Senders, they are a valuable feature. However, as currently implemented by Office 365, they are vulnerable to spoofing because they are simple string matches. Fortunately, as per above, we are making a change to not respect a user's safe sender if it fails authentication. Our recommendation is for users to add to safe senders when they want to receive email from someone specific. When a message skips filtering due to a user safe sender, it stamps SFV:SFE in the X-Forefront-Antispam-Report header. Be aware that it's fairly common for executives in organizations to safe sender each other, and spammers/phishers guess this and send malicious email which skips filtering. The option "Never block Sender's Domain (@example.com)" is never respected in Office 365. It is a no-operation. Making sure your domain is secure... or at least, authenticating so you can securely allow it to bypass filtering Okay, by now you've understood why you should only permit secure domains to skip filtering when sending into your organization (unless you know what you're doing and want to take a calculated risk). The next question is: How do I ensure my domain is secure?

First, if you're sending outbound email through Office 365, make sure you've set up SPF and DKIM so that it passes authentication when email originates outside of the service. . How Office 365 uses Sender Policy Framework (SPF) to prevent spoofing Use DKIM to validate outbound email sent from your custom domain in Office 365asdfasdf .asdfas.
Second, for each service you use that sends outbound email as your domain, make sure you add it either to your SPF record, or set up DKIM so that it signs as your domain.There is lots and lots of documentation on the Internet about how to go about doing this, and the best way to inventory them is to use DMARC (for example, see How we got to enforce DMARC for sub-domains of Microsoft’s largest consumer email brands and How we moved microsoft.com to a p=quarantine DMARC record). . However, there are a handful of large senders that a lot of large organizations use. Here's a quick list of their documentation on how to set up SPF and/or DKIM on them: . SalesForce SPF - What SPF records do I need to add to my domain for Salesforce? DKIM - DKIM Signing Outbound Email, Set up DKIM in SalesForce classic . SparkPost SPF - Sparkpost SPF builder DKIM - Sparkpost: DKIM explained . SendGrid SPF - SPF: Sendgrid documentation DKIM - DKIM: Sendgrid documentation . MailChimp SPF and DKIM - Set Up Custom Domain Authentication: DKIM and SPF . SailThru SPF and DKIM - Configure DNS records for your domain . This list is not meant to be exhaustive. If I haven't listed the service here, you can easily find it yourself by doing a search for "Set up SPF " or "Set up DKIM ".

By setting up authentication, you can send email to yourself (or anyone else) and have it authenticate as your domain, and then it's safe to add to a domain allow list. However, you must do this for each service you send out of, otherwise if you don't, it will go through the regular spam filtering stack where it may get marked as spam. Conclusion We understand the need to sometimes bypass spam filtering; sometimes filters make mistake and legitimate email gets marked as spam. However, you need to do it securely so you don't accidentally allow spam/phish/malware into your organization. We're making updates to the system so that we do the right thing even without you - as an administrator - having to specifically set it up. However, there will always be a place for admins to make some overrides. This post should help you do it.

Appendix - IP Allows There is another way to skip filtering in Office 365 and that's by setting up an IP Allow. There are advantages and disadvantages. On the plus side, they are easy to add. You just navigate through the UX (or use Powershell), and add a sending IP and it then skips all spam filtering (malware filtering is still applied). No fancy ETR syntax required. However, one drawback is that they are hard to manage. Whereas it's pretty easy to see what a domain is (e.g., microsoft.com is Microsoft, google.com is Google), it's difficult to look at an IP and see what it is without doing a reverse DNS lookup or a WHOIS lookup. "Just what was I thinking when I added 1.2.3.4 to my IP Allow?" A second drawback of IP allows is that you don't always know who sends the IP. Is it the one sender who you got a false positive from? Or is it shared IP space (such as a bulk email provider, or even Office 365)? That is, will adding an IP Allow skip filtering from many different senders, most of whom you didn't intend? Before adding it, you should verify whether the IP belongs to a single domain, or many. If you don't know how to do that, use an ETR per above and allow the sending domain securely. A third drawback of IP allows, related to the second drawback, is that you may add too many IPs. For example, if you add an entire /24 range, you may not know the ownership of that entire range. For many administrators who aren't technical enough to look up WHOIS records and interpret CIDR notation, this may end up permitting more than you thought. So, exercise caution when using this feature.

A short intro to how the Phishing Confidence Level (PCL) works

tzink — Fri, 24 Nov 2017 20:02:52 GMT

This is a rough description of how the Phishing Confidence Level (PCL) works in Office 365. Way back in the olden days - 2007 or so - Exchange server used to have its own spam filter, Smartscreen. This was more-or-less the same spam filter running in Outlook.com. But whereas Smartscreen in Outlook.com (then known as Hotmail) updated multiple times per day, the spam filter in Exchange only updated periodically. Smartscreen on Exchange was designed to communicate with Outlook the email client. When Smartscreen set the Spam Confidence Level (SCL), this was propagated to Outlook who would move messages SCL 5 and higher to the Junk Email folder. The SCL was stamped in the following header:

X-MS-Exchange-Organization-SCL: 5

Smartscreen could also detect if a message was phishing, and if so, it would set the Phishing Confidence Level (PCL) to 8. It would stamp this in the following header:

X-MS-Exchange-Organization-PCL: 8

Or, perhaps this:

X-MS-Exchange-Organization-PCL: Suspicious

That comes from Antispam Stamps, our documentation from Exchange on-prem's filtering (I'm also not totally sure on the format, I rarely work with the phishing filter for on-prem because I'm all about the Office 365 and Outlook.com services). But it's not enough to store these values in headers, Exchange also stores them in MAPI properties. Not every X-MS-Exchange-Organization-* header corresponds to a MAPI property, but these ones do (MAPI is a database-backed mailstore for Exchange; I've been working with it for years and still don't fully understand it). So, Outlook has access to those headers but also to those MAPI properties. Whenever Outlook sees PCL 8 in a MAPI property, it displays the following pink bar in the email client: You may have seen this in Outlook; if you do, it's because PCL 8 has been set. A couple years ago, we moved away from the native Outlook phishing bar because... well, it's aesthetically unpleasing and hard to update (we have to update the code and then get our users to update their software. That's an uphill battle.) In its place, we moved to Safety Tips. Where's what the new Safety Tip looks like for phishing: This new-and-improved Safety Tip is only available in Outlook.com and Office 365. It is not available in Exchange on-prem. Doesn't this look way better? The "Learn more" link can be controlled dynamically, that is, we can change where it points to. The problem is that all the code to show the pink bar still exists in Outlook, which means that if Office 365 (or Outlook.com) ever sees a phishing message and tries to set PCL 8, which is how we've defined phishing previously, that native pink bar will show up in the client. We don't want that, we only want to show the new Safety Tip. To solve this problem, we decided that we would reserve a new value for phishing: -9990. That way, we would always know that it was Office 365 that identified a message as phishing and not some other service, and Outlook would never show the native pink bar. It allows us to instead show only the newest Safety Tip. The PCL is supposed to be stamped in two places, the first is the X-Microsoft-Antispam header in the PCL property, and then additionally in the X-MS-Exchange-Organization-PCL header. But it also gets stamped sometimes in the X-Exchange-Antispam-CFA-Test header which is an internal header. But sometimes a message still gets PCL 8, despite my best efforts to purge PCL 8 from the system.

1. Exchange on-prem still sets the PCL value If you are running Exchange on-prem in front of Office 365, and it detects a message as phish and sets PCL 8, and you have a set of connectors to connect to Office 365 which propagates message properties forward, Office 365 will inherit and promote the previously existing PCL the same as it does a previously set SCL. In a case like this, PCL 8 will appear in the message headers. However, the X-MS-Exchange-Organization-PCL header will have a bunch of other properties that Office 365 does not use, and that allows us to differentiate between Exchange on-prem and Office 365's antiphishing. 2. Corner cases in our code Despite my super-best efforts, sometimes messages get PCL 8 because they have a "Mark as phish" rule somewhere in the system that does not have PCL -9990 assigned, and instead gets a default PCL 8. This one drives me crazy. Usually, although not always, when a message is marked as phish in our system, it's because the SFTY value is 9.x, and 9.1 is the most common reason (this property is not documented). But sometimes even in a default case, 9.1 is stamped. Those are difficult to track down. Fortunately, this doesn't happen that much anymore.

The Phishing Confidence Level is a little more complicated than that, because we have machine learning and some other technologies, and I don't quite know how they interact with the SFTY and PCL. I haven't worked on those features and I'm not so familiar with them. Still, this description - as of this writing - covers the majority of PCL and phishing cases that I have seen. I hope you find this useful.