Edit

Share via

Attack simulation training deployment considerations and FAQ

Tip

Did you know you can try the features in Microsoft Defender XDR for Office 365 Plan 2 for free? Use the 90-day Defender for Office 365 trial at the Microsoft Defender portal trials hub. Learn about who can sign up and trial terms here.

Attack simulation training enables Microsoft 365 E5 or Microsoft Defender for Office 365 Plan 2 organizations to measure and manage social engineering risk by allowing the creation and management of phishing simulations that are powered by real-world, harmless phishing payloads. Hyper-targeted training, delivered in partnership with Terranova security, helps improve knowledge and change employee behavior.

For more information about getting started with Attack simulation training, see Get started using Attack simulation training.

While the simulation creation and scheduling experience is designed to be free-flowing and frictionless, simulations at an enterprise scale require planning. This article helps address specific challenges that we see as our customers run simulations in their own environments.

Issues with end user experiences

Phishing simulation URLs blocked by Google Safe Browsing

A URL reputation service might identify one or more of the URLs that are used by Attack simulation training as unsafe. Google Safe Browsing in Google Chrome blocks some of the simulated phishing URLs with a Deceptive site ahead message. While we work with many URL reputation vendors to always allow our simulation URLs, we don't always have full coverage.

The Deceptive site ahead warning in Google Chrome

This issue doesn't affect Microsoft Edge.

As part of the planning phase, be sure to check the availability of the URL in your supported web browsers before you use the URL in a phishing campaign. If the URLs are blocked by Google Safe Browsing, follow this guidance from Google to allow access to the URLs.

Refer to Get started using Attack simulation training for the list of URLs that are currently used by Attack simulation training.

Phishing simulation and admin URLs blocked by network proxy solutions and filter drivers

Both phishing simulation URLs and admin URLs might be blocked or dropped by your intermediate security devices or filters. For example:

  • Firewalls
  • Web Application Firewall (WAF) solutions
  • Third-party filter drivers (for example, kernel mode filters)

While we have seen few customers being blocked at this layer, it does happen. If you encounter problems, consider configuring the following URLs to bypass scanning by your security devices or filters as required:

Simulation messages not delivered to all targeted users

It's possible that the number of users who actually receive the simulation email messages is less than the number of users who were targeted by the simulation. The following types of users are excluded as part of target validation:

  • Invalid recipient email addresses.
  • Guest users.
  • Users that are no longer active in Microsoft Entra ID.

If you use distribution groups or mail-enabled security groups to target users, you can use the Get-DistributionGroupMember cmdlet in Exchange Online PowerShell to view and validate distribution group members.

Trainings unexpectedly assigned or not assigned to users

The training threshold in training campaigns prevents users from having the same trainings assigned to them during a specific interval (90 days by default). For more information, see Set the training threshold.

If you created a simulation or a simulation automation with the training assignment value Assign training for me (Recommended), we assign training based on a user's previous simulation and training results. To assign training based on specific criteria, select Select training courses and modules myself.

What happens when a user replies to or forwards a simulation message?

If a user replies to or forwards a simulation message to another mailbox, the message is treated like a normal email message (including detonation by Safe Links or Safe Attachments). The Simulation report shows whether the simulation message was replied to or forwarded. Each URL in the simulation email is tied to an individual user, so Safe Links detonations are identified as clicks by the user.

If you use a dedicated security operations (SecOps) mailbox, be sure to identify it as a SecOps in the advanced delivery policy so messages are delivered unfiltered.

How can I stagger the delivery of simulation messages?

Either way, it's important to use different payloads to avoid discussion and identification among users.

Why are images in simulation messages being blocked by Outlook?

By default, Outlook is configured to block automatic image downloads in messages from the internet. Although you can configure Outlook to automatically download images, we don't recommend it due to the security implications (potential automatic download of malicious code or web bugs, also known as web beacons or tracking pixels).

Third-party filtering services might be to blame. For any non-Microsoft filtering systems that you use, you need to allow or exempt the following items:

  • All Attack simulation training URLs and the corresponding domains. Currently, we don't send simulation messages from a static list of IP addresses.
  • Any other domains that you use in custom payloads.

Can I add the External tag or safety tips to simulation messages?

Custom payloads have the option to add the External tag to messages. For more information, see Step 5 in Create payloads.

There are no built-in options to add safety tips to payloads, but you can use the following methods on the Configure payload page of the payload setup wizard::

  • Use an existing email message that contains the safety tip as a template. Save the message as HTML and copy the information.

  • Use the following sample code for the First contact safety tip:

    <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%;mso-cellspacing:0in;mso-yfti-tbllook:1184;
mso-table-lspace:2.25pt;mso-table-rspace:2.25pt;mso-table-anchor-vertical:
paragraph;mso-table-anchor-horizontal:column;mso-table-left:left;mso-padding-alt:
0in 0in 0in 0in">
<tbody><tr style="mso-yfti-irow:0;mso-yfti-firstrow:yes;mso-yfti-lastrow:yes">
  <td style="background:#A6A6A6;padding:5.25pt 1.5pt 5.25pt 1.5pt"></td>
  <td width="100%" style="width:100.0%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 11.25pt" cellpadding="7px 5px 7px 15px" color="#212121">
  <div>
  <p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;
  mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:
  column;mso-height-rule:exactly"><span style="font-size:9.0pt;font-family:
  wf_segoe-ui_normal;mso-fareast-font-family:&quot;Times New Roman&quot;;mso-bidi-font-family:
  Aptos;color:#212121;mso-ligatures:none">You don't often get email from
  this sender <a rel="noopener" href="https://aka.ms/LearnAboutSenderIdentification" tabindex="-1" target="_blank">Learn why
  this is important</a></span></p>
  </div>
  </td>
  <td width="75" style="width:56.25pt;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 3.75pt;
  align:left" cellpadding="7px 5px 7px 5px" color="#212121"></td>
</tr>
</tbody></table>
<div>
<p class="MsoNormal"><span lang="DA" style="font-size:12.0pt;font-family:&quot;Georgia&quot;,serif;
color:black;mso-ansi-language:DA">Insert payload content here,</span></p>
</div>

Can I assign training modules without putting users through a simulation?

Yes. For more information, see Training campaigns in Attack simulation training.

How do I find out about simulation messages that weren't delivered?

The Users tab for the simulation is filterable by Simulation message delivery: Failed to deliver.

If you own the sender domain, the undelivered simulation report is returned in a non-delivery report (also known as an NDR or bounce message). For more information about the codes in the NDR, see Email non-delivery reports and SMTP errors in Exchange Online.

Issues with Attack simulation training reporting

Tip

Simulation data recording start a few minutes after the simulation is launched and after users begin interacting with the simulation messages. There's no fixed start time. Events are still captured after the simulation ends.

Differences in user activity data from Attack simulation training reports and other reports

For reporting on user activity related to simulation messages, we recommend using the built-in simulation reports. Reports from other sources (for example, Advanced hunting) might not be accurate.

Simulation URLs aren't wrapped by Safe Links and are considered unwrapped links. Not all clicks on unwrapped links go through Safe Links, so user activity related to simulation messages might not be recorded in the UrlClickEvents logs.

Attack simulation training reports don't contain any activity details

Attack simulation training comes with rich, actionable insights that keep you informed of the threat readiness progress of your employees. If Attack simulation training reports aren't populated with data, verify that audit logging is turned on in your organization (it's on by default).

Audit logging is required by Attack simulation training so events can be captured, recorded, and read back. Turning off audit logging has the following consequences for Attack simulation training:

  • Reporting data isn't available across all reports. The reports appear empty.
  • Training assignments are blocked, because data isn't available.

To verify that audit logging is on or to turn it on, see Turn auditing on or off.

Tip

Empty activity details are also caused by no E5 licenses being assigned to users. Verify at least one E5 license is assigned to an active user to ensure that reporting events are captured and recorded.

User actions and admin actions are audited. In the Management Activity API, look for the AuditLogRecordType values 85, 88, and 218.

Some auditing information might also be available in the CloudAppEvents table in Microsoft Defender XDR Advanced hunting via the Defender portal or the Streaming API.

Reporting issues with on-premises mailboxes

Attack simulation training supports on-premises mailboxes, but with reduced reporting functionality:

  • Data on whether users read, forwarded, or deleted the simulation email isn't available for on-premises mailboxes.
  • The number of users who reported the simulation email isn't available for on-premises mailboxes.

Tip

Other than the simulation messages being sent via the transport pipeline vs. direct injection in Microsoft 365, the training, automation, and content management experiences are the same for on-premises mailboxes.

Simulation reports aren't updated immediately

Detailed simulation reports aren't updated immediately after you launch a campaign. Don't worry; this behavior is expected.

Every simulation campaign has a lifecycle. When first created, the simulation is in the Scheduled state. When the simulation starts, it transitions to the In progress state. When completed, the simulation transitions to the Completed state.

While a simulation is in the Scheduled state, the simulation reports are mostly empty. During this stage, the simulation engine is resolving the target user email addresses, expanding distribution groups, removing guest users from the list, etc.:

Simulation details showing the simulation in the Scheduled state

Once the simulation enters the In progress stage, information starts trickling into the reporting:

Simulation details showing the simulation in the In progress state

It can take up to 30 minutes for the individual simulation reports to update after the transition to the In progress state. The report data continues to build until the simulation reaches the Completed state. Reporting updates occur at the following intervals:

  • Every 10 minutes for the first 60 minutes.
  • Every 15 minutes after 60 minutes until two days.
  • Every 30 minutes after two days until seven days.
  • Every 60 minutes after seven days.

Widgets on the Overview page provide a quick snapshot of your organization's simulation-based security posture over time. Because these widgets reflect your overall security posture and journey over time, they're updated after each simulation campaign is completed.

Note

You can use the Export option on the various reporting pages to extract data.

Messages reported as phishing by users aren't appearing in simulation reports

Simulation reports in Attack simulator training provide details on user activity. For example:

  • Users who clicked on the link in the message.
  • Users who gave up their credentials.
  • Users who reported the message as phishing.

If messages that users reported as phishing aren't captured in Attack simulation training simulation reports, there might be an Exchange mail flow rule (also known as a transport rule) that's blocking the delivery of the reported messages to Microsoft. Verify that any mail flow rules aren't blocking delivery to the following email addresses:

  • junk@office365.microsoft.com
  • abuse@messaging.microsoft.com
  • phish@office365.microsoft.com
  • not_junk@office365.microsoft.com

Users are assigned training after they report a simulated message

If users are assigned training after they report a phishing simulation message, check to see if your organization uses a reporting mailbox to receive user reported messages at https://security.microsoft.com/securitysettings/userSubmission. The reporting mailbox needs to be configured to skip many security checks as described in the reporting mailbox prerequisites.

If you don't configure the required exclusions for the custom reporting mailbox, the messages might be detonated by Safe Links or Safe Attachments protection, which causes training assignments.

Other frequently asked questions

A: Several options are available to target users:

  • Include all users (currently available to organizations with less than 40,000 users).
  • Choose specific users.
  • Select users from a CSV file (one email address per line).
  • Microsoft Entra group-based targeting.

We find that campaigns where the targeted users are identified by Microsoft Entra groups are easier to manage.

Q: How many training modules are there?

Currently, there are 94 built-in trainings on the Training modules page.

Q: Are there any limits in targeting users while importing from a CSV or adding users?

A: The limit for importing recipients from a CSV file or adding individual recipients to a simulation is 40,000.

A recipient can be an individual user or a group. A group might contain hundreds or thousands of recipients. The upper limit on the number of users is 400,000, but we recommend a limit of 200,000 users for each simulation for best performance.

Managing a large CSV file or adding many individual recipients can be cumbersome. Using Microsoft Entra groups simplify the overall management of the simulation.

Tip

Currently, shared mailboxes aren't supported in Attack simulation training. Simulations should target user mailboxes or groups containing user mailboxes.

Distribution groups are expanded and the list of users is generated at the time of saving the simulation or simulation automation.

Q: Are the limits for the number of simulations that can be deployed during a specific time interval?

A. No, although you might experience slowness if you launch many parallel simulations. Message rates (including simulation message rates) are constrained by the message rate limits of the service.

Q: Does Microsoft provide payloads in other languages?

A: Currently, there are 40+ localized payloads available in 29+ languages: English, Spanish, German, Japanese, French, Portuguese, Dutch, Italian, Swedish, Chinese (Simplified), Norwegian Bokmål, Polish, Russian, Finnish, Korean, Turkish, Hungarian, Hebrew, Thai, Arabic, Vietnamese, Slovak, Greek, Indonesian, Romanian, Slovenian, Croatian, Catalan, and Other. We've determined that direct or machine translation of existing payloads to other languages leads to inaccuracies and decreased relevance.

That being said, you can create your own payload in the language of your choice using the custom payload authoring experience. We also strongly recommend that you harvest existing payloads that were used to target users in a specific geography. In other words, let the attackers localize the content for you.

Q: How many training videos are available?

A: Currently, there are more than 85 training modules available in the content library.

Q: How can I switch to other languages for my admin portal and training experience?

A: In Microsoft 365 or Office 365, language configuration is specific and centralized for each user account. For instructions on how to change your language setting, see Change your display language and time zone in Microsoft 365 for Business.

The configuration change might take up to 30 minutes to synchronize across all services.

Q: Can I trigger a test simulation to understand what it looks like prior to launching a full-fledged campaign?

A: Yes you can! On the last Review Simulation page in the new simulation wizard, select Send a test. This option sends a sample phishing simulation message to the currently logged in user. After you validate the phishing message in your Inbox, you can submit the simulation.

The Send a test button on the Review simulation page

Tip

You can also use Send a test from the Payloads page. But, if you ever use the selected payload in a simulation, the test message appears in the aggregate reports. You can export the results or use the Microsoft Graph API to filter the results.

Q: Can I target users that belong to a different tenant as part of the same simulation campaign?

A: No. Currently, cross-tenant simulations aren't supported. Verify that all of your targeted users are in the same tenant. Any cross-tenant users or guest users are excluded from the simulation campaign.

Q: How does region aware delivery work?

A: Region aware delivery uses the time zone attribute of the targeted user's mailbox to determine when to deliver the message. There might be a time difference of ± one hour in the email delivery based on the user's time zone. For example, consider the following scenario:

  • At 7:00 AM in the Pacific time zone (UTC-8), an admin creates and schedules a campaign to start at 9:00 AM on the same day.
  • UserA is in the Eastern time zone (UTC-5).
  • UserB is also in the Pacific time zone.

At 9:00 AM on the same day, the simulation message is sent to UserB. With region-aware delivery, the message isn't sent to UserA on the same day, because 9:00 AM Pacific time is 12:00 PM Eastern time. Instead, the message is sent to UserA at 9:00 AM Eastern time on the following day.

So, on the initial run of a campaign with region aware delivery enabled, it might appear that the simulation message was sent only to users in a specific time zone. But, as time passes and more users come into scope, the targeted users increase.

If you don't use region aware delivery, the campaign starts based on the time zone of the user who's setting it up.

Q: Does Microsoft collect or store any information that users enter at the Credential Harvest sign-in page, used in the Credential Harvest simulation technique?

A: No. Any information entered at the credential harvest sign-in page is discarded silently. Only the 'click' is recorded to capture the compromise event. Microsoft doesn't collect, log or store any details that users enter at this step.

Q: How long is simulation information retained? Can I delete simulation data?

A: See the following table:

Data type Retention
Simulation metadata 18 months unless the simulation is deleted sooner by an admin.
Simulation automation 18 months unless the simulation automation is deleted sooner by an admin.
Payload automation 18 months unless the payload automation is deleted sooner by an admin.
User activity in simulation metadata 18 months unless the simulation is deleted sooner by an admin.
Global payloads Persisted unless deleted by Microsoft.
Tenant payloads 18 months unless the archived payload is deleted sooner by an admin.
User activity in training metadata 18 months unless the simulation is deleted sooner by an admin.
MDO recommended payloads 6 months.
Global end user notifications Persisted unless deleted by Microsoft.
Tenant end user notifications 18 months unless the notification is deleted sooner by an admin.
Global login pages Persisted unless deleted by Microsoft.
Tenant login pages 18 months unless the login page is deleted sooner by an admin.
Global landing pages Persisted unless deleted by Microsoft
Tenant landing pages 18 months unless the landing page is deleted sooner by an admin.

If the entire tenant is deleted, attack simulation training data is deleted after 90 days.

For more information, see Data retention information for Microsoft Defender for Office 365.

Q: Can I create, view, and manage simulations using an API?

A: Yes. Read and write scenarios are supported using the Microsoft Graph API:

  • AttackSimulation.Read.All:
    • Read simulation metadata
    • Read user activity
    • Read training data
    • Read repeat offenders
  • AttackSimulation.ReadWrite.All: Run simulations using the specified payloads, notifications, and login pages.

For more information, see List simulations and Reports API overview for attack simulation training as part of Microsoft Defender for Office 365.

Q: Can I delete custom payloads?

A: Yes. First you archive the payload, then you delete the archived payload. For instructions, see Archive payloads.

Q: Can I modify the built-in payloads?

A: Not directly. You can copy the payload and then modify the copy. For instructions, see Copy payloads.