Create and publish analytics rules for Microsoft Sentinel solutions

Microsoft Sentinel analytics rules are sets of criteria. They define how data is monitored, what's detected, and what actions are taken when specific conditions are met. These rules help identify suspicious behavior, anomalies, and potential security threats by analyzing logs and signals from various data sources.

Microsoft Sentinel analytics rules are powerful tools for enhancing an organization's security posture because they proactively detect and respond to potential threats. By following a structured approach to creating and managing these rules, organizations can use the capabilities of Microsoft Sentinel to protect their digital assets and maintain a robust security infrastructure. For more information, see Threat detection in Microsoft Sentinel.

This article walks you through the process of creating and publishing analytics rules to Microsoft Sentinel solutions.

Use cases for Microsoft Sentinel analytics rules

Microsoft Sentinel analytics rules can be applied to a wide range of scenarios to enhance security monitoring and threat detection. Common use cases include:

  • Intrusion detection: Identify unauthorized access attempts or suspicious sign-in activities that could indicate a potential breach.
  • Malware detection: Monitor for known malware signatures or unusual behavior that might suggest the presence of malicious software.
  • Data exfiltration: Detect large or unusual data transfers that could signify that data is being exfiltrated from the network.
  • Insider threats: Identify anomalous behavior from internal users, such as accessing sensitive data outside of normal hours or patterns.
  • Compliance monitoring: Ensure adherence to regulatory requirements by monitoring for specific activities or access patterns.
  • Threat hunting: Proactively search for indicators of compromise or other signs of malicious activity within the network.
  • Account compromise: Detect signs that user accounts might be compromised, such as unusual geographic sign-in patterns or multiple failed sign-in attempts.
  • Network anomalies: Identify unusual network traffic patterns that could indicate the presence of a threat or misconfiguration.
  • Privilege escalation: Monitor for attempts to gain elevated privileges within the network, which can be a precursor to further malicious activity.
  • Endpoint security: Ensure that endpoints are secure by detecting deviations from normal behavior or the presence of unauthorized software.

Create and publish analytics rules

You create analytics rules in YAML format. Use these examples of analytics rules as references to create your own:

The following sections provide a detailed walkthrough of various attributes of an analytics rule.

Sample analytics rule

The following is a sample analytics rule in YAML format.

id: aaaaaaaa-0000-1111-2222-bbbbbbbbbbbb
name: Anomalous login from unknown IP
description: |
  Identifies login attempts from IP addresses not seen in the past 14 days.
severity: Medium
requiredDataConnectors:
  - connectorId: ContosoMyProduct
    dataTypes:
      - ContosoMyProduct_CL
queryFrequency: 1h
queryPeriod: 14d
triggerOperator: gt
triggerThreshold: 0
tactics:
  - InitialAccess
relevantTechniques:
  - T1078
status: Available
query: |
  ContosoMyProduct_CL
  | where TimeGenerated > ago(1h)
  | where EventType == "Login"
  | summarize count() by SrcIpAddr, UserName
  | extend Name = tostring(split(UserName, '@', 0)[0])
  | extend UPNSuffix = tostring(split(UserName, '@', 1)[0])
entityMappings:
  - entityType: Account
    fieldMappings:
      - identifier: Name
        columnName: Name
      - identifier: UPNSuffix
        columnName: UPNSuffix
  - entityType: IP
    fieldMappings:
      - identifier: Address
        columnName: SrcIpAddr
version: 1.0.0
kind: Scheduled

ID

The id attribute consists of a standard globally unique identifier (GUID). Generate it by using any development tool, an online generator, or the new PowerShell New-GUID cmdlet. It must be unique among other GUIDs.

This field is mandatory.

Kind

The kind attribute represents the type of rule.

There are two accepted values: scheduled and NRT (near-real time). The scheduled value requires that you define other properties, including queryFrequency, queryPeriod, triggerThreshold, and triggerOperator. For NRT rules, omit queryFrequency, queryPeriod, triggerOperator, and triggerThreshold.

This field is mandatory.

Name

The name attribute provides a brief label that summarizes the detection. Make sure the label is clear and concise to help users understand the purpose of the rule. Use alertDetailsOverride to generate dynamic names to help analysts understand the alert. This attribute:

  • Uses sentence-case capitalization.
  • Doesn't end in a period.
  • Has a recommended length of less than 50 characters, and a hard maximum of 100 characters.
  • Avoids the words "Suspicious" and "Suspect". Use "Unexpected", "Anomalous", or "Rare" instead.

This field is mandatory.

Description

The description attribute provides a detailed description of the detection. The description includes information about the behavior being detected, the potential effect, and any recommended actions. This attribute:

  • Uses sentence case capitalization.
  • Starts with "This query searches for" or "Identifies."
  • Is different from and more descriptive than the name field.
  • Has a maximum length of 255 characters.
  • Is five sentences or less.
  • Doesn't describe the data source (connector or data type).
  • Doesn't provide a technical explanation for the query language.
  • Uses ASCII characters only. Em dashes, smart quotes, and other non-ASCII characters fail validation.

This field is mandatory.

Severity

The severity attribute defines the severity level of the detection. Severity reflects the potential effect of the behavior being detected and the urgency of the response.

  • Informational: The incident might not directly represent a security threat, but might be of interest for follow-up investigation, or to add context or situational awareness to an analyst.
  • Low: Immediate effect is minimal, and a threat actor would need to conduct multiple steps before achieving an effect on an environment.
  • Medium: The threat actor could achieve some effect on the environment with this activity, but the effect would be limited in scope or require extra activity.
  • High: The identified activity provides the threat actor with wide-ranging access to conduct actions on the environment.

Note

Severity level defaults aren't a guarantee of the current or environment impact level. Severity level applies only to Microsoft Sentinel analytics templates. Otherwise, the security service that issued the alert controls the severity attribute in the Alerts table. You can use alertDetailsOverride to provide a dynamic severity attribute that depends on the actual outcome of the query.

Status

The status attribute indicates the production readiness of the rule:

  • Available: Production-ready rules.
  • InPreview: Beta rules.
  • Deprecated: Rules being phased out.

Required data connectors

The requiredDataConnectors attribute represents the list of data connectors that the rule needs to function correctly, including the data sources against which the rule queries. If there's no current data connector mapping, you must use an open brace: requiredDataConnectors: [].

The connectorId attribute specifies the ID of the data connector that you need so the query functions correctly. If your detection query depends on the data fetched from a specific connector, you must specify the connector ID here. For instance, if your analytics rule depends on the data from this connector, you must specify the connectorID as 1PasswordCCPDefinition.

The dataTypes attribute represents the data types that the analytics rule depends on and mentions the name of the data type referenced in the dataTypes section of the connector. For instance, if your hunting query depends on the data from this connector, you must specify the data type as OnePasswordEventLogs_CL. If the hunting query operates on a Kusto function/parser instead of the table (like Syslog, CommonEventFormat, or _CL), dataTypes is the Kusto function name/parser name and not the table name.

Query period

The queryPeriod attribute represents a specified period during which the query runs. For example: the last 3 days. For this attribute:

  • Use Kusto Query Language (KQL) TimeSpan format (for example, 3 days is 3d, 2 hours is 2h).
  • Ensure that any learning or reference period is within this time frame.
  • Don't use a value higher than the maximum supported value, which is 14d.

This field is mandatory for scheduled analytics rules.

Query frequency

The queryFrequency attribute represents the frequency at which the query runs. For this attribute:

  • Use KQL TimeSpan format (for example, 3 days is 3d, 2 hours is 2h).
  • Use a queryFrequency that is less than or equal to the queryPeriod.
  • Follow this rule: if the queryPeriod is greater than or equal to 2 days (2d), the queryFrequency value can't be less than 1 hour (1h) and is only used for high-severity detections.

This field is mandatory for scheduled analytics rules.

Trigger operator

The triggerOperator attribute indicates the mechanism that triggers the alert. For example: greater than (gt) the number set in the triggerThreshold attribute (see Trigger threshold).

  • gt: Greater than.
  • lt: Less than.
  • eq: Equal to.

This field is mandatory for scheduled analytics rules.

Trigger threshold

The triggerThreshold attribute represents the threshold that triggers the alert. Threshold is the value that the triggerOperator references. Supported values include any integer between 0 and 10,000.

For example, if the triggerOperator is set to gt and the triggerThreshold is 1, the alert triggers when a value is greater than 1.

This field is mandatory for scheduled analytics rules.

Tactics

The tactics attribute defines the MITRE ATT&CK tactics that the detection relates to. When you define the tactics, it helps users understand the context of the detection and how it fits into the overall threat landscape. For this attribute:

  • ATT&CK Framework v16 is supported.
  • Names use PascalCase and can't include spaces. For example: InitialAccess or LateralMovement.
  • A maximum of five tactics can be defined per rule.
  • Valid values are: Reconnaissance, ResourceDevelopment, InitialAccess, Execution, Persistence, PrivilegeEscalation, DefenseEvasion, CredentialAccess, Discovery, LateralMovement, Collection, CommandAndControl, Exfiltration, and Impact.

This field is mandatory.

Relevant techniques

The relevantTechniques attribute defines the MITRE ATT&CK techniques that the detection relates to. When you define the techniques, it helps users understand the context of the detection and how it fits into the overall threat landscape. For this attribute:

  • ATT&CK Framework v16 is supported.
  • Each technique must belong to at least one of the listed tactics.
  • Names can't include spaces. For example: T1078 or T1078.001.
  • A maximum of 10 techniques can be defined per rule.

This field is mandatory.

Query

The query attribute defines the detection logic. We recommend that you write the query in KQL and make sure that it's structured and easy to understand. We recommend that you create an efficient query that's optimized for performance to ensure it can be run against large datasets without affecting performance. Make sure that your query meets the following criteria.

Limit the query to 10,000 characters. If the query section exceeds this limit, consider reducing the number of characters. A static list of items used for comparison within the query body can cause you to go over the limit. We recommend that you move these lists to one of the following options:

Each line in the query body must have at least one space at the beginning, but two spaces are standard to support readability.

When you submit a query for a datatype that isn't present in the Detections or Hunting Queries folder, name the subfolder that contains the YAML files after the table being queried. For instance, if your query pertains to the AzureDevOpsAuditing table, create a folder named AzureDevOpsAuditing.

Define human-readable names for explicit constants:

  • let FailedLoginEventID = 4625;
  • let countThreshold = 6;

We highly recommend that you use comments to clarify the query. Avoid adding comments at the end of a query statement line. Instead, add your comments on a separate line. For example:

  // Removing noisy processes for an environment, adjust as needed

If you're referencing a parser instead of a table name, ensure clarity in the description by including a comment next to the parser function reference. The parser must be imported into the workspace first. Otherwise, the queries don't recognize it as valid.

Ensure that every available entity field is returned for mapping purposes. (Refer to the Entity mappings section.) Sanitize the returned table so that it provides only the properties that you need to investigate further. You don't need a TimeGenerated filter when you use a simple lookback command across the entire query. The queryPeriod value in the YAML controls this process.

For baselining or performing a historical comparison, such as comparing today to the previous seven days, include a time-bounded filter such as | where TimeGenerated >= ago(lookback), because the YAML template doesn't currently support multiple queryPeriod values. Avoid using time frames shorter than one day unless there's a specific reason. We don't recommend time frames longer than 14 days due to potential performance impacts.

Summarize when necessary. Ensure that you include the time field (usually TimeGenerated) because you need it in the entity field. Include both the min() and max() values as follows: | summarize StartTime = max(TimeGenerated), EndTime = min(TimeGenerated). Use the terms StartTime and EndTime exclusively. Don't assign the fields the names StartTimeUtc or EndTimeUtc, because these names can conflict with user experience preferences.

Additionally, include as many fields as possible to help the user understand the context of the alert. We recommend that you include at least one of the primary entities: Host, Account, or IP.

Column names in your custom table must use camelCase (for example, SrcIpAddr or UserName).

This field is mandatory.

Event grouping settings

The eventGroupingSettings attribute relates to alerts. An alert rule can generate a separate alert for each query result. For instance, a rule that identifies non-Microsoft alerts in the event stream could create a Microsoft Sentinel alert for each source alert.

  • To produce a single alert for all query results (the default), use:

    eventGroupingSettings:
aggregationKind: SingleAlert

  • To produce a separate alert for each query result, use:

    eventGroupingSettings:
aggregationKind: AlertPerResult

Entity mappings

The entityMappings attribute is integral when you configure scheduled analytics rules. It enriches the query's output (alerts and incidents) with essential information that serves as the building blocks of any investigative processes and remedial actions that follow.

The entityType represents the standard list of entities recognized by Microsoft Sentinel. Supported types include Account, Host, IP, URL, File, Process, DNS, AzureResource, FileHash, RegistryKey, RegistryValue, SecurityGroup, Mailbox, and MailMessage. See allowed values in the Entity type column in the Entity mapping table.

This field is mandatory.

Field mappings

The fieldMappings attribute represents the identifier of the field in the query output that corresponds to the entity type. See allowed values under the identifiers column value at Entity mapping table. For this attribute:

  • Each template can have up to 10 entity mappings.

  • Each entity mapping can have up to three field mappings (that is, identifiers).

    entityMappings:
- entityType: Account
  fieldMappings:
    - identifier: FullName
      columnName: AccountCustomEntity
- entityType: Host
  fieldMappings:
    - identifier: FullName
      columnName: HostCustomEntity
- entityType: IP
  fieldMappings:
    - identifier: Address
      columnName: ClientIP
- entityType: DNS
  fieldMappings:
    - identifier: DomainName
      columnName: Name

Custom details

The customDetails attribute integrates event data into alerts, making it visible in security incidents for faster triaging, investigation, and response. Custom details are key/value pairs of property and column names. For more information, see Surface custom event details in alerts in Microsoft Sentinel. Up to 20 custom details (that is, key/value pairs) can be defined per template. Key names must be 20 characters or fewer.

    customDetails:
      Computers: Computer
      IPs: ComputerIP

Alert details override

The alertDetailsOverride attribute is a dynamic field that you can use to override the alert details. You can use this attribute to provide more context or information to the analyst when the alert is triggered. When you use this feature, you ensure that analysts receive pertinent information, including relevant entity names, to facilitate a quicker, and more accurate understanding of the incident. Limitations include:

  • A maximum of three parameters can be included in either the name or description.

  • The name must not exceed 256 characters, while the description is limited to 5,000 characters.

  • The column name within the curly braces must precisely match the expected column name, without any leading or trailing white space (for example, {{columnName}}, not {{ columnName }}). In the following example, columnName1, columnName2, dynamicTactic, and dynamicSeverity are output fields of the scheduled alert query.

        alertDetailsOverride:
      alertDisplayNameFormat: free text with field names embedded using the format {{columnName}} # Up to 256 chars and 3 placeholders
      alertDescriptionFormat: free text with field names embedded using the format  {{columnName}} # Up to 5000 chars and 3 placeholders
      alertTacticsColumnName: dynamicTacticColumnName
      alertSeverityColumnName: dynamicSeverityColumnName


        alertDetailsOverride:
      alertDisplayNameFormat: rule {{columnName1}} display name
      alertDescriptionFormat: rule {{columnName2}} display name
      alertTacticsColumnName: dynamicTactic
      alertSeverityColumnName: dynamicSeverity

Incident configuration

The incidentConfiguration attribute controls incident creation and the grouping of alerts into incidents. Use it to define how alerts that this rule generates are aggregated into incidents.

Version

When a customer creates a new hunting query from the template, the template version is saved. If a new template version is published, customers are notified in the UX. Versions follow the format a, b, and c, in which a is the major version, b is the minor version, and c is the patch. The version field is the last line of the template.

This field is mandatory.

  • Last updated on