Migrate ArcSight detection rules to Microsoft Sentinel

Article
06/01/2022

This article describes how to identify, compare, and migrate your ArcSight detection rules to Microsoft Sentinel analytics rules.

Identify and migrate rules

Microsoft Sentinel uses machine learning analytics to create high-fidelity and actionable incidents, and some of your existing detections may be redundant in Microsoft Sentinel. Therefore, don't migrate all of your detection and analytics rules blindly. Review these considerations as you identify your existing detection rules.

Make sure to select use cases that justify rule migration, considering business priority and efficiency.
Check that you understand Microsoft Sentinel rule types.
Check that you understand the rule terminology.
Review any rules that haven't triggered any alerts in the past 6-12 months, and determine whether they're still relevant.
Eliminate low-level threats or alerts that you routinely ignore.
Use existing functionality, and check whether Microsoft Sentinel’s built-in analytics rules might address your current use cases. Because Microsoft Sentinel uses machine learning analytics to produce high-fidelity and actionable incidents, it’s likely that some of your existing detections won’t be required anymore.
Confirm connected data sources and review your data connection methods. Revisit data collection conversations to ensure data depth and breadth across the use cases you plan to detect.
Explore community resources such as the SOC Prime Threat Detection Marketplace to check whether your rules are available.
Consider whether an online query converter such as Uncoder.io might work for your rules.
If rules aren’t available or can’t be converted, they need to be created manually, using a KQL query. Review the rules mapping to create new queries.

Learn more about best practices for migrating detection rules.

To migrate your analytics rules to Microsoft Sentinel:

Verify that you have a testing system in place for each rule you want to migrate.
1. Prepare a validation process for your migrated rules, including full test scenarios and scripts.
2. Ensure that your team has useful resources to test your migrated rules.
3. Confirm that you have any required data sources connected, and review your data connection methods.
Verify whether your detections are available as built-in templates in Microsoft Sentinel:
- If the built-in rules are sufficient, use built-in rule templates to create rules for your own workspace.
  
  In Microsoft Sentinel, go to the Configuration > Analytics > Rule templates tab, and create and update each relevant analytics rule.
  
  For more information, see Detect threats out-of-the-box.
- If you have detections that aren't covered by Microsoft Sentinel's built-in rules, try an online query converter, such as Uncoder.io to convert your queries to KQL.
  
  Identify the trigger condition and rule action, and then construct and review your KQL query.
- If neither the built-in rules nor an online rule converter is sufficient, you'll need to create the rule manually. In such cases, use the following steps to start creating your rule:
  1. Identify the data sources you want to use in your rule. You'll want to create a mapping table between data sources and data tables in Microsoft Sentinel to identify the tables you want to query.
  2. Identify any attributes, fields, or entities in your data that you want to use in your rules.
  3. Identify your rule criteria and logic. At this stage, you may want to use rule templates as samples for how to construct your KQL queries.
    
    Consider filters, correlation rules, active lists, reference sets, watchlists, detection anomalies, aggregations, and so on. You might use references provided by your legacy SIEM to understand how to best map your query syntax.
  4. Identify the trigger condition and rule action, and then construct and review your KQL query. When reviewing your query, consider KQL optimization guidance resources.
Test the rule with each of your relevant use cases. If it doesn't provide expected results, you may want to review the KQL and test it again.
When you're satisfied, you can consider the rule migrated. Create a playbook for your rule action as needed. For more information, see Automate threat response with playbooks in Microsoft Sentinel.

Learn more about analytics rules:

Create custom analytics rules to detect threats. Use alert grouping to reduce alert fatigue by grouping alerts that occur within a given timeframe.
Map data fields to entities in Microsoft Sentinel to enable SOC engineers to define entities as part of the evidence to track during an investigation. Entity mapping also makes it possible for SOC analysts to take advantage of an intuitive [investigation graph (investigate-cases.md#use-the-investigation-graph-to-deep-dive) that can help reduce time and effort.
Investigate incidents with UEBA data, as an example of how to use evidence to surface events, alerts, and any bookmarks associated with a particular incident in the incident preview pane.
Kusto Query Language (KQL), which you can use to send read-only requests to your Log Analytics database to process data and return results. KQL is also used across other Microsoft services, such as Microsoft Defender for Endpoint and Application Insights.

Compare rule terminology

This table helps you to clarify the concept of a rule in Microsoft Sentinel compared to ArcSight.

	ArcSight	Microsoft Sentinel
Rule type	• Filter rule • Join rule • Active list rule • And more	• Scheduled query • Fusion • Microsoft Security • Machine Learning (ML) Behavior Analytics
Criteria	Define in rule conditions	Define in KQL
Trigger condition	• Define in action • Define in aggregation (for event aggregation)	Threshold: Number of query results
Action	• Set event field • Send notification • Create new case • Add to active list • And more	• Create alert or incident • Integrates with Logic Apps

Map and compare rule samples

Use these samples to compare and map rules from ArcSight to Microsoft Sentinel in various scenarios.

Rule	Description	Sample detection rule (ArcSight)	Sample KQL query	Resources
Filter (`AND`)	A sample rule with `AND` conditions. The event must match all conditions.	Filter (AND) example	Filter (AND) example	String filter: • String operators Numerical filter: • Numerical operators Datetime filter: • ago • Datetime • between • now Parsing: • parse • extract • parse_json • parse_csv • parse_path • parse_url
Filter (`OR`)	A sample rule with `OR` conditions. The event can match any of the conditions.	Filter (OR) example	Filter (OR) example	• String operators • in
Nested filter	A sample rule with nested filtering conditions. The rule includes the `MatchesFilter` statement, which also includes filtering conditions.	Nested filter example	Nested filter example	• Sample KQL function • Sample parameter function • join • where
Active list (lookup)	A sample lookup rule that uses the `InActiveList` statement.	Active list (lookup) example	Active list (lookup) example	• A watchlist is the equivalent of the active list feature. Learn more about watchlists. • Other ways to implement lookups
Correlation (matching)	A sample rule that defines a condition against a set of base events, using the `Matching Event` statement.	Correlation (matching) example	Correlation (matching) example	join operator: • join • join with time window • shuffle • Broadcast • Union define statement: • let Aggregation: • make_set • make_list • make_bag • pack
Correlation (time window)	A sample rule that defines a condition against a set of base events, using the `Matching Event` statement, and uses the `Wait time` filter condition.	Correlation (time window) example	Correlation (time window) example	• join • Microsoft Sentinel rules and join statement

Filter (AND) example: ArcSight

Here's a sample filter rule with AND conditions in ArcSight.

Filter (AND) example: KQL

Here's the filter rule with AND conditions in KQL.

SecurityEvent
| where EventID == 4728
| where SubjectUserName =~ "AutoMatedService"
| where isnotempty(SubjectDomainName)

This rule assumes that Microsoft Monitoring Agent (MMA) or Azure Monitoring Agent (AMA) collect the Windows Security Events. Therefore, the rule uses the Microsoft Sentinel SecurityEvent table.

Consider these best practices:

To optimize your queries, avoid case-insensitive operators when possible: =~.
Use == if the value isn't case-sensitive.
Order the filters by starting with the where statement, which filters out the most data.

Filter (OR) example: ArcSight

Here's a sample filter rule with OR conditions in ArcSight.

Diagram illustrating a sample filter rule (or).

Filter (OR) example: KQL

Here are a few ways to write the filter rule with OR conditions in KQL.

As a first option, use the in statement:

SecurityEvent
| where SubjectUserName in
 ("Adm1","ServiceAccount1","AutomationServices")

As a second option, use the or statement:

SecurityEvent
| where SubjectUserName == "Adm1" or 
SubjectUserName == "ServiceAccount1" or 
SubjectUserName == "AutomationServices"

While both options are identical in performance, we recommend the first option, which is easier to read.

Nested filter example: ArcSight

Here's a sample nested filter rule in ArcSight.

Diagram illustrating a sample nested filter rule.

Here's a rule for the /All Filters/Soc Filters/Exclude Valid Users filter.

Diagram illustrating an Exclude Valid Users filter.

Nested filter example: KQL

Here are a few ways to write the filter rule with OR conditions in KQL.

As a first option, use a direct filter with a where statement:

SecurityEvent
| where EventID == 4728 
| where isnotempty(SubjectDomainName) or 
isnotempty(TargetDomainName) 
| where SubjectUserName !~ "AutoMatedService"

As a second option, use a KQL function:

Save the following query as a KQL function with the ExcludeValidUsers alias.

    SecurityEvent
    | where EventID == 4728
    | where isnotempty(SubjectDomainName)
    | where SubjectUserName =~ "AutoMatedService"
    | project SubjectUserName

Use the following query to filter the ExcludeValidUsers alias.

    SecurityEvent    
    | where EventID == 4728
    | where isnotempty(SubjectDomainName) or 
    isnotempty(TargetDomainName)
    | where SubjectUserName !in (ExcludeValidUsers)

As a third option, use a parameter function:

Create a parameter function with ExcludeValidUsers as the name and alias.

Define the parameters of the function. For example:

    Tbl: (TimeGenerated:datatime, Computer:string, 
    EventID:string, SubjectDomainName:string, 
    TargetDomainName:string, SubjectUserName:string)

The parameter function has the following query:

    Tbl
    | where SubjectUserName !~ "AutoMatedService"

Run the following query to invoke the parameter function:

    let Events = (
    SecurityEvent 
    | where EventID == 4728
    );
    ExcludeValidUsers(Events)

As a fourth option, use the join function:

let events = (
SecurityEvent
| where EventID == 4728
| where isnotempty(SubjectDomainName) 
or isnotempty(TargetDomainName)
);
let ExcludeValidUsers = (
SecurityEvent
| where EventID == 4728
| where isnotempty(SubjectDomainName)
| where SubjectUserName =~ "AutoMatedService"
);
events
| join kind=leftanti ExcludeValidUsers on 
$left.SubjectUserName == $right.SubjectUserName

Considerations:

We recommend that you use a direct filter with a where statement (first option) due to its simplicity. For optimized performance, avoid using join (fourth option).
To optimize your queries, avoid the =~ and !~ case-insensitive operators when possible. Use the == and != operators if the value isn't case-sensitive.

Active list (lookup) example: ArcSight

Here's an active list (lookup) rule in ArcSight.

Diagram illustrating a sample active list rule (lookup).

Active list (lookup) example: KQL

This rule assumes that the Cyber-Ark Exception Accounts watchlist exists in Microsoft Sentinel with an Account field.

let Activelist=(
_GetWatchlist('Cyber-Ark Exception Accounts')
| project Account );
CommonSecurityLog
| where DestinationUserName in (Activelist)
| where DeviceVendor == "Cyber-Ark"
| where DeviceAction == "Get File Request"
| where DeviceCustomNumber1 != ""
| project DeviceAction, DestinationUserName, 
TimeGenerated,SourceHostName, 
SourceUserName, DeviceEventClassID

Order the filters by starting with the where statement that filters out the most data.

Correlation (matching) example: ArcSight

Here's a sample ArcSight rule that defines a condition against a set of base events, using the Matching Event statement.

Diagram illustrating a sample correlation rule (matching).

Correlation (matching) example: KQL

let event1 =(
SecurityEvent
| where EventID == 4728
);
let event2 =(
SecurityEvent
| where EventID == 4729
);
event1
| join kind=inner event2 
on $left.TargetUserName==$right.TargetUserName

Best practices:

To optimize your query, ensure that the smaller table is on the left side of the join function.
If the left side of the table is relatively small (up to 100 K records), add hint.strategy=broadcast for better performance.

Correlation (time window) example: ArcSight

Here's a sample ArcSight rule that defines a condition against a set of base events, using the Matching Event statement, and uses the Wait time filter condition.

Diagram illustrating a sample correlation rule (time window).

Correlation (time window) example: KQL

let waittime = 10m;
let lookback = 1d;
let event1 = (
SecurityEvent
| where TimeGenerated > ago(waittime+lookback)
| where EventID == 4728
| project event1_time = TimeGenerated, 
event1_ID = EventID, event1_Activity= Activity, 
event1_Host = Computer, TargetUserName, 
event1_UPN=UserPrincipalName, 
AccountUsedToAdd = SubjectUserName 
);
let event2 = (
SecurityEvent
| where TimeGenerated > ago(waittime)
| where EventID == 4729
| project event2_time = TimeGenerated, 
event2_ID = EventID, event2_Activity= Activity, 
event2_Host= Computer, TargetUserName, 
event2_UPN=UserPrincipalName,
 AccountUsedToRemove = SubjectUserName 
);
 event1
| join kind=inner event2 on TargetUserName
| where event2_time - event1_time < lookback
| where tolong(event2_time - event1_time ) >=0
| project delta_time = event2_time - event1_time,
 event1_time, event2_time,
 event1_ID,event2_ID,event1_Activity,
 event2_Activity, TargetUserName, AccountUsedToAdd,
 AccountUsedToRemove,event1_Host,event2_Host, 
 event1_UPN,event2_UPN

Aggregation example: ArcSight

Here's a sample ArcSight rule with aggregation settings: three matches within 10 minutes.

Diagram illustrating a sample aggregation rule.

Aggregation example: KQL

SecurityEvent
| summarize Count = count() by SubjectUserName, 
SubjectDomainName
| where Count >3

Next steps

In this article, you learned how to map your migration rules from ArcSight to Microsoft Sentinel.

Migrate your SOAR automation