Manage Advanced Security Information Model (ASIM) parsers (Public preview)

Advanced Security Information Model (ASIM) users use unifying parsers instead of table names in their queries, to view data in a normalized format and get all the data relevant to the schema in a single query. Each unifying parser uses multiple source-specific parsers that handle each source's specific details.

To understand how parsers fit within the ASIM architecture, refer to the ASIM architecture diagram.

You may need to manage the source-specific parsers used by each unifying parser to:

• Add a custom, source-specific parser to a unifying parser.

• Replace a built-in, source-specific parser that's used by a unifying parser with a custom, source-specific parser. Replace built-in parsers when you want to:

  • Use a version of the built-in parser other than the one used by default in the unifying parser.

  • Prevent automated updates by preserving the version of the source-specific parser used by the unifying parser.

  • Use a modified version of a built-in parser.

• Configure a source-specific parser, for example to define the sources that send information relevant to the parser.

This article guides you through managing your parsers, whether using built-in, unifying ASIM parsers or workspace-deployed unifying parsers.

Important

ASIM is currently in PREVIEW. The Azure Preview Supplemental Terms include additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Prerequisites

The procedures in this article assume that all source-specific parsers have already been deployed to your Microsoft Sentinel workspace.

For more information, see Develop ASIM parsers.

Manage built-in unifying parsers

Set up your workspace

Microsoft Sentinel users cannot edit built-in unifying parsers. Instead, use the following mechanisms to modify the behavior of built-in unifying parsers:

• To support adding source-specific parsers, ASIM uses unifying, custom parsers. These custom parsers are workspace-deployed, and therefore editable. Built-in, unifying parsers automatically pick up these custom parsers, if they exist.

  You can deploy initial, empty, unifying custom parsers to your Microsoft Sentinel workspace for all supported schemas, or individually for specific schemas. For more information, see Deploy initial ASIM empty custom unifying parsers in the Microsoft Sentinel GitHub repository.

• To support excluding built-in source-specific parsers, ASIM uses a watchlist. Deploy the watchlist to your Microsoft Sentinel workspace from the Microsoft Sentinel GitHub repository.

• To define source type for built-in and custom parsers, ASIM uses a watchlist. Deploy the watchlist to your Microsoft Sentinel workspace from the Microsoft Sentinel GitHub repository.

Add a custom parser to a built-in unifying parser

To add a custom parser, insert a line to the custom unifying parser to reference the new, custom parser.

Make sure to add both a filtering custom parser and a parameter-less custom parser. To learn more about how to edit parsers, refer to the document Functions in Azure Monitor log queries.

The syntax of the line to add is different for each schema:

Schema	Parser	Line to add
DNS	Im_DnsCustom	_parser_name_ (starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)
NetworkSession	Im_NetworkSessionCustom	_parser_name_ (starttime, endtime, srcipaddr_has_any_prefix, dstipaddr_has_any_prefix, dstportnumber, hostname_has_any, dvcaction, eventresult)
WebSession	Im_WebSessionCustom	_parser_name_ (starttime, endtime, srcipaddr_has_any_prefix, url_has_any, httpuseragent_has_any, eventresultdetails_in, eventresult)

When adding an additional parser to a unifying custom parser that already references parsers, make sure you add a comma at the end of the previous line.

For example, the following code shows a custom unifying parser after having added the added_parser:

union isfuzzy=true
existing_parser(starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype),
added_parser(starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)

Use a modified version of a built-in parser

To modify an existing, built-in source-specific parser:

1. Create a custom parser based on the original parser and add it to the built-in parser.

2. Add a record to the ASim Disabled Parsers watchlist.

3. Define the CallerContext value as Exclude<parser name>, where <parser name> is the name of the unifying parsers you want to exclude the parser from.

4. Define the SourceSpecificParser value Exclude<parser name>, where <parser name>is the name of the parser you want to exclude, without a version specifier.

For example, to exclude the Azure Firewall DNS parser, add the following record to the watchlist:

CallerContext	SourceSpecificParser
Exclude_Im_Dns	Exclude_Im_Dns_AzureFirewall

Prevent an automated update of a built-in parser

Use the following process to prevent automatic updates for built-in, source-specific parsers:

1. Add the built-in parser version you want to use, such as _Im_Dns_AzureFirewallV02, to the custom unifying parser. For more information, see above, Add a custom parser to a built-in unifying parser.

2. Add an exception for the built-in parser. For example, when you want to entirely opt out from automatic updates, and therefore exclude a large number of built-in parsers, add:

• A record with Any as the SourceSpecificParser field, to exclude all parsers for the CallerContext.
• A record for Any in the CallerContext and the SourceSpecificParser fields to exclude all built-in parsers.

For more information, see Use a modified version of a built-in parser.

Manage workspace-deployed unifying parsers

Add a custom parser to a workspace-deployed unifying parser

To add a custom parser, insert a line to the union statement in the workspace-deployed unifying parser that references the new custom parser.

Make sure to add both a filtering custom parser and a parameter-less custom parser. The syntax of the line to add is different for each schema:

Schema	Parser	Line to add
Authentication	ImAuthentication	_parser_name_ (starttime, endtime, targetusername_has)
DNS	ImDns	_parser_name_ (starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)
File Event	imFileEvent	_parser_name_
Network Session	imNetworkSession	_parser_name_ (starttime, endtime, srcipaddr_has_any_prefix, dstipaddr_has_any_prefix, dstportnumber, url_has_any, httpuseragent_has_any, hostname_has_any, dvcaction, eventresult)
Process Event	- imProcess - imProcessCreate - imProcessTerminate	_parser_name_
Registry Event	imRegistry	_parser_name_
Web Session	imWebSession	_parser_name_ parser (starttime, endtime, srcipaddr_has_any, url_has_any, httpuseragent_has_any, eventresultdetails_in, eventresult)

When adding an additional parser to a unifying parser, make sure you add a comma at the end of the previous line.

For example, the following example shows the DNS filtering unifying parser, after having added the custom added_parser:

  let Generic=(starttime:datetime=datetime(null), endtime:datetime=datetime(null) , srcipaddr:string='*' , domain_has_any:dynamic=dynamic([]) , responsecodename:string='*', response_has_ipv4:string='*' , response_has_any_prefix:dynamic=dynamic([]) , eventtype:string='lookup' ){
  let DisabledParsers=materialize(_GetWatchlist('ASimDisabledParsers') | where SearchKey in ('Any', 'imDns') | extend SourceSpecificParser=column_ifexists('SourceSpecificParser','') | distinct SourceSpecificParser);
  let imDnsBuiltInDisabled=toscalar('imDnsBuiltIn' in (DisabledParsers) or 'Any' in (DisabledParsers)); 
  union isfuzzy=true
      vimDnsEmpty
    , vimDnsCiscoUmbrella  ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsCiscoUmbrella'   in (DisabledParsers) )))
    , vimDnsInfobloxNIOS   ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsInfobloxNIOS'    in (DisabledParsers) )))
    ...
    , vimDnsAzureFirewall  ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsAzureFirewall'   in (DisabledParsers) )))
    , vimDnsMicrosoftNXlog ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsMicrosoftNXlog'  in (DisabledParsers) ))),
    added_parser ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)
     };
  Generic( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)

Use a modified version of a workspace-deployed parser

Microsoft Sentinel users can directly modify workspace-deployed parsers. Create a parser based on the original, comment out the original, and then add your modified version to the workspace-deployed unifying parser.

For example, the following code shows a DNS filtering unifying parser, having replaced the vimDnsAzureFirewall parser with a modified version:

  let Generic=(starttime:datetime=datetime(null), endtime:datetime=datetime(null) , srcipaddr:string='*' , domain_has_any:dynamic=dynamic([]) , responsecodename:string='*', response_has_ipv4:string='*' , response_has_any_prefix:dynamic=dynamic([]) , eventtype:string='lookup' ){
  let DisabledParsers=materialize(_GetWatchlist('ASimDisabledParsers') | where SearchKey in ('Any', 'imDns') | extend SourceSpecificParser=column_ifexists('SourceSpecificParser','') | distinct SourceSpecificParser);
  let imDnsBuiltInDisabled=toscalar('imDnsBuiltIn' in (DisabledParsers) or 'Any' in (DisabledParsers)); 
  union isfuzzy=true
      vimDnsEmpty
    , vimDnsCiscoUmbrella  ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsCiscoUmbrella'   in (DisabledParsers) )))
    , vimDnsInfobloxNIOS   ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsInfobloxNIOS'    in (DisabledParsers) )))
    ...
    // , vimDnsAzureFirewall  ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsAzureFirewall'   in (DisabledParsers) )))
    , vimDnsMicrosoftNXlog ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsMicrosoftNXlog'  in (DisabledParsers) ))),
    modified_vimDnsAzureFirewall ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)
     };
  Generic( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)

Configure the sources relevant to a source-specific parser

Some parsers requires you to update the list of sources that are relevant to the parser. For example, a parser that uses Syslog data, may not be able to determine what Syslog events are relevant to the parser. Such a parser may use the Sources_by_SourceType watchlist to determine which sources send information relevant to the parser. For such parses add a record for each relevant source to the watchlist:

• Set the SourceType field to the parser specific value specified in the parser documentation.
• Set the Source field to the identifier of the source used in the events. You may need to query the original table, such as Syslog, to determine the correct value.

If you system does not have the Sources_by_SourceType watchlist deployed, deploy the watchlist to your Microsoft Sentinel workspace from the Microsoft Sentinel GitHub repository.

Next steps

This article discusses managing the Advanced Security Information Model (ASIM) parsers.

Learn more about ASIM parsers:

• ASIM parsers overview
• Use ASIM parsers
• Develop custom ASIM parsers
• The ASIM parsers list

Learn more about the ASIM in general:

• Watch the Deep Dive Webinar on Microsoft Sentinel Normalizing Parsers and Normalized Content or review the slides
• Advanced Security Information Model (ASIM) overview
• Advanced Security Information Model (ASIM) schemas
• Advanced Security Information Model (ASIM) content