Manage Advanced Security Information Model (ASIM) parsers (Public preview)
Advanced Security Information Model (ASIM) users use unifying parsers instead of table names in their queries, to view data in a normalized format and get all the data relevant to the schema in a single query. Each unifying parser uses multiple source-specific parsers that handle each source's specific details.
To understand how parsers fit within the ASIM architecture, refer to the ASIM architecture diagram.
You may need to manage the source-specific parsers used by each unifying parser to:
Add a custom, source-specific parser to a unifying parser.
Replace a built-in, source-specific parser that's used by a unifying parser with a custom, source-specific parser. Replace built-in parsers when you want to:
Use a version of the built-in parser other than the one used by default in the unifying parser.
Prevent automated updates by preserving the version of the source-specific parser used by the unifying parser.
Use a modified version of a built-in parser.
Configure a source-specific parser, for example to define the sources that send information relevant to the parser.
This article guides you through managing your parsers, whether using built-in, unifying ASIM parsers or workspace-deployed unifying parsers.
Important
ASIM is currently in PREVIEW. The Azure Preview Supplemental Terms include additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
Prerequisites
The procedures in this article assume that all source-specific parsers have already been deployed to your Microsoft Sentinel workspace.
For more information, see Develop ASIM parsers.
Manage built-in unifying parsers
Set up your workspace
Microsoft Sentinel users cannot edit built-in unifying parsers. Instead, use the following mechanisms to modify the behavior of built-in unifying parsers:
To support adding source-specific parsers, ASIM uses unifying, custom parsers. These custom parsers are workspace-deployed, and therefore editable. Built-in, unifying parsers automatically pick up these custom parsers, if they exist.
You can deploy initial, empty, unifying custom parsers to your Microsoft Sentinel workspace for all supported schemas, or individually for specific schemas. For more information, see Deploy initial ASIM empty custom unifying parsers in the Microsoft Sentinel GitHub repository.
To support excluding built-in source-specific parsers, ASIM uses a watchlist. Deploy the watchlist to your Microsoft Sentinel workspace from the Microsoft Sentinel GitHub repository.
To define source type for built-in and custom parsers, ASIM uses a watchlist. Deploy the watchlist to your Microsoft Sentinel workspace from the Microsoft Sentinel GitHub repository.
Add a custom parser to a built-in unifying parser
To add a custom parser, insert a line to the custom unifying parser to reference the new, custom parser.
Make sure to add both a filtering custom parser and a parameter-less custom parser. To learn more about how to edit parsers, refer to the document Functions in Azure Monitor log queries.
The syntax of the line to add is different for each schema:
Schema | Parser | Line to add |
---|---|---|
DNS | Im_DnsCustom |
_parser_name_ (starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype) |
NetworkSession | Im_NetworkSessionCustom |
_parser_name_ (starttime, endtime, srcipaddr_has_any_prefix, dstipaddr_has_any_prefix, dstportnumber, hostname_has_any, dvcaction, eventresult) |
WebSession | Im_WebSessionCustom |
_parser_name_ (starttime, endtime, srcipaddr_has_any_prefix, url_has_any, httpuseragent_has_any, eventresultdetails_in, eventresult) |
When adding an additional parser to a unifying custom parser that already references parsers, make sure you add a comma at the end of the previous line.
For example, the following code shows a custom unifying parser after having added the added_parser
:
union isfuzzy=true
existing_parser(starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype),
added_parser(starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)
Use a modified version of a built-in parser
To modify an existing, built-in source-specific parser:
Create a custom parser based on the original parser and add it to the built-in parser.
Add a record to the
ASim Disabled Parsers
watchlist.Define the
CallerContext
value asExclude<parser name>
, where<parser name>
is the name of the unifying parsers you want to exclude the parser from.Define the
SourceSpecificParser
valueExclude<parser name>
, where<parser name>
is the name of the parser you want to exclude, without a version specifier.
For example, to exclude the Azure Firewall DNS parser, add the following record to the watchlist:
CallerContext | SourceSpecificParser |
---|---|
Exclude_Im_Dns |
Exclude_Im_Dns_AzureFirewall |
Prevent an automated update of a built-in parser
Use the following process to prevent automatic updates for built-in, source-specific parsers:
Add the built-in parser version you want to use, such as
_Im_Dns_AzureFirewallV02
, to the custom unifying parser. For more information, see above, Add a custom parser to a built-in unifying parser.Add an exception for the built-in parser. For example, when you want to entirely opt out from automatic updates, and therefore exclude a large number of built-in parsers, add:
- A record with
Any
as theSourceSpecificParser
field, to exclude all parsers for theCallerContext
. - A record for
Any
in the CallerContext and theSourceSpecificParser
fields to exclude all built-in parsers.
For more information, see Use a modified version of a built-in parser.
Manage workspace-deployed unifying parsers
Add a custom parser to a workspace-deployed unifying parser
To add a custom parser, insert a line to the union
statement in the workspace-deployed unifying parser that references the new custom parser.
Make sure to add both a filtering custom parser and a parameter-less custom parser. The syntax of the line to add is different for each schema:
Schema | Parser | Line to add |
---|---|---|
Authentication | ImAuthentication |
_parser_name_ (starttime, endtime, targetusername_has) |
DNS | ImDns |
_parser_name_ (starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype) |
File Event | imFileEvent |
_parser_name_ |
Network Session | imNetworkSession |
_parser_name_ (starttime, endtime, srcipaddr_has_any_prefix, dstipaddr_has_any_prefix, dstportnumber, url_has_any, httpuseragent_has_any, hostname_has_any, dvcaction, eventresult) |
Process Event | - imProcess - imProcessCreate - imProcessTerminate |
_parser_name_ |
Registry Event | imRegistry |
_parser_name_ |
Web Session | imWebSession |
_parser_name_ parser (starttime, endtime, srcipaddr_has_any, url_has_any, httpuseragent_has_any, eventresultdetails_in, eventresult) |
When adding an additional parser to a unifying parser, make sure you add a comma at the end of the previous line.
For example, the following example shows the DNS filtering unifying parser, after having added the custom added_parser
:
let Generic=(starttime:datetime=datetime(null), endtime:datetime=datetime(null) , srcipaddr:string='*' , domain_has_any:dynamic=dynamic([]) , responsecodename:string='*', response_has_ipv4:string='*' , response_has_any_prefix:dynamic=dynamic([]) , eventtype:string='lookup' ){
let DisabledParsers=materialize(_GetWatchlist('ASimDisabledParsers') | where SearchKey in ('Any', 'imDns') | extend SourceSpecificParser=column_ifexists('SourceSpecificParser','') | distinct SourceSpecificParser);
let imDnsBuiltInDisabled=toscalar('imDnsBuiltIn' in (DisabledParsers) or 'Any' in (DisabledParsers));
union isfuzzy=true
vimDnsEmpty
, vimDnsCiscoUmbrella ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsCiscoUmbrella' in (DisabledParsers) )))
, vimDnsInfobloxNIOS ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsInfobloxNIOS' in (DisabledParsers) )))
...
, vimDnsAzureFirewall ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsAzureFirewall' in (DisabledParsers) )))
, vimDnsMicrosoftNXlog ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsMicrosoftNXlog' in (DisabledParsers) ))),
added_parser ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)
};
Generic( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)
Use a modified version of a workspace-deployed parser
Microsoft Sentinel users can directly modify workspace-deployed parsers. Create a parser based on the original, comment out the original, and then add your modified version to the workspace-deployed unifying parser.
For example, the following code shows a DNS filtering unifying parser, having replaced the vimDnsAzureFirewall
parser with a modified version:
let Generic=(starttime:datetime=datetime(null), endtime:datetime=datetime(null) , srcipaddr:string='*' , domain_has_any:dynamic=dynamic([]) , responsecodename:string='*', response_has_ipv4:string='*' , response_has_any_prefix:dynamic=dynamic([]) , eventtype:string='lookup' ){
let DisabledParsers=materialize(_GetWatchlist('ASimDisabledParsers') | where SearchKey in ('Any', 'imDns') | extend SourceSpecificParser=column_ifexists('SourceSpecificParser','') | distinct SourceSpecificParser);
let imDnsBuiltInDisabled=toscalar('imDnsBuiltIn' in (DisabledParsers) or 'Any' in (DisabledParsers));
union isfuzzy=true
vimDnsEmpty
, vimDnsCiscoUmbrella ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsCiscoUmbrella' in (DisabledParsers) )))
, vimDnsInfobloxNIOS ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsInfobloxNIOS' in (DisabledParsers) )))
...
// , vimDnsAzureFirewall ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsAzureFirewall' in (DisabledParsers) )))
, vimDnsMicrosoftNXlog ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype, (imDnsBuiltInDisabled or('vimDnsMicrosoftNXlog' in (DisabledParsers) ))),
modified_vimDnsAzureFirewall ( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)
};
Generic( starttime, endtime, srcipaddr, domain_has_any, responsecodename, response_has_ipv4, response_has_any_prefix, eventtype)
Configure the sources relevant to a source-specific parser
Some parsers requires you to update the list of sources that are relevant to the parser. For example, a parser that uses Syslog data, may not be able to determine what Syslog events are relevant to the parser. Such a parser may use the Sources_by_SourceType
watchlist to determine which sources send information relevant to the parser. For such parses add a record for each relevant source to the watchlist:
- Set the
SourceType
field to the parser specific value specified in the parser documentation. - Set the
Source
field to the identifier of the source used in the events. You may need to query the original table, such as Syslog, to determine the correct value.
If you system does not have the Sources_by_SourceType
watchlist deployed, deploy the watchlist to your Microsoft Sentinel workspace from the Microsoft Sentinel GitHub repository.
Next steps
This article discusses managing the Advanced Security Information Model (ASIM) parsers.
Learn more about ASIM parsers:
Learn more about the ASIM in general: