Parsing multiline data in Sentinel

Question

Parsing multiline data in Sentinel

Gaurav Ramteke 1

Hi,

We are getting data into LogAnalytics in following format;
One RaW logs has multiple lines in the log file and they repeat for each operations performed in the same log file

Reverse date time = 487930544  
Sequence number = 2147488705  
Component = BSS  
Security event = FALSE  
Event number = 2177  
Event name = Export Configuration Completed  
Event class = Software  
Event severity = Info  
Alarm event = FALSE  
Server name = System Management  
Function Name =   
Hostname = servername  
Browser Hostname =   
Browser Address =   
Operator = software_owner  
Session ID = a/dXlasjndlnojqpjorn[wkeokjwpeownokansdlk  
Date time = 03/08/22 22:54:08  
Alarm status = None  
Alarm distribution status = test_DIST_N_A  
Alarm date time = 01/01/70 00:00:00  
Alarm operator =   
Configuration change = FALSE  
Display text = Export Successfully Completed for entities: Operator. Total number of occurrence  
Length = 95  
Record version = 0  
Merged text = Export Successfully Completed for entities: Operator. Total number of occurrences exported: 24

In the similar manner the lines will repeat itself for each of the activity in the same file

How do we collate the above lines as one entry for a log
Break the lines before "Reverse date time"
Parse it as a Key = Value pairs meaning Rows and Colums

2 answers

Your answer

Answer 1

Hi @Gaurav Ramteke

I have two ways I can achieve this (I am sure I can do a few more, with things building the dynamic type by replacing strings). Both my examples provide two sets of events within a single record, and therefore using an arbitrary "Event Id" to denote the difference records. You may have different ids you wish to use to identify unique records. I've changed the reverse date time and sequence numbers to identify different activities within the same record.

First way is by parsing the keys value pairs using the parse operator. This makes the assumption that all the keys are static and will always be provided. It allows the data to be typed easily while parsing. the \n forces the parse to check at the start of a new line, though may not be needed.

print source =   
   
   Reverse date time = 487930544  
       Sequence number = 2147488705  
       Component = BSS  
       Security event = FALSE  
       Event number = 2177  
       Event name = Export Configuration Completed  
       Event class = Software  
       Event severity = Info  
       Alarm event = FALSE  
       Server name = System Management  
       Function Name =   
       Hostname = servername  
       Browser Hostname =   
       Browser Address =   
       Operator = software_owner  
       Session ID = a/dXlasjndlnojqpjorn[wkeokjwpeownokansdlk  
       Date time = 03/08/22 22:54:08  
       Alarm status = None  
       Alarm distribution status = test_DIST_N_A  
       Alarm date time = 01/01/70 00:00:00  
       Alarm operator =   
       Configuration change = FALSE  
       Display text = Export Successfully Completed for entities: Operator. Total number of occurrence  
       Length = 95  
       Record version = 0  
       Merged text = Export Successfully Completed for entities: Operator. Total number of occurrences exported: 24  
       Reverse date time = 487930545  
       Sequence number = 2147488706  
       Component = BSS  
       Security event = FALSE  
       Event number = 2177  
       Event name = Export Configuration Completed  
       Event class = Software  
       Event severity = Info  
       Alarm event = FALSE  
       Server name = System Management  
       Function Name =   
       Hostname = servername  
       Browser Hostname =   
       Browser Address =   
       Operator = software_owner  
       Session ID = a/dXlasjndlnojqpjorn[wkeokjwpeownokansdlk  
       Date time = 03/08/22 22:54:08  
       Alarm status = None  
       Alarm distribution status = test_DIST_N_A  
       Alarm date time = 01/01/70 00:00:00  
       Alarm operator =   
       Configuration change = FALSE  
       Display text = Export Successfully Completed for entities: Operator. Total number of occurrence  
       Length = 95  
       Record version = 0  
       Merged text = Export Successfully Completed for entities: Operator. Total number of occurrences exported: 24  

| extend source = split(source, "Reverse date time = ")  
| mv-expand source to typeof(string)   
| where isnotempty(source)  
| extend source = strcat("Reverse date time = ", source)  
| parse source with  "Reverse date time = " ['Reverse date time']:long   
"\nSequence number = " ['Sequence number']:long  
 "\nComponent = " Component:string  
 "\nSecurity event = " ['Security event']:bool  
 "\nEvent number = " ['Event number']:int  
 "\nEvent name = " ['Event name']:string  
 "\nEvent class = " ['Event class']:string  
 "\nEvent severity = " ['Event severity']:string  
 "\nAlarm event = " ['Alarm event']:bool  
 "\nServer name = " ['Server name']:string  
 "\nFunction Name = " ['Function Name']:string  
 "\nHostname = " Hostname:string  
 "\nBrowser Hostname = " ['Browser Hostname']:string  
 "\nBrowser Address = " ['Browser Address']:string  
 "\nOperator = " Operator:string  
 "\nSession ID = " ['Session ID']:string  
 "\nDate time = " ['Date time']:datetime   
 "\nAlarm status = " ['Alarm status']:string  
 "\nAlarm distribution status = " ['Alarm distribution status']:string  
 "\nAlarm date time = " ['Alarm date time']:datetime   
 "\nAlarm operator = " ['Alarm operator']:string  
 "\nConfiguration change = " ['Configuration change']:bool  
 "\nDisplay text = " ['Display text']:string  
 "\nLength = " Length:int  
 "\nRecord version = " ['Record version']:int  
 "\nMerged text = " ['Merged text']:string  
| project-away source

GauravRamteke 1 Reputation point

2022-08-08T12:16:05.6+00:00

Thank you | AlistairRoss-msft,

I think both examples does not fit to my scenario because,
As you ingest the logs from the file we need to select the line break delimiter to ingest the logs now;
When you search for the logs - you could see each log contains only one line from the sample that i have submitted and not all the lines as a log
I hope you understand and if not let me attach the snippet for your reference
Alistair Ross 7,466 Reputation points Microsoft Employee

2022-08-08T12:57:51.313+00:00

Hi @GauravRamteke

Thanks for sharing the screenshot, this makes sense now. I have a few questions

How are you ingesting these logs into Sentinel? I am assuming Log Analytics custom logs with the RawData Column Name

Can you configure the logs to start with one of the following formats. This would simplify collection

YYYY-MM-DD HH:MM:SS

M/D/YYYY HH:MM:SS AM/PM

Mon DD, YYYY HH:MM:SS

yyMMdd HH:mm:ss

ddMMyy HH:mm:ss

MMM d hh:mm:ss

dd/MMM/yyyy:HH:mm:ss zzz

yyyy-MM-ddTHH:mm:ssK

What we need to do is pack the rows into a single object with a unique identifier, the issue here is because your logs are multi line, and are separated by the line, not a date time stamp, it makes it much harder (regardless of the SIEM tool used). The only unique identifier across the rows is the TimeGenerated column (based on your snippet), but if that time stamp is the same across multiple records, then we run the high likelihood of merging event data with another.
Alistair Ross 7,466 Reputation points Microsoft Employee

2022-08-08T12:58:09.883+00:00

If the TimeGenerated field is unique for the one record, then you can use this

datatable (TimeGenerated:datetime , RawData:string)[ datetime('2022-08-08T12:42:58.753447Z'), "Reverse date time = 487930544", datetime('2022-08-08T12:42:58.753447Z'), "Sequence number = 2147488705", datetime('2022-08-08T12:42:58.753447Z'), "Component = BSS", datetime('2022-08-08T12:44:58.753447Z'), "Reverse date time = 487930546", datetime('2022-08-08T12:44:58.753447Z'), "Sequence number = 2147488706", datetime('2022-08-08T12:44:58.753447Z'), "Component = BSS", ] | extend RawData = split(RawData," = ") | extend RawData = pack(tostring(RawData[0]),RawData[1]) | summarize RawData = make_bag(RawData) by TimeGenerated //, Computer | evaluate bag_unpack(RawData)

Otherwise you will need to either reformat the logs or parse the logs client side, using Fluentd or a custom solution. The problem you have is if the logs are received into log analytics out of order or there is latency between records, the individual lines for the activity may not get aggregated into the single record.
GauravRamteke 1 Reputation point

2022-08-09T05:47:04.883+00:00

Thanks AlistairRoss

Yes, you are right I have tried doing the summarization on Time Generated but it does not work as required
secondly, we do not have option to bag all those lines before Reverse Date Time option using regex or something else

Regards
Gaurav

Answer 2

The second method (assuming the first answer has made it out of review, otherwise this is the first) makes the assumption that the keys may change and therefore you need to convert it to a JSON object and then unpack the object.

print EventId = 1, source =   
   
   Reverse date time = 487930544  
        Sequence number = 2147488705  
        Component = BSS  
        Security event = FALSE  
        Event number = 2177  
        Event name = Export Configuration Completed  
        Event class = Software  
        Event severity = Info  
        Alarm event = FALSE  
        Server name = System Management  
        Function Name =   
        Hostname = servername  
        Browser Hostname =   
        Browser Address =   
        Operator = software_owner  
        Session ID = a/dXlasjndlnojqpjorn[wkeokjwpeownokansdlk  
        Date time = 03/08/22 22:54:08  
        Alarm status = None  
        Alarm distribution status = test_DIST_N_A  
        Alarm date time = 01/01/70 00:00:00  
        Alarm operator =   
        Configuration change = FALSE  
        Display text = Export Successfully Completed for entities: Operator. Total number of occurrence  
        Length = 95  
        Record version = 0  
        Merged text = Export Successfully Completed for entities: Operator. Total number of occurrences exported: 24  
        Reverse date time = 487930545  
        Sequence number = 2147488706  
        Component = BSS  
        Security event = FALSE  
        Event number = 2177  
        Event name = Export Configuration Completed  
        Event class = Software  
        Event severity = Info  
        Alarm event = FALSE  
        Server name = System Management  
        Function Name =   
        Hostname = servername  
        Browser Hostname =   
        Browser Address =   
        Operator = software_owner  
        Session ID = a/dXlasjndlnojqpjorn[wkeokjwpeownokansdlk  
        Date time = 03/08/22 22:54:08  
        Alarm status = None  
        Alarm distribution status = test_DIST_N_A  
        Alarm date time = 01/01/70 00:00:00  
        Alarm operator =   
        Configuration change = FALSE  
        Display text = Export Successfully Completed for entities: Operator. Total number of occurrence  
        Length = 95  
        Record version = 0  
        Merged text = Export Successfully Completed for entities: Operator. Total number of occurrences exported: 24  

| extend source = split(source, "Reverse date time = ")  
| mv-expand with_itemindex=EventIdIndex source to typeof(string)   
| where isnotempty(source)  
| extend source = strcat("Reverse date time = ", source)  
| extend source = split(source,"\n")  
| mv-expand source to typeof(string)  
| extend source = split(source," = ")  
| extend source = pack(tostring(source[0]),source[1])  
| summarize source = make_bag(source) by EventIdIndex, EventId  
| evaluate bag_unpack(source)

Share via

Parsing multiline data in Sentinel

2 answers

Your answer