diffpatterns_text plugin

Applies to: ✅ Microsoft Fabric ✅ Azure Data Explorer

Compares two datasets of string values and finds text patterns that characterize differences between the two datasets. The plugin is invoked with the evaluate operator.

The diffpatterns_text returns a set of text patterns that capture different portions of the data in the two sets. For example, a pattern capturing a large percentage of the rows when the condition is true and low percentage of the rows when the condition is false. The patterns are built from consecutive tokens separated by white space, with a token from the text column or a * representing a wildcard. Each pattern is represented by a row in the results.

Syntax

T | evaluate diffpatterns_text(TextColumn, BooleanCondition [, MinTokens, Threshold , MaxTokens])

Learn more about syntax conventions.

Parameters

Name	Type	Required	Description
TextColumn	string	✔️	The text column to analyze.
BooleanCondition	string	✔️	An expression that evaluates to a boolean value. The algorithm splits the query into the two datasets to compare based on this expression.
MinTokens	int		An integer value between 0 and 200 that represents the minimal number of non-wildcard tokens per result pattern. The default is 1.
Threshold	decimal		A decimal value between 0.015 and 1 that sets the minimal pattern ratio difference between the two sets. Default is 0.05. See diffpatterns.
MaxTokens	int		An integer value between 0 and 20 that sets the maximal number of tokens per result pattern, specifying a lower limit decreases the query runtime.

Returns

The result of diffpatterns_text returns the following columns:

• Count_of_True: The number of rows matching the pattern when the condition is true.
• Count_of_False: The number of rows matching the pattern when the condition is false.
• Percent_of_True: The percentage of rows matching the pattern from the rows when the condition is true.
• Percent_of_False: The percentage of rows matching the pattern from the rows when the condition is false.
• Pattern: The text pattern containing tokens from the text string and '*' for wildcards.

Note

The patterns aren't necessarily distinct and may not provide full coverage of the dataset. The patterns may be overlapping and some rows may not match any pattern.

Example

The examples in this tutorial use the StormEvents table, which is publicly available in the Weather analytics sample data.

StormEvents     
| where EventNarrative != "" and monthofyear(StartTime) > 1 and monthofyear(StartTime) < 9
| where EventType == "Drought" or EventType == "Extreme Cold/Wind Chill"
| evaluate diffpatterns_text(EpisodeNarrative, EventType == "Extreme Cold/Wind Chill", 2)

Output

Count_of_True	Count_of_False	Percent_of_True	Percent_of_False	Pattern
11	0	6.29	0	Winds shifting northwest in * wake * a surface trough brought heavy lake effect snowfall downwind * Lake Superior from
9	0	5.14	0	Canadian high pressure settled * * region * produced the coldest temperatures since February * 2006. Durations * freezing temperatures
0	34	0	6.24	* * * * * * * * * * * * * * * * * * West Tennessee,
0	42	0	7.71	* * * * * * caused * * * * * * * * across western Colorado. *
0	45	0	8.26	* * below normal *
0	110	0	20.18	Below normal *