Hunt for threats with Microsoft Sentinel
As security analysts and investigators, you want to be proactive about looking for security threats, but your various systems and security appliances generate mountains of data that can be difficult to parse and filter into meaningful events. Microsoft Sentinel has powerful hunting search and query tools to hunt for security threats across your organization's data sources. To help security analysts look proactively for new anomalies that weren't detected by your security apps or even by your scheduled analytics rules, Microsoft Sentinel's built-in hunting queries guide you into asking the right questions to find issues in the data you already have on your network.
For example, one built-in query provides data about the most uncommon processes running on your infrastructure. You wouldn't want an alert about each time they are run - they could be entirely innocent - but you might want to take a look at the query on occasion to see if there's anything unusual.
For information about feature availability in US Government clouds, see the Microsoft Sentinel tables in Cloud feature availability for US Government customers.
Use built-in queries
The hunting dashboard provides ready-made query examples designed to get you started and get you familiar with the tables and the query language. Queries run on data stored in log tables, such as for process creation, DNS events, or other event types.
Built-in hunting queries are developed by Microsoft security researchers on a continuous basis, both adding new queries and fine-tuning existing queries to provide you with an entry point to look for new detections and figure out where to start hunting for the beginnings of new attacks.
Use queries before, during, and after a compromise to take the following actions:
Before an incident occurs: Waiting on detections is not enough. Take proactive action by running any threat-hunting queries related to the data you're ingesting into your workspace at least once a week.
Results from your proactive hunting provide early insight into events that may confirm that a compromise is in process, or will at least show weaker areas in your environment that are at risk and need attention.
During a compromise: Use livestream to run a specific query constantly, presenting results as they come in. Use livestream when you need to actively monitor user events, such as if you need to verify whether a specific compromise is still taking place, to help determine a threat actor's next action, and towards the end of an investigation to confirm that the compromise is indeed over.
After a compromise: After a compromise or an incident has occurred, make sure to improve your coverage and insight to prevent similar incidents in the future.
Modify your existing queries or create new ones to assist with early detection, based on insights you've gained from your compromise or incident.
If you've discovered or created a hunting query that provides high value insights into possible attacks, create custom detection rules based on that query and surface those insights as alerts to your security incident responders.
View the query's results, and select New alert rule > Create Microsoft Sentinel alert. Use the Analytics rule wizard to create a new rule based on your query. For more information, see Create custom analytics rules to detect threats.
You can also create hunting and livestream queries over data stored in Azure Data Explorer. For more information, see details of constructing cross-resource queries in the Azure Monitor documentation.
Use community resources, such as the Microsoft Sentinel GitHub repository to find additional queries and data sources.
Use the hunting dashboard
The hunting dashboard enables you to run all your queries, or a selected subset, in a single selection. In the Microsoft Sentinel portal, select Hunting.
The table shown lists all the queries written by Microsoft's team of security analysts and any extra query you created or modified. Each query provides a description of what it hunts for, and what kind of data it runs on. These queries are grouped by their MITRE ATT&CK tactics. The icons on the right categorize the type of threat, such as initial access, persistence, and exfiltration. MITRE ATT&CK techniques are shown in the Techniques column and describe the specific behavior identified by the hunting query.
Use the hunting dashboard to identify where to start hunting, by looking at result count, spikes, or the change in result count over a 24-hour period. Sort and filter by favorites, data source, MITRE ATT&CK tactic or technique, results, results delta, or results delta percentage. View queries that still need data sources connected, and get recommendations on how to enable these queries.
The following table describes detailed actions available from the hunting dashboard:
|See how queries apply to your environment||Select the Run all queries button, or select a subset of queries using the check boxes to the left of each row and select the Run selected queries button.
Running your queries can take anywhere from a few seconds to many minutes, depending on how many queries are selected, the time range, and the amount of data that is being queried.
|View the queries that returned results||After your queries are done running, view the queries that returned results using the Results filter:
- Sort to see which queries had the most or fewest results.
- View the queries that are not at all active in your environment by selecting N/A in the Results filter.
- Hover over the info icon (i) next to the N/A to see which data sources are required to make this query active.
|Identify spikes in your data||Identify spikes in the data by sorting or filtering on Results delta or Results delta percentage.
This compares the results of the last 24 hours against the results of the previous 24-48 hours, highlighting any large differences or relative difference in volume.
|View queries mapped to the MITRE ATT&CK tactic||The MITRE ATT&CK tactic bar, at the top of the table, lists how many queries are mapped to each MITRE ATT&CK tactic. The tactic bar gets dynamically updated based on the current set of filters applied.
This enables you to see which MITRE ATT&CK tactics show up when you filter by a given result count, a high result delta, N/A results, or any other set of filters.
|View queries mapped to MITRE ATT&CK techniques||Queries can also be mapped to MITRE ATT&CK techniques. You can filter or sort by MITRE ATT&CK techniques using the Technique filter. By opening a query, you will be able to select the technique to see the MITRE ATT&CK description of the technique.|
|Save a query to your favorites||Queries saved to your favorites automatically run each time the Hunting page is accessed. You can create your own hunting query or clone and customize an existing hunting query template.|
|Run queries||Select Run Query in the hunting query details page to run the query directly from the hunting page. The number of matches is displayed within the table, in the Results column. Review the list of hunting queries and their matches.|
|Review an underlying query||Perform a quick review of the underlying query in the query details pane. You can see the results by clicking the View query results link (below the query window) or the View Results button (at the bottom of the pane). The query will open in the Logs (Log Analytics) blade, and below the query, you can review the matches for the query.|
Create a custom hunting query
Create or modify a query and save it as your own query or share it with users who are in the same tenant.
To create a new query:
Select New query.
Fill in all the blank fields and select Create.
To clone and modify an existing query:
From the table, select the hunting query you want to modify.
Select the ellipsis (...) in the line of the query you want to modify, and select Clone query.
Modify the query and select Create.
To modify an existing custom query:
From the table, select the hunting query that you wish to modify. Note that only queries that from a custom content source can be edited. Other content sources have to be edited at that source.
Select the ellipsis (...) in the line of the query you want to modify, and select Edit query.
Modify the Custom query field with the updated query. You can also modify the entity mapping and techniques as explained in the "To create a new query" section of this documentation.
A typical query starts with a table or parser name followed by a series of operators separated by a pipe character ("|").
In the example above, start with the table name SecurityEvent and add piped elements as needed.
Define a time filter to review only records from the previous seven days.
Add a filter in the query to only show event ID 4688.
Add a filter in the query on the command line to contain only instances of cscript.exe.
Project only the columns you're interested in exploring and limit the results to 1000 and select Run query.
Select the green triangle and run the query. You can test the query and run it to look for anomalous behavior.
We recommend that your query uses an Advanced Security Information Model (ASIM) parser and not a built-in table. This ensures that the query will support any current or future relevant data source rather than a single data source.
During the hunting and investigation process, you may come across query results that may look unusual or suspicious. Bookmark these items to refer back to them in the future, such as when creating or enriching an incident for investigation. Events such as potential root causes, indicators of compromise, or other notable events should be raised as a bookmark. If a key event you've bookmarked is severe enough to warrant an investigation, escalate it to an incident.
In your results, mark the checkboxes for any rows you want to preserve, and select Add bookmark. This creates for a record for each marked row - a bookmark - that contains the row results as well as the query that created the results. You can add your own tags and notes to each bookmark.
- As with scheduled analytics rules, you can enrich your bookmarks with entity mappings to extract multiple entity types and identifiers, and MITRE ATT&CK mappings to associate particular tactics and techniques.
- Bookmarks will default to use the same entity and MITRE ATT&CK technique mappings as the hunting query that produced the bookmarked results.
View all the bookmarked findings by clicking on the Bookmarks tab in the main Hunting page. Add tags to bookmarks to classify them for filtering. For example, if you're investigating an attack campaign, you can create a tag for the campaign, apply the tag to any relevant bookmarks, and then filter all the bookmarks based on the campaign.
Investigate a single bookmarked finding by selecting the bookmark and then clicking Investigate in the details pane to open the investigation experience. You can also directly select a listed entity to view that entity’s corresponding entity page.
You can also create an incident from one or more bookmarks, or add one or more bookmarks to an existing incident. Select a checkbox to the left of any bookmarks you want to use, and then select Incident actions > Create new incident or Add to existing incident. Triage and investigate the incident like any other.
For more information, see Use bookmarks in hunting.
Use notebooks to power investigations
When your hunting and investigations become more complex, use Microsoft Sentinel notebooks to enhance your activity with machine learning, visualizations, and data analysis.
Notebooks provide a kind of virtual sandbox, complete with its own kernel, where you can carry out a complete investigation. Your notebook can include the raw data, the code you run on that data, the results, and their visualizations. Save your notebooks so that you can share it with others to reuse in your organization.
Notebooks may be helpful when your hunting or investigation becomes too large to remember easily, view details, or when you need to save queries and results. To help you create and share notebooks, Microsoft Sentinel provides Jupyter Notebooks, an open-source, interactive development and data manipulation environment, integrated directly in the Microsoft Sentinel Notebooks page.
For more information, see:
- Use Jupyter Notebook to hunt for security threats
- The Jupyter Project documentation
- Jupyter introductory documentation.
- The Infosec Jupyter Book
- Real Python tutorials
The following table describes some methods of using Jupyter notebooks to help your processes in Microsoft Sentinel:
|Data persistence, repeatability, and backtracking||If you're working with many queries and results sets, you're likely to have some dead ends. You'll need to decide which queries and results to keep, and how to accumulate the useful results in a single report.
Use Jupyter Notebooks to save queries and data as you go, use variables to rerun queries with different values or dates, or save your queries to rerun on future investigations.
|Scripting and programming||Use Jupyter Notebooks to add programming to your queries, including:
- Declarative languages like Kusto Query Language (KQL) or SQL, to encode your logic in a single, possibly complex, statement.
- Procedural programming languages, to run logic in a series of steps.
Splitting your logic into steps can help you see and debug intermediate results, add functionality that might not be available in the query language, and reuse partial results in later processing steps.
|Links to external data||While Microsoft Sentinel tables have most telemetry and event data, Jupyter Notebooks can link to any data that's accessible over your network or from a file. Using Jupyter Notebooks allows you to include data such as:
- Data in external services that you don't own, such as geolocation data or threat intelligence sources
- Sensitive data that's stored only within your organization, such as human resource databases or lists of high-value assets
- Data that you haven't yet migrated to the cloud.
|Specialized data processing, machine learning, and visualization tools||Jupyter Notebooks provides additional visualizations, machine learning libraries, and data processing and transformation features.
For example, use Jupyter Notebooks with the following Python capabilities:
- pandas for data processing, cleanup, and engineering
- Matplotlib, HoloViews, and Plotly for visualization
- NumPy and SciPy for advanced numerical and scientific processing
- scikit-learn for machine learning
- TensorFlow, PyTorch, and Keras for deep learning
MSTIC, Jupyter, and Python security tools
The Microsoft Threat Intelligence Center (MSTIC) is a team of Microsoft security analysts and engineers who author security detections for several Microsoft platforms and work on threat identification and investigation.
MSTIC built MSTICPy, a library for information security investigations and hunting in Jupyter Notebooks. MSTICPy provides reusable functionality that aims to speed up notebook creation, and make it easier for users to read notebooks in Microsoft Sentinel.
For example, MSTICPy can:
- Query log data from multiple sources.
- Enrich the data with threat intelligence, geolocations, and Azure resource data.
- Extract Indicators of Activity (IoA) from logs, and unpack encoded data.
- Do sophisticated analyses such as anomalous session detection and time series decomposition.
- Visualize data using interactive timelines, process trees, and multi-dimensional Morph Charts.
MSTICPy also includes some time-saving notebook tools, such as widgets that set query time boundaries, select and display items from lists, and configure the notebook environment.
For more information, see:
- MSTICPy documentation
- Tutorial: Get started with Jupyter notebooks and MSTICPy in Microsoft Sentinel
- Advanced configurations for Jupyter notebooks and MSTICPy in Microsoft Sentinel
Useful operators and functions
Hunting queries are built in Kusto Query Language (KQL), a powerful query language with IntelliSense language that gives you the power and flexibility you need to take hunting to the next level.
It's the same language used by the queries in your analytics rules and elsewhere in Microsoft Sentinel. For more information, see Query Language Reference.
The following operators are especially helpful in Microsoft Sentinel hunting queries:
where - Filter a table to the subset of rows that satisfy a predicate.
summarize - Produce a table that aggregates the content of the input table.
join - Merge the rows of two tables to form a new table by matching values of the specified column(s) from each table.
count - Return the number of records in the input record set.
top - Return the first N records sorted by the specified columns.
limit - Return up to the specified number of rows.
project - Select the columns to include, rename or drop, and insert new computed columns.
extend - Create calculated columns and append them to the result set.
makeset - Return a dynamic (JSON) array of the set of distinct values that Expr takes in the group
find - Find rows that match a predicate across a set of tables.
adx() - This function performs cross-resource queries of Azure Data Explorer data sources from the Microsoft Sentinel hunting experience and Log Analytics. For more information, see Cross-resource query Azure Data Explorer by using Azure Monitor.
In this article, you learned how to run a hunting investigation with Microsoft Sentinel.
For more information, see:
- Use notebooks to run automated hunting campaigns
- Use bookmarks to save interesting information while hunting