Jupyter notebooks with Microsoft Sentinel hunting capabilities

• Applies to:

  Microsoft Sentinel in the Azure portal, Microsoft Sentinel in the Microsoft Defender portal

Jupyter notebooks combine full programmability with a huge collection of libraries for machine learning, visualization, and data analysis. These attributes make Jupyter a compelling tool for security investigation and hunting.

The foundation of Microsoft Sentinel is the data store; it combines high-performance querying, dynamic schema, and scales to massive data volumes. The Azure portal and all Microsoft Sentinel tools use a common API to access this data store. The same API is also available for external tools such as Jupyter notebooks and Python.

Important

Microsoft Sentinel is available as part of the public preview for the unified security operations platform in the Microsoft Defender portal. For more information, see Microsoft Sentinel in the Microsoft Defender portal.

When to use Jupyter notebooks

While many common tasks can be carried out in the portal, Jupyter extends the scope of what you can do with this data.

For example, use notebooks to:

• Perform analytics that aren't provided out-of-the-box in Microsoft Sentinel, such as some Python machine learning features
• Create data visualizations that aren't provided out-of-the-box in Microsoft Sentinel, such as custom timelines and process trees
• Integrate data sources outside of Microsoft Sentinel, such as an on-premises data set.

We integrated the Jupyter experience into the Azure portal, making it easy for you to create and run notebooks to analyze your data. The Kqlmagic library provides the glue that lets you take Kusto Query Language (KQL) queries from Microsoft Sentinel and run them directly inside a notebook.

Several notebooks, developed by some of Microsoft's security analysts, are packaged with Microsoft Sentinel:

• Some of these notebooks are built for a specific scenario and can be used as-is.
• Others are intended as samples to illustrate techniques and features that you can copy or adapt for use in your own notebooks.

Import other notebooks from the Microsoft Sentinel GitHub repository.

How Jupyter notebooks work

Notebooks have two components:

• The browser-based interface, where you enter and run queries and code, and where the results of the execution are displayed.
• A kernel that is responsible for parsing and executing the code itself.

The Microsoft Sentinel notebook's kernel runs on an Azure virtual machine (VM). The VM instance can support running many notebooks at once. If your notebooks include complex machine learning models, several licensing options exist to use more powerful virtual machines.

Understand Python packages

The Microsoft Sentinel notebooks use many popular Python libraries such as pandas, matplotlib, bokeh, and others. There are a great many other Python packages for you to choose from, covering areas such as:

• Visualizations and graphics
• Data processing and analysis
• Statistics and numerical computing
• Machine learning and deep learning

To avoid having to type or paste complex and repetitive code into notebook cells, most Python notebooks rely on third-party libraries called packages. To use a package in a notebook, you need to both install and import the package. Azure Machine Learning Compute has most common packages pre-installed. Make sure that you import the package, or the relevant part of the package, such as a module, file, function, or class.

Microsoft Sentinel notebooks use a Python package called MSTICPy, which is a collection of cybersecurity tools for data retrieval, analysis, enrichment, and visualization.

MSTICPy tools are designed specifically to help with creating notebooks for hunting and investigation and we're actively working on new features and improvements. For more information, see:

• MSTIC Jupyter and Python Security Tools documentation
• Get started with Jupyter notebooks and MSTICPy in Microsoft Sentinel
• Advanced configurations for Jupyter notebooks and MSTICPy in Microsoft Sentinel

Find notebooks

In Microsoft Sentinel, select Notebooks to see notebooks that Microsoft Sentinel provides. Learn more about using notebooks in threat hunting and investigation by exploring notebook templates like Credential Scan on Azure Log Analytics and Guided Investigation - Process Alerts.

For more notebooks built by Microsoft or contributed from the community, go to Microsoft Sentinel GitHub repository. Use notebooks shared in the Microsoft Sentinel GitHub repository as useful tools, illustrations, and code samples that you can use when developing your own notebooks.

• The Sample-Notebooks directory includes sample notebooks that are saved with data that you can use to show intended output.

• The HowTos directory includes notebooks that describe concepts such as setting your default Python version, creating Microsoft Sentinel bookmarks from a notebook, and more.

Manage access to Microsoft Sentinel notebooks

To use Jupyter notebooks in Microsoft Sentinel, you must first have the right permissions, depending on your user role.

While you can run Microsoft Sentinel notebooks in JupyterLab or Jupyter classic, in Microsoft Sentinel, notebooks are run on an Azure Machine Learning platform. To run notebooks in Microsoft Sentinel, you must have appropriate access to both Microsoft Sentinel workspace and an Azure Machine Learning workspace.

Permission	Description
Microsoft Sentinel permissions	Like other Microsoft Sentinel resources, to access notebooks in Microsoft Sentinel, a Microsoft Sentinel Reader, Microsoft Sentinel Responder, or Microsoft Sentinel Contributor role is required. For more information, see Permissions in Microsoft Sentinel.
Azure Machine Learning permissions	An Azure Machine Learning workspace is an Azure resource. Like other Azure resources, when a new Azure Machine Learning workspace is created, it comes with default roles. You can add users to the workspace and assign them to one of these built-in roles. For more information, see Azure Machine Learning default roles and Azure built-in roles. Important: Role access can be scoped to multiple levels in Azure. For example, someone with owner access to a workspace might not have owner access to the resource group that contains the workspace. For more information, see How Azure RBAC works. If you're an owner of an Azure Machine Learning workspace, you can add and remove roles for the workspace and assign roles to users. For more information, see: - Azure portal - PowerShell - Azure CLI - REST API - Azure Resource Manager templates - Azure Machine Learning CLI If the built-in roles are insufficient, you can also create custom roles. Custom roles might have read, write, delete, and compute resource permissions in that workspace. You can make the role available at a specific workspace level, a specific resource group level, or a specific subscription level. For more information, see Create custom role.

Submit feedback for a notebook

Submit feedback, requests for features, bug reports, or improvements to existing notebooks. Go to the Microsoft Sentinel GitHub repository to create an issue, or fork and upload a contribution.

Related content

• Hunt for security threats with Jupyter notebooks
• Get started with Jupyter notebooks and MSTICPy in Microsoft Sentinel
• Proactively hunt for threats
• Keep track of data during hunting with Microsoft Sentinel

For blogs, videos, and other resources, see:

• Create your first Microsoft Sentinel notebook (Blog series)
• Tutorial: Microsoft Sentinel notebooks - Getting started (Video)
• Tutorial: Edit and run Jupyter notebooks without leaving Azure Machine Learning studio (Video)
• Detect Credential Leaks using Azure Sentinel Notebooks (Video)
• Webinar: Microsoft Sentinel notebooks fundamentals (Video)
• Jupyter, msticpy, and Microsoft Sentinel