Breyta

Deila með


Advanced configurations for Jupyter notebooks and MSTICPy in Microsoft Sentinel

This article describes advanced configurations for working with Jupyter notebooks and MSTICPy in Microsoft Sentinel.

For more information, see Use Jupyter notebooks to hunt for security threats and Tutorial: Get started with Jupyter notebooks and MSTICPy in Microsoft Sentinel.

Prerequisites

This article is a continuation on from Tutorial: Get started with Jupyter notebooks and MSTICPy in Microsoft Sentinel. We recommend that you perform the tutorial before continuing with the advanced procedures described below.

Specify authentication parameters for Azure and Microsoft Sentinel APIs

This procedure describes how to configure authentication parameters for Microsoft Sentinel and other Azure API resources in your msticpyconfig.yaml file.

To add Azure authentication and Microsoft Sentinel API settings in the MSTICPy settings editor:

  1. Proceed to the next cell, with the following code, and run it:

    mpedit.set_tab("Data Providers")
    mpedit
    
  2. In the Data Providers tab, select AzureCLI > Add.

  3. Select the authentication methods to use:

    • While you can use a different set of methods from the Azure defaults, this usage isn't a typical configuration.
    • Unless you want to use the env (environment variable) authentication, leave the clientId, tenantiId, and clientSecret fields empty.
    • While not recommended, MSTICPy also supports using client app IDs and secrets for your authentication. In such cases, define your clientId, tenantId, and clientSecret fields directly in the Data Providers tab.
  4. Select Save File to save your changes.

Define autoloading query providers

Define any query providers that you want to MSTICPy to load automatically when you run the nbinit.init_notebook function.

When you frequently author new notebooks, autoloading query providers can save you time by ensuring that required providers are loaded before other components, such as pivot functions and notebooklets.

To add autoloading query providers:

  1. Proceed to the next cell, with the following code, and run it:

    mpedit.set_tab("Autoload QueryProvs")
    mpedit
    
  2. In the Autoload QueryProv tab:

    • For Microsoft Sentinel providers, specify both the provider name and the workspace name you want to connect to.
    • For other query providers, specify the provider name only.

    Each provider also has the following optional values:

    • Auto-connect: This option is defined as True by default, and MSTICPy tries to authenticate to the provider immediately after loading. MSTICPy assumes that you've configured credentials for the provider in your settings.

    • Alias: When MSTICPy loads a provider, it assigns the provider to a Python variable name. By default, the variable name is qryworkspace_name for Microsoft Sentinel providers and qryprovider_name for other providers.

      For example, if you load a query provider for the ContosoSOC workspace, this query provider will be created in your notebook environment with the name qry_ContosoSOC. Add an alias if you want to use something shorter or easier to type and remember. The provider variable name will be qry_<alias>, where <alias> is replaced by the alias name that you provided.

      Providers you load by this mechanism are also added to the MSTICPy current_providers attribute, which is used, for example, in the following code:

      import msticpy
      msticpy.current_providers
      
  3. Select Save Settings to save your changes.

Define autoloaded MSTICPy components

This procedure describes how to define other components that are automatically loaded by MSTICPy when you run the nbinit.init_notebook function.

Supported components include, in the following order:

  1. TILookup: The TI provider library
  2. GeoIP: The GeoIP provider you want to use
  3. AzureData: The module you use to query details about Azure resources
  4. AzureSentinelAPI: The module you use to query the Microsoft Sentinel API
  5. Notebooklets: Notebooklets from the msticnb package
  6. Pivot: Pivot functions

Note

The components load in this order because the Pivot component needs query and other providers loaded to find the pivot functions that it attaches to entities. For more information, see MSTICPy documentation.

To define auto-loaded MSTICPy components:

  1. Proceed to the next cell, with the following code, and run it:

    mpedit.set_tab("Autoload Components")
    mpedit
    
  2. In the Autoload Components tab, define any parameter values as needed. For example:

    • GeoIpLookup. Enter the name of the GeoIP provider you want to use, either GeoLiteLookup or IPStack. For more information, see Add GeoIP provider settings.

    • AzureData and AzureSentinelAPI components. Define the following values:

      • auth_methods: Override the default settings for AzureCLI, and connect using the selected methods.
      • Auto-connect: Set to false to load without connecting.

      For more information, see Specify authentication parameters for Azure and Microsoft Sentinel APIs.

    • Notebooklets. The Notebooklets component has a single parameter block: AzureSentinel.

      Specify your Microsoft Sentinel workspace using the following syntax: workspace:\<workspace name>. The workspace name must be one of the workspaces defined in the Microsoft Sentinel tab.

      If you want to add more parameters to send to the notebooklets init function, specify them as key:value pairs, separated by newlines. For example:

      workspace:<workspace name>
      providers=["LocalData","geolitelookup"]
      

      For more information, see the MSTICNB (MSTIC Notebooklets) documentation.

    Some components, like TILookup and Pivot, don't require any parameters.

  3. Select Save Settings to save your changes.

Switch between Python 3.6 and 3.8 kernels

If you're switching between Python 3.65 and 3.8 kernels, you may find that MSTICPy and other packages don't get installed as expected.

This may happen when the !pip install pkg command will install correctly in the first environment, but then doesn't install correctly in the second. This creates a situation where the second environment can't import or use the package.

We recommend that you don't use !pip install... to install packages in Azure ML notebooks. Instead, use one of the following options:

  • Use the %pip line magic within a notebook. Run:

    
    %pip install --upgrade msticpy
    
  • Install from a terminal:

    1. Open a terminal in Azure ML notebooks and run the following commands:

      conda activate azureml_py38
      pip install --upgrade msticpy
      
    2. Close the terminal and restart the kernel.

Set an environment variable for your msticpyconfig.yaml file

If you are running in Azure ML and have your msticpyconfig.yaml file in the root of your user folder, MSTICPy will automatically find these settings. However, if you are running the notebooks in another environment, follow the instructions in this section to set an environment variable that points to the location of your configuration file.

Defining the path to your msticpyconfig.yaml file in an environment variable allows you to store your file in a known location and make sure that you always load the same settings.

Use multiple configuration files, with multiple environment variables, if you want to use different settings for different notebooks.

  1. Decide on a location for your msticpyconfig.yaml file, such as in ~/.msticpyconfig.yaml or %userprofile%/msticpyconfig.yaml.

    Azure ML users: If you store your configuration file in your Azure ML user folder, the MSTICPy init_notebook function (run in the initialization cell) will automatically find and use the file, and you do not need to set a MSTICPYCONFIG environment variable.

    However, if you also have secrets stored in the file, we recommend storing the configuration file on the compute local drive. The compute internal storage is accessible only to the person who created the compute, whereas the shared storage is accessible to anyone with access to your Azure ML workspace.

    For more information, see What is an Azure Machine Learning compute instance?.

  2. If needed, copy your msticpyconfig.yaml file to your selected location.

  3. Set the MSTICPYCONFIG environment variable to point to that location.

Use one of the following procedures to define the MSTICPYCONFIG environment variable.

For example, to set the MSTICPYCONFIG environment variable on Windows systems:

  1. Move the msticpyconfig.yaml file to the Compute instance as needed.

  2. Open the System Properties dialog box to the Advanced tab.

  3. Select Environment Variables... to open the Environment Variables dialog.

  4. In the System variables area, select New..., and define the values as follows:

    • Variable name: Define as MSTICPYCONFIG
    • Variable value: Enter the path to your msticpyconfig.yaml file

Note

For the Linux and Windows options, you'll need to restart your Jupyter server for it to pick up the environment variable that you defined.

Next steps

For more information, see:

Subject More references
MSTICPy - MSTICPy Package Configuration
- MSTICPy Settings Editor
- Configuring Your Notebook Environment.
- MPSettingsEditor notebook.

Note: The Azure-Sentinel-Notebooks GitHub repo also contains a template msticpyconfig.yaml file with commented-out sections, which might help you understand the settings.
Microsoft Sentinel and Jupyter notebooks - Create your first Microsoft Sentinel notebook (Blog series)
- Jupyter Notebooks: An Introduction
- MSTICPy documentation
- Microsoft Sentinel Notebooks documentation
- The Infosec Jupyterbook
- Linux Host Explorer Notebook walkthrough
- Why use Jupyter for Security Investigations
- Security Investigations with Microsoft Sentinel & Notebooks
- Pandas Documentation
- Bokeh Documentation