Workspace managed network isolation (preview)

APPLIES TO: Azure CLI ml extension v2 (current)

APPLIES TO: Python SDK azure-ai-ml v2 (current)

Azure Machine Learning provides preview support for managed virtual network (VNet) isolation. Managed VNet isolation streamlines and automates your network isolation configuration with a built-in, workspace-level Azure Machine Learning managed virtual network.

Important

This feature is currently in public preview. This preview version is provided without a service-level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Managed virtual network architecture

When you enable managed virtual network isolation, a managed VNet is created for the workspace. Managed compute resources (compute clusters and compute instances) for the workspace automatically use this managed VNet. The managed VNet can use private endpoints for Azure resources that are used by your workspace, such as Azure Storage, Azure Key Vault, and Azure Container Registry.

The following diagram shows a managed virtual network uses private endpoints to communicate with the storage, key vault, and container registry used by the workspace.

Diagram of managed virtual network isolation.

There are two different configuration modes for outbound traffic from the managed virtual network:

Tip

Regardless of the outbound mode you use, traffic to Azure resources can be configured to use a private endpoint. For example, you may allow all outbound traffic to the internet, but restrict communication with Azure resources by creating a private endpoint for that resource in the managed VNet

Outbound mode Description Scenarios
Allow internet outbound Allow all internet outbound traffic from the managed VNet. Recommended if you need access to machine learning artifacts on the Internet, such as python packages or pretrained models.
Allow only approved outbound Outbound traffic is allowed by specifying service tags. Recommended if you want to minimize the risk of data exfiltration but you need to prepare all required machine learning artifacts in your private locations.

The managed virtual network is preconfigured with required default rules. It's also configured for private endpoint connections to your workspace default storage, container registry and key vault if they're configured as private. After choosing the isolation mode, you only need to consider other outbound requirements you may need to add.

Supported scenarios in preview and to be supported scenarios

Scenarios Supported in preview To be supported
Isolation Mode • Allow internet outbound
• Allow only approved outbound
Compute Compute Instance
Compute Cluster
Serverless
Serverless spark
• New managed online endpoint creation
• Migration of existing managed online endpoint
• No Public IP option of Compute Instance, Compute Cluster and Serverless
Outbound • Private Endpoint
• Service Tag
• FQDN

Prerequisites

Before following the steps in this article, make sure you have the following prerequisites:

Important

To use the information in this article, you must enable this preview feature for your subscription. To check whether it has been registered, or to register it, use the steps in the Set up preview features in Azure subscription. Depending on whether you use the Azure portal, Azure CLI, or Azure PowerShell, you may need to register the feature with a different name. Use the following table to determine the name of the feature to register:

Registration method Feature name
Azure portal Azure Machine Learning Managed Network
Azure CLI AMLManagedNetworkEnabled
Azure PowerShell AMLManagedNetworkEnabled
  • An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the free or paid version of Azure Machine Learning.

  • The Azure CLI and the ml extension to the Azure CLI. For more information, see Install, set up, and use the CLI (v2).

    Tip

    Azure Machine Learning managed virtual network was introduced on May 23rd, 2023. If you have an older version of the ml extension, you may need to update it for the examples in this article work. To update the extension, use the following Azure CLI command:

    az extension update -n ml
    
  • The CLI examples in this article assume that you're using the Bash (or compatible) shell. For example, from a Linux system or Windows Subsystem for Linux.

  • The Azure CLI examples in this article use ws to represent the name of the workspace, and rg to represent the name of the resource group. Change these values as needed when using the commands with your Azure subscription.

Configure a managed virtual network to allow internet outbound

Important

The creation of the managed virtual network is deferred until a compute resource is created or provisioning is manually started. Manually start provisioning if you plan to submit serverless spark jobs.

To configure a managed VNet that allows internet outbound communications, you can use either the --managed-network allow_internet_outbound parameter or a YAML configuration file that contains the following entries:

managed_network:
  isolation_mode: allow_internet_outbound

You can also define outbound rules to other Azure services that the workspace relies on. These rules define private endpoints that allow an Azure resource to securely communicate with the managed VNet. The following rule demonstrates adding a private endpoint to an Azure Blob resource.

managed_network:
  isolation_mode: allow_internet_outbound
  outbound_rules:
  - name: added-perule
    destination:
      service_resource_id: /subscriptions/{subscription ID}/resourceGroups/{resource group name}/providers/Microsoft.Storage/storageAccounts/{storage account name}
      spark_enabled: true
      subresource_target: blob
    type: private_endpoint

You can configure a managed VNet using either the az ml workspace create or az ml workspace update commands:

  • Create a new workspace:

    Tip

    before creating a new workspace, you must create an Azure Resource Group to contain it. For more information, see Manage Azure Resource Groups.

    The following example creates a new workspace. The --managed-network allow_internet_outbound parameter configures a managed VNet for the workspace:

    az ml workspace create --name ws --resource-group rg --managed-network allow_internet_outbound
    

    To create a workspace using a YAML file instead, use the --file parameter and specify the YAML file that contains the configuration settings:

    az ml workspace create --file workspace.yaml --resource-group rg --name ws
    

    The following YAML example defines a workspace with a managed VNet:

    name: myworkspace
    location: EastUS
    managed_network:
    isolation_mode: allow_internet_outbound
    
  • Update an existing workspace:

    Warning

    Before updating an existing workspace to use a managed virtual network, you must delete all computing resources for the workspace. This includes compute instance, compute cluster, and managed online endpoints.

    The following example updates an existing workspace. The --managed-network allow_internet_outbound parameter configures a managed VNet for the workspace:

    az ml workspace update --name ws --resource-group rg --managed-network allow_internet_outbound
    

    To Update an existing workspace using the YAML file, use the --file parameter and specify the YAML file that contains the configuration settings:

    az ml workspace update --file workspace.yaml --name ws --resource-group MyGroup
    

    The following YAML example defines a managed VNet for the workspace. It also demonstrates how to add a private endpoint connection to a resource used by the workspace; in this example, a private endpoint for a blob store:

    name: myworkspace
    managed_network:
      isolation_mode: allow_internet_outbound
      outbound_rules:
      - name: added-perule
        destination:
          service_resource_id: /subscriptions/{subscription ID}/resourceGroups/{resource group name}/providers/Microsoft.Storage/storageAccounts/{storage account name}
          spark_enabled: true
          subresource_target: blob
        type: private_endpoint
    

Configure a managed virtual network to allow only approved outbound

Important

The creation of the managed virtual network is deferred until a compute resource is created or provisioning is manually started. Manually start provisioning if you plan to submit serverless spark jobs.

To configure a managed VNet that allows only approved outbound communications, you can use either the --managed-network allow_only_approved_outbound parameter or a YAML configuration file that contains the following entries:

managed_network:
  isolation_mode: allow_only_approved_outbound

You can also define outbound rules to define approved outbound communication. An outbound rule can be created for a type of service_tag. You can also define private endpoints that allow an Azure resource to securely communicate with the managed VNet. The following rule demonstrates adding a private endpoint to an Azure Blob resource, a service tag to Azure Data Factory:

Tip

Adding an outbound for a service tag is only valid when the managed VNet is configured to allow_only_approved_outbound.

managed_network:
  isolation_mode: allow_only_approved_outbound
  outbound_rules:
  - name: added-servicetagrule
    destination:
      port_ranges: 80, 8080
      protocol: TCP
      service_tag: DataFactory
    type: service_tag
  - name: added-perule
    destination:
      service_resource_id: /subscriptions/{subscription ID}/resourceGroups/{resource group name}/providers/Microsoft.Storage/storageAccounts/{storage account name}
      spark_enabled: true
      subresource_target: blob
    type: private_endpoint

You can configure a managed VNet using either the az ml workspace create or az ml workspace update commands:

  • Create a new workspace:

    Tip

    Before creating a new workspace, you must create an Azure Resource Group to contain it. For more information, see Manage Azure Resource Groups.

    The following example uses the --managed-network allow_only_approved_outbound parameter to configure the managed VNet:

    az ml workspace create --name ws --resource-group rg --managed-network allow_only_approved_outbound
    

    The following YAML file defines a workspace with a managed virtual network:

    name: myworkspace
    location: EastUS
    managed_network:
    isolation_mode: allow_only_approved_outbound
    

    To create a workspace using the YAML file, use the --file parameter:

    az ml workspace create --file workspace.yaml --resource-group rg --name ws
    
  • Update an existing workspace

    Warning

    Before updating an existing workspace to use a managed virtual network, you must delete all computing resources for the workspace. This includes compute instance, compute cluster, and managed online endpoints.

    The following example uses the --managed-network allow_only_approved_outbound parameter to configure the managed VNet for an existing workspace:

    az ml workspace update --name ws --resource-group rg --managed-network allow_only_approved_outbound
    

    The following YAML file defines a managed VNet for the workspace. It also demonstrates how to add an approved outbound to the managed VNet. In this example, an outbound rule is added for both a service tag:

    name: myworkspace_dep
    managed_network:
      isolation_mode: allow_only_approved_outbound
      outbound_rules:
      - name: added-servicetagrule
        destination:
          port_ranges: 80, 8080
          protocol: TCP
          service_tag: DataFactory
        type: service_tag
      - name: added-perule
        destination:
          service_resource_id: /subscriptions/{subscription ID}/resourceGroups/{resource group name}/providers/Microsoft.Storage/storageAccounts/{storage account name}
          spark_enabled: true
          subresource_target: blob
        type: private_endpoint
    

Configure for serverless spark jobs

Tip

The steps in this section are only needed for Spark serverless. If you are using serverless compute cluster, you can skip this section.

To enable the serverless spark jobs for the managed VNet, you must perform the following actions:

  • Configure a managed VNet for the workspace and add an outbound private endpoint for the Azure Storage Account.
  • After you configure the managed VNet, provision it and flag it to allow spark jobs.
  1. Configure an outbound private endpoint.

    Use a YAML file to define the managed VNet configuration and add a private endpoint for the Azure Storage Account. Also set spark_enabled: true:

    Tip

    This example is for a managed VNet configured to allow internet traffic. If you want to allow only approved outbound traffic, set isolation_mode: allow_only_approved_outbound instead.

    name: myworkspace
    managed_network:
      isolation_mode: allow_internet_outbound
      outbound_rules:
      - name: added-perule
        destination:
          service_resource_id: /subscriptions/{subscription ID}/resourceGroups/{resource group name}/providers/Microsoft.Storage/storageAccounts/{storage account name}
          spark_enabled: true
          subresource_target: blob
        type: private_endpoint
    

    You can use a YAML configuration file with the az ml workspace update command by specifying the --file parameter and the name of the YAML file. For example, the following command updates an existing workspace using a YAML file named workspace_pe.yml:

    az ml workspace update --file workspace_pe.yml --resource_group rg --name ws
    
  2. Provision the managed VNet.

    Note

    If your workspace is already configured for a public endpoint (for example, with an Azure Virtual Network), and has public network access enabled, you must disable it before provisioning the managed virtual network. If you don't disable public network access when provisioning the managed virtual network, the private endpoints for the managed endpoint may not be created successfully.

    The following example shows how to provision a managed VNet for serverless spark jobs by using the --include-spark parameter.

    az ml workspace provision-network -g my_resource_group -n my_workspace_name --include-spark
    

Manage outbound rules

To list the managed VNet outbound rules for a workspace, use the following command:

az ml workspace outbound-rule list --workspace-name ws --resource-group rg

To view the details of a managed VNet outbound rule, use the following command:

az ml workspace outbound-rule show --rule rule-name --workspace-name ws --resource-group rg

To remove an outbound rule from the managed VNet, use the following command:

az ml workspace outbound-rule remove --rule rule-name --workspace-name ws --resource-group rg

List of required rules

Tip

These rules are automatically added to the managed VNet.

Outbound rules:

  • AzureActiveDirectory
  • AzureMachineLearning
  • BatchNodeManagement.region
  • AzureResourceManager
  • AzureFrontDoor
  • MicrosoftContainerRegistry
  • AzureMonitor

Inbound rules:

  • AzureMachineLearning

Currently we don't have any recommended outbound rules.

Limitations

  • Once you enable managed virtual network isolation of your workspace, you can't disable it.
  • Managed virtual network uses private endpoint connection to access your private resources. You can't have a private endpoint and a service endpoint at the same time for your Azure resources, such as a storage account. We recommend using private endpoints in all scenarios.

Next steps