Workspace Managed Virtual Network Isolation
APPLIES TO:
Azure CLI ml extension v2 (current)
Python SDK azure-ai-ml v2 (current)
Azure Machine Learning provides support for managed virtual network (managed VNet) isolation. Managed VNet isolation streamlines and automates your network isolation configuration with a built-in, workspace-level Azure Machine Learning managed VNet. The managed VNet secures your managed Azure Machine Learning resources, such as compute instances, compute clusters, serverless compute, and managed online endpoints.
Securing your workspace with a managed network provides network isolation for outbound access from the workspace and managed computes. An Azure Virtual Network that you create and manage is used to provide network isolation inbound access to the workspace. For example, a private endpoint for the workspace is created in your Azure Virtual Network. Any clients connecting to the virtual network can access the workspace through the private endpoint. When running jobs on managed computes, the managed network restricts what the compute can access.
Managed Virtual Network Architecture
When you enable managed virtual network isolation, a managed VNet is created for the workspace. Managed compute resources you create for the workspace automatically use this managed VNet. The managed VNet can use private endpoints for Azure resources that are used by your workspace, such as Azure Storage, Azure Key Vault, and Azure Container Registry.
There are two different configuration modes for outbound traffic from the managed VNet:
Tip
Regardless of the outbound mode you use, traffic to Azure resources can be configured to use a private endpoint. For example, you may allow all outbound traffic to the internet, but restrict communication with Azure resources by adding outbound rules for the resources.
Outbound mode | Description | Scenarios |
---|---|---|
Allow internet outbound | Allow all internet outbound traffic from the managed VNet. | You want unrestricted access to machine learning resources on the internet, such as python packages or pretrained models.1 |
Allow only approved outbound | Outbound traffic is allowed by specifying service tags. | * You want to minimize the risk of data exfiltration, but you need to prepare all required machine learning artifacts in your private environment. * You want to configure outbound access to an approved list of services, service tags, or FQDNs. |
Disabled | Inbound and outbound traffic isn't restricted or you're using your own Azure Virtual Network to protect resources. | You want public inbound and outbound from the workspace, or you're handling network isolation with your own Azure VNet. |
1: You can use outbound rules with allow only approved outbound mode to achieve the same result as using allow internet outbound. The differences are:
- You must add rules for each outbound connection you need to allow.
- Adding FQDN outbound rules increase your costs as this rule type uses Azure Firewall. For more information, see Pricing
- The default rules for allow only approved outbound are designed to minimize the risk of data exfiltration. Any outbound rules you add may increase your risk.
The managed VNet is preconfigured with required default rules. It's also configured for private endpoint connections to your workspace, workspace's default storage, container registry and key vault if they're configured as private or the workspace isolation mode is set to allow only approved outbound. After choosing the isolation mode, you only need to consider other outbound requirements you may need to add.
The following diagram shows a managed VNet configured to allow internet outbound:
The following diagram shows a managed VNet configured to allow only approved outbound:
Note
In this configuration, the storage, key vault, and container registry used by the workspace are flagged as private. Since they are flagged as private, a private endpoint is used to communicate with them.
Note
Once a managed VNet workspace is configured to allow internet outbound, the workspace cannot be reconfigured to disabled. Similarily, once a managed VNet workspace is configured to allow only approved outbound, the workspace cannot be reconfigured to allow internet outbound. Please keep this in mind when selecting the isolation mode for managed VNet in your workspace.
Azure Machine Learning studio
If you want to use the integrated notebook or create datasets in the default storage account from studio, your client needs access to the default storage account. Create a private endpoint or service endpoint for the default storage account in the Azure Virtual Network that the clients use.
Part of Azure Machine Learning studio runs locally in the client's web browser, and communicates directly with the default storage for the workspace. Creating a private endpoint or service endpoint (for the default storage account) in the client's virtual network ensures that the client can communicate with the storage account.
For more information on creating a private endpoint or service endpoint, see the Connect privately to a storage account and Service Endpoints articles.
Prerequisites
Before following the steps in this article, make sure you have the following prerequisites:
An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the free or paid version of Azure Machine Learning.
The Microsoft.Network resource provider must be registered for your Azure subscription. This resource provider is used by the workspace when creating private endpoints for the managed virtual network.
For information on registering resource providers, see Resolve errors for resource provider registration.
The Azure identity you use when deploying a managed network requires the following Azure role-based access control (Azure RBAC) actions to create private endpoints:
Microsoft.MachineLearningServices/workspaces/privateEndpointConnections/read
Microsoft.MachineLearningServices/workspaces/privateEndpointConnections/write
The Azure CLI and the
ml
extension to the Azure CLI. For more information, see Install, set up, and use the CLI (v2).Tip
Azure Machine Learning managed VNet was introduced on May 23rd, 2023. If you have an older version of the ml extension, you may need to update it for the examples in this article work. To update the extension, use the following Azure CLI command:
az extension update -n ml
The CLI examples in this article assume that you're using the Bash (or compatible) shell. For example, from a Linux system or Windows Subsystem for Linux.
The Azure CLI examples in this article use
ws
to represent the name of the workspace, andrg
to represent the name of the resource group. Change these values as needed when using the commands with your Azure subscription.
Note
If you are using UAI workspace please make sure to add the Azure AI Enterprise Network Connection Approver role to your identity. For more information, see User-assigned managed identity.
Configure a managed virtual network to allow internet outbound
Tip
The creation of the managed VNet is deferred until a compute resource is created or provisioning is manually started. When allowing automatic creation, it can take around 30 minutes to create the first compute resource as it is also provisioning the network. For more information, see Manually provision the network.
Important
If you plan to submit serverless Spark jobs, you must manually start provisioning. For more information, see the configure for serverless Spark jobs section.
To configure a managed VNet that allows internet outbound communications, you can use either the --managed-network allow_internet_outbound
parameter or a YAML configuration file that contains the following entries:
managed_network:
isolation_mode: allow_internet_outbound
You can also define outbound rules to other Azure services that the workspace relies on. These rules define private endpoints that allow an Azure resource to securely communicate with the managed VNet. The following rule demonstrates adding a private endpoint to an Azure Blob resource.
managed_network:
isolation_mode: allow_internet_outbound
outbound_rules:
- name: added-perule
destination:
service_resource_id: /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT_NAME>
spark_enabled: true
subresource_target: blob
type: private_endpoint
You can configure a managed VNet using either the az ml workspace create
or az ml workspace update
commands:
Create a new workspace:
The following example creates a new workspace. The
--managed-network allow_internet_outbound
parameter configures a managed VNet for the workspace:az ml workspace create --name ws --resource-group rg --managed-network allow_internet_outbound
To create a workspace using a YAML file instead, use the
--file
parameter and specify the YAML file that contains the configuration settings:az ml workspace create --file workspace.yaml --resource-group rg --name ws
The following YAML example defines a workspace with a managed VNet:
name: myworkspace location: EastUS managed_network: isolation_mode: allow_internet_outbound
Update an existing workspace:
Warning
Before updating an existing workspace to use a managed virtual network, you must delete all computing resources for the workspace. This includes compute instance, compute cluster, and managed online endpoints.
The following example updates an existing workspace. The
--managed-network allow_internet_outbound
parameter configures a managed VNet for the workspace:az ml workspace update --name ws --resource-group rg --managed-network allow_internet_outbound
To update an existing workspace using the YAML file, use the
--file
parameter and specify the YAML file that contains the configuration settings:az ml workspace update --file workspace.yaml --name ws --resource-group MyGroup
The following YAML example defines a managed VNet for the workspace. It also demonstrates how to add a private endpoint connection to a resource used by the workspace; in this example, a private endpoint for a blob store:
name: myworkspace managed_network: isolation_mode: allow_internet_outbound outbound_rules: - name: added-perule destination: service_resource_id: /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT_NAME> spark_enabled: true subresource_target: blob type: private_endpoint
Configure a managed virtual network to allow only approved outbound
Tip
The managed VNet is automatically provisioned when you create a compute resource. When allowing automatic creation, it can take around 30 minutes to create the first compute resource as it is also provisioning the network. If you configured FQDN outbound rules, the first FQDN rule adds around 10 minutes to the provisioning time. For more information, see Manually provision the network.
Important
If you plan to submit serverless Spark jobs, you must manually start provisioning. For more information, see the configure for serverless Spark jobs section.
To configure a managed VNet that allows only approved outbound communications, you can use either the --managed-network allow_only_approved_outbound
parameter or a YAML configuration file that contains the following entries:
managed_network:
isolation_mode: allow_only_approved_outbound
You can also define outbound rules to define approved outbound communication. An outbound rule can be created for a type of service_tag
, fqdn
, and private_endpoint
. The following rule demonstrates adding a private endpoint to an Azure Blob resource, a service tag to Azure Data Factory, and an FQDN to pypi.org
:
Important
- Adding an outbound for a service tag or FQDN is only valid when the managed VNet is configured to
allow_only_approved_outbound
. - If you add outbound rules, Microsoft can't guarantee data exfiltration.
Warning
FQDN outbound rules are implemented using Azure Firewall. If you use outbound FQDN rules, charges for Azure Firewall are added to your billing. For more information, see Pricing.
managed_network:
isolation_mode: allow_only_approved_outbound
outbound_rules:
- name: added-servicetagrule
destination:
port_ranges: 80, 8080
protocol: TCP
service_tag: DataFactory
type: service_tag
- name: add-fqdnrule
destination: 'pypi.org'
type: fqdn
- name: added-perule
destination:
service_resource_id: /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT_NAME>
spark_enabled: true
subresource_target: blob
type: private_endpoint
You can configure a managed VNet using either the az ml workspace create
or az ml workspace update
commands:
Create a new workspace:
The following example uses the
--managed-network allow_only_approved_outbound
parameter to configure the managed VNet:az ml workspace create --name ws --resource-group rg --managed-network allow_only_approved_outbound
The following YAML file defines a workspace with a managed VNet:
name: myworkspace location: EastUS managed_network: isolation_mode: allow_only_approved_outbound
To create a workspace using the YAML file, use the
--file
parameter:az ml workspace create --file workspace.yaml --resource-group rg --name ws
Update an existing workspace
Warning
Before updating an existing workspace to use a managed virtual network, you must delete all computing resources for the workspace. This includes compute instance, compute cluster, and managed online endpoints.
The following example uses the
--managed-network allow_only_approved_outbound
parameter to configure the managed VNet for an existing workspace:az ml workspace update --name ws --resource-group rg --managed-network allow_only_approved_outbound
The following YAML file defines a managed VNet for the workspace. It also demonstrates how to add an approved outbound to the managed VNet. In this example, an outbound rule is added for both a service tag:
Warning
FQDN outbound rules are implemented using Azure Firewall. If you use outbound FQDN rules, charges for Azure Firewall are added to your billing.For more information, see Pricing.
name: myworkspace_dep managed_network: isolation_mode: allow_only_approved_outbound outbound_rules: - name: added-servicetagrule destination: port_ranges: 80, 8080 protocol: TCP service_tag: DataFactory type: service_tag - name: add-fqdnrule destination: 'pypi.org' type: fqdn - name: added-perule destination: service_resource_id: /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT_NAME> spark_enabled: true subresource_target: blob type: private_endpoint
Configure for serverless Spark jobs
Tip
The steps in this section are only needed if you plan to submit serverless Spark jobs. If you aren't going to be submitting serverless Spark jobs, you can skip this section.
To enable the serverless Spark jobs for the managed VNet, you must perform the following actions:
- Configure a managed VNet for the workspace and add an outbound private endpoint for the Azure Storage Account.
- After you configure the managed VNet, provision it and flag it to allow Spark jobs.
Configure an outbound private endpoint.
Use a YAML file to define the managed VNet configuration and add a private endpoint for the Azure Storage Account. Also set
spark_enabled: true
:Tip
This example is for a managed VNet configured using
isolation_mode: allow_internet_outbound
to allow internet traffic. If you want to allow only approved outbound traffic, useisolation_mode: allow_only_approved_outbound
.name: myworkspace managed_network: isolation_mode: allow_internet_outbound outbound_rules: - name: added-perule destination: service_resource_id: /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.Storage/storageAccounts/<STORAGE_ACCOUNT_NAME> spark_enabled: true subresource_target: blob type: private_endpoint
You can use a YAML configuration file with the
az ml workspace update
command by specifying the--file
parameter and the name of the YAML file. For example, the following command updates an existing workspace using a YAML file namedworkspace_pe.yml
:az ml workspace update --file workspace_pe.yml --resource_group rg --name ws
Note
When Allow Only Approved Outbound is enabled (
isolation_mode: allow_only_approved_outbound
), conda package dependencies defined in Spark session configuration will fail to install. To resolve this problem, upload a self-contained Python package wheel with no external dependencies to an Azure storage account and create private endpoint to this storage account. Use the path to Python package wheel aspy_files
parameter in your Spark job. Setting an FQDN outbound rule will not bypass this issue as FQDN rule propagation is not supported by Spark.Provision the managed VNet.
Note
If your workspace is already configured for a public endpoint (for example, with an Azure Virtual Network), and has public network access enabled, you must disable it before provisioning the managed VNet. If you don't disable public network access when provisioning the managed VNet, the private endpoints for the managed endpoint may not be created successfully.
The following example shows how to provision a managed VNet for serverless Spark jobs by using the
--include-spark
parameter.az ml workspace provision-network -g my_resource_group -n my_workspace_name --include-spark
Manually provision a managed VNet
The managed VNet is automatically provisioned when you create a compute instance. When you rely on automatic provisioning, it can take around 30 minutes to create the first compute instance as it is also provisioning the network. If you configured FQDN outbound rules (only available with allow only approved mode), the first FQDN rule adds around 10 minutes to the provisioning time. If you have a large set of outbound rules to be provisioned in the managed network, it can take longer for provisioning to complete. The increased provisioning time can cause your first compute instance creation to time out.
To reduce the wait time and avoid potential timeout errors, we recommend manually provisioning the managed network. Then wait until the provisioning completes before you create a compute instance.
Note
To create an online deployment, you must manually provision the managed network, or create a compute instance first which will automatically provision it.
The following example shows how to provision a managed VNet.
Tip
If you plan to submit serverless Spark jobs, add the --include-spark
parameter.
az ml workspace provision-network -g my_resource_group -n my_workspace_name
To verify that the provisioning has completed, use the following command:
az ml workspace show -n my_workspace_name -g my_resource_group --query managed_network
Configure image builds
When the Azure Container Registry for your workspace is behind a virtual network, it can't be used to directly build Docker images. Instead, configure your workspace to use a compute cluster or compute instance to build images.
Important
The compute resource used to build Docker images needs to be able to access the package repositories that are used to train and deploy your models. If you're using a network configured to allow only approved outbound, you may need to add rules that allow access to public repos or use private Python packages.
To update a workspace to use a compute cluster or compute instance to build Docker images, use the az ml workspace update
command with the --image-build-compute
parameter:
az ml workspace update --name ws --resource-group rg --image-build-compute mycompute
Manage outbound rules
To list the managed VNet outbound rules for a workspace, use the following command:
az ml workspace outbound-rule list --workspace-name ws --resource-group rg
To view the details of a managed VNet outbound rule, use the following command:
az ml workspace outbound-rule show --rule rule-name --workspace-name ws --resource-group rg
To remove an outbound rule from the managed VNet, use the following command:
az ml workspace outbound-rule remove --rule rule-name --workspace-name ws --resource-group rg
List of required rules
Tip
These rules are automatically added to the managed VNet.
Private endpoints:
- When the isolation mode for the managed VNet is
Allow internet outbound
, private endpoint outbound rules are automatically created as required rules from the managed VNet for the workspace and associated resources with public network access disabled (Key Vault, Storage Account, Container Registry, Azure Machine Learning workspace). - When the isolation mode for the managed VNet is
Allow only approved outbound
, private endpoint outbound rules are automatically created as required rules from the managed VNet for the workspace and associated resources regardless of public network access mode for those resources (Key Vault, Storage Account, Container Registry, Azure Machine Learning workspace).
Outbound service tag rules:
AzureActiveDirectory
AzureMachineLearning
BatchNodeManagement.region
AzureResourceManager
AzureFrontDoor.FirstParty
MicrosoftContainerRegistry
AzureMonitor
Inbound service tag rules:
AzureMachineLearning
List of scenario specific outbound rules
Scenario: Access public machine learning packages
To allow installation of Python packages for training and deployment, add outbound FQDN rules to allow traffic to the following host names:
Warning
FQDN outbound rules are implemented using Azure Firewall. If you use outbound FQDN rules, charges for Azure Firewall are added to your billing. For more information, see Pricing.
Note
This is not a complete list of the hosts required for all Python resources on the internet, only the most commonly used. For example, if you need access to a GitHub repository or other host, you must identify and add the required hosts for that scenario.
Host name | Purpose |
---|---|
anaconda.com *.anaconda.com |
Used to install default packages. |
*.anaconda.org |
Used to get repo data. |
pypi.org |
Used to list dependencies from the default index, if any, and the index isn't overwritten by user settings. If the index is overwritten, you must also allow *.pythonhosted.org . |
pytorch.org *.pytorch.org |
Used by some examples based on PyTorch. |
*.tensorflow.org |
Used by some examples based on Tensorflow. |
Scenario: Use Visual Studio Code desktop or web with compute instance
If you plan to use Visual Studio Code with Azure Machine Learning, add outbound FQDN rules to allow traffic to the following hosts:
Warning
FQDN outbound rules are implemented using Azure Firewall. If you use outbound FQDN rules, charges for Azure Firewall are added to your billing. For more information, see Pricing.
*.vscode.dev
vscode.blob.core.windows.net
*.gallerycdn.vsassets.io
raw.githubusercontent.com
*.vscode-unpkg.net
*.vscode-cdn.net
*.vscodeexperiments.azureedge.net
default.exp-tas.com
code.visualstudio.com
update.code.visualstudio.com
*.vo.msecnd.net
marketplace.visualstudio.com
vscode.download.prss.microsoft.com
Scenario: Use batch endpoints or ParallelRunStep
If you plan to use Azure Machine Learning batch endpoints for deployment or ParallelRunStep, add outbound private endpoint rules to allow traffic to the following sub resources for the default storage account:
queue
table
Scenario: Use prompt flow with Azure OpenAI, content safety, and Azure AI Search
- Private endpoint to Azure AI Services
- Private endpoint to Azure AI Search
Scenario: Use HuggingFace models
If you plan to use HuggingFace models with Azure Machine Learning, add outbound FQDN rules to allow traffic to the following hosts:
Warning
FQDN outbound rules are implemented using Azure Firewall. If you use outbound FQDN rules, charges for Azure Firewall are added to your billing. For more information, see Pricing.
docker.io
*.docker.io
*.docker.com
production.cloudflare.docker.com
cdn.auth0.com
cdn-lfs.huggingface.co
Scenario: Enable access from selected IP Addresses
If you want to enable access from specific IP addresses, use the following actions:
Add an outbound private endpoint rule to allow traffic to the Azure Machine Learning workspace. This allows compute instances created in the managed virtual network to access the workspace.
Tip
You can't add this rule during workspace creation, as the workspace doesn't exist yet.
Enable public network access to the workspace. For more information, see public network access enabled.
Add your IP addresses to the firewall for Azure Machine Learning. For more information, see enable access only from IP ranges.
Note
Only IPv4 addresses are supported.
Private endpoints
Private endpoints are currently supported for the following Azure services:
- Azure Machine Learning
- Azure Machine Learning registries
- Azure Storage (all sub resource types)
- Azure Container Registry
- Azure Key Vault
- Azure AI services
- Azure AI Search (formerly Cognitive Search)
- Azure SQL Server
- Azure Data Factory
- Azure Cosmos DB (all sub resource types)
- Azure Event Hubs
- Azure Redis Cache
- Azure Databricks
- Azure Database for MariaDB
- Azure Database for PostgreSQL Single Server
- Azure Database for PostgreSQL Flexible Server
- Azure Database for MySQL
- Azure API Management
When you create a private endpoint, you provide the resource type and subresource that the endpoint connects to. Some resources have multiple types and subresources. For more information, see what is a private endpoint.
When you create a private endpoint for Azure Machine Learning dependency resources, such as Azure Storage, Azure Container Registry, and Azure Key Vault, the resource can be in a different Azure subscription. However, the resource must be in the same tenant as the Azure Machine Learning workspace.
Important
When configuring private endpoints for an Azure Machine Learning managed VNet, the private endpoints are only created when created when the first compute is created or when managed VNet provisioning is forced. For more information on forcing the managed VNet provisioning, see Configure for serverless Spark jobs.
Pricing
The Azure Machine Learning managed VNet feature is free. However, you're charged for the following resources that are used by the managed VNet:
Azure Private Link - Private endpoints used to secure communications between the managed VNet and Azure resources relies on Azure Private Link. For more information on pricing, see Azure Private Link pricing.
FQDN outbound rules - FQDN outbound rules are implemented using Azure Firewall. If you use outbound FQDN rules, charges for Azure Firewall are added to your billing. The Azure Firewall (standard SKU) is provisioned by Azure Machine Learning.
Important
The firewall isn't created until you add an outbound FQDN rule. For more information on pricing, see Azure Firewall pricing and view prices for the standard version.
Limitations
- Azure AI Studio doesn't support using your own Azure Virtual Network to secure the hub, project, or compute resources. You can only use the managed network feature to secure these resources.
- Once you enable managed VNet isolation of your workspace (either allow internet outbound or allow only approved outbound), you can't disable it.
- Managed VNet uses private endpoint connection to access your private resources. You can't have a private endpoint and a service endpoint at the same time for your Azure resources, such as a storage account. We recommend using private endpoints in all scenarios.
- The managed VNet is deleted when the workspace is deleted.
- Data exfiltration protection is automatically enabled for the only approved outbound mode. If you add other outbound rules, such as to FQDNs, Microsoft can't guarantee that you're protected from data exfiltration to those outbound destinations.
- Creating a compute cluster in a different region than the workspace isn't supported when using a managed VNet.
- Kubernetes and attached VMs aren't supported in an Azure Machine Learning managed VNet.
- Using FQDN outbound rules increases the cost of the managed VNet because FQDN rules use Azure Firewall. For more information, see Pricing.
- FQDN outbound rules only support ports 80 and 443.
- If your compute instance is in a managed network and is configured for no public IP, use the
az ml compute connect-ssh
command to connect to it using SSH. - When using Managed Vnet, you can't deploy compute resources inside your custom Vnet. Compute resources can only be created inside the managed Vnet.
- Managed network isolation cannot establish a private connection from the managed virtual network to a user's on-premises resources. For the list of supported private connections, see Private Endpoints.
Migration of compute resources
If you have an existing workspace and want to enable managed VNet for it, there's currently no supported migration path for existing manged compute resources. You'll need to delete all existing managed compute resources and recreate them after enabling the managed VNet. The following list contains the compute resources that must be deleted and recreated:
- Compute cluster
- Compute instance
- Kubernetes clusters
- Managed online endpoints