Use network isolation with managed online endpoints (preview)

When deploying a machine learning model to a managed online endpoint, you can secure communication with the online endpoint by using private endpoints. Using a private endpoint with online endpoints is currently a preview feature.

Important

This feature is currently in public preview. This preview version is provided without a service-level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

You can secure the inbound scoring requests from clients to an online endpoint. You can also secure the outbound communications between a deployment and the Azure resources used by the deployment. Security for inbound and outbound communication is configured separately. For more information on endpoints and deployments, see What are endpoints and deployments.

The following diagram shows how communications flow through private endpoints to the managed online endpoint. Incoming scoring requests from clients are received through the workspace private endpoint from your virtual network. Outbound communication with services is handled through private endpoints to those service instances from the deployment:

Diagram of overall ingress/egress communication.

Prerequisites

  • To use Azure machine learning, you must have an Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the free or paid version of Azure Machine Learning today.

  • You must install and configure the Azure CLI and ML extension. For more information, see Install, set up, and use the CLI (v2).

  • You must have an Azure Resource Group, in which you (or the service principal you use) need to have Contributor access. You'll have such a resource group if you configured your ML extension per the above article.

  • You must have an Azure Machine Learning workspace, and the workspace must use a private endpoint. If you don't have one, the steps in this article create an example workspace, VNet, and VM. For more information, see Configure a private endpoint for Azure Machine Learning workspace.

    The workspace can be configured to allow or disallow public network access. If you plan on using managed online endpoint deployments that use public outbound, then you must also configure the workspace to allow public access.

    Outbound communication from managed online endpoint deployment is to the workspace API. When the endpoint is configured to use public outbound, then the workspace must be able to accept that public communication (allow public access).

  • When the workspace is configured with a private endpoint, the Azure Container Registry for the workspace must be configured for Premium tier. For more information, see Azure Container Registry service tiers.

  • The Azure Container Registry and Azure Storage Account must be in the same Azure Resource Group as the workspace.

Important

The end-to-end example in this article comes from the files in the azureml-examples GitHub repository. To clone the samples repository and switch to the repository's cli/ directory, use the following commands:

git clone https://github.com/Azure/azureml-examples
cd azureml-examples/cli

Limitations

  • The v1_legacy_mode flag must be disabled (false) on your Azure Machine Learning workspace. If this flag is enabled, you won't be able to create a managed online endpoint. For more information, see Network isolation with v2 API.

  • If your Azure Machine Learning workspace has a private endpoint that was created before May 24, 2022, you must recreate the workspace's private endpoint before configuring your online endpoints to use a private endpoint. For more information on creating a private endpoint for your workspace, see How to configure a private endpoint for Azure Machine Learning workspace.

  • Secure outbound communication creates three private endpoints per deployment. One to Azure Blob storage, one to Azure Container Registry, and one to your workspace.

  • Azure Log Analytics and Application Insights aren't supported when using network isolation with a deployment. To see the logs for the deployment, use the az ml online-deployment get_logs command instead.

  • You can configure public access to a managed online endpoint (inbound and outbound). You can also configure public access to an Azure Machine Learning workspace.

    Outbound communication from managed online endpoint deployment is to the workspace API. When the endpoint is configured to use public outbound, then the workspace must be able to accept that public communication (allow public access).

Note

Requests to create, update, or retrieve the authentication keys are sent to the Azure Resource Manager over the public network.

Inbound (scoring)

To secure scoring requests to the online endpoint to your virtual network, set the public_network_access flag for the endpoint to disabled:

az ml online-endpoint create -f endpoint.yml --set public_network_access=disabled

When public_network_access is disabled, inbound scoring requests are received using the private endpoint of the Azure Machine Learning workspace and the endpoint can't be reached from public networks.

Outbound (resource access)

To restrict communication between a deployment and the Azure resources used to by the deployment, set the egress_public_network_access flag to disabled. Use this flag to ensure that the download of the model, code, and images needed by your deployment are secured with a private endpoint.

The following are the resources that the deployment communicates with over the private endpoint:

  • The Azure Machine Learning workspace.
  • The Azure Storage blob that is the default storage for the workspace.
  • The Azure Container Registry for the workspace.

When you configure the egress_public_network_access to disabled, a new private endpoint is created per deployment, per service. For example, if you set the flag to disabled for three deployments to an online endpoint, nine private endpoints are created. Each deployment would have three private endpoints that are used to communicate with the workspace, blob, and container registry.

az ml online-deployment create -f deployment.yml --set egress_public_network_access=disabled

Scenarios

The following table lists the supported configurations when configuring inbound and outbound communications for an online endpoint:

Configuration Inbound
(Endpoint property)
Outbound
(Deployment property)
Supported?
secure inbound with secure outbound public_network_access is disabled egress_public_network_access is disabled Yes
secure inbound with public outbound public_network_access is disabled
The workspace must also allow public access.
egress_public_network_access is enabled Yes
public inbound with secure outbound public_network_access is enabled egress_public_network_access is disabled Yes
public inbound with public outbound public_network_access is enabled
The workspace must also allow public access.
egress_public_network_access is enabled Yes

Important

Outbound communication from managed online endpoint deployment is to the workspace API. When the endpoint is configured to use public outbound, then the workspace must be able to accept that public communication (allow public access).

End-to-end example

Use the information in this section to create an example configuration that uses private endpoints to secure online endpoints.

Tip

In this example, and Azure Virtual Machine is created inside the VNet. You connect to the VM using SSH, and run the deployment from the VM. This configuration is used to simplify the steps in this example, and does not represent a typical secure configuration. For example, in a production environment you would most likely use a VPN client or Azure ExpressRoute to directly connect clients to the virtual network.

Create workspace and secured resources

The steps in this section use an Azure Resource Manager template to create the following Azure resources:

  • Azure Virtual Network
  • Azure Machine Learning workspace
  • Azure Container Registry
  • Azure Key Vault
  • Azure Storage account (blob & file storage)

Public access is disabled for all the services. While the Azure Machine Learning workspace is secured behind a vnet, it's configured to allow public network access. For more information, see CLI 2.0 secure communications. A scoring subnet is created, along with outbound rules that allow communication with the following Azure services:

  • Azure Active Directory
  • Azure Resource Manager
  • Azure Front Door
  • Microsoft Container Registries

The following diagram shows the different components created in this architecture:

The following diagram shows the overall architecture of this example:

Diagram of the services created.

To create the resources, use the following Azure CLI commands. Replace <UNIQUE_SUFFIX> with a unique suffix for the resources that are created.

# SUFFIX will be used as resource name suffix in created workspace and related resources
export SUFFIX="<UNIQUE_SUFFIX>"
az deployment group create --template-file endpoints/online/managed/vnet/setup_ws/main.bicep --parameters suffix=$SUFFIX
# Note: if you get an error that appinsights is not available in your current location, use optional parameter to the above script: appinsightsLocation=<location> (e.g. westus2)

Create the virtual machine jump box

To create an Azure Virtual Machine that can be used to connect to the VNet, use the following command. Replace <your-new-password> with the password you want to use when connecting to this VM:

# create vm
az vm create --name test-vm --vnet-name vnet-$SUFFIX --subnet snet-scoring --image UbuntuLTS --admin-username azureuser --admin-password <your-new-password>

Important

The VM created by these commands has a public endpoint that you can connect to over the public network.

The response from this command is similar to the following JSON document:

{
  "fqdns": "",
  "id": "/subscriptions/<GUID>/resourceGroups/<my-resource-group>/providers/Microsoft.Compute/virtualMachines/test-vm",
  "location": "westus",
  "macAddress": "00-0D-3A-ED-D8-E8",
  "powerState": "VM running",
  "privateIpAddress": "192.168.0.12",
  "publicIpAddress": "20.114.122.77",
  "resourceGroup": "<my-resource-group>",
  "zones": ""
}

Use the following command to connect to the VM using SSH. Replace publicIpAddress with the value of the public IP address in the response from the previous command:

ssh azureusere@publicIpAddress

When prompted, enter the password you used when creating the VM.

Configure the VM

  1. Use the following commands from the SSH session to install the CLI and Docker:

    # setup docker
    sudo apt-get update -y && sudo apt install docker.io -y && sudo snap install docker && docker --version && sudo usermod -aG docker $USER
    # setup az cli and ml extension
    curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash && az extension add --upgrade -n ml -y
    
  2. To create the environment variables used by this example, run the following commands. Replace <YOUR_SUBSCRIPTION_ID> with your Azure subscription ID. Replace <YOUR_RESOURCE_GROUP> with the resource group that contains your workspace. Replace <SUFFIX_USED_IN_SETUP> with the suffix you provided earlier. Replace <LOCATION> with the location of your Azure workspace. Replace <YOUR_ENDPOINT_NAME> with the name to use for the endpoint.

    Tip

    Use the tabs to select whether you want to perform a deployment using an MLflow model or generic ML model.

    export SUBSCRIPTION="<YOUR_SUBSCRIPTION_ID>"
    export RESOURCE_GROUP="<YOUR_RESOURCE_GROUP>"
    export LOCATION="<LOCATION>"
    
    # SUFFIX that was used when creating the workspace resources. Alternatively the resource names can be looked up from the resource group after the vnet setup script has completed.
    export SUFFIX="<SUFFIX_USED_IN_SETUP>"
    
    # SUFFIX used during the initial setup. Alternatively the resource names can be looked up from the resource group after the  setup script has completed.
    export WORKSPACE=mlw-$SUFFIX
    export ACR_NAME=cr$SUFFIX
    
    # provide a unique name for the endpoint
    export ENDPOINT_NAME="<YOUR_ENDPOINT_NAME>"
    
    # name of the image that will be built for this sample and pushed into acr - no need to change this
    export IMAGE_NAME="img"
    
    # Yaml files that will be used to create endpoint and deployment. These are relative to azureml-examples/cli/ directory. Do not change these
    export ENDPOINT_FILE_PATH="endpoints/online/managed/vnet/sample/endpoint.yml"
    export DEPLOYMENT_FILE_PATH="endpoints/online/managed/vnet/sample/blue-deployment-vnet.yml"
    export SAMPLE_REQUEST_PATH="endpoints/online/managed/vnet/sample/sample-request.json"
    export ENV_DIR_PATH="endpoints/online/managed/vnet/sample/environment"
    
  3. To sign in to the Azure CLI in the VM environment, use the following command:

    az login
    
  4. To configure the defaults for the CLI, use the following commands:

    # configure cli defaults
    az account set --subscription $SUBSCRIPTION
    az configure --defaults group=$RESOURCE_GROUP workspace=$WORKSPACE location=$LOCATION
    
  5. To clone the example files for the deployment, use the following command:

    sudo mkdir -p /home/samples; sudo git clone -b main --depth 1 https://github.com/Azure/azureml-examples.git /home/samples/azureml-examples
    
  6. To build a custom docker image to use with the deployment, use the following commands:

    # Navigate to the samples
    cd /home/samples/azureml-examples/cli/$ENV_DIR_PATH
    # login to acr. Optionally, to avoid using sudo, complete the docker post install steps: https://docs.docker.com/engine/install/linux-postinstall/
    sudo az acr login -n $ACR_NAME
    # Build the docker image with the sample docker file
    sudo docker build -t $ACR_NAME.azurecr.io/repo/$IMAGE_NAME:v1 .
    # push the image to the ACR
    sudo docker push $ACR_NAME.azurecr.io/repo/$IMAGE_NAME:v1
    # check if the image exists in acr
    az acr repository show -n $ACR_NAME --repository repo/$IMAGE_NAME
    

    Tip

    In this example, we build the Docker image before pushing it to Azure Container Registry. Alternatively, you can build the image in your vnet by using an Azure Machine Learning compute cluster and environments. For more information, see Secure Azure Machine Learning workspace.

Create a secured managed online endpoint

  1. To create a managed online endpoint that is secured using a private endpoint for inbound and outbound communication, use the following commands:

    Tip

    You can test or debug the Docker image locally by using the --local flag when creating the deployment. For more information, see the Deploy and debug locally article.

    # navigate to the cli directory in the azurem-examples repo
    cd /home/samples/azureml-examples/cli/
    
    # create endpoint
    az ml online-endpoint create --name $ENDPOINT_NAME -f $ENDPOINT_FILE_PATH --set public_network_access="disabled"
    # create deployment in managed vnet
    az ml online-deployment create --name blue --endpoint $ENDPOINT_NAME -f $DEPLOYMENT_FILE_PATH --all-traffic --set environment.image="$ACR_NAME.azurecr.io/repo/$IMAGE_NAME:v1" egress_public_network_access="disabled"
    
  2. To make a scoring request with the endpoint, use the following commands:

    # Try scoring using the CLI
    az ml online-endpoint invoke --name $ENDPOINT_NAME --request-file $SAMPLE_REQUEST_PATH
    
    # Try scoring using curl
    ENDPOINT_KEY=$(az ml online-endpoint get-credentials -n $ENDPOINT_NAME -o tsv --query primaryKey)
    SCORING_URI=$(az ml online-endpoint show -n $ENDPOINT_NAME -o tsv --query scoring_uri)
    curl --request POST "$SCORING_URI" --header "Authorization: Bearer $ENDPOINT_KEY" --header 'Content-Type: application/json' --data @$SAMPLE_REQUEST_PATH
    

Cleanup

To delete the endpoint, use the following command:

az ml online-endpoint delete --name $ENDPOINT_NAME --yes --no-wait

To delete the VM, use the following command:

az vm delete -n $VM_NAME -y --no-wait

To delete all the resources created in this article, use the following command. Replace <resource-group-name> with the name of the resource group used in this example:

az group delete --resource-group <resource-group-name>

Troubleshooting

Online endpoint creation fails with a V1LegacyMode == true message

The Azure Machine Learning workspace can be configured for v1_legacy_mode, which disables v2 APIs. Managed online endpoints are a feature of the v2 API platform, and won't work if v1_legacy_mode is enabled for the workspace.

Important

Check with your network security team before disabling v1_legacy_mode. It may have been enabled by your network security team for a reason.

For information on how to disable v1_legacy_mode, see Network isolation with v2.

Online endpoint creation with key-based authentication fails

Use the following command to list the network rules of the Azure Key Vault for your workspace. Replace <keyvault-name> with the name of your key vault:

az keyvault network-rule list -n <keyvault-name>

The response for this command is similar to the following JSON document:

{
    "bypass": "AzureServices",
    "defaultAction": "Deny",
    "ipRules": [],
    "virtualNetworkRules": []
}

If the value of bypass isn't AzureServices, use the guidance in the Configure key vault network settings to set it to AzureServices.

Online deployments fail with an image download error

  1. Check if the egress-public-network-access flag is disabled for the deployment. If this flag is enabled, and the visibility of the container registry is private, then this failure is expected.

  2. Use the following command to check the status of the private endpoint connection. Replace <registry-name> with the name of the Azure Container Registry for your workspace:

    az acr private-endpoint-connection list -r <registry-name> --query "[?privateLinkServiceConnectionState.description=='Egress for Microsoft.MachineLearningServices/workspaces/onlineEndpoints'].{Name:name, status:privateLinkServiceConnectionState.status}"
    

    In the response document, verify that the status field is set to Approved. If it isn't approved, use the following command to approve it. Replace <private-endpoint-name> with the name returned from the previous command:

    az network private-endpoint-connection approve -n <private-endpoint-name>
    

Scoring endpoint can't be resolved

  1. Verify that the client issuing the scoring request is a virtual network that can access the Azure Machine Learning workspace.

  2. Use the nslookup command on the endpoint hostname to retrieve the IP address information:

    nslookup endpointname.westcentralus.inference.ml.azure.com
    

    The response contains an address. This address should be in the range provided by the virtual network.

  3. If the host name isn't resolved by the nslookup command, check if an A record exists in the private DNS zone for the virtual network. To check the records, use the following command:

    az network private-dns record-set list -z privatelink.api.azureml.ms -o tsv --query [].name
    

    The results should contain an entry that is similar to *.<GUID>.inference.<region>.

  4. If no inference value is returned, delete the private endpoint for the workspace and then recreate it. For more information, see How to configure a private endpoint.

Online deployments can't be scored

  1. Use the following command to see if the deployment was successfully deployed:

    az ml online-deployment show -e <endpointname> -n <deploymentname> --query '{name:name,state:provisioning_state}' 
    

    If the deployment completed successfully, the value of state will be Succeeded.

  2. If the deployment was successful, use the following command to check that traffic is assigned to the deployment. Replace <endpointname> with the name of your endpoint:

    az ml online-endpoint show -n <endpointname>  --query traffic
    

    Tip

    This step isn't needed if you are using the azureml-model-deployment header in your request to target this deployment.

    The response from this command should list percentage of traffic assigned to deployments.

  3. If the traffic assignments (or deployment header) are set correctly, use the following command to get the logs for the endpoint. Replace <endpointname> with the name of the endpoint, and <deploymentname> with the deployment:

    az ml online-deployment get-logs  -e <endpointname> -n <deploymentname> 
    

    Look through the logs to see if there's a problem running the scoring code when you submit a request to the deployment.

Next steps