How do I access AutoML pretrained models when Azure Machine Learning Workspace is set to "allow only approved outbound"?

Question

I have an Azure Machine Learning Workspace, where the networking settings are as follows:

Public access is set to "Enabled from selected IP addresses" to only open for relevant, internal IP addresses.

Workspace managed outbound access is set to "Allow only approved outbound".

Upon creation of the workspace, the setup creates a number of required outbound rules, with the relevant endpoints and service tags, to connect to other internal Azure services, as expected.

My machine learning model relies on AutoML, and part of the model initialization and training, requires access to pretrained model weights that are provided by Microsoft. However, when I try to execute a training run of my model pipeline, the pipeline fails with the following:

Downloading: "https://aka.ms/automl-resources/data/models-vision-pretrained/mobilenet_v2-b0353104.pth"
Encountered URLError error while loading pretrained model. Retries left: 4. Error details: <urlopen error EOF occurred in violation of protocol (_ssl.c:1129)>

To me, this suggests that my model code cannot reach the model repository, hosted by Microsoft. I suspect that this is somehow related to the networking settings of my ML Workspace, as I have no such issue when I use a workspace that is wide open, connection-wise.

I have tried added an FQDM rule to the workspace managed outbound access settings, for "aka.ms", but that has not made any difference. The code still fails the same way, when attempting to download the model weights. The ML code executes on a compute cluster created in the workspace, with a system assigned managed identity.

How do I set up my environment to allow my code to download the required model weights when I execute it?

Answer 1

Hello aot,

Thank you for posting your question in the Microsoft Q&A forum. 

When working with Azure Machine Learning (AML) workspaces configured with strict network security policies, users often encounter challenges accessing external resources required for model training. This situation becomes particularly apparent when attempting to download pretrained model weights from Microsoft-hosted repositories during AutoML workflows. The error message indicating a URL fetch failure with SSL protocol violations typically points to network restrictions blocking outbound connections to essential endpoints. 

Few important things to keep in mind about keeping things secure while still making sure everything works properly. Right now, your workspace is set to "Allow only approved outbound" access, which means it needs specific permission for certain Microsoft services. The automatic setup does include most of the important connections, but it might miss some resources that AutoML needs - especially those coming from aka.ms links, which point to files stored in Azure Blob Storage. 

To resolve this while preserving security, administrators should implement a multi-layered approach. The solution begins with expanding the workspace's outbound rules as below: 

Allow Outbound Access to Microsoft Model Repositories 

Add the following service tags to the workspace’s outbound rules: 

• AzureActiveDirectory (for authentication). 
• AzureResourceManager (for AML service dependencies). 
• AzureFrontDoor.Frontend (for aka.ms redirections). 
• Storage (for Blob Storage hosting model weights). 
• FQDN Rules: Explicitly allow: 
  • *.aka.ms (redirects to Azure Storage). 
  • *.blob.core.windows.net (actual model storage). 
  • *.azureml.ms (AML-specific endpoints). 

For compute clusters running the training jobs, network security groups must permit HTTPS traffic on port 443 to these endpoints. The system-assigned managed identity used by the cluster also requires proper role assignments, particularly Storage Blob Data Reader permissions on the workspace's default storage account. These configurations work together to create a secure but functional environment where AutoML can access necessary pretrained models while maintaining overall network security posture. 

If outbound rules remain problematic, below are few alternate solutions you may try: 

• Download Weights Offline: Manually download the model weights to your workspace’s default Blob Storage or a Datastore, then modify the AutoML code to use the local path. 
• Use Azure Private Endpoints: If strict isolation is required, configure Private Endpoints for Azure Storage and AML services (though this may not resolve aka.ms access). 

Some Microsoft refereence documentation may help: 

• Secure workspace resources using virtual networks (VNets) - Azure Machine Learning | Microsoft Learn 
• Announcing Azure Machine Learning managed network isolation | Microsoft Community Hub 

If the above answer helped, please do not forget to "Accept Answer" as this may help other community members to refer the info if facing a similar issue. Your contribution to the Microsoft Q&A community is highly appreciated.