Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,579 questions
Container does not exist error when trying to invoke ML Batch Endpoint
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 92%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETS%3C/text%3E%3C/svg%3E)
Tom Smith
0
Reputation points
So I have a Batch Endpoint deployed in ML studio.
I am able to run a job successfully when manually creating a job for this endpoint from within the portal using a datastore, which is just a mounted container of images in ML studio.
I have been attempting to invoke this endpoint using both the python SDK as well as through the rest API with .NET, however in both scenarios the job fails with the following error.
Failed to download snapshot from storage using SAS url. Error message: The specified container does not exist.
RequestId:563bdada-501e-0078-4ee1-8e27c8000000
Time:2023-05-25T08:21:05.1296437Z, Storage account name: mystorageaccount, Storage error code: ContainerNotFound, Storage path: https://mystorageaccount.blob.core.windows.net/132fbb23-7-a6c501d9-9a17-5ec2-a46a-80dee62dee15/script/__pycache__/score.cpython-37.pyc. Storage container does not exist for storage account name: mystorageaccount, container name: 132fbb23-7-a6c501d9-9a17-5ec2-a46a-80dee62dee14 Tip: Please try to re-upload the snapshot. To do so, edit any content in the local source directory and resubmit the run.
To clarify, the container referenced in the url above is nothing like the url I send.
I am invoking the endpoint as per documentation like this:
job = ml_client.batch_endpoints.invoke(
endpoint_name=endpoint_name,
input= Input(type=AssetTypes.URI_Folder, path='https://mystorageaccount.blob.core.windows.net/test-batch-images')
)
The container does exist, and I have tried this on multiple containers with the same error. I have also tried setting the public access level to both blob as well as container and still receive the same error.
I have ensured I have followed the authentication principles lined out here:
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-data-batch-endpoints-jobs?view=azureml-api-2&tabs=cli under 'Security considerations when reading data', where my compute cluster has a managed Identity associated with it that has the correct access to the storage account.
I have also tried generating an sas url for the container and passing that as an argument, but this returns an error when invoking saying 'URL contains secrets, must store secrets in datastore.'
There's nothing in the documentation about invoking an endpoint with a secret for the attached data. I would have assumed this was managed behind the scenes given that you have to add a managed identity to the compute cluster to access containers.
AM I missing something? is there a way to separately authenticate the created docker container running the batch?
{count} votes
-
Tom Smith 0 Reputation points
2023-05-25T11:59:23.9166667+00:00 Embarrassingly after two days of trying to fix this issue I have solved it an hour after posting in the forum. This error was caused due to some kind of corrupt authentication as the batch endpoint spins up when mounting the scoring script.
I simply reuploaded the deployment and it worked.
-
romungi-MSFT 42,311 Reputation points Microsoft Employee2023-05-25T12:08:09.42+00:00 @Tom Smith I think you are using the same storage account for input as the one that is attached to your Azure ML workspace. The container name that the error is referring to is probably an artifact of the model that it is trying to invoke but is not available. Is it possible to upload the same data from your input container to the default data store in the Azure Machine Learning workspace and then run a batch deployment on it.
Something similar to calling the endpoint as mentioned in this section?
default_ds = ml_client.datastores.get_default() data_path = "heart-disease-uci-unlabeled" input = Input(type=AssetTypes.URI_FOLDER, path=f"{default_ds.id}/paths/{data_path}) job = ml_client.batch_endpoints.invoke( endpoint_name=endpoint.name, input=input)Could you also try using a different container for inputs?
If it still fails this could be a deployment issue of the endpoint rather than an issue with passing the input.
-
romungi-MSFT 42,311 Reputation points Microsoft Employee
2023-05-31T11:19:54.6066667+00:00 @Tom Smith Did you get a chance to review the suggestion above and check if a different container works?
Sign in to comment