Machine Learning CI Pipeline: Submitting an experiment failed

Question

Hello everybody.

I am currently working on a CI Pipeline for a Machine Learning Model.

At the Task where the training should happen my code fails at submitting an experiment.

I get following Error:

  File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\azureml\_restclient\snapshots_client.py", line 137, in create_snapshot
    return self.create_snapshot(file_or_folder_path, retry_on_failure=False)
  File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\azureml\_restclient\snapshots_client.py", line 139, in create_snapshot
    raise SnapshotException(get_http_exception_response_string(response))
azureml.exceptions._azureml_exception.SnapshotException: SnapshotException:
    Message: {
    "error_details": {
        "componentName": "project",
        "correlation": {
            "operation": "dsda3eb68326da4fa76b73560e39c8ac7",
            "request": "23876ee7f425484f"
        },
        "environment": "westeurope",
        "error": {
            "code": "UserError",
            "innerError": {
                "code": "NotFoundError"
            },
            "message": "Unable to find storage with address: Primary = 'https://blobstorage354836.blob.core.windows.net/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d'; Secondary = 'https://blobstorage354836-secondary.blob.core.windows.net/snapshots/b8f82531-be8a-44dc-b851-780e7a05486d'"
        },
        "location": "northeurope",
        "time": "2021-07-07T11:39:05.0508466+00:00"
    },
    "status_code": 404,
    "url": "https://westeurope.experiments.azureml.net/content/v2.0/subscriptions/13f6ec8e-c4c1-4b2e-9f8b-80e2f17b0306/resourceGroups/ZA-GR-Prod-RPD/providers/Microsoft.MachineLearningServices/workspaces/Project_X/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d"
}
    InnerException None
    ErrorResponse 
{
    "error": {
        "message": "{\n    \"error_details\": {\n        \"componentName\": \"project\",\n        \"correlation\": {\n            \"operation\": \"d636a3eb634da4fa76b7230e39c8ac7\",\n            \"request\": \"40f26ee7f456384f\"\n        },\n        \"environment\": \"northeurope\",\n        \"error\": {\n            \"code\": \"UserError\",\n            \"innerError\": {\n                \"code\": \"NotFoundError\"\n            },\n            \"message\": \"Unable to find storage with address: Primary = 'https://blobstorage354836.blob.core.windows.net/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d'; Secondary = 'https://blobstorage354836-secondary.blob.core.windows.net/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d'\"\n        },\n        \"location\": \"westeurope\",\n        \"time\": \"2021-07-07T11:39:05.0508466+00:00\"\n    },\n    \"status_code\": 404,\n    \"url\": \"https://westeurope.experiments.azureml.net/content/v2.0/subscriptions/13f6ec8e-c4c1-4b2e-9f8b-80e2f17b0306/resourceGroups/Ressource_cencored/providers/Microsoft.MachineLearningServices/workspaces/Projet_X/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d\"\n}"

Can anyone help me?

Thanks.

Answer 1

Hello,

Hope you have solved this issue already. For issue like client application receives an HTTP 404 (Not found) message from the server, this implies that the object the client was attempting to use (such as an entity, table, blob, container, or queue) does not exist in the storage service. There are a number of possible reasons for this, such as:

The client or another process previously deleted the object : https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#client-previously-deleted-the-object

A Shared Access Signature (SAS) authorization issue: https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#SAS-authorization-issue

Client-side JavaScript code does not have permission to access the object: https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#JavaScript-code-does-not-have-permission

Network failure: https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#network-failure

SAS issue will be the most possible reason based on my experience. Please do let us know if you are still blocked by this issue.

Regards,
Yutong