Machine Learning CI Pipeline: Submitting an experiment failed

Haider Stefan 6 Reputation points
2021-07-07T11:54:21.767+00:00

Hello everybody.

I am currently working on a CI Pipeline for a Machine Learning Model.

At the Task where the training should happen my code fails at submitting an experiment.

I get following Error:

  File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\azureml\_restclient\snapshots_client.py", line 137, in create_snapshot
    return self.create_snapshot(file_or_folder_path, retry_on_failure=False)
  File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\azureml\_restclient\snapshots_client.py", line 139, in create_snapshot
    raise SnapshotException(get_http_exception_response_string(response))
azureml.exceptions._azureml_exception.SnapshotException: SnapshotException:
    Message: {
    "error_details": {
        "componentName": "project",
        "correlation": {
            "operation": "dsda3eb68326da4fa76b73560e39c8ac7",
            "request": "23876ee7f425484f"
        },
        "environment": "westeurope",
        "error": {
            "code": "UserError",
            "innerError": {
                "code": "NotFoundError"
            },
            "message": "Unable to find storage with address: Primary = 'https://blobstorage354836.blob.core.windows.net/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d'; Secondary = 'https://blobstorage354836-secondary.blob.core.windows.net/snapshots/b8f82531-be8a-44dc-b851-780e7a05486d'"
        },
        "location": "northeurope",
        "time": "2021-07-07T11:39:05.0508466+00:00"
    },
    "status_code": 404,
    "url": "https://westeurope.experiments.azureml.net/content/v2.0/subscriptions/13f6ec8e-c4c1-4b2e-9f8b-80e2f17b0306/resourceGroups/ZA-GR-Prod-RPD/providers/Microsoft.MachineLearningServices/workspaces/Project_X/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d"
}
    InnerException None
    ErrorResponse 
{
    "error": {
        "message": "{\n    \"error_details\": {\n        \"componentName\": \"project\",\n        \"correlation\": {\n            \"operation\": \"d636a3eb634da4fa76b7230e39c8ac7\",\n            \"request\": \"40f26ee7f456384f\"\n        },\n        \"environment\": \"northeurope\",\n        \"error\": {\n            \"code\": \"UserError\",\n            \"innerError\": {\n                \"code\": \"NotFoundError\"\n            },\n            \"message\": \"Unable to find storage with address: Primary = 'https://blobstorage354836.blob.core.windows.net/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d'; Secondary = 'https://blobstorage354836-secondary.blob.core.windows.net/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d'\"\n        },\n        \"location\": \"westeurope\",\n        \"time\": \"2021-07-07T11:39:05.0508466+00:00\"\n    },\n    \"status_code\": 404,\n    \"url\": \"https://westeurope.experiments.azureml.net/content/v2.0/subscriptions/13f6ec8e-c4c1-4b2e-9f8b-80e2f17b0306/resourceGroups/Ressource_cencored/providers/Microsoft.MachineLearningServices/workspaces/Projet_X/snapshots/b8f73541-be8a-44dc-b851-780e7a05486d\"\n}"

Can anyone help me?

Thanks.

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,540 questions
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
7,064 questions
{count} votes

1 answer

Sort by: Most helpful
  1. YutongTie-MSFT 45,906 Reputation points
    2021-08-05T03:00:44.467+00:00

    Hello,

    Hope you have solved this issue already. For issue like client application receives an HTTP 404 (Not found) message from the server, this implies that the object the client was attempting to use (such as an entity, table, blob, container, or queue) does not exist in the storage service. There are a number of possible reasons for this, such as:

    The client or another process previously deleted the object : https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#client-previously-deleted-the-object

    A Shared Access Signature (SAS) authorization issue: https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#SAS-authorization-issue

    Client-side JavaScript code does not have permission to access the object: https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#JavaScript-code-does-not-have-permission

    Network failure: https://learn.microsoft.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#network-failure

    SAS issue will be the most possible reason based on my experience. Please do let us know if you are still blocked by this issue.

    Regards,
    Yutong

    0 comments No comments