Azure Machine Learning resync keys not working - no longer able to access files/submit experiments to ML service

Jim 1 Reputation point
2020-06-11T16:31:55.17+00:00

I followed this tutorial: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-change-storage-access-key - to update the access keys of my default blob datastore linked to my Machine Learning service. This worked a month ago. After doing it today, every time when submitting an experiment I get the error:
"Job preparation failed: HTTP Error 403: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature"

When opening log files for an experiment I get:
"403: You are not authorized to access this resource."

The resync keys operation from the command line works, as I can see in the Machine Learning service Activity log. The Python SDK code also works, as I tried updating the resource group name or default datastore and this gets through.

Any ideas on how to fix this? I am at a loss.

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,714 questions
{count} votes

1 answer

Sort by: Most helpful
  1. romungi-MSFT 43,691 Reputation points Microsoft Employee
    2020-06-16T10:35:20.733+00:00

    Updating this thread with the resolution steps performed by our team in this scenario.

    Issue:

    After updating the storage access keys, the job submission was failing with 403

    Cause:

    The values for the storage keys was cached in the storage connection strings at the backend

    Resolution:

    The engineering team manually deleted the cached storage keys from the backend

    @Jim-9799 Please feel free to accept this as answer so any other user can follow the steps.

    0 comments No comments