All of our Azure pipelines are failing - Datastore connection issue

Brittany Perry 0

Our pipelines have been running in the Azure sandbox environment for years - about a week ago all of the pipelines began failing with the following error:

Error Code: ScriptExecution.DatastoreResolution.Authentication Failed Step: d19eb5a7-7859-462f-8652-60104d71b7bd Error Message: ScriptExecutionException was caused by DatastoreResolutionException. DatastoreResolutionException was caused by AuthenticationException. Not authorized to fetch info for the Datastore 'cspo_storage_datastore' for the subscription: e321efb2-bff7-4ec0-a4a7-c5f95abef641, resource group: ctc-sandbox-spanlytics-cc-rg, workspace: cspo-ml-testing. Using base service url: https://canadacentral.api.azureml.ms. Status code: Forbidden. Reason phrase: Authorization failed for cspo-ml-testing. Request id: ca1245c706b31ce65aed514c49dc593d.

The connection has been tested and shows that it is successful. The storage account access key has been updated as well - not sure what else to try or why this error would randomly start appearing?

BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2024-04-17T20:30:04.3733333+00:00

Hello Brittany Perry,

Welcome to the Microsoft Q&A forum.

Per the error message "Authorization failed for cspo-ml-testing", this is due to change in the authentication credentials or permissions for the Datastore.

Could you please check the authentication credentials and permissions for the Datastore 'cspo_storage_datastore'?

also, since you mentioned the keys are updated, Can you please follow the below document and see if you have updated the keys correctly? (to sync the keys)

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-change-storage-access-key?view=azureml-api-2#update-the-access-key
Brittany Perry 0 Reputation points

2024-04-18T12:56:12+00:00

Hello @BhargavaGunnam-MSFT ,

The authentication credentials or permissions have not been changed for the Datastore which is part of why I was so confused when the pipelines began failing out of nowhere. The authentication we use for the Datastore is our storage account key - this has been rotated and I've replaced the old key in the Datastore authentication tab (after the pipelines stopped working) and tested it to show that the connection is successful. That link you provided is the documentation I followed to update the keys - the only thing is I waited the hour for them to sync automatically because due to organization limits, I can't download Azure Cloud Shell to force the sync immediately. I'm not sure what else to try or if there is another place I'm missing where the authentication or key needs to be updated?
BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2024-04-18T22:33:06.2633333+00:00

If the authentication credentials and permissions for the Datastore haven't been changed, and you've already updated the storage account key, then it's possible that the issue is related to the sync delay you mentioned.

Since it has been more than an hour since you updated the storage account key, and the pipelines are still failing with the same error message, there may be another issue causing the authentication to fail.

Could you please check the logs to see if they provide any additional information about the cause of the authentication failure? Additionally, verify the permissions for the storage account that the Datastore is using.

You can also try creating a new Datastore with the updated storage account key and update the pipelines to use the new Datastore. This will help determine if the issue lies with the Datastore or the pipelines themselves.

If the above steps don't resolve the issue, I recommend logging a support case. If you don't have a support case, please let me know so that I can provide a one-time free support request to address this matter.

I hope this information helps! Feel free to reach out if you have any further questions.