Databricks clusters suddenly cannot run notebooks 'Cloud provider storage unavailable'

Question

Hello.

Our three environments have been running notebooks and jobs fine for some time. But suddenly as of last week, we are only able to run notebooks manually. Running even the most basic notebook in a job (i.e. a notebook that just prints 'hello world') produces the error:

“Run failed with error message Cloud provider storage unavailable”

This affects all three environments and no changes to the clusters were made on our part. We use Unity Catalog for external storage access and had recently made changes to add external tables to the catalog but don't think this is the issue as the other two environments haven't had these changes added yet and they also produce the same error.

I know that when you run a notebook manually vs running a job, the cluster accesses the credentials differently. Is there anything obvious we can test to get to the bottom of this?

Many thanks

Answer 1

Adam Peel - Thanks for the question and using MS Q&A platform.

This error usually points to an issue with cloud storage credentials or access policies when jobs run on clusters. You’re correct that interactive notebooks vs. jobs use different credential passthrough mechanisms in Databricks:

• Interactive notebooks often use your personal identity (via Azure AD passthrough or AWS STS).
• Jobs typically rely on service principals or cluster-level credentials configured in the workspace.

Since this affects all environments and started suddenly, here are the most likely causes and steps to troubleshoot:

Step1: Verify External Location Config in Unity Catalog

Go to Data → External Locations in Databricks.

• Check:
  • Credential type: Managed Identity or Service Principal.
  • Storage URL: abfss://<container>@<storageaccount>.dfs.core.windows.net/ Credential status: Should show Active.

Run this SQL in a notebook:

DESCRIBE EXTERNAL LOCATION <location_name>;

Confirm the credential_name is correct.

Step2: Check Managed Identity or Service Principal Permissions

• If using Managed Identity:
  • In Azure Portal → Storage Account → IAM, ensure the identity has:
  • Storage Blob Data Contributor
  • Storage Blob Data Reader
• If using Service Principal:
  • Validate the secret hasn’t expired.
  • Check RBAC roles on the storage account.

Step3: Validate Cluster Access Mode

• For Unity Catalog jobs:
  • Cluster must be Single User or Shared with UC enabled. If using Credential Passthrough, confirm Enable Credential Passthrough is ON.

Jobs fail if cluster is No Isolation Shared or missing UC config.

Step4: Test Storage Access from Job Cluster

Create a test job with this command:

dbutils.fs.ls("abfss://<container>@<storageaccount>.dfs.core.windows.net/")

If this fails, it’s a credential or network issue.

Step5: Check Network Restrictions

• If storage account has firewall/VNet rules, ensure:
  • Databricks subnets are allowed. Managed Identity can access via Private Endpoint if configured.

Step6: Inspect Job Logs

Look for:

AuthenticationFailed → Credential expired or missing.

403 Forbidden → RBAC issue.

Timeout → Firewall/VNet issue.

✅ Common Fixes

• Rotate Service Principal secret if expired.
• Reassign Managed Identity roles if removed.
• Update External Location credential in Unity Catalog.
• Switch cluster to Single User mode for testing.

Hope this helps. Let me know if you have any further questions or need additional assistance. Also, if these answer your query, do click the "Upvote" and click "Accept the answer" of which might be beneficial to other community members reading this thread.

---

𝘛𝘰 𝘴𝘵𝘢𝘺 𝘪𝘯𝘧𝘰𝘳𝘮𝘦𝘥 𝘢𝘣𝘰𝘶𝘵 𝘵𝘩𝘦 𝘭𝘢𝘵𝘦𝘴𝘵 𝘶𝘱𝘥𝘢𝘵𝘦𝘴 𝘢𝘯𝘥 𝘪𝘯𝘴𝘪𝘨𝘩𝘵𝘴 𝘰𝘯 𝘈𝘻𝘶𝘳𝘦 𝘋𝘢𝘵𝘢𝘣𝘳𝘪𝘤𝘬𝘴, 𝘥𝘢𝘵𝘢 𝘦𝘯𝘨𝘪𝘯𝘦𝘦𝘳𝘪𝘯𝘨, 𝘢𝘯𝘥 Data & AI 𝘪𝘯𝘯𝘰𝘷𝘢𝘵𝘪𝘰𝘯𝘴, 𝘧𝘰𝘭𝘭𝘰𝘸 𝘮𝘦 𝘰𝘯 𝘓𝘪𝘯𝘬𝘦𝘥𝘐𝘯.