ServicePrincipalAuthentication no longer working in Databricks

asked 2021-04-12T16:59:03.07+00:00
AlexW.admin 21 Reputation points

Hi all.

I've had this problem for MONTHS now and, not having the option to give up, I'm getting desperate.

I have a databricks set up where an azure file-share is mounted, and this is used to extract data and read it into a database. Up until Monday it was been working fine.

Recently, although nothing has changed about the way the drive is mounted (via Azure ML libraries):

sp = ServicePrincipalAuthentication(tenant_id="x", # tenantID service_principal_id="y", clientId service_principal_password="z")

clientSecret ws = Workspace.get(name="wsname", auth=sp, subscription_id="a")

Listing the contents of a directory suddenly takes an enormous amount of time to finish (50 minutes), before no longer being able to find the folder. Essentially, it repeatedly tries to switch back to interactive authentication before failing altogether, saying [Errno22]: Invalid Argument.

import os

folder = "/mnt/tmp/xx/a/b/c"

patient_names = os.listdir("/mnt/tmp/xx/a/b/c") print(patient_names)

I'm lost, is there anywhere I should be looking to try and find out what's wrong?

It works fine using Interactive authentication and WAS working with SPA, but suddenly does not.
I've tried:

  • Recreating the datastore and dataset in ML
  • Creating new databricks clusters on which to run the code.
  • Creating another service principal
  • Running the code on my windows work machine with Pycharm and Python 3.7
  • Creating an Ubuntu environment with Pycharm and Python 3.7
  • Creating a new machine learning environment
  • Running the code with the python logging module to see if anything useful has come back.
  • Trying several different versions of azureml-sdk[databricks]
  • Trying yet another service principal. Nothing has worked. I don't understand how/why service principals are suddenly being ignored, why there are no error messages or useful information of any kind, or how nobody else has come across this problem?

Please can someone help?

Thank you.

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
1,619 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,199 questions
{count} votes