Azure Databrick workspace linking to Azure Machine Learning workspace error unable to get Workspace.from_config()

Question

We are in a POC of Azure Databricks and Azure Machine Learning integration so that we can advance our MLOps practice.

We have been dealing the with workspace linking issue for the past few weeks and have hit a wall. We worked with Databricks and search the internet for solution but were not successful. As AML is still in Preview, documentation may not be all there.

I think this example below is the most recent and calls out how the linking should work. Is the statement highlighted below correct and if so, is there detail configuration that we are missing besides clicking the "Link Azure ML Workspace" button?

[https://tsmatz.github.io/azure-databricks-exercise/exercise10-mlflow.html][1]

Note : Here (in this hands-on) we connect to an Azure Machine Learning workspace by running Python code, however, you can now use the following "Link Azure ML Workspace" button (simplified integrated experience) in Azure Databricks launcher page to connect a new or existing workspace. Once you have linked with this experience, you don't need to run the following ws.write_config(), Workspace.from_config(), and mlflow.set_tracking_uri().

We have done the following.

1. Linked the workspace via "Link Azure ML Workspace" button
2. Run the attached MLFLow + AML – Combined.dbc
3. Failure in Cmd 15
   ws = Workspace.from_config()

UserErrorException: UserErrorException:

UserErrorException Traceback (most recent call last)
<command-4338604541924184> in <module>
3
4
----> 5 ws = Workspace.from_config()
6 exp = "/adb/XXXXXXXXXXXXXXX/XXXXXXXXXXXX/Users/######@#######.com/MLFlow + AML - Combined"
7 runs = list(exp.get_runs())

/databricks/python/lib/python3.7/site-packages/azureml/core/workspace.py in from_config(path, auth, _logger, _file_name)
272 'We could not find config.json in: {} or in its parent directories. '
273 'Please provide the full path to the config file or ensure that '
--> 274 'config.json exists in the parent directories.'.format(normalized_path))
275
276 subscription_id, resource_group, workspace_name = project_info.get_workspace_info(

UserErrorException: UserErrorException:
Message: We could not find config.json in: /databricks/driver or in its parent directories. Please provide the full path to the config file or ensure that config.json exists in the parent directories.
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "We could not find config.json in: /databricks/driver or in its parent directories. Please provide the full path to the config file or ensure that config.json exists in the parent directories."
}
}
4. Config.json manually place into directory path specified in error message

5. Perform workspace write_config(). But, forces interactive authentication which is blocked by our organization policy.

Besides performing the "Link Azure ML Workspace" via the button from Databricks, we have not performed any other configuration. Based on the documentation, it should just work. Appreciate any assistance.

Answer 1

@tkdebug-4414 Thanks for the details. Can you please try to authenticate successfully using the following approach.

from azureml.core.authentication import InteractiveLoginAuthentication
interactive_auth = InteractiveLoginAuthentication(tenant_id="<tenantID>")

ws = Workspace(subscription_id="<>",
resource_group="rg-test-auseast-aidf-ml",
workspace_name="mlworkspace-test-auseast-aidf",
auth=interactive_auth)