How can I fix strange magic command behaviour in Databricks notebooks?

Question

How can I fix strange magic command behaviour in Databricks notebooks?

Joseph Sharman 1

Hi,

I have encountered a strange issue when using magic commands on my Azure Databricks workspace.

When I create a cell and type the magic command for a language which is not default in the notebook, the code is instead interpreted as the default notebook language.

Here you can see a notebook with python as the default language, with a sql magic command:

And here, the same notebook. But with python code written in the cell.

However, if I create a cell at the end of the notebook (not in between existing cells). And then set the language to SQL using this button, the code will be correctly interpreted as SQL.

This issue first arose this week on Monday the 26th of September. It has broken a lot of my existing notebooks which use "%sql".

I have tried restarting my cluster, creating a new cluster, creating a new notebook, and changing my notebook default language. But none of these have fixed the issue.

Thanks,
Joe

Edit:
This is the Log4j output when I try and run a %sql cell when I have typed "%sql" rather than using the UI to select sql.
"ERROR WSFSDriverManager$: Files only supported within a repository. Invalid base notebook path: /Users/{email}/{folder}{notebook}"

SSingh-MSFT 16,371 Reputation points Moderator

2022-10-04T09:50:06.68+00:00

Hi @Joseph Sharman ,

Welcome to Microsoft Q&A portal and thanks for using Azure Services.

As I understood from the issue mentioned, I tried to execute SQL magic command in between cells in Python default notebook and gives me output as shown below:

Could you please check if the python file is saved as a Workspace notebook and not under DBFS?

Anticipating your response. Thanks!

2 answers

Your answer

SSingh-MSFT 16,371 Reputation points Moderator

2022-10-04T09:50:06.68+00:00

Hi @Joseph Sharman ,

Welcome to Microsoft Q&A portal and thanks for using Azure Services.

As I understood from the issue mentioned, I tried to execute SQL magic command in between cells in Python default notebook and gives me output as shown below:

Could you please check if the python file is saved as a Workspace notebook and not under DBFS?

Anticipating your response. Thanks!

Answer 1

Joseph Sharman 1

Hi @SSingh-MSFT

Thanks for your reply!

That output you have is what I had previously (before I started having issues).

I believe the notebook in in workspace, not DBFS. I wasn't entirely sure how to check, but from this I believe it is in Workspace?

Thanks,
Joe

SSingh-MSFT 16,371 Reputation points Moderator

2022-10-04T10:26:26.413+00:00
Hi @Joseph Sharman ,

This looks strange.

If we focus on "ERROR WSFSDriverManager$: Files only supported within a repository. Invalid base notebook path: /Users/{email}/{folder}{notebook}", could you please check the Git repository as below:

You can only link a notebook to an initialized Git repository that isn’t empty. Test the URL in a web browser.

The GitHub personal access token must be active.

If a notebook is linked to a GitHub branch that is renamed, the change is not automatically reflected in Databricks. You must re-link the notebook to the branch manually.

Could you please check if above pointers are intact. Please reply in case of any relevant information you may want to share for better investigation.

Answer 2

Joseph Sharman 1

Hi,

I don't have Git repositories linked to Databricks, and I have not had one set up in the past.

Therefore I think none of these points are an issue.

SSingh-MSFT 16,371 Reputation points Moderator

2022-10-06T06:02:57.69+00:00

Hi @Joseph Sharman ,

Thanks for the confirmation about Git.

Could you please check if this issue exists in other Notebooks too within the Workspace? If there are other Environments, I request you to check there too the same so that we can narrow down the spectrum and troubleshoot.

Thanks for your patience.
Joseph Sharman 1 Reputation point

2022-10-06T10:12:14.51+00:00

Hi,

Yes the problem persists in every notebook in the workspace. Existing notebooks and newly creates ones. However, the problem does not exist in other databricks environments I have.
SSingh-MSFT 16,371 Reputation points Moderator

2022-10-06T11:53:12.843+00:00

Hi @Joseph Sharman ,

Thanks for replying with details.

Since notebooks are working fine in other environment, please check your network settings, in case you have any firewall or rules that may be blocking the connection or some issue in spark config.

Could you please share any relevant information about the recent changes made into this environment?
SSingh-MSFT 16,371 Reputation points Moderator

2022-10-10T07:09:11.05+00:00

Hi @Joseph Sharman ,

Just checking if you are still facing the issue or it is resolved.

Please let us know with your findings on the above details regarding your environment which is showing error to further troubleshoot.

Thanks!
Joseph Sharman 1 Reputation point

2022-10-10T14:33:41.08+00:00

Hi,

The issue is still unresolved. I haven't made any recent changes to my databricks environment. Where should I be looking to check my networking settings?
SSingh-MSFT 16,371 Reputation points Moderator

2022-10-11T10:48:21.8+00:00

Hi @Joseph Sharman ,

Sorry about the inconvenience you are facing.

Could you please check things mentioned below:

1). In Databricks workspace, under Admin Console, please check User Status:

2). Please check if Databricks service is Active in Portal:

3). Check Access in IAM:

4). Check Workspace Settings are proper in Admin Console:

5). Please refer to the Manage Virtual Network page for further checks related to Network settings.

Let me know if above settings help in your issue.

Thanks.
Joseph Sharman 1 Reputation point

2022-10-11T14:42:20.36+00:00

Hi,

All of the settings are the same on my side, apart from "Workspace Access Control". I cannot enable this as I am not on the premium plan.
SSingh-MSFT 16,371 Reputation points Moderator

2022-10-13T08:55:02.953+00:00

Hi @Joseph Sharman ,

I agree that this issue looks strange, and I wasn't able to reproduce this issue. If you have a support plan, could you please file a support ticket for deeper investigation and in case if you don't have a support plan, do let us know here so that we can check on other options.

Thanks

Share via

How can I fix strange magic command behaviour in Databricks notebooks?

2 answers

Your answer