When I create a notebook in Databricks workplace, I can get its Filepath by right clicking on the notebook name and clicking on "Copy file path". However, I don't understand well where it is located. Could you give me some insight? Also, can I get a list of all notebooks' filepath using "%fs ls" command?

Hello @Mohammad Saber , Thanks for the question and using MS Q&A platform. Unfortuantely, you cannot use %fs ls or dbutils.fs.ls to get the list of all notebooks. Note: You can get what you need via the the REST API or Databricks CLI. REST API: List command of Workspaces REST API Databricks CLI: List objects in a workspace Hope this will help. Please let us know if any further queries. ------------------------------ Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how Want a reminder to come back and check responses? Here is how to subscribe to a notification If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators

When I create a notebook in Databricks workplace, where is located?

Accepted answer

PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee

2022-09-22T05:59:06.93+00:00
Hello @Mohammad Saber ,

Thanks for the question and using MS Q&A platform.

Unfortuantely, you cannot use %fs ls or dbutils.fs.ls to get the list of all notebooks.

Note: You can get what you need via the the REST API or Databricks CLI.

REST API: List command of Workspaces REST API

Databricks CLI: List objects in a workspace

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators
Please sign in to rate this answer.
Mohammad Saber 586 Reputation points

2022-09-22T08:50:10.223+00:00

Thanks.

Are all notebooks stored on DBFS?

If I have multi nodes, is notebook stored on Driver node or Worker node?

PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee

2022-09-22T09:10:00.923+00:00

Hello @Mohammad Saber ,

Notebook results are stored in workspace system data storage, which is not accessible by users.

Mohammad Saber 586 Reputation points

2022-09-22T23:14:44.997+00:00

Hi @PRADEEPCHEEKATLA-MSFT

Thanks for your help.

I didn't understand well.

"Notebook results are stored in workspace system data storage".

"Notebook" & "Notebook results" are stored in different location?

Is " workspace system data storage" different from DBFS? If this concept is explained in the documentation, could you please share it with me? I couldn't find good explanation about it.

Thanks.

PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee

2022-09-23T06:38:22.757+00:00

Hello @Mohammad Saber ,

Yes, workspace storage is different from DBFS.

DBFS is an abstraction over the cloud storage implementations that allow you to work with files in cloud storage using simple paths instead of full URLs. From documentation:

Databricks File System (DBFS) is a distributed file system mounted into a Databricks workspace and available on Databricks clusters. DBFS is an abstraction on top of scalable object storage and offers the following benefits:

Allows you to mount storage objects so that you can seamlessly access data without requiring credentials.

Allows you to interact with object storage using directory and file semantics instead of storage URLs.

Persists files to object storage, so you won’t lose data after you terminate a cluster.

Under the hood, on Azure it uses the same ADLS.

Please note that there is so-called DBFS Root - created from the storage account that is created automatically during workspace creation, and DBFS mounts to "external" storage accounts. It's generally recommended to use DBFS Root only for temporary files, because if you delete workspace, that storage account will be removed as well.

Mohammad Saber 586 Reputation points

2022-09-23T07:17:37.29+00:00

@PRADEEPCHEEKATLA-MSFT

Thanks a lot.

The first image shows the Cloud Storage that DBFS is abstracted from. Is that correct?

Is there any way that I can see the content of Workspace System Data Storage in Azure portal?

Mohammad Saber 586 Reputation points

2022-09-23T07:29:16.38+00:00

And there is still something confusing which is storage location when I have Driver and Worker nodes.

Could you let me know if Workspace System Data Storage is located on Driver node? Or Azure creates a cloud storage along with cluster nodes as Workspace System Data Storage?

What about Tables and datasets? Where are these objects are stored?

Thanks for your help.

PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee

2022-09-26T05:46:33.607+00:00

Hello @Mohammad Saber ,

Is there any way that I can see the content of Workspace System Data Storage in Azure portal?

No, you can get what you need via the the REST API or Databricks CLI.

Could you let me know if Workspace System Data Storage is located on Driver node? Or Azure creates a cloud storage along with cluster nodes as Workspace System Data Storage?

Workspace system data storage is stored in the Azure Storage account as which is created in the managed resource group as shown above.

What about Tables and datasets? Where are these objects are stored?

Tables and datasets are stored in the DBFS which is on the same storage account and accesses able via DBFS Explorer.
Sign in to comment

When I create a notebook in Databricks workplace, where is located?

0 additional answers