How do we get the folder path of the file inside the workspace - Azure Databricks

Sharath 20 Reputation points
2023-06-29T21:36:03.5333333+00:00

I'm using the Databricks REST API to create the folder under the workspace, so that the folder will be accessed controlled as shown below

URL - {{baseUrl}}/2.0/workspace/mkdirs

Body -

{"path":"/dde9540f/cert"}

It's getting successfully created, I can see under the workspace and also list the files under the newly created folder using Databricks List REST API

List API - {{baseUrl}}/2.0/workspace/list?path=/dde9540f

Response -

{

"objects": [ {

         "object_type": "DIRECTORY",
         "path": "/dde9540f/cert",
         "object_id": 142492279484620
     }
 ] }

Then I will pass the path of the certificate to the file system, so that spark can use the provided path to read the certificate under /Workspace/dde9540f/cert/client-keystore.p12 and authenticate with destination file system to read the contents.

What is the valid path for the files under workspace? I tried using the below path, where I have given the read access to all the users.

/Workspace/dde9540f/cert/client-keystore.p12

But ended with the error shown below

Caused by: java.io.IOException: java.nio.file.FileSystemException: /Workspace/dde9540f/cert/client-keystore.p12: Operation not permitted

When I was using the dbfs file system, I was providing the file system path as below and I was able to read certificate successfully. I don't want to use dbfs approach cause DBFS is accessible by all users

/dbfs/dde9540f/cert/client-keystore.p12

Any help would be appreciated

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,517 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.