I need to download files from databricks onto my local machine

Question

My attempt at using the url method returns a 401 error "credential was not sent or was of an unsupported type for this API"

My attempt as using the cdi is also unsuccessful.

I do not have the download option available in the dbfs, I have the crednetials and the dbfs browser option is toggled on.

What am I missing here?

Shouldn't be this complex to download a file.

Answer 1

Hi @Jaden Materi

Thanks for posting your query!

A 401 Unauthorized error usually occurs when the API request doesn’t include valid authentication credentials. This can happen if the Bearer token is missing, malformed, or incorrect. To resolve this, ensure you're using the correct personal access token in the Authorization header when making your API request. The token should be passed like this: Authorization: Bearer <your_token>

When using the DBFS paths, ensure you're using the correct format. Databricks DBFS paths are prefixed with dbfs:/. Ensure that when you reference the file, you are not mistakenly using a local file path instead of DBFS paths.

Double-check your permissions to ensure you have the necessary role to download files from DBFS. Your Databricks user may not have permission to download the files, which can be restricted by the workspace admin.

Check the CLI setup and token configuration. Ensure the CLI is installed correctly and is properly configured with your token.

you should be able to use commands like below to copy files from DBFS to your local machine.

databricks fs cp dbfs:/path/to/your/file /local/path –recursive

You access the file through the FileStore URL, your browser will automatically trigger the download. Ensure that FileStore is enabled and that you have the correct permissions to access the file.You can check the below commands:

 
Move the file to FileStore:

dbutils.fs.cp("dbfs:/path/to/your/file", "dbfs:/FileStore/your_file")

Access the file via URL:

https://<databricks-instance-url>/files/your_file

Ref:
https://docs.databricks.com/aws/en/archive/dev-tools/cli/dbfs-cli

https://docs.databricks.com/aws/en/archive/legacy/filestore

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.