Use the Files API with Unity Catalog volumes

Use the Files API to read, write, list, and delete files and directories by path. It works with files in Unity Catalog volumes, which you refer to by their absolute volume path, such as /Volumes/<catalog>/<schema>/<volume>/<path>. Files API 3.0 requests use the /api/3.0/fs endpoint prefix and go to your workspace's regional hostname. For every operation, parameter, and header, see the Files API 3.0 reference.

Use of the Files API might incur Azure Databricks data transfer charges.

Send requests to the regional hostname

Important

This feature is in Beta. To use it, a workspace admin must turn on Files API V3 with Large File Upload Support from the Previews page. See Manage Azure Databricks previews.

Send Files API 3.0 requests to the regional hostname of the workspace that the requesting user belongs to. The regional hostname has the following format:

<workspace-id>.storage.<workspace-region>.<databricks-domain>

For example, 1234567890123456.storage.westus.azuredatabricks.net.

To find your workspace ID and region, use the account console. See Workspace instance names, URLs, and IDs. You can also find the workspace ID in the deployment URL: in https://<databricks-instance>/?o=1234567890123456, the number after o= is the workspace ID. The workspace UI shows the region below the workspace name.

Configure network connectivity

Important

This feature is in Beta. To use it, a workspace admin must turn on Files API V3 with Large File Upload Support from the Previews page. See Manage Azure Databricks previews.

Files API 3.0 requires network connectivity from your network to Azure Databricks and from Azure Databricks to your storage.

From your network to Azure Databricks

If you allow public internet access to your workspace but restrict it with a firewall, allowlist the control plane IP ranges for your region.

Include the IPs under Control Plane IPs, including default storage and webapp. See Inbound to Azure Databricks control plane.

If you don't allow internet access to Azure Databricks and connect through a front-end private link, create a second front-end private link for performance-intensive services. See Configure inbound Private Link for performance-intensive services.

From Azure Databricks to your storage

If you allow public access to your storage but apply network restrictions, for in-region storage, add your storage accounts to an Azure network security perimeter (NSP), then allowlist the AzureDatabricksServerless service tag. The network security perimeter must stay in Transition mode. Otherwise, allowlist the Azure Databricks outbound IPs.

If you don't allow public access to your storage, configure a private link to your Azure storage. See Configure private connectivity to Azure resources.

Note

Serverless network billing applies. See Understand Databricks networking costs.

Upload large files with multipart upload

Important

This feature is in Beta. To use it, a workspace admin must turn on Files API V3 with Large File Upload Support from the Previews page. See Manage Azure Databricks previews.

A single upload request accepts files up to 5000 MiB. To upload a larger file, or a file whose size is unknown when the upload starts, use multipart upload: you upload the file in parts, and Azure Databricks assembles them into a single file.

Part numbers range from 1 to 10000. Every part except the last must be at least 5 MiB, and each part can be at most 4000 MiB. Azure Databricks assembles the parts in ascending part-number order. For every parameter, see Upload a file and Perform an action on a file.

Multipart upload example

The following example uploads a file in two parts.

To start the upload session, initiate the upload. The response contains the upload_id that identifies the session:

POST /api/3.0/fs/files/Volumes/c/s/v/file?action=initiate-upload

200 OK
Content-Type: application/json
{
  "upload_id": "dvOR6FE39mNb"
}

To upload the first part, send its content with the upload_id and part_number. The response includes the part's ETag:

PUT /api/3.0/fs/files/Volumes/c/s/v/file?upload_id=dvOR6FE39mNb&part_number=1
Content-Length: 268435456
Content-Type: application/octet-stream
<binary data>

200 OK
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

To upload the second part, repeat the request with the next part_number:

PUT /api/3.0/fs/files/Volumes/c/s/v/file?upload_id=dvOR6FE39mNb&part_number=2
Content-Length: 268435456
Content-Type: application/octet-stream
<binary data>

200 OK
ETag: "954fb694be262821177cfbc47f9d4a0aa06c349b"

To finish, complete the upload with a body that lists every part and its stored ETag:

POST /api/3.0/fs/files/Volumes/c/s/v/file?action=complete-upload&upload_id=dvOR6FE39mNb
Content-Type: application/json
{
  "parts": [
    { "part_number": 1, "etag": "\"33a64df551425fcc55e4d42a148795d9f25f89d4\"" },
    { "part_number": 2, "etag": "\"954fb694be262821177cfbc47f9d4a0aa06c349b\"" }
  ]
}

200 OK
ETag: "b3f1c2a94d6e7085f2a1c0b9e8d7654321fedcba"

The ETag in this response belongs to the whole assembled file, not to a part. You can use it in later conditional requests, for example to overwrite the file only if it hasn't changed since you uploaded it.

To cancel the upload instead of completing it, send the abort-upload action with the upload_id:

POST /api/3.0/fs/files/Volumes/c/s/v/file?action=abort-upload&upload_id=dvOR6FE39mNb

200 OK

Download large files with range requests

Use range requests to download a large file in several smaller requests, or to resume an interrupted download.

Request a range

Send the Range header with the zero-based positions of the first and last bytes you want. The range includes both positions. For example, bytes=0-1048575 requests the first 1 MiB of a 10 MiB file:

GET /api/3.0/fs/files/Volumes/main/default/my-volume/large-file.bin
Range: bytes=0-1048575

206 Partial Content
Accept-Ranges: bytes
Content-Range: bytes 0-1048575/10485760
Content-Length: 1048576
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"
<binary data>

The 206 Partial Content response contains only the requested bytes, and Content-Range shows which bytes it contains and the file's total size. To read to the end of the file, omit the last position, as in bytes=1048576-. If the last position is past the end of the file, the response contains the bytes up to the end. For other Range syntax, see Download a file.

A HEAD request with a Range header returns the same status and headers as the matching GET request, without the body.

Resume an interrupted download

To resume a download, request the remaining bytes with a range that starts at the first byte you haven't received. Also send If-Match with the ETag from your first response, so you don't combine bytes from two versions of the file:

GET /api/3.0/fs/files/Volumes/main/default/my-volume/large-file.bin
Range: bytes=1048576-
If-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"

206 Partial Content
Accept-Ranges: bytes
Content-Range: bytes 1048576-10485759/10485760
Content-Length: 9437184
<binary data>

If the file changed, the server returns 412 Precondition Failed, and you must start the download again.

Use conditional requests

Important

This feature is in Beta. To use it, a workspace admin must turn on Files API V3 with Large File Upload Support from the Previews page. See Manage Azure Databricks previews.

A conditional request takes effect only if the file is in the state you expect. Use conditional requests to avoid repeat downloads, to keep concurrent writers from overwriting each other's changes, and to create or delete files safely. Files API 3.0 supports the four standard HTTP conditional headers from RFC 9110:

Header Compares against Honored on
If-Match The file's ETag Reads, writes, and deletes
If-None-Match The file's ETag Reads and writes
If-Modified-Since The file's Last-Modified time Reads only
If-Unmodified-Since The file's Last-Modified time Reads only

When the condition isn't met, the server doesn't perform the operation and returns one of the following statuses:

  • 304 Not Modified, when a GET or HEAD request with If-None-Match or If-Modified-Since fails. Your cached copy is still valid.
  • 412 Precondition Failed, when any other condition fails.

Validators: ETag and Last-Modified

Conditional headers compare a value you supply with one of the file's validators. The Files API returns both validators as response headers on GET and HEAD:

  • ETag: an opaque identifier for the file's content, also returned on PUT and the multipart complete-upload response. Store it exactly as returned, including the surrounding quotes, and don't parse or construct it.
  • Last-Modified: the file's last modification time, as an HTTP-date. Its precision is one second, so If-Modified-Since and If-Unmodified-Since can't detect several changes within the same second. For concurrency control, use ETag with If-Match or If-None-Match.

Conditional reads

The most common conditional read is cache revalidation. Send If-None-Match with the ETag of your cached copy, or If-Modified-Since with the time you last fetched it. If the file is unchanged, the server returns 304 Not Modified with no body. Otherwise, it returns 200 OK with the new content:

GET /api/3.0/fs/files/Volumes/main/default/reports/report.csv
If-None-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"

304 Not Modified

If-Match and If-Unmodified-Since work the other way: the server returns the file only if it hasn't changed, and otherwise returns 412. A read can include at most two conditional headers, and their precedence follows RFC 9110.

Conditional writes

On PUT, use If-None-Match and If-Match to avoid overwriting changes you haven't seen, also called the lost update problem:

  • If-None-Match: * uploads the file only if no file exists at the path. Otherwise, the request returns 412 Precondition Failed.
  • If-Match: <etag> uploads the file only if its current ETag matches the one you supply. Otherwise, the request returns 412 Precondition Failed, or 404 if no file exists at the path.

To create a file only if it doesn't already exist, send If-None-Match: *:

PUT /api/3.0/fs/files/Volumes/main/default/reports/report.csv
If-None-Match: *
Content-Type: application/octet-stream
<binary data>

200 OK

To overwrite a file only if it's still the version you last fetched, send its ETag with If-Match:

PUT /api/3.0/fs/files/Volumes/main/default/reports/report.csv
If-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"
Content-Type: application/octet-stream
<binary data>

200 OK

For a multipart upload, send the conditional header on the initiate-upload call. The Files API evaluates it when the upload completes, so the condition applies to the assembled file. For the header forms that each call supports, see Upload a file and Perform an action on a file.

Conditional deletes

A plain DELETE succeeds even if the file doesn't exist, so it can't tell you whether you removed the version you intended. To make a delete conditional, send If-Match:

  • If-Match: <etag> deletes the file only if its current ETag matches the one you supply. If someone changed the file after you fetched it, the request returns 412 Precondition Failed. If the file no longer exists, it returns 404.
  • If-Match: * deletes the file only if it exists, and returns 404 if it doesn't.
DELETE /api/3.0/fs/files/Volumes/main/default/reports/report.csv
If-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"

204 No Content

Files API versions

Files API 2.0 sends requests to your workspace hostname with the /api/2.0/fs endpoint prefix. It doesn't support multipart upload, and its only conditional header is If-Unmodified-Since on downloads. See the Files API 2.0 reference.

To move code from Files API 2.0 to 3.0, make the following changes:

  • Last updated on