Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
This feature is in Beta. To use it, a workspace admin must turn on Files API V3 with Large File Upload Support from the Previews page. See Manage Azure Databricks previews.
Multipart upload lets you upload large files to Unity Catalog volumes by splitting them into smaller parts that upload independently and are then assembled into a single file. This approach follows the same model used by major cloud providers.
A single-request upload, which uploads a file in a single operation, accepts files up to 5000 MiB. To upload a larger file, or a file whose total size is unknown when the upload starts, use multipart upload.
Requirements
- Files API 3.0. Files API 2.0 does not support multipart upload. See Files API versions.
- Send requests to your workspace's regional hostname. See Use the Files API with the regional hostname.
- Network connectivity to Azure Databricks and to your storage. See Network connectivity for the Files API.
How multipart upload works
Multipart upload has three steps:
- Initiate the upload to start a session and receive an upload ID.
- Upload parts, sending each part's content with the upload ID and a part number.
- Complete the upload by providing the list of uploaded parts.
You can also cancel an in-progress upload at any time with the abort-upload action.
Part numbers range from 1 to 10000. Every part except the last must be at least 5 MiB, and each part can be at most 4000 MiB. Azure Databricks assembles the parts in ascending part-number order when the upload completes.
Example
The following example uploads a file in two parts.
To start the upload session, initiate the upload. The response contains the upload_id that identifies the session:
POST /api/3.0/fs/files/Volumes/c/s/v/file?action=initiate-upload
200 OK
Content-Type: application/json
{
"upload_id": "dvOR6FE39mNb"
}
To upload the first part, send its content with the upload_id and part_number. The response includes the part's ETag:
PUT /api/3.0/fs/files/Volumes/c/s/v/file?upload_id=dvOR6FE39mNb&part_number=1
Content-Length: 268435456
Content-Type: application/octet-stream
<binary data>
200 OK
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"
To upload the second part, repeat the request with the next part_number:
PUT /api/3.0/fs/files/Volumes/c/s/v/file?upload_id=dvOR6FE39mNb&part_number=2
Content-Length: 268435456
Content-Type: application/octet-stream
<binary data>
200 OK
ETag: "954fb694be262821177cfbc47f9d4a0aa06c349b"
To finish, complete the upload with a body that lists every part and its stored ETag:
POST /api/3.0/fs/files/Volumes/c/s/v/file?action=complete-upload&upload_id=dvOR6FE39mNb
Content-Type: application/json
{
"parts": [
{ "part_number": 1, "etag": "\"33a64df551425fcc55e4d42a148795d9f25f89d4\"" },
{ "part_number": 2, "etag": "\"954fb694be262821177cfbc47f9d4a0aa06c349b\"" }
]
}
200 OK
ETag: "b3f1c2a94d6e7085f2a1c0b9e8d7654321fedcba"
The ETag in this response belongs to the whole assembled file, not to a part. You can use it in later conditional requests, for example to overwrite the file only if it has not changed since you uploaded it.
To cancel the upload instead of completing it, send the abort-upload action with the upload_id:
POST /api/3.0/fs/files/Volumes/c/s/v/file?action=abort-upload&upload_id=dvOR6FE39mNb
200 OK