Use the Databricks REST API

Article
05/11/2023

The Databricks REST APIs enable you to automate Azure Databricks by using common REST API utilities and libraries.

REST API reference

Azure Databricks has three REST APIs that perform different tasks:

2.0 and 2.1 for general administration
1.2 for running commands directly on Azure Databricks

For the latest version of all REST APIs, see REST API (latest). You can also jump directly to the REST API home pages for each version: 2.1, 2.0, or 1.2.

Important

To access Databricks REST APIs, you must authenticate.

Account API 2.0
Clusters API 2.0
Cluster Policies API 2.0
Data Lineage API 2.0
Databricks SQL Queries, Dashboards, and Alerts API 2.0
Databricks SQL Query History API 2.0
Databricks SQL Statement Execution API 2.0
Databricks SQL Warehouses API 2.0
DBFS API 2.0
Delta Live Tables API 2.0
Git Credentials API 2.0
Global Init Scripts API 2.0
Groups API 2.0
Instance Pools API 2.0
IP Access List API 2.0
Jobs API 2.1, 2.0
Libraries API 2.0
MLflow API 2.0
Permissions API 2.0
Repos API 2.0
SCIM API 2.0
Secrets API 2.0
Serving Endpoints API 2.0
Token API 2.0
Token Management API 2.0
Unity Catalog API 2.1
Workspace API 2.0
API 1.2

Call the REST API

Databricks REST API calls typically include the following components:

The workspace URL of your Azure Databricks deployment.
The REST API operation type, such as GET, POST, PATCH, or DELETE.
The REST API operation path, such as /api/2.0/clusters/get to get information for a specified cluster .
Azure Databricks authentication information, such as an access token.
Any request body (typically in JSON format) or request query parameters that are required by the REST API operation.

The preceding REST API reference articles provide information about the type, path, any necessary request body or request query parameters, and the response (typically in JSON format), for each supported operation. Most of these articles also provide request body and response examples.

A few examples of how to use curl to specify these components are provided later in this article. To adapt these examples for other common REST API utilities and libraries, see the provider’s documentation.

Authentication

Before you can call the REST API, you must be able to authenticate with Azure Databricks. To authenticate, you typically reference an Azure Databricks personal access token or an Azure Active Directory (Azure AD) token from each REST API call. See also how to use tokens.

Rate limits

To ensure high quality of service under heavy load, Databricks enforces rate limits for all REST API calls. Limits are set per endpoint and per workspace to ensure fair usage and high availability.

Requests that exceed the rate limit return a 429 response status code.

For information on rate limits for API requests, see API rate limits.

Use curl to call the REST API

curl is a popular utility for transferring data to and from servers.

While most API calls require that you specify a JSON body, for GET calls you can specify a query string by appending it after ? and surrounding the URL in quotes. If you use curl, you can specify --get (or -G) and --data (or -d) along with the query string; you do not need to surround the URL or the query string in quotes.

In the following examples, replace https://adb-1234567890123456.7.azuredatabricks.net with the workspace URL of your Azure Databricks deployment. The per-workspace instance ID should start with adb-. Do not use the deprecated regional URL starting with <azure-region-name>. It may not work for new workspaces, will be less reliable, and will exhibit lower performance than per-workspace URLs.

This example prints information about the specified cluster. This example uses a .netrc file.

Using ?:

curl --netrc 'https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/get?cluster_id=1234-567890-patch123'

Using --get and --data:

curl --netrc --get \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/get \
--data cluster_id=1234-567890-batch123

Response:

{
  "cluster_id": "1234-567890-batch123",
  "driver": {
    "node_id": "123ab456789012345cd67e8e90123f45",
    "instance_id": "234ab456789012345cd67e8e90123f45",
    "start_timestamp": 1618448261567,
    "host_private_ip": "10.0.0.0",
    "private_ip": "10.0.0.0"
  },
  ...
}

This example lists the contents of the DBFS root. This example uses a .netrc file.

curl --netrc --get \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/dbfs/list \
--data path=/

Response:

"files": [
  {
    "path": "/tmp",
    "is_dir": true,
    "file_size": 0,
    "modification_time": 1547078156000
  },
  {
    "path": "/my_file.txt",
    "is_dir": false,
    "file_size": 40,
    "modification_time": 1541374426000
  },
  ...
]

Runtime version strings

Many API calls require you to specify a Databricks runtime version string. This section describes the structure of a version string in the Databricks REST API.

<M>.<F>.x[-cpu][-esr][-gpu][-ml][-photon]-scala<scala-version>

where

M: Databricks Runtime major release
F: Databricks Runtime feature release
cpu: CPU version (with -ml only)
esr: Extended Support
gpu: GPU-enabled
ml: Machine learning
photon: Photon
scala-version: version of Scala used to compile Spark: 2.10, 2.11, or 2.12

For example:

7.6.x-gpu-ml-scala2.12 represents Databricks Runtime 7.6 for Machine Learning, is GPU-enabled, and uses Scala version 2.12 to compile Spark version 3.0.1

The Supported Azure Databricks runtime releases and support schedule and Unsupported releases tables map Databricks Runtime versions to the Spark version contained in the runtime.

You can get a list of available Azure Databricks runtime version strings by calling the Runtime versions API.

Databricks Light

apache-spark.<M>.<F>.x-scala<scala-version>