Clusters API 2.0

  • Article
  • 28 minutes to read

The Clusters API allows you to create, start, edit, list, terminate, and delete clusters. The maximum allowed size of a request to the Clusters API is 10MB.

Cluster lifecycle methods require a cluster ID, which is returned from Create. To obtain a list of clusters, invoke List.

Azure Databricks maps cluster node instance types to compute units known as DBUs. See the instance type pricing page for a list of the supported instance types and their corresponding DBUs. For instance provider information, see Azure instance type specifications and pricing.

Azure Databricks always provides one year’s deprecation notice before ceasing support for an instance type.

Warning

You should never hard code secrets or store them in plain text. Use the Secrets API 2.0 to manage secrets in the Databricks CLI setup & documentation. Use the Secrets utility (dbutils.secrets) to reference secrets in notebooks and jobs.

Important

To access Databricks REST APIs, you must authenticate.

Create

Endpoint HTTP Method
2.0/clusters/create POST

Create a new Apache Spark cluster. This method acquires new instances from the cloud provider if necessary. This method is asynchronous; the returned cluster_id can be used to poll the cluster state. When this method returns, the cluster is in a PENDING state. The cluster is usable once it enters a RUNNING state. See ClusterState.

Note

Azure Databricks may not be able to acquire some of the requested nodes, due to cloud provider limitations or transient network issues. If it is unable to acquire a sufficient number of the requested nodes, cluster creation will terminate with an informative error message.

Examples

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/create \
--data @create-cluster.json

create-cluster.json:

{
  "cluster_name": "my-cluster",
  "spark_version": "7.3.x-scala2.12",
  "node_type_id": "Standard_D3_v2",
  "spark_conf": {
    "spark.speculation": true
  },
  "num_workers": 25
}

{ "cluster_id": "1234-567890-undid123" }

Here is an example for an autoscaling cluster. This cluster will start with two nodes, the minimum.

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/create \
--data @create-cluster.json

create-cluster.json:

{
  "cluster_name": "autoscaling-cluster",
  "spark_version": "7.3.x-scala2.12",
  "node_type_id": "Standard_D3_v2",
  "autoscale" : {
    "min_workers": 2,
    "max_workers": 50
  }
}

{ "cluster_id": "1234-567890-hared123" }

This example creates a Single Node cluster. To create a Single Node cluster:

  • Set spark_conf and custom_tags to the exact values in the example.
  • Set num_workers to 0.
curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/create \
--data @create-cluster.json

create-cluster.json:

{
  "cluster_name": "single-node-cluster",
  "spark_version": "7.6.x-scala2.12",
  "node_type_id": "Standard_DS3_v2",
  "num_workers": 0,
  "spark_conf": {
    "spark.databricks.cluster.profile": "singleNode",
    "spark.master": "[*, 4]"
  },
  "custom_tags": {
    "ResourceClass": "SingleNode"
  }
}

{ "cluster_id": "1234-567890-pouch123" }

To create a job or submit a run with a new cluster using a policy and the policy’s default values, set policy_id to the policy ID and apply_policy_default_values to true:

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/create \
--data @create-cluster.json

create-cluster.json:


{
    "num_workers": null,
    "autoscale": {
        "min_workers": 2,
        "max_workers": 8
    },
    "cluster_name": "my-cluster",
    "spark_version": "7.3.x-scala2.12",
    "spark_conf": {},
    "node_type_id": "Standard_D3_v2",
    "custom_tags": {},
    "spark_env_vars": {
        "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
    },
    "autotermination_minutes": 120,
    "init_scripts": [],
    "policy_id": "C65B864F02000008",
    "apply_policy_default_values": true
}

To create a new cluster, define the cluster’s properties in new_cluster:

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/job/create \
--data @create-job.json

create-job.json:

{
  "run_name": "my spark task",
  "new_cluster": {
    "spark_version": "7.3.x-scala2.12",
    "node_type_id": "Standard_D3_v2",
    "num_workers": 10,
    "policy_id": "ABCD000000000000"
  },
  "libraries": [
    {
      "jar": "dbfs:/my-jar.jar"
    },
    {
      "maven": {
        "coordinates": "org.jsoup:jsoup:1.7.2"
      }
    }
  ],
  "spark_jar_task": {
    "main_class_name": "com.databricks.ComputeModels"
  }
}

Request structure of the cluster definition

Field Name Type Description
num_workers OR autoscale INT32 OR AutoScale If num_workers, number of worker nodes that this cluster should have. A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in executors will gradually increase from 5 to 10 as the new nodes are provisioned.

If autoscale, parameters needed in order to automatically scale clusters up and down based on load.
cluster_name STRING Cluster name requested by the user. This doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.
spark_version STRING The runtime version of the cluster. You can retrieve a list of available runtime versions by using the Runtime versions API call. This field is required.
spark_conf SparkConfPair An object containing a set of optional, user-specified Spark configuration key-value pairs. You can also pass in a string of extra JVM options to the driver and the executors via
spark.driver.extraJavaOptions and spark.executor.extraJavaOptions respectively.

Example Spark confs:
{"spark.speculation": true, "spark.streaming.ui.retainedBatches": 5} or
{"spark.driver.extraJavaOptions": "-verbose:gc -XX:+PrintGCDetails"}
node_type_id STRING This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads A list of available node types can be retrieved by using the List node types API call. This field is required.
driver_node_type_id STRING The node type of the Spark driver. This field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.
custom_tags ClusterTag An object containing a set of tags for cluster resources. Databricks tags all cluster resources (such as VMs) with these tags in addition to default_tags.

Note:

* Azure Databricks allows at most 43 custom tags.
* If the cluster is created on an instance pool, the cluster tags are not copied to the cluster resources. To tag resources for an instance pool, see the custom_tags field in the Instance Pools API 2.0.
cluster_log_conf ClusterLogConf The configuration for delivering Spark logs to a long-term storage destination. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every
5 mins. The destination of driver logs is <destination>/<cluster-ID>/driver, while the destination of executor logs is <destination>/<cluster-ID>/executor.
init_scripts An array of InitScriptInfo The configuration for storing init scripts. Any number of scripts can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to
<destination>/<cluster-ID>/init_scripts.
docker_image DockerImage Docker image for a custom container.
spark_env_vars SparkEnvPair An object containing a set of optional, user-specified environment variable key-value pairs. Key-value pairs of the form (X,Y) are exported as is (that is,
export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the following example. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables:
{"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or
{"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}
autotermination_minutes INT32 Automatically terminates the cluster after it is inactive for this time in minutes. If not set, this cluster will not be automatically terminated. If specified, the threshold must be between 10 and 10000 minutes. You can also set this value to 0 to explicitly disable automatic termination.
driver_instance_pool_id STRING The ID of the instance pool to use for drivers. You must also specify instance_pool_id. Refer to Instance Pools API 2.0 for details.
instance_pool_id STRING The optional ID of the instance pool to use for cluster nodes. If driver_instance_pool_id is present,
instance_pool_id is used for worker nodes only. Otherwise, it is used for both the driver and the worker nodes. Refer to Instance Pools API 2.0 for details.
idempotency_token STRING An optional token that can be used to guarantee the idempotency of cluster creation requests. If the idempotency token is assigned to a cluster that is not in the TERMINATED state, the request does not create a new cluster but instead returns the ID of the existing cluster. Otherwise, a new cluster is created. The idempotency token is cleared when the cluster is terminated

If you specify the idempotency token, upon failure you can retry until the request succeeds. Azure Databricks will guarantee that exactly one cluster will be launched with that idempotency token.

This token should have at most 64 characters.
apply_policy_default_values BOOL Whether to use policy default values for missing cluster attributes.
enable_local_disk_encryption BOOL Whether encryption of disks locally attached to the cluster is enabled.
azure_attributes AzureAttributes Attributes related to clusters running on Azure. If not specified at cluster creation, a set of default values is used.
runtime_engine STRING The type of runtime engine to use. If not specified, the runtime engine type is inferred based on the
spark_version value. Allowed values include:

* PHOTON: Use the Photon runtime engine type.
* STANDARD: Use the standard runtime engine type.

This field is optional.

Response structure

Field Name Type Description
cluster_id STRING Canonical identifier for the cluster.

Edit

Endpoint HTTP Method
2.0/clusters/edit POST

Edit the configuration of a cluster to match the provided attributes and size.

You can edit a cluster if it is in a RUNNING or TERMINATED state. If you edit a cluster while it is in a RUNNING state, it will be restarted so that the new attributes can take effect. If you edit a cluster while it is in a TERMINATED state, it will remain TERMINATED. The next time it is started using the clusters/start API, the new attributes will take effect. An attempt to edit a cluster in any other state will be rejected with an INVALID_STATE error code.

Clusters created by the Databricks Jobs service cannot be edited.

Example

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/edit \
--data @edit-cluster.json

edit-cluster.json:

{
  "cluster_id": "1202-211320-brick1",
  "num_workers": 10,
  "spark_version": "7.3.x-scala2.12",
  "node_type_id": "Standard_D3_v2"
}

{}

Request structure

Change owner

Endpoint HTTP Method
2.0/clusters/change-owner POST

Changes a cluster’s owner. The new owner must be an admin.

Example

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/change-owner \
--data '{ "cluster_id": "1234-567890-reef123", "owner_username": "someone@example.com" }'

{}

Request structure

Field Name Type Description
cluster_id STRING The cluster whose owner you want to change. This field is required.
owner_username STRING The username of the cluster’s new owner. This field is required.

Response structure

If the request succeeds, an empty response is returned.

Response errors

Error Description
INVALID_PARAMETER_VALUE The cluster ID is not valid.
PERMISSION_DENIED The new owner is not an administrator.
RESOURCE_DOES_NOT_EXIST The username is not valid.

Other errors return BAD_REQUEST.

Start

Endpoint HTTP Method
2.0/clusters/start POST

Start a terminated cluster given its ID. This is similar to createCluster, except:

  • The terminated cluster ID and attributes are preserved.
  • The cluster starts with the last specified cluster size. If the terminated cluster is an autoscaling cluster, the cluster starts with the minimum number of nodes.
  • If the cluster is in the RESTARTING state, a 400 error is returned.
  • You cannot start a cluster launched to run a job.

Example

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/start \
--data '{ "cluster_id": "1234-567890-reef123" }'

{}

Request structure

Field Name Type Description
cluster_id STRING The cluster to be started. This field is required.

Restart

Endpoint HTTP Method
2.0/clusters/restart POST

Restart a cluster given its ID. The cluster must be in the RUNNING state.

Example

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/restart \
--data '{ "cluster_id": "1234-567890-reef123" }'

{}

Request structure

Field Name Type Description
cluster_id STRING The cluster to be started. This field is required.

Resize

Endpoint HTTP Method
2.0/clusters/resize POST

Resize a cluster to have a desired number of workers. The cluster must be in the RUNNING state.

Example

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/resize \
--data '{ "cluster_id": "1234-567890-reef123", "num_workers": 30 }'

{}

Request structure

Field Name Type Description
num_workers OR autoscale INT32 OR AutoScale If num_workers, number of worker nodes that this cluster should have. A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in executors will gradually increase from 5 to 10 as the new nodes are provisioned.

If autoscale, parameters needed in order to automatically scale clusters up and down based on load.
cluster_id STRING The cluster to be resized. This field is required.

Delete (terminate)

Endpoint HTTP Method
2.0/clusters/delete POST

Terminate a cluster given its ID. The cluster is removed asynchronously. Once the termination has completed, the cluster will be in the TERMINATED state. If the cluster is already in a TERMINATING or TERMINATED state, nothing will happen.

Unless a cluster is pinned, 30 days after the cluster is terminated, it is permanently deleted.

Example

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/delete \
--data '{ "cluster_id": "1234-567890-frays123" }'

{}

Request structure

Field Name Type Description
cluster_id STRING The cluster to be terminated. This field is required.

Permanent delete

Endpoint HTTP Method
2.0/clusters/permanent-delete POST

Permanently delete a cluster. If the cluster is running, it is terminated and its resources are asynchronously removed. If the cluster is terminated, then it is immediately removed.

You cannot perform any action, including retrieve the cluster’s permissions, on a permanently deleted cluster. A permanently deleted cluster is also no longer returned in the cluster list.

Example

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/permanent-delete \
--data '{ "cluster_id": "1234-567890-frays123" }'

{}

Request structure

Field Name Type Description
cluster_id STRING The cluster to be permanently deleted. This field is required.

Get

Endpoint HTTP Method
2.0/clusters/get GET

Retrieve the information for a cluster given its identifier. Clusters can be described while they are running or up to 30 days after they are terminated.

Example

curl --netrc -X GET \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/get \
--data '{ "cluster_id": "1234-567890-reef123" }' \
| jq .

{
  "cluster_id": "1234-567890-reef123",
  "driver": {
    "node_id": "dced0ce388954c38abef081f54c18afd",
    "instance_id": "c69c0b119a2a499d8a2843c4d256136a",
    "start_timestamp": 1619718438896,
    "host_private_ip": "10.0.0.1",
    "private_ip": "10.0.0.2"
  },
  "spark_context_id": 5631707659504820000,
  "jdbc_port": 10000,
  "cluster_name": "my-cluster",
  "spark_version": "8.2.x-scala2.12",
  "node_type_id": "Standard_L4s",
  "driver_node_type_id": "Standard_L4s",
  "custom_tags": {
    "ResourceClass": "SingleNode"
  },
  "autotermination_minutes": 0,
  "enable_elastic_disk": true,
  "disk_spec": {},
  "cluster_source": "UI",
  "enable_local_disk_encryption": false,
  "azure_attributes": {
    "first_on_demand": 1,
    "availability": "ON_DEMAND_AZURE",
    "spot_bid_max_price": -1
  },
  "instance_source": {
    "node_type_id": "Standard_L4s"
  },
  "driver_instance_source": {
    "node_type_id": "Standard_L4s"
  },
  "state": "RUNNING",
  "state_message": "",
  "start_time": 1610745129764,
  "last_state_loss_time": 1619718513513,
  "num_workers": 0,
  "cluster_memory_mb": 32768,
  "cluster_cores": 4,
  "default_tags": {
    "Vendor": "Databricks",
    "Creator": "someone@example.com",
    "ClusterName": "my-cluster",
    "ClusterId": "1234-567890-reef123"
  },
  "creator_user_name": "someone@example.com",
  "pinned_by_user_name": "3401478490056118",
  "init_scripts_safe_mode": false
}

Request structure

Field Name Type Description
cluster_id STRING The cluster about which to retrieve information. This field is required.

Response structure

Pin

Note

You must be an Azure Databricks administrator to invoke this API.

Endpoint HTTP Method
2.0/clusters/pin POST

Ensure that an all-purpose cluster configuration is retained even after a cluster has been terminated for more than 30 days. Pinning ensures that the cluster is always returned by the List API. Pinning a cluster that is already pinned has no effect.

Example

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/pin \
--data '{ "cluster_id": "1234-567890-reef123" }'

{}

Request structure

Field Name Type Description
cluster_id STRING The cluster to pin. This field is required.

Unpin

Note

You must be an Azure Databricks administrator to invoke this API.

Endpoint HTTP Method
2.0/clusters/unpin POST

Allows the cluster to eventually be removed from the list returned by the List API. Unpinning a cluster that is not pinned has no effect.

Example

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/unpin \
--data '{ "cluster_id": "1234-567890-reef123" }'

{}

Request structure

Field Name Type Description
cluster_id STRING The cluster to unpin. This field is required.

List

Endpoint HTTP Method
2.0/clusters/list GET

Return information about all pinned clusters, active clusters, up to 200 of the most recently terminated all-purpose clusters in the past 30 days, and up to 30 of the most recently terminated job clusters in the past 30 days. For example, if there is 1 pinned cluster, 4 active clusters, 45 terminated all-purpose clusters in the past 30 days, and 50 terminated job clusters in the past 30 days, then this API returns the 1 pinned cluster, 4 active clusters, all 45 terminated all-purpose clusters, and the 30 most recently terminated job clusters.

Example

curl --netrc -X GET \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/list \
| jq .

{
  "clusters": [
    {
      "cluster_id": "1234-567890-reef123",
      "driver": {
        "node_id": "dced0ce388954c38abef081f54c18afd",
        "instance_id": "c69c0b119a2a499d8a2843c4d256136a",
        "start_timestamp": 1619718438896,
        "host_private_ip": "10.0.0.1",
        "private_ip": "10.0.0.2"
      },
      "spark_context_id": 5631707659504820000,
      "jdbc_port": 10000,
      "cluster_name": "my-cluster",
      "spark_version": "8.2.x-scala2.12",
      "node_type_id": "Standard_L4s",
      "driver_node_type_id": "Standard_L4s",
      "custom_tags": {
        "ResourceClass": "SingleNode"
      },
      "autotermination_minutes": 0,
      "enable_elastic_disk": true,
      "disk_spec": {},
      "cluster_source": "UI",
      "enable_local_disk_encryption": false,
      "azure_attributes": {
        "first_on_demand": 1,
        "availability": "ON_DEMAND_AZURE",
        "spot_bid_max_price": -1
      },
      "instance_source": {
        "node_type_id": "Standard_L4s"
      },
      "driver_instance_source": {
        "node_type_id": "Standard_L4s"
      },
      "state": "RUNNING",
      "state_message": "",
      "start_time": 1610745129764,
      "last_state_loss_time": 1619718513513,
      "num_workers": 0,
      "cluster_memory_mb": 32768,
      "cluster_cores": 4,
      "default_tags": {
        "Vendor": "Databricks",
        "Creator": "someone@example.com",
        "ClusterName": "my-cluster",
        "ClusterId": "1234-567890-reef123"
      },
      "creator_user_name": "someone@example.com",
      "pinned_by_user_name": "3401478490056118",
      "init_scripts_safe_mode": false
    },
    {
      "..."
    }
  ]
}

Response structure

Field Name Type Description
clusters An array of ClusterInfo A list of clusters.

List node types

Endpoint HTTP Method
2.0/clusters/list-node-types GET

Return a list of supported Spark node types. These node types can be used to launch a cluster.

Example

curl --netrc -X GET \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/list-node-types \
| jq .

{
  "node_types": [
    {
      "node_type_id": "Standard_L80s_v2",
      "memory_mb": 655360,
      "num_cores": 80,
      "description": "Standard_L80s_v2",
      "instance_type_id": "Standard_L80s_v2",
      "is_deprecated": false,
      "category": "Storage Optimized",
      "support_ebs_volumes": true,
      "support_cluster_tags": true,
      "num_gpus": 0,
      "node_instance_type": {
        "instance_type_id": "Standard_L80s_v2",
        "local_disks": 1,
        "local_disk_size_gb": 800,
        "instance_family": "Standard LSv2 Family vCPUs",
        "local_nvme_disk_size_gb": 1788,
        "local_nvme_disks": 10,
        "swap_size": "10g"
      },
      "is_hidden": false,
      "support_port_forwarding": true,
      "display_order": 0,
      "is_io_cache_enabled": true,
      "node_info": {
        "available_core_quota": 350,
        "total_core_quota": 350
      }
    },
    {
      "..."
    }
  ]
}

Response structure

Field Name Type Description
node_types An array of NodeType The list of available Spark node types.

Runtime versions

Endpoint HTTP Method
2.0/clusters/spark-versions GET

Return the list of available runtime versions. These versions can be used to launch a cluster.

Example

curl --netrc -X GET \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/spark-versions \
| jq .

{
  "versions": [
    {
      "key": "8.2.x-scala2.12",
      "name": "8.2 (includes Apache Spark 3.1.1, Scala 2.12)"
    },
    {
      "..."
    }
  ]
}

Response structure

Field Name Type Description
versions An array of SparkVersion All the available runtime versions.

Events

Endpoint HTTP Method
2.0/clusters/events POST

Retrieve a list of events about the activity of a cluster. You can retrieve events from active clusters (running, pending, or reconfiguring) and terminated clusters within 30 days of their last termination. This API is paginated. If there are more events to read, the response includes all the parameters necessary to request the next page of events.

Example:

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/events \
--data @list-events.json \
| jq .

list-events.json:

{
  "cluster_id": "1234-567890-reef123",
  "start_time": 1617238800000,
  "end_time": 1619485200000,
  "order": "DESC",
  "offset": 5,
  "limit": 5,
  "event_types": [ "RUNNING" ]
}

{
  "events": [
    {
      "cluster_id": "1234-567890-reef123",
      "timestamp": 1619471498409,
      "type": "RUNNING",
      "details": {
        "current_num_workers": 2,
        "target_num_workers": 2
      }
    },
    {
      "..."
    }
  ],
  "next_page": {
    "cluster_id": "1234-567890-reef123",
    "start_time": 1617238800000,
    "end_time": 1619485200000,
    "order": "DESC",
    "offset": 10,
    "limit": 5
  },
  "total_count": 25
}

Example request to retrieve the next page of events:

curl --netrc -X POST \
https://adb-1234567890123456.7.azuredatabricks.net/api/2.0/clusters/events \
--data @list-events.json \
| jq .

list-events.json:

{
  "cluster_id": "1234-567890-reef123",
  "start_time": 1617238800000,
  "end_time": 1619485200000,
  "order": "DESC",
  "offset": 10,
  "limit": 5,
  "event_types": [ "RUNNING" ]
}

{
  "events": [
    {
      "cluster_id": "1234-567890-reef123",
      "timestamp": 1618330776302,
      "type": "RUNNING",
      "details": {
        "current_num_workers": 2,
        "target_num_workers": 2
      }
    },
    {
      "..."
    }
  ],
  "next_page": {
    "cluster_id": "1234-567890-reef123",
    "start_time": 1617238800000,
    "end_time": 1619485200000,
    "order": "DESC",
    "offset": 15,
    "limit": 5
  },
  "total_count": 25
}

Request structure

Retrieve events pertaining to a specific cluster.

Field Name Type Description
cluster_id STRING The ID of the cluster to retrieve events about. This field is required.
start_time INT64 The start time in epoch milliseconds. If empty, returns events starting from the beginning of time.
end_time INT64 The end time in epoch milliseconds. If empty, returns events up to the current time.
order ListOrder The order to list events in; either ASC or DESC. Defaults to DESC.
event_types An array of ClusterEventType An optional set of event types to filter on. If empty, all event types are returned.
offset INT64 The offset in the result set. Defaults to 0 (no offset). When an offset is specified and the results are requested in descending order, the end_time field is required.
limit INT64 The maximum number of events to include in a page of events. Defaults to 50, and maximum allowed value is 500.

Response structure

Field Name Type Description
events An array of ClusterEvent This list of matching events.
next_page Request structure The parameters required to retrieve the next page of events. Omitted if there are no more events to read.
total_count INT64 The total number of events filtered by the start_time, end_time, and event_types.

Data structures

In this section:

AutoScale

Range defining the min and max number of cluster workers.

Field Name Type Description
min_workers INT32 The minimum number of workers to which the cluster can scale down when underutilized. It is also the initial number of workers the cluster will have after creation.
max_workers INT32 The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.

ClusterInfo

Metadata about a cluster.

ClusterEvent

Cluster event information.

Field Name Type Description
cluster_id STRING Canonical identifier for the cluster. This field is required.
timestamp INT64 The timestamp when the event occurred, stored as the number of milliseconds since the unix epoch. Assigned by the Timeline service.
type ClusterEventType The event type. This field is required.
details EventDetails The event details. This field is required.

ClusterEventType

Type of a cluster event.

Event Type Description
CREATING Indicates that the cluster is being created.
DID_NOT_EXPAND_DISK Indicates that a disk is low on space, but adding disks would put it over the max capacity.
EXPANDED_DISK Indicates that a disk was low on space and the disks were expanded.
FAILED_TO_EXPAND_DISK Indicates that a disk was low on space and disk space could not be expanded.
INIT_SCRIPTS_STARTING Indicates that the cluster scoped init script has started.
INIT_SCRIPTS_FINISHED Indicates that the cluster scoped init script has finished.
STARTING Indicates that the cluster is being started.
RESTARTING Indicates that the cluster is being started.
TERMINATING Indicates that the cluster is being terminated.
EDITED Indicates that the cluster has been edited.
RUNNING Indicates the cluster has finished being created. Includes the number of nodes in the cluster and a failure reason if some nodes could not be acquired.
RESIZING Indicates a change in the target size of the cluster (upsize or downsize).
UPSIZE_COMPLETED Indicates that nodes finished being added to the cluster. Includes the number of nodes in the cluster and a failure reason if some nodes could not be acquired.
NODES_LOST Indicates that some nodes were lost from the cluster.
DRIVER_HEALTHY Indicates that the driver is healthy and the cluster is ready for use.
DRIVER_UNAVAILABLE Indicates that the driver is unavailable.
SPARK_EXCEPTION Indicates that a Spark exception was thrown from the driver.
DRIVER_NOT_RESPONDING Indicates that the driver is up but is not responsive, likely due to GC.
DBFS_DOWN Indicates that the driver is up but DBFS is down.
METASTORE_DOWN Indicates that the driver is up but the metastore is down.
NODE_BLACKLISTED Indicates that a node is not allowed by Spark.
PINNED Indicates that the cluster was pinned.
UNPINNED Indicates that the cluster was unpinned.

EventDetails

Details about a cluster event.

Field Name Type Description
current_num_workers INT32 The number of nodes in the cluster.
target_num_workers INT32 The targeted number of nodes in the cluster.
previous_attributes ClusterAttributes The cluster attributes before a cluster was edited.
attributes ClusterAttributes * For created clusters, the attributes of the cluster.
* For edited clusters, the new attributes of the cluster.
previous_cluster_size ClusterSize The size of the cluster before an edit or resize.
cluster_size ClusterSize The cluster size that was set in the cluster creation or edit.
cause ResizeCause The cause of a change in target size.
reason TerminationReason A termination reason:

* On a TERMINATED event, the reason for the termination.
* On a RESIZE_COMPLETE event, indicates the reason that we failed to acquire some nodes.
user STRING The user that caused the event to occur. (Empty if it was done by Azure Databricks.)

ClusterAttributes

Common set of attributes set during cluster creation. These attributes cannot be changed over the lifetime of a cluster.

ClusterSize

Cluster size specification.

Field Name Type Description
num_workers OR autoscale INT32 OR AutoScale If num_workers, number of worker nodes that this cluster should have. A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field is updated to reflect the target size of 10 workers, whereas the workers listed in executors gradually increase from 5 to 10 as the new nodes are provisioned.

If autoscale, parameters needed in order to automatically scale clusters up and down based on load.

ListOrder

Generic ordering enum for list-based queries.

Order Description
DESC Descending order.
ASC Ascending order.

ResizeCause

Reason why a cluster was resized.

Cause Description
AUTOSCALE Automatically resized based on load.
USER_REQUEST User requested a new size.
AUTORECOVERY Autorecovery monitor resized the cluster after it lost a node.

ClusterLogConf

Path to cluster log.

Field Name Type Description
dbfs DbfsStorageInfo DBFS location of cluster log. Destination must be provided. For example,
{ "dbfs" : { "destination" : "dbfs:/home/cluster_log" } }

InitScriptInfo

Path to an init script. For instructions on using init scripts with Databricks Container Services, see Use an init script.

Note

The file storage type is only available for clusters set up using Databricks Container Services.

Field Name Type Description
dbfs OR file DbfsStorageInfo

FileStorageInfo		 DBFS location of init script. Destination must be provided. For example,
{ "dbfs" : { "destination" : "dbfs:/home/init_script" } }

File location of init script. Destination must be provided. For example,
{ "file" : { "destination" : "file:/my/local/file.sh" } }

ClusterTag

Cluster tag definition.

Type Description
STRING The key of the tag. The key must:

* Be between 1 and 512 characters long
* Not contain any of the characters <>%*&+?\\/
* Not begin with azure, microsoft, or windows
STRING The value of the tag. The value length must be less than or equal to 256 UTF-8 characters.

DbfsStorageInfo

DBFS storage information.

Field Name Type Description
destination STRING DBFS destination. Example: dbfs:/my/path

FileStorageInfo

File storage information.

Note

This location type is only available for clusters set up using Databricks Container Services.

Field Name Type Description
destination STRING File destination. Example: file:/my/file.sh

DockerImage

Docker image connection information.

Field Type Description
url string URL for the Docker image.
basic_auth DockerBasicAuth Basic authentication information for Docker repository.

DockerBasicAuth

Docker repository basic authentication information.

Field Description
username User name for the Docker repository.
password Password for the Docker repository.

LogSyncStatus

Log delivery status.

Field Name Type Description
last_attempted INT64 The timestamp of last attempt. If the last attempt fails, last_exception contains the exception in the last attempt.
last_exception STRING The exception thrown in the last attempt, it would be null (omitted in the response) if there is no exception in last attempted.

NodeType

Description of a Spark node type including both the dimensions of the node and the instance type on which it will be hosted.

Field Name Type Description
node_type_id STRING Unique identifier for this node type. This field is required.
memory_mb INT32 Memory (in MB) available for this node type. This field is required.
num_cores FLOAT Number of CPU cores available for this node type. This can be fractional if the number of cores on a machine instance is not divisible by the number of Spark nodes on that machine. This field is required.
description STRING A string description associated with this node type. This field is required.
instance_type_id STRING An identifier for the type of hardware that this node runs on. This field is required.
is_deprecated BOOL Whether the node type is deprecated. Non-deprecated node types offer greater performance.
node_info ClusterCloudProviderNodeInfo Node type info reported by the cloud provider.

ClusterCloudProviderNodeInfo

Information about an instance supplied by a cloud provider.

Field Name Type Description
status ClusterCloudProviderNodeStatus Status as reported by the cloud provider.
available_core_quota INT32 Available CPU core quota.
total_core_quota INT32 Total CPU core quota.

ClusterCloudProviderNodeStatus

Status of an instance supplied by a cloud provider.

Status Description
NotEnabledOnSubscription Node type not available for subscription.
NotAvailableInRegion Node type not available in region.

ParameterPair

Parameter that provides additional information about why a cluster was terminated.

Type Description
TerminationParameter Type of termination information.
STRING The termination information.

SparkConfPair

Spark configuration key-value pairs.

Type Description
STRING A configuration property name.
STRING The configuration property value.

SparkEnvPair

Spark environment variable key-value pairs.

Important

When specifying environment variables in a job cluster, the fields in this data structure accept only Latin characters (ASCII character set). Using non-ASCII characters will return an error. Examples of invalid, non-ASCII characters are Chinese, Japanese kanjis, and emojis.

Type Description
STRING An environment variable name.
STRING The environment variable value.

SparkNode

Spark driver or executor configuration.

Field Name Type Description
private_ip STRING Private IP address (typically a 10.x.x.x address) of the Spark node. This is different from the private IP address of the host instance.
public_dns STRING Public DNS address of this node. This address can be used to access the Spark JDBC server on the driver node.
node_id STRING Globally unique identifier for this node.
instance_id STRING Globally unique identifier for the host instance from the cloud provider.
start_timestamp INT64 The timestamp (in millisecond) when the Spark node is launched.
host_private_ip STRING The private IP address of the host instance.

SparkVersion

Databricks Runtime version of the cluster.

Field Name Type Description
key STRING Databricks Runtime version key, for example 7.3.x-scala2.12. The value that should be provided as the spark_version when creating a new cluster. The exact runtime version may change over time for a “wildcard” version (that is, 7.3.x-scala2.12 is a “wildcard” version) with minor bug fixes.
name STRING A descriptive name for the runtime version, for example “Databricks Runtime 7.3 LTS”.

TerminationReason

Reason why a cluster was terminated.

Field Name Type Description
code TerminationCode Status code indicating why a cluster was terminated.
type TerminationType Reason indicating why a cluster was terminated.
parameters ParameterPair Object containing a set of parameters that provide information about why a cluster was terminated.

PoolClusterTerminationCode

Status code indicating why the cluster was terminated due to a pool failure.

Code Description
INSTANCE_POOL_MAX_CAPACITY_FAILURE The pool max capacity has been reached.
INSTANCE_POOL_NOT_FOUND_FAILURE The pool specified by the cluster is no longer active or doesn’t exist.

ClusterSource

Service that created the cluster.

Service Description
UI Cluster created through the UI.
JOB Cluster created by the Databricks job scheduler.
API Cluster created through an API call.

ClusterState

State of a cluster. The allowable state transitions are as follows:

  • PENDING -> RUNNING
  • PENDING -> TERMINATING
  • RUNNING -> RESIZING
  • RUNNING -> RESTARTING
  • RUNNING -> TERMINATING
  • RESTARTING -> RUNNING
  • RESTARTING -> TERMINATING
  • RESIZING -> RUNNING
  • RESIZING -> TERMINATING
  • TERMINATING -> TERMINATED
State Description
PENDING Indicates that a cluster is in the process of being created.
RUNNING Indicates that a cluster has been started and is ready for use.
RESTARTING Indicates that a cluster is in the process of restarting.
RESIZING Indicates that a cluster is in the process of adding or removing nodes.
TERMINATING Indicates that a cluster is in the process of being destroyed.
TERMINATED Indicates that a cluster has been successfully destroyed.
ERROR This state is no longer used. It was used to indicate a cluster that failed to be created.
TERMINATING and TERMINATED are used instead.
UNKNOWN Indicates that a cluster is in an unknown state. A cluster should never be in this state.

TerminationCode

Status code indicating why the cluster was terminated.

TerminationType

Reason why the cluster was terminated.

Type Description
SUCCESS Termination succeeded.
CLIENT_ERROR Non-retriable. Client must fix parameters before reattempting the cluster creation.
SERVICE_FAULT Azure Databricks service issue. Client can retry.
CLOUD_FAILURE Cloud provider infrastructure issue. Client can retry after the underlying issue is resolved.

TerminationParameter

Key that provides additional information about why a cluster was terminated.

Key Description
username The username of the user who terminated the cluster.
databricks_error_message Additional context that may explain the reason for cluster termination.
inactivity_duration_min An idle cluster was shut down after being inactive for this duration.
instance_id The ID of the instance that was hosting the Spark driver.
azure_error_code The Azure provided error code describing why cluster nodes could not be provisioned. For reference, see: https://learn.microsoft.com/azure/virtual-machines/windows/error-messages.
azure_error_message Human-readable context of various failures from Azure. This field is unstructured, and its exact format is subject to change.
instance_pool_id The ID of the instance pool the cluster is using.
instance_pool_error_code The error code for cluster failures specific to a pool.

AzureAttributes

Attributes set during cluster creation related to Azure.

Field Name Type Description
first_on_demand INT32 The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value must be greater than 0, or else cluster creation validation fails. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. This value does not affect cluster size and cannot be mutated over the lifetime of a cluster.
availability AzureAvailability Availability type used for all subsequent nodes past the first_on_demand ones.
spot_bid_max_price DOUBLE The max bid price used for Azure spot instances. You can set this to greater than or equal to the current spot price. You can also set this to -1 (the default), which specifies that the instance cannot be evicted on the basis of price. The price for the instance will be the current price for spot instances or the price for a standard instance. You can view historical pricing and eviction rates in the Azure portal.

AzureAvailability

The Azure instance availability type behavior.

Type Description
SPOT_AZURE Use spot instances.
ON_DEMAND_AZURE Use on-demand instances.
SPOT_WITH_FALLBACK_AZURE Preferably use spot instances, but fall back to on-demand instances if spot instances cannot be acquired (for example, if Azure spot prices are too high or out of quota). Does not apply to pool availability.