Interactive Cluster update api, is removing unity access for a cluster which has unity enabled

Question

Interactive Cluster update api, is removing unity access for a cluster which has unity enabled

Siddharth Choudhary 40

Hi, when i'm changing my interactive cluster config using databricks rest api
/api/2.1/clusters/edit , the cluster which is already unity enabled , after hitting the api the unity enable tag is gone the api is changing other config but why it is removing unity access.
api body

{ "cluster_id": "",

"cluster_name": "",

"num_workers": 1,

"use_photon": true,

"autotermination_minutes": 10,

"azure_attributes": {

"availability": "SPOT_WITH_FALLBACK_AZURE",

"spot_bid_max_price": "-1.0"

},

"autoscale":{

"min_workers":1,

"max_workers":8

},

"spark_version": "14.3.x-scala2.12",

"node_type_id": "Standard_DS3_v2",

"driver_node_type_id": "Standard_DS4_v2"}

can you please look into the issue

Accepted answer

0 additional answers

Your answer

Answer 1

Venkat Reddy Navari 2,975 Microsoft External Staff Moderator

Hi Siddharth Choudhary

Looking at the API body you shared, everything seems pretty standard updating the number of workers, auto scale settings, Photon, and node types shouldn’t, in theory, affect the Unity Catalog enablement. Since you mentioned the cluster is already Unity-enabled before the API call, it’s strange that the “unity enable tag” is being stripped out after the update. It almost sounds like the API might be resetting some hidden or required field that Unity Catalog depends on, even though it’s not explicitly mentioned in your request.

Here are a few key areas to investigate:

Full Cluster Config: When you hit the /api/2.1/clusters/edit endpoint, it’s a full replacement of the cluster’s configuration (not a partial update). If the Unity Catalog setting—like "data_security_mode": "SINGLE_USER" or a similar flag—isn’t included in your API payload, it could be getting reset to a default (disabled) state.
Cluster Access Mode: Unity Catalog usually requires the cluster to be in single-user or shared mode. Your original payload doesn’t specify the access mode, so it might be reverting to a mode that doesn’t support Unity Catalog. The updated query includes "data_security_mode": "SINGLE_USER" and "single_user_name" to lock it in.

Here’s the updated API body with Unity Catalog settings added:

{
  "cluster_id": "",
  "cluster_name": "",
  "num_workers": 1,
  "use_photon": true,
  "autotermination_minutes": 10,
  "azure_attributes": {
    "availability": "SPOT_WITH_FALLBACK_AZURE",
    "spot_bid_max_price": "-1.0"
  },
  "autoscale": {
    "min_workers": 1,
    "max_workers": 8
  },
  "spark_version": "14.3.x-scala2.12",
  "node_type_id": "Standard_DS3_v2",
  "driver_node_type_id": "Standard_DS4_v2",
  "data_security_mode": "SINGLE_USER",
  "single_user_name": "<your-username>"
}

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Siddharth Choudhary 40 Reputation points

2025-03-25T11:41:27.7933333+00:00

Thank you for you answer, it seems that was the issue, one more thing if my cluster has some cluster configuration added , so do i need to add them in payload also?
or they will be kept same?
Venkat Reddy Navari 2,975 Reputation points Microsoft External Staff Moderator

2025-03-26T10:49:25+00:00

Hi Siddharth Choudhary

Regarding your question about cluster configurations

If you're modifying an existing interactive cluster, any settings not explicitly included in the API payload may reset to their defaults. To retain all configurations, it's best to first retrieve the current cluster settings using the /api/2.1/clusters/get API and include the necessary parameters in your update request.

If you're creating a new job cluster dynamically, you must specify all required configurations in the payload, as Databricks does not retain settings from previous clusters.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Venkat Reddy Navari 2,975 Reputation points Microsoft External Staff Moderator

2025-03-27T13:38:53.64+00:00

Hi Siddharth Choudhary Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Siddharth Choudhary 40 Reputation points

2025-03-28T06:13:30.8033333+00:00

Hi Venkat,
thank you for your answer, one thing i have to update only nodetype , termination time and maybe 1-2 more things,
so instead of using this api , can partial update api can be used for that?
Venkat Reddy Navari 2,975 Reputation points Microsoft External Staff Moderator

2025-03-28T13:55:42.2733333+00:00
Siddharth Choudhary Currently, Databricks does not support a partial update API for modifying cluster configurations. The /api/2.1/clusters/edit API performs a full replacement of the cluster configuration. Any settings not included in the API request might reset to their default values. To avoid losing existing configurations while making updates (such as node type and termination time), we recommend the following approach:

Retrieve the existing cluster configuration using the /api/2.1/clusters/get API.

Modify only the required fields (e.g., node_type_id, autotermination_minutes).

Send the full updated configuration back using /api/2.1/clusters/edit.

For more details, you can refer to Databricks’ official documentation:

Get Cluster Configuration: https://docs.databricks.com/api/workspace/clusters/get

Edit Cluster Configuration: https://docs.databricks.com/api/workspace/clusters/edit

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Siddharth Choudhary 40 Reputation points

2025-03-29T06:44:02.31+00:00

hi, i found this api on databricks only
https://docs.databricks.com/api/workspace/clusters/update
it says it partially update the cluster
can you please check
Venkat Reddy Navari 2,975 Reputation points Microsoft External Staff Moderator

2025-03-31T14:22:02.4+00:00
Hi Siddharth Choudhary, you’re right that Databricks provides /api/2.1/clusters/update, which allows partial updates for certain cluster attributes. Previously, we mentioned that Databricks does not support partial updates because /api/2.1/clusters/edit fully replaces the configuration, and not all settings can be updated using /update.

The /update API is primarily used for modifying runtime attributes (e.g., tags, libraries), whereas changes to core cluster configurations—like node_type_id or autotermination_minutes—are typically done via /edit. However, if /update works for your use case, that’s a great alternative!

To use the /api/2.1/clusters/update API effectively, you can specify only the fields you want to change, and the rest of the cluster configuration will remain intact. This should help you avoid the issue of losing any existing configurations, including the Unity Catalog settings.

Here’s example of how you might structure your request using the partial update API:

{ "cluster_id": "<your-cluster-id>", "node_type_id": "Standard_DS4_v2", "autotermination_minutes": 15 }

Make sure to replace <your-cluster-id> with the actual ID of your cluster.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Siddharth Choudhary 40 Reputation points

2025-04-01T05:20:21.81+00:00

Hi, I want to update my termination time , node type, autoscaling nodes, num of workers, azure_attributes,

these above things i want to update, so **/api/2.1/clusters/update **this work will work right for this case?
Venkat Reddy Navari 2,975 Reputation points Microsoft External Staff Moderator

2025-04-02T10:35:19.4+00:00
Hi @Siddharth Choudhary
Yes, the /api/2.1/clusters/update API is designed for partial updates, allowing you to modify specific attributes without affecting the rest of the cluster configuration. However, there are some limitations on what can be updated.

Based on your requirements (termination time, node type, autoscaling nodes, number of workers, and azure_attributes), here’s what you need to consider:

Termination Time & Node Type: These settings are typically managed via the /api/2.1/clusters/edit API. While /update may allow modifications for some attributes, it's best to verify the specific fields supported by the API documentation.

Autoscaling & Number of Workers: Changes to num_workers and autoscale settings may be supported by /update, but again, please check the API documentation to confirm.

Azure Attributes: Updating azure_attributes (e.g., spot pricing, availability settings) might not be directly supported through /update. You may need to use /edit for these changes.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Interactive Cluster update api, is removing unity access for a cluster which has unity enabled

0 additional answers

Your answer