Azure Spark Configuration (Environment) Documentation

David Beavon 976

In the environment tab of my cluster, there are lots of properties starting with "spark.databricks.clusterUsageTags".

However if I google for them I cannot find any information.

Is there any documentation or reference sheet for these values? Can someone please point me to it? Some of them are self evident and others are not.

I'm particularly interested in the last seven below.

spark.databricks.clusterUsageTags.autoTerminationMinutes
spark.databricks.clusterUsageTags.azureSubscriptionId
spark.databricks.clusterUsageTags.clusterAllTags
spark.databricks.clusterUsageTags.clusterAvailability
spark.databricks.clusterUsageTags.clusterCreator
spark.databricks.clusterUsageTags.clusterFirstOnDemand
spark.databricks.clusterUsageTags.clusterGeneration
spark.databricks.clusterUsageTags.clusterLogDeliveryEnabled
spark.databricks.clusterUsageTags.clusterLogDestination
spark.databricks.clusterUsageTags.clusterMaxWorkers
spark.databricks.clusterUsageTags.clusterMetastoreAccessType
spark.databricks.clusterUsageTags.clusterMinWorkers
spark.databricks.clusterUsageTags.clusterName
spark.databricks.clusterUsageTags.clusterTargetWorkers
spark.databricks.clusterUsageTags.clusterWorkers
spark.databricks.clusterUsageTags.containerType
spark.databricks.clusterUsageTags.dataPlaneRegion
spark.databricks.clusterUsageTags.driverContainerId
spark.databricks.clusterUsageTags.driverContainerPrivateIp
spark.databricks.clusterUsageTags.driverInstanceId
spark.databricks.clusterUsageTags.driverInstancePrivateIp
spark.databricks.clusterUsageTags.driverNodeType
spark.databricks.clusterUsageTags.driverPublicDns
spark.databricks.clusterUsageTags.region
spark.databricks.clusterUsageTags.sparkVersion
spark.databricks.clusterUsageTags.workerEnvironmentId

spark.databricks.clusterUsageTags.clusterNodeType
spark.databricks.clusterUsageTags.clusterState
spark.databricks.clusterUsageTags.clusterStateMessage
spark.databricks.clusterUsageTags.clusterId
spark.databricks.clusterUsageTags.cloudProvider
spark.databricks.clusterUsageTags.isSingleUserCluster
spark.databricks.clusterUsageTags.clusterNoDriverDaemon

Saurabh Sharma 23,761 Reputation points Microsoft Employee

2021-04-28T18:35:41.237+00:00

Hi @David Beavon ,

Thanks for using Microsoft Q&A !!

Unfortunately, I do not see any documentation on these properties but I am checking internally if I can get any more information around the same. I will get back to you on the same.

Thanks
Saurabh
David Beavon 976 Reputation points

2021-04-28T19:57:13.123+00:00

@Saurabh Sharma In particular I'm hoping there is a way to create a "normal" cluster, that has multiple driver processes rather than one massive driver daemon.

In databricks this property always seems to be false:
spark.databricks.clusterUsageTags.clusterNoDriverDaemon

Perhaps there is a cluster type where databricks behaves normally. We are running into a bottleneck as a result of our IPC between the driver and external processes running on the driver node.
Saurabh Sharma 23,761 Reputation points Microsoft Employee

2021-04-29T14:32:04.207+00:00

@David Beavon Thanks for sharing details. I am still waiting for products team response. I will provide the context to them as well.

Thanks
Saurabh
Nico 0 Reputation points

2023-10-11T18:34:25.71+00:00

I found this post while also googling for the same information (also could not find anything). It feels like there definitely should be some type of documentation for this stuff, as it is the only way to get information about the environment a notebook is running in (as far as I know).

On another now, where did you find that list for the spark.databricks.clusterUsageTags properties?

Share via

Azure Spark Configuration (Environment) Documentation