How to provide job cluster log read access to other users in Azure data bricks for which the cluster is created through spn

Question

Hi,

We are creating Job clusters through SPN for our custom code to run, the driver logs are not accessible to other users apart from admin .

We would like to create a script to provide access to users via access groups to read logs.

Request you to provide a solution to provide access(not from admin console since we run the creation scripts in parallel multiple times and providing it manually is not possible)

Answer 1

Hello @GIRISH KUMAR,

Welcome to the MS Q&A platform.

If you do not want to manage job access control from the admin console, you can use Terraform integration to manage permissions in a fully automated setup using Databricks Terraform provider and databricks_permissions

Code from the documentation page:

resource "databricks_group" "auto" {
  display_name = "Automation"
}

resource "databricks_group" "eng" {
  display_name = "Engineering"
}

data "databricks_spark_version" "latest" {}

data "databricks_node_type" "smallest" {
  local_disk = true
}

resource "databricks_job" "this" {
  name                = "Featurization"
  max_concurrent_runs = 1

  new_cluster {
    num_workers   = 300
    spark_version = data.databricks_spark_version.latest.id
    node_type_id  = data.databricks_node_type.smallest.id
  }

  notebook_task {
    notebook_path = "/Production/MakeFeatures"
  }
}

resource "databricks_permissions" "job_usage" {
  job_id = databricks_job.this.id

  access_control {
    group_name       = "users"
    permission_level = "CAN_VIEW"
  }

  access_control {
    group_name       = databricks_group.auto.display_name
    permission_level = "CAN_MANAGE_RUN"
  }

  access_control {
    group_name       = databricks_group.eng.display_name
    permission_level = "CAN_MANAGE"
  }
}

I hope this helps. Please let us know if you have any further questions.