Share via

Set Permissions on new Databricks jobs created by Data Factory

P 101 Reputation points
2021-11-29T06:14:50.607+00:00

Following the guide here https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook#create-linked-services I have created a new Databricks Linked Service in my ADF instance that creates a new Job Cluster.

153179-image.png

I then create a pipeline and create a Databricks Activity within that pipeline to run a notebook in Databricks and schedule the pipeline to run daily. Each time the pipeline run a new job is created on Databricks for the pipeline execution for the data.

On occasion this daily job fail and I want my support team to take a look at the Databricks job logs to see what went wrong. Is there a way I can configure the Databricks pipeline/Activity such that members of a particular Databricks Group have CAN VIEW permission on all jobs created via ADF? At the moment I need to give my support team Databricks Admin/Contributor access on the Databricks resource so that they can view the logs which from a security perspective is not a very good practice.

I have been told that the Databricks Jobs API was recently updated so that it accepted a access_control_list parameter where the permissions of a new job can be specified but it is unclear how I can set this parameter via either the Databricks LinkedService or Activity in ADF.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,080 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,199 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HimanshuSinha-msft 19,386 Reputation points Microsoft Employee
    2021-11-29T21:14:36.507+00:00

    Hello @,
    Thanks for the ask and using Microsoft Q&A platform .

    Is there a way I can configure the Databricks pipeline/Activity such that members of a particular Databricks Group have CAN VIEW permission on all jobs created via ADF?

    The currently the way things are implemented ADF is just orchestrator, and I don’t think it’s possible to implement the current ask using ADF .
    As I understand you do not want to give “more access” to a particular due to security reasons . Have you tried looking into the using log analytics ? The below link should help .
    https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagnostic-logs#diagnostic-log-delivery
    Please do let me know how it goes.
    Thanks
    Himanshu

    -------------------------------------------------------------------------------------------------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators