Setting permission for Databricks Jobs log without admin access

Question

I have a group of users who need permission to view the logs of Databricks jobs that are started by Azure Data Factory (ADF). However, I want to avoid giving them admin access.

I’ve only found documentation on how to set permissions for individual jobs, but I would like to know if it’s possible to set permissions for all existing and future jobs at once.

Additionally, I have already enabled the "Job Visibility Control" feature.

Any guidance on how to achieve this would be greatly appreciated!

 

Answer 1

Hi @Gabriel-2005

Welcome to Microsoft Q&A platform and thanks for posting your query here.

If you're using Azure Data Factory (ADF) to trigger Databricks jobs, permissions for those job runs are only granted to users with the right permissions on the job itself. This happens because ADF still uses the older Jobs API 2.0 (Submit Run function), which doesn’t support setting permissions for job runs dynamically. The newer API 2.1 does allow this, but ADF hasn’t adopted it yet.

Possible solutions:

• Use a Script to Grant Permissions Automatically You can create a script to check for jobs run in the last 24 hours and grant view access to specific user groups. Here's the main idea:
  • Fetch job runs from the past day using the API.
  • For each job, use the permissions/jobs endpoint to assign CAN_VIEW permissions to the required groups.

Here’s an example script:

 # Get jobs run in the last 24 hours
end_date = datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0)
start_date = end_date - timedelta(days=1)
params = { 
    "active_only": "true",
    "completed_only": "true",
    "start_time": str(start_date.timestamp()),
    "end_time": str(end_date.timestamp())
}
headers = {
    "Authorization": "Bearer " + api_databricks_token,
    "Content-Type": "application/json"
}
url = f"{api_databricks_base_url}/api/2.0/jobs/runs/list"
response = requests.get(url, headers=headers, params=params)
job_ids = [job['job_id'] for job in response.json().get('runs', [])]
group_names = ["Group1", "Group2"]
# Grant view access for each group
for job_id in job_ids:
    for group_name in group_names:
        permissions_payload = {
            'access_control_list': [
                {'group_name': group_name, 'permission_level': 'CAN_VIEW'}
            ]
        }
        response = requests.patch(
            f"{api_databricks_base_url}/api/2.0/permissions/jobs/{job_id}",
            headers=headers, json=permissions_payload
        )

• Trigger Jobs Directly from Databricks Instead of triggering jobs via ADF activities, use the ADF Web Activity to trigger jobs already defined inside Databricks. This way, permissions are managed directly within Databricks and not tied to ADF’s limitations.

For more details on this, you can refer to this article.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.