Not able to run databricks notebook using Databricks API, getting permission issue.

Question

Not able to run databricks notebook using Databricks API, getting permission issue.

Abhishek Singh 46

I’m trying to run a notebook using the Databricks Azure REST API. I’ve granted my Service Principal (SPN) "Can Manage" permissions for REPOS, clusters, and the workspace. Despite this, I’m still encountering an error when using Postman:

{
    "error_code": "PERMISSION_DENIED",
    "message": "User "my-spn" does not have Manage Run or Owner or Admin permissions on job 246372968680205"
}

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-19T21:00:44.3433333+00:00
Hello Abhishek Singh,

Welcome to the Microsoft Q&A forum.

The error you're encountering is related to permission settings on the job in Databricks. Here are some key points to understand:

Job Ownership: The creator of a job automatically has "Is Owner" permission. A job can only have one owner, and groups cannot be assigned as owners.

Permissions and Job Execution: When a job is triggered through the "Run Now" API, the job assumes the permissions of the job owner, not the user who triggered the run. For example, if a job is set to run on a cluster accessible only to the job owner (user A), another user (user B) with "Can Manage Run" permission can still trigger the job, but the job will execute under the permissions of user A.

Issue with Service Principal: In your case, even though your Service Principal (SPN) has "Can Manage" permissions for REPOS, clusters, and the workspace, it doesn't have ownership of the job in question. Since jobs triggered via API calls assume the permissions of the job owner, your SPN needs to either:

Be granted ownership of the job.

Or, the job owner must ensure that the permissions are sufficient for the job to run as expected.

To resolve this issue, you need to transfer the ownership of the job to your SPN or ensure the current job owner's permissions allow the job to run under the SPN's context.

Please check the below official document.

https://learn.microsoft.com/en-us/azure/databricks/security/auth/access-control/#jobs
Abhishek Singh 46 Reputation points

2024-08-20T12:01:50.3766667+00:00
Thank you for your response. I've added the ACL permissions to my API call, but I’m still unable to execute the created job. I'm creating a job and generating a bearer token using an SPN, which I then use for all subsequent API calls. I also see permission are getting attached but still. I am getting the issue:

Error: run failed with error message Unable to access the notebook "/Workspace/Repos/test.sql". Either it does not exist, or the identity used to run this job, my-spn (id), lacks the required permissions.

{ "max_concurrent_runs": 1, "name": "NEW_TEST", "tasks": [ { "notebook_task": { "notebook_path": "/Workspace/Repos/test.sql" }, "task_key": "test_123456", "existing_cluster_id": "poc-cluster" } ], "access_control_list": [ // { // "user_name": "my-name", // "permission_level": "IS_OWNER" // }, { "service_principal_name": "my-spn", "permission_level": "IS_OWNER" } ] }
Abhishek Singh 46 Reputation points

2024-08-20T13:52:19.9066667+00:00

Closing the ticket. Issued Resolved
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-20T14:59:38.97+00:00

Thank you, Abhishek Singh, for the confirmation.

If you could share the resolution here, it would be helpful for community members looking for answers to similar issues.
Abhishek Singh 46 Reputation points

2024-08-20T18:42:03.5966667+00:00

I initially thought it was a permission issue, but upon closer inspection, I realized I was running a Python file instead of a notebook. In my API, I was executing it as a notebook, which is why it couldn't find the file. I’m not entirely sure how it works behind the scenes, but after changing the notebook_task to spark_python_task, I was able to run it successfully.

{ "spark_python_task": { "python_file": "/Workspace/Repos/test.py" }
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-20T21:47:06.0566667+00:00

Hi Abhishek Singh,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

3 answers

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-19T21:00:44.3433333+00:00

Hello Abhishek Singh,

Welcome to the Microsoft Q&A forum.

The error you're encountering is related to permission settings on the job in Databricks. Here are some key points to understand:

Job Ownership: The creator of a job automatically has "Is Owner" permission. A job can only have one owner, and groups cannot be assigned as owners.

Permissions and Job Execution: When a job is triggered through the "Run Now" API, the job assumes the permissions of the job owner, not the user who triggered the run. For example, if a job is set to run on a cluster accessible only to the job owner (user A), another user (user B) with "Can Manage Run" permission can still trigger the job, but the job will execute under the permissions of user A.

Issue with Service Principal: In your case, even though your Service Principal (SPN) has "Can Manage" permissions for REPOS, clusters, and the workspace, it doesn't have ownership of the job in question. Since jobs triggered via API calls assume the permissions of the job owner, your SPN needs to either:

Be granted ownership of the job.

Or, the job owner must ensure that the permissions are sufficient for the job to run as expected.

To resolve this issue, you need to transfer the ownership of the job to your SPN or ensure the current job owner's permissions allow the job to run under the SPN's context.

Please check the below official document.

https://learn.microsoft.com/en-us/azure/databricks/security/auth/access-control/#jobs
Abhishek Singh 46 Reputation points

2024-08-20T12:01:50.3766667+00:00

Thank you for your response. I've added the ACL permissions to my API call, but I’m still unable to execute the created job. I'm creating a job and generating a bearer token using an SPN, which I then use for all subsequent API calls. I also see permission are getting attached but still. I am getting the issue:

Error: run failed with error message Unable to access the notebook "/Workspace/Repos/test.sql". Either it does not exist, or the identity used to run this job, my-spn (id), lacks the required permissions.

{ "max_concurrent_runs": 1, "name": "NEW_TEST", "tasks": [ { "notebook_task": { "notebook_path": "/Workspace/Repos/test.sql" }, "task_key": "test_123456", "existing_cluster_id": "poc-cluster" } ], "access_control_list": [ // { // "user_name": "my-name", // "permission_level": "IS_OWNER" // }, { "service_principal_name": "my-spn", "permission_level": "IS_OWNER" } ] }
Abhishek Singh 46 Reputation points

2024-08-20T13:52:19.9066667+00:00

Closing the ticket. Issued Resolved
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-20T14:59:38.97+00:00

Thank you, Abhishek Singh, for the confirmation.

If you could share the resolution here, it would be helpful for community members looking for answers to similar issues.
Abhishek Singh 46 Reputation points

2024-08-20T18:42:03.5966667+00:00

I initially thought it was a permission issue, but upon closer inspection, I realized I was running a Python file instead of a notebook. In my API, I was executing it as a notebook, which is why it couldn't find the file. I’m not entirely sure how it works behind the scenes, but after changing the notebook_task to spark_python_task, I was able to run it successfully.

{ "spark_python_task": { "python_file": "/Workspace/Repos/test.py" }
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-08-20T21:47:06.0566667+00:00

Hi Abhishek Singh,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Answer 1

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more

Answer 2

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more

Answer 3

Hi Abhishek Singh,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer

Error Message:

"error_code": "PERMISSION_DENIED",
  "message": "User "my-spn" does not have Manage Run or Owner or Admin permissions on job 246372968680205"

Issue:

unable to run databricks notebook using Databricks API, getting permission error.

Solution:

The problem was related to the task type in the API call. The error occurred because a Python file was being executed as a notebook. This was resolved by changing the task type to spark_python_task and specifying the correct path to the Python file.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Share via

Not able to run databricks notebook using Databricks API, getting permission issue.

3 answers

Your answer