Hello Team, I'm having below error,I have already verified all the connection which is correct I'm using existing cluster of databricks in my free subscription account. Errror: run failed with error message Unable to access the notebook "/PySpark_Scripts/xyz.py". Either it does not exist, or the identity used to run this job, Bramha Naskari, lacks the required permissions

Bramha Naskari - Thanks for the question and using MS Q&A platform. It seems like the error message is indicating that the identity used to run the job lacks the required permissions to access the notebook. You may want to check if the user account you are using has the necessary permissions to access the notebook. You can try the following steps to troubleshoot the issue: Check if the notebook exists in the specified location "/PySpark_Scripts/xyz.py". If it does not exist, you may want to create the notebook or update the path to the correct location. Check if the user account you are using has the necessary permissions to access the notebook. You can verify this by checking the access control list (ACL) for the notebook. Make sure that the user account has the "Read" permission for the notebook. If you are using an existing cluster, make sure that the cluster has the necessary libraries installed to run the notebook. You can check the cluster configuration to verify if the required libraries are installed. If you are still facing issues, please confirm if you have configured with any source controller like Github/Devops and share more details about the job configuration for futher investigation. I hope this helps you resolve the issue. Let me know if you have any further questions.

Azure Data Factory Aazure Databricks pipeline error

Accepted answer

PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2024-01-16T01:48:27.3433333+00:00

Bramha Naskari - Thanks for the question and using MS Q&A platform.

It seems like the error message is indicating that the identity used to run the job lacks the required permissions to access the notebook. You may want to check if the user account you are using has the necessary permissions to access the notebook.

You can try the following steps to troubleshoot the issue:

Check if the notebook exists in the specified location "/PySpark_Scripts/xyz.py". If it does not exist, you may want to create the notebook or update the path to the correct location.

Check if the user account you are using has the necessary permissions to access the notebook. You can verify this by checking the access control list (ACL) for the notebook. Make sure that the user account has the "Read" permission for the notebook.

If you are using an existing cluster, make sure that the cluster has the necessary libraries installed to run the notebook. You can check the cluster configuration to verify if the required libraries are installed.

If you are still facing issues, please confirm if you have configured with any source controller like Github/Devops and share more details about the job configuration for futher investigation.

I hope this helps you resolve the issue. Let me know if you have any further questions.
Please sign in to rate this answer.
Bramha Naskari 20 Reputation points

2024-01-16T07:37:19.8233333+00:00

Thanks for responding ! But I have verified and test the connection which was successful (Linked service) I'm using free tier account and using existing cluster which I have created on databricks single node still haing this error as I'm the only owner of this free tier account pipeline suppose to run I'm not sure where I missed the configuration please provide me more assistance on that.

PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2024-01-17T04:43:02.22+00:00

Bramha Naskari - I understand that you have already verified the connection and checked the permissions. In that case, you may want to check the following:

Check if the notebook path is correct: Ensure that the notebook path specified in the job configuration is correct. You can do this by navigating to the notebook in your Databricks workspace and copying the path from the address bar.

Check if the user account is correct: Ensure that you are using the correct user account to run the job. You can do this by checking the user account specified in the job configuration.

Check if the cluster is running: Ensure that the cluster you are using is running. You can do this by checking the cluster status in your Databricks workspace.

Check if the cluster has the necessary libraries: Ensure that the cluster you are using has the necessary libraries installed to run the notebook. You can do this by checking the cluster configuration and verifying if the required libraries are installed.

Check if the job is using the correct cluster: Ensure that the job is using the correct cluster to run the notebook. You can do this by checking the cluster specified in the job configuration.

If you have checked all of the above and are still experiencing issues, you may want to try creating a new notebook and running the job with the new notebook to see if the issue persists.

Hope this helps.

Bramha Naskari 20 Reputation points

2024-01-21T08:16:53.2733333+00:00

Well, Thanks for responding really grate experience to learn azure as I'm getting answers from community. It was something related to access issue

PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2024-01-22T03:23:58.67+00:00

Bramha Naskari - Glad to know it helped. Do continue to use MS Q&A platform for any questions related to Azure.

AJB 0 Reputation points

2024-03-21T14:26:07.4166667+00:00

@Bramha Naskari Hello, How did you resolve the access issue ? where did you grant permissions ? Thanks in advance
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure Data Factory Aazure Databricks pipeline error

0 additional answers

Your answer