Can't run Azure Synapse Notebook in Pipeline

Question

I am running a Pyspark notebook in Synapse to save data in data lake gen2. I can run the notebook without any issues, but when I run the notebook using Pipeline I got an error :

Error
{
"errorCode": "6002",
"message": "AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.nio.file.AccessDeniedException Operation failed: \"This request is not authorized to perform this operation using this permission.\", 403, HEAD, https://dstetl01devadls.dfs.core.windows.net/dstetl01devadlsfs/synapse/workspaces/dst-etl01-dev-ws/warehouse?upn=false&action=getStatus&timeout=90)\nTraceback (most recent call last):\n\n File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py\", line 723, in sql\n return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)\n\n File \"/home/trusted-service-user/cluster-env/env/lib/python3.8/site-packages/py4j/java_gateway.py\", line 1304, in call\n return_value = get_return_value(\n\n File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 117, in deco\n raise converted from None\n\npyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.nio.file.AccessDeniedException Operation failed: \"This request is not authorized to perform this operation using this permission.\", 403, HEAD, https://dstetl01devadls.dfs.core.windows.net/dstetl01devadlsfs/synapse/workspaces/dst-etl01-dev-ws/warehouse?upn=false&action=getStatus&timeout=90)\n",
"failureType": "UserError",
"target": "Pyspark_etl",
"details": []
}

[183826-synapse-error.png][1]:

Answer 1

Hello @Engy Tawadros ,

Thanks for the question and using MS Q&A platform.

To resolve the issue, I would suggest you to add the RBAC Storage Blob Data Contributor to the user that is running the notebook, or your user.

Note: Storage Blob Data Contributor: Use to grant read/write/delete permissions to Blob storage resources.

If you are not assigning Storage Blob Data Contributor to users who are accessing the storage account, they will be not able to access the data from ADLS gen2 due to the lack of permission on the storage account.

If they try to access data from ADLS gen2 without the "Storage Blob Data Contributor" role on the storage account, they will receive the error message: Operation failed: "This request is not authorized to perform this operation.",403.

Once the storage account is created, select Access control (IAM) from the left navigation. Then assign the following roles or ensure they are already assigned.
Assign yourself to the Storage Blob Data Owner role on the Storage Account

For more details, refer to Notebook - This request is not authorized to perform this operation. , 403 and also you may checkout the MS Q&A thread addressing similar issue.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators