Can't run Azure Synapse Notebook in Pipeline

Engy Tawadros 46 Reputation points
2022-03-16T17:21:32.35+00:00

I am running a Pyspark notebook in Synapse to save data in data lake gen2. I can run the notebook without any issues, but when I run the notebook using Pipeline I got an error :

Error
{
"errorCode": "6002",
"message": "AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.nio.file.AccessDeniedException Operation failed: \"This request is not authorized to perform this operation using this permission.\", 403, HEAD, https://dstetl01devadls.dfs.core.windows.net/dstetl01devadlsfs/synapse/workspaces/dst-etl01-dev-ws/warehouse?upn=false&action=getStatus&timeout=90)\nTraceback (most recent call last):\n\n File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py\", line 723, in sql\n return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)\n\n File \"/home/trusted-service-user/cluster-env/env/lib/python3.8/site-packages/py4j/java_gateway.py\", line 1304, in call\n return_value = get_return_value(\n\n File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 117, in deco\n raise converted from None\n\npyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.nio.file.AccessDeniedException Operation failed: \"This request is not authorized to perform this operation using this permission.\", 403, HEAD, https://dstetl01devadls.dfs.core.windows.net/dstetl01devadlsfs/synapse/workspaces/dst-etl01-dev-ws/warehouse?upn=false&action=getStatus&timeout=90)\n",
"failureType": "UserError",
"target": "Pyspark_etl",
"details": []
}

[183826-synapse-error.png][1]:

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,460 questions
0 comments No comments
{count} vote

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 79,376 Reputation points Microsoft Employee
    2022-03-17T02:14:04.437+00:00

    Hello @Engy Tawadros ,

    Thanks for the question and using MS Q&A platform.

    To resolve the issue, I would suggest you to add the RBAC Storage Blob Data Contributor to the user that is running the notebook, or your user.

    Note: Storage Blob Data Contributor: Use to grant read/write/delete permissions to Blob storage resources.

    If you are not assigning Storage Blob Data Contributor to users who are accessing the storage account, they will be not able to access the data from ADLS gen2 due to the lack of permission on the storage account.

    If they try to access data from ADLS gen2 without the "Storage Blob Data Contributor" role on the storage account, they will receive the error message: Operation failed: "This request is not authorized to perform this operation.",403.

    Once the storage account is created, select Access control (IAM) from the left navigation. Then assign the following roles or ensure they are already assigned.
    Assign yourself to the Storage Blob Data Owner role on the Storage Account

    67282-image.png

    For more details, refer to Notebook - This request is not authorized to perform this operation. , 403 and also you may checkout the MS Q&A thread addressing similar issue.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful