Share via

Error while implementing File Notification Mode: com.microsoft.azure.storage.StorageException: This request is not authorized to perform this operation using this permission.

Hiran Amarathunga 25 Reputation points
2024-06-21T06:16:38.9166667+00:00

While implementing File Notification Mode in Autoloader, I get the following error. Has anyone faced the similar issue?

Note: The Databricks Service Principal is having Contributor role to Storage account.

com.microsoft.azure.storage.StorageException: This request is not authorized to perform this operation using this permission.

at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87) at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:196) at com.microsoft.azure.storage.queue.CloudQueue.retrieveMessages(CloudQueue.java:1281) at com.databricks.sql.aqs.AQSClientImpl.$anonfun$fetch$1(AQSClientImpl.scala:175) at com.databricks.sql.aqs.AQSClientImpl.withTokenAccess(AQSClientImpl.scala:255) at com.databricks.sql.aqs.AQSClientImpl.fetch(AQSClientImpl.scala:186) at com.databricks.sql.fileNotification.AsyncFetchMixin.fetchParallelSync(QueueClient.scala:211) at com.databricks.sql.fileNotification.AsyncFetchMixin.fetchParallelSync$(QueueClient.scala:203) at com.databricks.sql.aqs.AQSClientImpl.fetchParallelSync(AQSClientImpl.scala:47) at com.databricks.sql.fileNotification.autoIngest.NotificationFileEventFetcher.fetch(FileEventWorkerThread.scala:302) at com.databricks.sql.fileNotification.autoIngest.NotificationFileEventFetcher.run(FileEventWorkerThread.scala:259)

Thanks

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,036 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 83,646 Reputation points Microsoft Employee
    2024-06-21T08:10:39.9633333+00:00

    @Hiran Amarathunga - Thanks for the question and using MS Q&A platform.

    It seems like you are facing an issue while implementing File Notification Mode in Autoloader. The error message suggests that the request is not authorized to perform this operation using this permission.

    You mentioned that the Databricks Service Principal has the Contributor role to the Storage account. However, it is possible that the service principal may not have the required permissions to perform the operation.

    To resolve this issue, you can try the following steps:

    • Verify that the Databricks Service Principal has the required permissions to access the Storage account. Make sure to assign the Storage Blob Data Contributor role to the service principal.
    • Check if the Storage account firewall is enabled. If it is enabled, make sure that the Databricks cluster IP address is added to the allowed IP addresses list.
    • Verify that the Storage account connection string is correct, and that the Storage account is accessible from the Databricks cluster.

    If the issue persists, please provide more information about your setup and the steps you have taken so far. Sharing the code and the screenshot of the error message helps to assist better.

    For more details, refer to MS Q&A thread addressing similar issue.

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.