Synapse Spark - AuthorizationPermissionMismatch

Ryan Abbey 1,171

We have a storage account that uses CDM based files (D365 synapse link), we are trying to use the CDMUtil to create Spark based tables to enable us to reference these storage account files, however, we are getting the below error when running the utility

"This request is not authorized to perform this operation using this permission.", 403, GET, https://<XX>.dfs.core.windows.net/dd?upn=false&resource=filesystem&maxResults=500&directory=dd/Tables/Finance/FixedAssets/Main/AssetBook&timeout=90&recursive=false, AuthorizationPermissionMismatch, "This request is not authorized to perform this operation using this permission. RequestId:e2cbf764-401f-006a-192a-35320c000000 Time:2023-01-31T04:16:40.8880756Z"

The synapse managed identity has Storage Blob Data Contributor role on the storage account; as it's creating a Spark table, I'm not sure if there's anything that needs to be done on the target side (we have already created delta based tables on the target side but no external table like this)

Any ideas what else we are missing?

BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-02-01T20:06:55.31+00:00

Hello @Ryan Abbey ,

Welcome to the MS Q&A platform.

Please correct me if my understanding is wrong. You are trying to access the storage account using the CDMUtil utility and getting an error. Incorrect permissions on the storage account usually cause this error.

To resolve this, the managed identity of the Synapse workspace has the correct permissions on the storage account. In this case, you have mentioned that the managed identity has the Storage Blob Data Contributor role, which is correct for creating a Spark table. However, you also need to ensure that the managed identity has the correct permissions on the specific folder or container where the CDM files are stored. You can grant Storage Blob Data Reader permission to synapse managed identity on the specific folder or container to allow it to read the CDM files.

Also, you can try using SAS token authentication to access the storage account. With SAS token authentication, you can specify the permissions (read/write) at the folder or container level. This will allow you to grant the appropriate permissions to the managed identity without granting access to the entire storage account.

I hope this helps. Please let me know if you have any further questions.
Ryan Abbey 1,171 Reputation points

2023-02-02T20:20:57.32+00:00

Thanks for the response, it was discovered that the CDMUtil is not using the managed instance user but rather my own ID, when we gave permission for me to access the storage account, it worked
BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-02-06T19:54:40.53+00:00

Hello @Ryan Abbey ,

Thank you for letting us know the solution. Sorry, I can't reproduce the issue from my end as I don't have the D365 synapse link.

As per the prerequisites mentioned in this document, We need to grant synapse workspace managed identity, blob data contributor access to the data lake.

But I can check with my internal team and get back to you on the behavior,

I appreciate your patience.
BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee

2023-02-13T23:29:17.4166667+00:00

Hello @Ryan Abbey ,

I am still waiting for a response from my internal team. If this is a priority request, please submit a support request so the support engineer can directly work with you to collect the logs to troubleshoot the issue further. Please let me know if you need any help with the support request.