Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,599 questions
Unable to list the directory when create view for delta
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 28.999999999999996%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYC%3C/text%3E%3C/svg%3E)
Yap Chun Hing
5
Reputation points
Assuming, I have the a single spark dataframe outputted as DELTA into two folders using pyspark: folder A and folder B in the same blob storage Gen2
For Folder A and Folder B, both have external data source created with same credentials using sql script.
When creating a view for delta at Folder A, it is successful. [Folder A is the old folder that have been used.]
However, when creating a view for delta at Folder B, there is listing error.
/_delta_log/.' cannot be listed.
Since the view can be created for delta at Folder A, it means there is no issues on the delta files itself. The view for Folder B is to be created with a different name, so it shall not be a problem of conflicting view name.
It seems like permissions issues, but how could it happen since both folder A and B are within the same blob storage (same container), just different path.
Could it be firewall? How should I move on from here? Please advice.
{count} vote
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT 28,446 Reputation points Microsoft Employee2024-06-18T15:51:34.0166667+00:00 Hello
ACLs, permissions would have to be given on all folders in the hierarchy including the root.
Regarding the issue with creating a view for delta at Folder B, it is possible that there is an issue with the parquet files themselves. Applying a powerful RBAC permission at the storage account level (Storage Blob Data Contributor) can help rule out any issues with corrupt parquet files.
I recommend checking the permissions for all folders in the hierarchy, including the root, and validating the parquet files to ensure that they are not corrupt. Applying the Storage Blob Data Contributor permission at the storage account level can help rule out any issues with permissions.
I hope this helps
-
Yap Chun Hing 5 Reputation points
2024-06-19T05:30:24.6633333+00:00 I have tried on manually config the ACL permission of folder B to mimic folder A. It doesn't work. I try to recreate the delta again in folder B, it still doesn't work. By the way, folder A and folder B both contains the same delta outputed from spark. It shall not be the parquet corrupt issues, since the parquet in folder A can be listed.
Hmm... I know that my organisation is in the mid of implementing the purview, will it have any chance that affect this?
-
BhargavaGunnam-MSFT 28,446 Reputation points Microsoft Employee
2024-06-19T19:38:43.4266667+00:00 I don't believe implementing Purview will cause an issue here. If you have a support plan, I suggest opening a support case so that a support engineer can further troubleshoot the issue in a screen-sharing session. In case you don't have a support plan, please let me know so that I can provide a one-time free support request to work on this issue.
I am looking forward to hearing from you.
Sign in to comment