Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,474 questions
1 answer
About right naming convention on paths and blobs
Hi, I'm used to design ADLS partition in the following format: source={source}/table={table}/date={date}/content.csv When a table in any source doesn't have data in certain date, that partition won't exists. In the other hand, not every table will…
2 answers
One of the answers was accepted by the question author.
Azure Datalake query accelaration error XML specified is not syntactically valid.\nRequestId:e3204a59-901e-005f-7612-d2ee5f000000\nTime:2021-11-05T06:58:32.8652708Z\r\nStatus: 400 (XML specified is not syntactically valid.
I am using query acceleration to access and filter data lake- https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-query-acceleration-how-to?tabs=dotnet%2Cazure-cli I followed the same steps as per the document to the letter. but…
asked 2021-11-05T07:07:13.077+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 17%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
DevBuster007
186
Reputation points
commented 2024-10-17T09:58:52.64+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 99%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
James Anthony
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Power BI JSON file 1000 row limit
Hi all, I am facing issues with Power BI only pulling in 1000 rows. I have exported JSON data into azure data lake storage gen 2, which then our business analyst was able to pull in the files and create separate tables in a pbix file. The issue he has…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 17%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Why is the activity log stored in the adls gen2 container not a valid json
Why is the activity log stored in the adls gen2 container not a valid json? It's line by line, and the outer layer is missing '[]' and comma separation
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 53%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZ%3C/text%3E%3C/svg%3E)
1 answer
partitioning of Data lake
I have just the below link on data partitioning of Azure data lake : https://learn.microsoft.com/en-us/azure/architecture/best-practices/data-partitioning But it is too brief and does not cover this important topic extensively. I request you to please…
asked 2024-10-08T10:09:02.7433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 88%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azure_learner
280
Reputation points
commented 2024-10-15T18:56:04.92+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 6%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKD%3C/text%3E%3C/svg%3E)
Keshavulu Dasari
765
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
DataLakeServiceClient.DeleteFileSystem() method on non-ADLSv2 upgraded storage accounts
The .NET codebase I work on interacts with some storage accounts that are upgraded to ADLSv2 and some that are not. I noticed that the DataLakeServiceClient.DeleteFileSystem() method works in both cases. Can I get a confirmation that the method's…
asked 2024-10-08T00:52:39.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 39%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Max Morrison
30
Reputation points Microsoft Employee
accepted 2024-10-15T00:58:33.5466667+00:00
Max Morrison
30
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure datalake and data consistency
Hi Experts, Azure Data Lake Storage (ADLS) does not natively provide full ACID (Atomicity, Consistency, Isolation, Durability) transaction support unlike traditional relational databases designed to support ACID transactions. This raises the following…
asked 2024-10-12T09:44:01.3866667+00:00
azure_learner
280
Reputation points
accepted 2024-10-13T11:24:21.8266667+00:00
azure_learner
280
Reputation points
1 answer
Best way to grant SQL DB (serverless sql pool) access to end user.
What is the best way to grant access to users for SQL DB(serverless sql pool) build on top of synapse azure lake db. How should SQL DB (views or external tables) access the lake database (storage container). Should we use RBAC, Managed Identity, ACLS,…
asked 2024-10-08T00:04:18.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 44%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAF%3C/text%3E%3C/svg%3E)
Abbas Furniturewala
0
Reputation points
commented 2024-10-10T07:05:46.4733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
10,740
Reputation points Microsoft Vendor
0 answers
Data lake performance and improvement
The data lake performance depends on critical factors such as partitioning on date and if available in data region etc. I have already partitioned the data lake in a hierarchal structure LandingZone Subject area YYYY MM …
asked 2024-10-07T16:48:20.0033333+00:00
azure_learner
280
Reputation points
commented 2024-10-10T01:18:42.5566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 31%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Vinod Kumar Reddy Chilupuri
485
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
can abac be applied to a data lake?
Can abac be applied to a data lake storage account or does it have to be rbac with potential ACL?
asked 2024-09-13T04:26:59.0133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 22%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Michael Liveri
40
Reputation points
accepted 2024-10-03T23:50:18.7066667+00:00
Michael Liveri
40
Reputation points
2 answers
Azure data lake folder structure
Hi, the link provided in the below thread is not working, any way to find the information or a new URL/link: FAQs About Organizing a Data…
asked 2024-09-19T08:43:49.7+00:00
azure_learner
280
Reputation points
commented 2024-10-03T09:16:32.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
7,811
Reputation points Microsoft Vendor
2 answers
Copy Activity very slow for Azure data factory
Hi All, we are having issue where copy activity from Azure Table storage to data factory is taking over 50mins on daily basis. it used to take 15-20mins until a week back. What's the source of the slow copy activity? Azure Table storage What's the sink…
asked 2024-09-13T03:50:38.0833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 74%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mayuri Shah
0
Reputation points
commented 2024-10-03T05:41:48.32+00:00
AnnuKumari-MSFT
32,991
Reputation points Microsoft Employee
0 answers
Filepath() ,metadata functions in serverless view on delta format
Hi , Earlier i was creating serverless views with FORMAT='PARQUET' using openrowset wildcard paths ("/synapsetesing/test/*.parquet"), so i was able to use metadata functions like filepath etc , now when i create views on FORMAT='DELTA' ,…
asked 2021-11-08T07:20:29.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
commented 2024-10-03T01:27:05.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 0%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGK%3C/text%3E%3C/svg%3E)
Gilbert, Keith
0
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF pipeline to read the data from UC table to adls gen2 account
Hello Team, We have a requirement to create Azure Datafactory pipeline to read the data from UC table, access on the table is provided ( to Azure Datafactory Managed Identity) and copy the data into adls gen2. Is there a way or article to implement this?…
asked 2024-04-11T19:05:05.9733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 77%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
130
Reputation points
edited a comment 2024-10-01T08:27:53.21+00:00
Kuldeep Singh
16
Reputation points
1 answer
One of the answers was accepted by the question author.
Read Delta Tables from ADLS Gen2 in Power BI: Exploring Direct Access Methods
As part of our data pipeline, I'm using Azure Blob Storage as the source, Azure Data Factory (ADF) for processing, and Azure Data Lake Storage Gen2 (ADLS Gen2) as the destination. Here's the process I've followed: Step 1: I read a CSV file from Azure…
asked 2024-08-23T13:46:12.77+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 16%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAP%3C/text%3E%3C/svg%3E)
Akshay Patel
45
Reputation points
commented 2024-09-23T12:02:50.7433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 3%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDG%3C/text%3E%3C/svg%3E)
Daniel Gil
0
Reputation points
1 answer
One of the answers was accepted by the question author.
SQL Server Integration Services and Azure Data Lake Store Connection Manager
Is it possible a SQL Server Integration Services (SSIS) package can use the Azure Data Lake Store Connection Manager to connect to an Azure Data Lake Storage Gen2 ?? From what I read the article below refers only for Gen 1 but and about the Gen 2? Link…
asked 2024-09-18T14:52:20.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 74%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELB%3C/text%3E%3C/svg%3E)
Leonardo Bruno Pereira de Araujo
20
Reputation points
commented 2024-09-19T11:29:24.41+00:00
Leonardo Bruno Pereira de Araujo
20
Reputation points
2 answers
Unable to connect the ADLS to snowflake
I am not able to connect the ADLS to snowflake. I requested azure to create the connection but I don't see the user that starts with Snowflake... when I create a role in IAM.
asked 2024-09-05T18:47:27.9966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Purnima Sharma
0
Reputation points
commented 2024-09-19T08:09:02.9566667+00:00
AnnuKumari-MSFT
32,991
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to overwrite a table from SQL Server to SQL Dedicated Pool using Copy Data Activity in Azure Synapse Analytics
I am trying to use the Copy Data Activity in Azure Synapse Analytics to copy data from SQL Server to SQL Dedicated Pool. However, I want to ensure that every time I run the Copy Data Activity, the table in SQL Dedicated Pool is overwritten. How can I…
asked 2024-09-18T08:58:39.26+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 69%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Aditya Singh
160
Reputation points
accepted 2024-09-19T05:17:18.4233333+00:00
Aditya Singh
160
Reputation points
0 answers
Slow performance reading parquet files from Data Lake Gen2
I'm using Azure Data Lake Gen2 to store data in parquet file format. I have split the data using partitions by year, month and day to benefit from the filtering functionality. I'm reading the data from python directly (no Spark or Synapse here) using the…
asked 2024-02-15T14:36:23.51+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 30%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJL%3C/text%3E%3C/svg%3E)
Jorge Lopez
36
Reputation points
edited the question 2024-09-17T17:55:48.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 76%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
VarunTha
8,935
Reputation points Microsoft Vendor
0 answers
I got this error when debug my copy data on azure df 7cf2da46-b65b-4098-b8ec-d9c7858c190e
after connecting my Azure data factory to MsSqL, I tried to copy my data into azure data factory and I got this error 7cf2da46-b65b-4098-b8ec-d9c7858c190e
asked 2024-09-13T14:44:21.2833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 72%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECO%3C/text%3E%3C/svg%3E)
Charles Onyebuchi
20
Reputation points
commented 2024-09-17T15:42:37.1166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 46%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENN%3C/text%3E%3C/svg%3E)
NIKHILA NETHIKUNTA
2,790
Reputation points Microsoft Vendor