question

DevBuster-3739 avatar image
0 Votes"
DevBuster-3739 asked ShaikMaheer-MSFT commented

How to read data from file residing in subfolder in azure data lake storage in .net core

I have a .net core application where i need to read data from ADLS file. Issue is file is residing in subfolders. For example- Container/Folder1/Folder2/File.json

I am using below code to get the filtered data:

         var connectionString = this.accessKey;
         var blobServiceClient = new BlobServiceClient(connectionString);
         var containerClient = blobServiceClient.GetBlobContainerClient("ods");

         await foreach (var blobItem in containerClient.GetBlobsAsync(BlobTraits.Metadata, BlobStates.None, "File Prefix"))
         {
             var blobClient = containerClient.GetBlockBlobClient(blobItem.Name);

             var options = new BlobQueryOptions
             {
                 InputTextConfiguration = new BlobQueryJsonTextOptions(),
                 OutputTextConfiguration = new BlobQueryJsonTextOptions()
             };

             var result = await blobClient.QueryAsync(@"SELECT * FROM BlobStorage WHERE measuringpointid = 547", options);

             var jsonString = await new StreamReader(result.Value.Content).ReadToEndAsync();

             Console.WriteLine(jsonString);
             Console.ReadLine();
         }
azure-data-lake-storage
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @DevBuster-3739 ,

Thank you for posting query in Microsoft Q&A Platform.

I guess you can create a DataLakeFileClient instance that represents the file that you want to read and use the DataLakeFileClient.ReadAsync method.

DataLakeDirectoryClient class may help to point to you directory path(full path including subfolders) and from there you can create DataLakeFileClient instance by pointing to your file.
146743-image.png

Click here to know about reading content of file from ADLS Gen1 using .NET
Click here to know about read and download file from a directory in ADLS Gen2 using .NET

Please try and let us know how it goes. Thank you.

0 Votes 0 ·
image.png (23.8 KiB)

Thanks for the reply..
Datalake file which i have is 10GB. I just need to execute the query as stated in the question.
For example: out of 10 records just filter whose name == 'test'

0 Votes 0 ·

also when i try to run the above query i get below error:
XML specified is not syntactically valid.\nRequestId:e3204a59-901e-005f-7612-d2ee5f000000\nTime:2021-11-05T06:58:32.8652708Z\r\nStatus: 400 (XML specified is not syntactically valid.

0 Votes 0 ·

0 Answers