Process parquet files in azure function

Question

Process parquet files in azure function

Suraj Somani 1

I have a requirement- Create azure function API to read multiple parquet format file from datalake, apply some filter on data and respond with the result. But I am not able to serialize parquet file content. I tried using NuGet package- Parquet.Net, below is the code used. It results in empty records object. IMO, problem is that the stream I received from datalake is not seakable, thus when I mapped it to local stream, no content was mapped, thus results in empty. Kindly suggest if anything is missing here-

            if (fileFormat == "parquet")  
            {  
              // as stream read from azure datalake is not seakable, storing it in local memory stream and then process  
                var ms = new MemoryStream();  

                data.Content.CopyTo(ms);  
                ms.Position = 0;  
                  
                var records = (await ParquetConvert.DeserializeAsync

P a u l 10,761 Reputation points

2023-01-09T14:05:06.857+00:00

Could you show us your localClassName definition & a sample of the data from your data.Content?
Suraj Somani 1 Reputation point

2023-01-10T13:56:31.653+00:00

My Local class will look something like this-

Data.content is parquet file content which doesnt look like human readable format to me. File content schema correctly is matching to the class definition. parquet file is created from below csv file

P a u l 10,761

The code works for me if I use it with a sample parquet file I found online:


using Parquet;

using var data = new FileStream(&#34;userdata1.parquet&#34;, FileMode.Open, FileAccess.Read);

var ms = new MemoryStream();
data.CopyTo(ms);
ms.Position = 0;

var records = (await ParquetConvert.DeserializeAsync&lt;localClassName&gt;(ms));

Console.WriteLine(records);

class localClassName {
	public DateTimeOffset registration_dttm { get; set; }
	public int id { get; set; }
	public string first_name { get; set; }
	public string last_name { get; set; }
	public string email { get; set; }
	public string gender { get; set; }
	public string ip_address { get; set; }
	public string cc { get; set; }
	public string country { get; set; }
	public string birthdate { get; set; }
	public double salary { get; set; }
	public string title { get; set; }
	public string comments { get; set; }
}

So surely it's related to the format of the parquet file? If you save the file locally & try to deserialise it like I'm doing above do you still get no results?

Suraj Somani 1 Reputation point

2023-01-11T10:52:22.8333333+00:00

You are right, code is fine. It was issue with my parquet file. When tested with another file locally, then it worked. Thanks for suggestions, I have corrected file and issue is resolved.
Suraj Somani 1 Reputation point

2023-01-11T12:59:32.81+00:00
Issue in serializing data-

I am able to deserialize data, apply filters and get the required records in a list. When I try to serialize it back to parquet-

If I use FileStream and write the serialized/parquet stream to a file, it works perfectly fine. I can open this parquet file in parquet views and able to read the data.

When I use memoryStream and send it back as the API response to postman (API caller), the response received is not a valid parquet file, unable to read the content.

My requirement is to send the filtered records as parquet file in API response. Can you please suggest If I am making any mistake. Below is my code snippet. filteredRecords is an List<LocalClassName>.

D:\env4.parquet--> this file is readable parquet file.

return new OkObjectResult(await R.ReadToEndAsync()) --> the response that I am sending back to azure function caller is resulting in invalid parquet file.

else if (fileFormat == "parquet") { using (Stream fileStream = File.OpenWrite("D:\\env4.parquet")) using (var mem = new MemoryStream()) { await ParquetConvert.SerializeAsync(filteredRecords, mem); await ParquetConvert.SerializeAsync(filteredRecords, fileStream); mem.Position = 0; StreamReader R = new StreamReader(mem, true); return new OkObjectResult(await R.ReadToEndAsync()); } }

Your answer

P a u l 10,761 Reputation points

2023-01-09T14:05:06.857+00:00

Could you show us your localClassName definition & a sample of the data from your data.Content?
Suraj Somani 1 Reputation point

2023-01-10T13:56:31.653+00:00

My Local class will look something like this-

Data.content is parquet file content which doesnt look like human readable format to me. File content schema correctly is matching to the class definition. parquet file is created from below csv file
P a u l 10,761 Reputation points

2023-01-10T19:20:10.327+00:00

The code works for me if I use it with a sample parquet file I found online:

using Parquet; using var data = new FileStream("userdata1.parquet", FileMode.Open, FileAccess.Read); var ms = new MemoryStream(); data.CopyTo(ms); ms.Position = 0; var records = (await ParquetConvert.DeserializeAsync<localClassName>(ms)); Console.WriteLine(records); class localClassName { public DateTimeOffset registration_dttm { get; set; } public int id { get; set; } public string first_name { get; set; } public string last_name { get; set; } public string email { get; set; } public string gender { get; set; } public string ip_address { get; set; } public string cc { get; set; } public string country { get; set; } public string birthdate { get; set; } public double salary { get; set; } public string title { get; set; } public string comments { get; set; } }

So surely it's related to the format of the parquet file? If you save the file locally & try to deserialise it like I'm doing above do you still get no results?
Suraj Somani 1 Reputation point

2023-01-11T10:52:22.8333333+00:00

You are right, code is fine. It was issue with my parquet file. When tested with another file locally, then it worked. Thanks for suggestions, I have corrected file and issue is resolved.
Suraj Somani 1 Reputation point

2023-01-11T12:59:32.81+00:00

Issue in serializing data-

I am able to deserialize data, apply filters and get the required records in a list. When I try to serialize it back to parquet-

If I use FileStream and write the serialized/parquet stream to a file, it works perfectly fine. I can open this parquet file in parquet views and able to read the data.

When I use memoryStream and send it back as the API response to postman (API caller), the response received is not a valid parquet file, unable to read the content.

My requirement is to send the filtered records as parquet file in API response. Can you please suggest If I am making any mistake. Below is my code snippet. filteredRecords is an List<LocalClassName>.

D:\env4.parquet--> this file is readable parquet file.

return new OkObjectResult(await R.ReadToEndAsync()) --> the response that I am sending back to azure function caller is resulting in invalid parquet file.

else if (fileFormat == "parquet") { using (Stream fileStream = File.OpenWrite("D:\\env4.parquet")) using (var mem = new MemoryStream()) { await ParquetConvert.SerializeAsync(filteredRecords, mem); await ParquetConvert.SerializeAsync(filteredRecords, fileStream); mem.Position = 0; StreamReader R = new StreamReader(mem, true); return new OkObjectResult(await R.ReadToEndAsync()); } }

Share via

Process parquet files in azure function

Your answer