Azure data lake storage Java SDK to read from blob line by line

Question

Azure data lake storage Java SDK to read from blob line by line

Anonymous

Hi,

I am starting to get used to Azure Data lake storage. We are developing analytics application using Java with massive amount of data being stored in the data lake. The data is in the form of Json structure i.e. each event is a Json with different format and we store this as part of single blob file with the events separated by newline character or in another format we could just separate the events using commas. e.g.

{event1Json}

{event2Json}

OR

{event1Json},{event2Json}

This basically causes single blob storage file to store many thousands of events.

The real challenge is when we want to read this data using DataLakeFileClient Java SDK. This class provides read() API that reads entire blob file into an OutputStream which may crash the JVM due to large data being loaded.

The question I have is:

Is it possible to read event json data line by line from single blob file without loading entire blob into OutputStream?
Or do we need to store the events in small chunks in several multiple blob storage files if #1 is not possible using DataLake File client Java SDK?

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-09T23:51:57.6266667+00:00
Hello Deole, Pushkar (Pushkar),

Using python SDK, you can use get_blob_to_text method to read the blob line by line.

#read the content of the blob(assume it's a .txt file) str1 = block_blob_service.get_blob_to_text(container_name,blob_name) #split the string str1 with newline. arr1 = str1.content.splitlines() #read the one line each time. for a1 in arr1: print(a1)

This is discussed here: https://stackoverflow.com/questions/55968716/read-the-text-file-in-azure-storage-blob-line-by-line-using-python

If the blobs are too large to fit into memory, you might need to consider splitting the events into smaller chunks and storing them in multiple blobs. This would allow you to read each blob individually, reducing the memory footprint.

I hope this helps.
Anonymous

2024-01-10T05:55:22.2733333+00:00

Hi @Bhargava-MSFT ,

I am using Java SDK and looking for help on Java SDK API. Also, we have to store all data as one single blob file with events separated by newline character so we are looking for API that would read from that single blob without loading entire blob in memory.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-17T16:26:08.45+00:00

Hello Deole, Pushkar (Pushkar), I am checking to see if you had a chance to look into my above answer. Please let me know if you have any further questions.

1 answer

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-09T23:51:57.6266667+00:00

Hello Deole, Pushkar (Pushkar),

Using python SDK, you can use get_blob_to_text method to read the blob line by line.

#read the content of the blob(assume it's a .txt file) str1 = block_blob_service.get_blob_to_text(container_name,blob_name) #split the string str1 with newline. arr1 = str1.content.splitlines() #read the one line each time. for a1 in arr1: print(a1)

This is discussed here: https://stackoverflow.com/questions/55968716/read-the-text-file-in-azure-storage-blob-line-by-line-using-python

If the blobs are too large to fit into memory, you might need to consider splitting the events into smaller chunks and storing them in multiple blobs. This would allow you to read each blob individually, reducing the memory footprint.

I hope this helps.
Anonymous

2024-01-10T05:55:22.2733333+00:00

Hi @Bhargava-MSFT ,

I am using Java SDK and looking for help on Java SDK API. Also, we have to store all data as one single blob file with events separated by newline character so we are looking for API that would read from that single blob without loading entire blob in memory.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-17T16:26:08.45+00:00

Hello Deole, Pushkar (Pushkar), I am checking to see if you had a chance to look into my above answer. Please let me know if you have any further questions.

Answer 1

Hello Deole, Pushkar (Pushkar), I am not an expert on the Java SDK, but after further research, I found the below Microsoft document for your requirement. https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-java-sdk#read-a-file

// Read File
InputStream in = client.getReadStream(filename);
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
String line;
while ( (line = reader.readLine()) != null) {
    System.out.println(line);
}
reader.close();
System.out.println();
System.out.println("File contents read.");

The above code reads data from a single blob file line by line without loading the entire blob into memory. The getReadStream method returns an InputStream for the blob, and the BufferedReader reads the blob line by line. I hope this helps.

Share via

Azure data lake storage Java SDK to read from blob line by line

1 answer

Your answer