Read Parquet directly from blob storage ADLS in Python Azure Function.

Question

Hi, I have a PyArrow table (parquet file) in an ADLS storage account. Using a Python function, I need to query that Parquet file and return a value. The way I see it is by downloading the file and performing the filter. However, there seems to be a way to do it directly in the storage account by using PyArrowfs_adlgen2. In the following example: https://pypi.org/project/pyarrowfs-adlgen2/, it uses 'azure.identity.DefaultAzureCredential()' which I do not have. I would like to use the connection string. Is this possible?

Answer 1

Hi Javier Guerrero, Thank you for posting query in Microsoft Q&A Platform.

Looks like, pyarrowfs-adlsgen2 library cannot directly take connection string. It will uses azure.identity.DefaultAzureCredential. As per azure.identity.DefaultAzureCredential it will looks for credentials at many placee including Environment Variables. To know more about it check this link.In your case, since you wanted to work with connection string only, I would suggest you to consider using azure.storage.filedatalake.DataLakeFileContent and pandas. Below is the code.

import io
from azure.storage.filedatalake import DataLakeFileClient
import pandas as pd

# Replace these values with your own
account_url = "https://accountName.dfs.core.windows.net"
file_system_name = "containerName"
file_path = "folder/sample.parquet"
credential = "accountkey"

# Create a DataLakeFileClient object for the specified file
file_client = DataLakeFileClient(account_url=account_url, file_system_name=file_system_name, file_path=file_path, credential=credential)

# # Download the parquet file as a stream
# with file_client.download_file() as stream:
#     # Read the parquet file into a pandas DataFrame
#     df = pd.read_parquet(stream, engine='pyarrow')


# Download the parquet file as a stream
stream = file_client.download_file()
data = stream.readall()

# Read the parquet file into a pandas DataFrame
df = pd.read_parquet(io.BytesIO(data), engine='pyarrow')
print(df)

In the above code, we are not downloading file to local. We are downloading file directly to stream and reading from there.
Hope this helps. Please let me know how it goes.

---

Please consider hitting Accept Answer button. Accepted answers help community as well.