python app function error when reading a parquet file

LEANDRO DARIVA PINTO 11 Reputation points

I am developing a python script that will run as a app function. It should read one parquet file from our gen1 datalake and do some processing over it.
When running in debug mode in VS Code it works perfectly but when I deploy the script to the app function it retrieve a error with a not very meaninfull message.

Executed 'Functions.get_warehouse_from_sap' (Failed, Id=227a48b8-0486-4c3f-8758-1f6298afaf68, Duration=9122ms)

This happens when it tries to read the parquet file. I tried to use pyarrow and pandas.read_parquet function but both give the same error. I tried to put try execept aroung this particular point of the code but none excepetion is retrieved. To read the datalake I am using AzureDLFileSystem from python libray. Here is part of my code.

from import lib  
from import AzureDLFileSystem  
import pandas as pd  
adlCreds = lib.auth(tenant_id=tenant_id,  
                               resource = '')  
 adlsFileSystemClient = AzureDLFileSystem(adlCreds, store_name='<repository name>')'<path to my file>')  
#until here it works fine. It can open the file  

#here is where the problem happens.  
except Exception as e:  

Any idea?

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
3,005 questions
{count} votes

1 answer

Sort by: Most helpful
  1. LEANDRO DARIVA PINTO 11 Reputation points

    I finally manage to solve the problem. The problem was memory consumption. As I am still running the app in a dev enviroment it has just 1.5Gb of memory. I was reading the whole parquet file which was consuming approximatelly 2.5Gb. I changed my code so I use the pyarrow.read_table with filter option and reading just the necessary columns from the parquet file. This reduced the memory consumption and the app function started working.

    1 person found this answer helpful.
    0 comments No comments