MemoryError: Unable to allocate 84.4 MiB for an array with shape (11056593,) and data type int64

Question

My Azure web app isn't using all of the available memory. I am trying to read a 2 Gb .csv file into a pandas data frame but the web server says I am out of memory. However I am currently supposed to have 3.5 Gb of memory on the web service.

Why can't I use all of my memory for python?

accountURLPerformance.csv 11/11/2021, 7:18:48 PM 2142424 KB

[11/12/2021 20:12:57 > a76a50: ERR ] File "URL_syndicationSpike_v2.py", line 32, in readDataframe
[11/12/2021 20:12:57 > a76a50: ERR ] df = pd.read_csv(file)
[11/12/2021 20:12:57 > a76a50: ERR ] File "D:\home\python364x86\lib\site-packages\pandas\io\parsers.py", line 688, in read_csv
[11/12/2021 20:12:57 > a76a50: ERR ] return _read(filepath_or_buffer, kwds)
[11/12/2021 20:12:57 > a76a50: ERR ] File "D:\home\python364x86\lib\site-packages\pandas\io\parsers.py", line 460, in _read
[11/12/2021 20:12:57 > a76a50: ERR ] data = parser.read(nrows)
[11/12/2021 20:12:57 > a76a50: ERR ] File "D:\home\python364x86\lib\site-packages\pandas\io\parsers.py", line 1198, in read
[11/12/2021 20:12:57 > a76a50: ERR ] ret = self._engine.read(nrows)
[11/12/2021 20:12:57 > a76a50: ERR ] File "D:\home\python364x86\lib\site-packages\pandas\io\parsers.py", line 2157, in read
[11/12/2021 20:12:57 > a76a50: ERR ] data = self._reader.read(nrows)
[11/12/2021 20:12:57 > a76a50: ERR ] File "pandas_libs\parsers.pyx", line 847, in pandas._libs.parsers.TextReader.read
[11/12/2021 20:12:57 > a76a50: ERR ] File "pandas_libs\parsers.pyx", line 890, in pandas._libs.parsers.TextReader._read_low_memory
[11/12/2021 20:12:57 > a76a50: ERR ] File "pandas_libs\parsers.pyx", line 2074, in pandas._libs.parsers._concatenate_chunks
[11/12/2021 20:12:57 > a76a50: ERR ] File "<array_function internals>", line 6, in concatenate
[11/12/2021 20:12:57 > a76a50: ERR ] MemoryError: Unable to allocate 84.4 MiB for an array with shape (11056593,) and data type int64
[11/12/2021 20:12:58 > a76a50: SYS INFO] Status changed to Failed
[11/12/2021 20:12:58 > a76a50: SYS ERR ] Job failed due to exit code 1

Answer 1

We found that @Elliott Harrison was using a 32bit process python application

"The maximum available amount of memory for a 32-bit process (even on a 64-bit operating system) is 2 GB. By default, the worker process is set to 32-bit in App Service (for compatibility with legacy web applications)."

• https://learn.microsoft.com/en-us/azure/app-service/faq-availability-performance-application-issues#i-see-the-message--worker-process-requested-recycle-due-to--percent-memory--limit---how-do-i-address-this-issue-
• https://stackoverflow.com/questions/639540/how-much-memory-can-a-32-bit-process-access-on-a-64-bit-operating-system

Resolution
Followed this article to update the python app to a 64bit process and that resolved their issue.
https://learn.microsoft.com/en-us/visualstudio/python/managing-python-on-azure-app-service?view=vs-2022

Additional Resources
https://learn.microsoft.com/en-us/windows/win32/memory/memory-limits-for-windows-releases?redirectedfrom=MSDN