Why is the task taking much more RAM memory when running on Azure batch node than on my laptop ?

ajb cryptocapital 0 Reputation points
2023-12-21T21:17:22.1433333+00:00

Hello,

I want to run on azure batch a python script that takes around 10GB of RAM (mainly from dataframe) on my laptop and more than 32GB on Azure nodes. I am wondering how it can increase so much from my laptop to Azure machine. I tried to run it on different type of machine (A, B, D, 16GB, 32GB of RAM) but each time I have the exit code 137 (out-of-memory). When I ssh to the node I see that the consumption of memory is huge, and I even dump logs of memory to see what takes a lot of RAM. At each loop of my script adding a column in a dataframe takes twice or three times more memory than on my laptop.

Did you ever encounter such a difference between laptop RAM usage and Azure nodes ? If yes what is the solution ?

Thanks a lot

Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
305 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. vipullag-MSFT 24,206 Reputation points Microsoft Employee
    2023-12-27T10:38:01.82+00:00

    Hello ajb cryptocapital

    Welcome to Microsoft Q&A Platform, thanks for posting your query here.
    It seems like you are facing memory issues while running your Python script on Azure Batch. The difference in memory usage between your laptop and Azure nodes could be due to various factors such as differences in hardware, operating system, and software configurations.

    To troubleshoot the issue, you can try the following steps:

    1. Check the memory usage of your Python script on your laptop and compare it with the memory usage on Azure nodes. This will help you identify if there is a significant difference in memory usage.
    2. Check if there are any memory leaks in your Python script. You can use memory profiling tools to identify memory leaks in your code.
    3. Check if you are using the latest version of Python and the required packages. Upgrading to the latest version of Python and packages can sometimes help resolve memory issues.
    4. Try optimizing your code to reduce memory usage.
    5. Consider using a larger Azure node with more memory to run your Python script. You can also try increasing the memory limit for your Azure Batch job.

    I hope these steps help you resolve the memory issues you are facing while running your Python script on Azure Batch.

    0 comments