How to fix OutOfMemoryException thrown on Azure AKS?

Rehaan 26

I have deployed a application pod in Azure AKS which accepts a parameter of type byte array. I am sending a file byte array of 120 MB but it throws OutOfMemoryException.

Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMO8F59CUA9H", Request id "0HMO8F59CUA9H:00000002": An unhandled exception was thrown by the application.
      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags)

My node pool has 16 GB RAM.

Container config:

resources:  requests:    memory: "200Mi"    cpu: "100m"  limits:    memory: "200Mi"    cpu: "100m"

Rehaan 26 Reputation points

2023-02-07T09:07:59.81+00:00
I changed the config to the below but the error persists.

resources: requests: memory: "400Mi" cpu: "300m" limits: memory: "400Mi" cpu: "300m"
Rehaan 26 Reputation points

2023-02-07T09:09:33.1333333+00:00
I changed the config as per below but the error persists.

resources: requests: memory: "500Mi" cpu: "300m" limits: memory: "500Mi" cpu: "300m"
Rehaan 26 Reputation points

2023-02-07T09:29:33.6466667+00:00
Akram Kathimi 1,041 Reputation points Microsoft Employee

2023-02-07T09:59:38.04+00:00

If the pod is not facing any resource issues and you are not seeing OOM killed errors in kubernetes, then the issue would be in your code. you can find many references to this error online, for example check this.
Rehaan 26 Reputation points

2023-02-07T11:22:22.2666667+00:00

Here is my scenario:

I am running a an integration test where I read a file (120MB) in byte array. and assigns it to a ViewModel.
It goes through proxy where i am using httpClient and it sends the viewModel with byteArray in json Body to an API. The API sends the ViewModel to another API in json Body via httpClient. And the second API uploads the file to Blob Storage.

The error is coming from the first API right now.

Test -> Proxy -> Api -> 2nd Api -> Blob Storage

Any insights?
Rehaan 26 Reputation points

2023-02-07T11:23:08.73+00:00

Here is my scenario:
I am running a an integration test where I read a file (120MB) in byte array. and assigns it to a ViewModel.
It goes through proxy where i am using httpClient and it sends the viewModel with byteArray in json Body to an API. The API sends the ViewModel to another API in json Body via httpClient. And the second API uploads the file to Blob Storage.
The error is coming from the first API right now.
Test -> Proxy -> Api -> 2nd Api -> Blob Storage
Any insights?
Akram Kathimi 1,041 Reputation points Microsoft Employee

2023-02-08T01:18:34.7333333+00:00

Hi,

From the container config, we can see that the memory limit for the container is 200Mi. Given that the file is 120 MB (which is about 114 Mi) and the pod's own process memory consumption, and the error. It is clear that the pod is struggling with memory resources.

Please try to increase the memory limits for the container, and see if the error persists. Check this doc for more info.

Also, you can check the current memory consumption for the running containers by using the Insights tab in the AKS cluster from the portal. as shown here, and in the below screenshot.

Please Accept the answer if the information helped you. This will help us and others in the community as well.
Swampillai, Quintus (Omaha) 0 Reputation points

2023-10-10T17:57:29.9966667+00:00

We are running into same issue. Did anyone find a solution for it.

1 answer

AirGordon 7,105 Reputation points

2023-11-07T07:54:04.4+00:00

I'd suggest you start by significantly bumping the request for memory and removing the limit from the pod in your dev environment. This will allow your code to run, and for you to better troubleshoot without adding the Kubernetes complexities. Limits for memory tell Kubernetes to kill the pod if it reaches that value.

After you have your code working properly, put your resource limit back.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment