AKS-Deployed Azure Function Services – Expected Memory Growth and Pod Restart Behavior ?

Question

We are running multiple Azure Function Services deployed on Azure Kubernetes Service (AKS), processing messages from Azure Service Bus and Event Hub.

We consistently observe the following behavior across Function Services:

• After pod startup, container memory usage gradually increases

CPU usage remains low and stable

When the container memory limit is reached, Kubernetes restarts the pod

• After restart:
  • Memory usage resets
  • The Function Service resumes message processing normally
• The same behavior is observed across multiple Function Services but the same functions hosted in ACA there we don't see this kind of memory usage behavior.

We would like Microsoft to clarify the following:

Whether this gradual memory growth under sustained Service Bus / Event Hub message processing is expected behavior for Azure Function Services running on AKS.

Whether 2 Gi memory should be considered a minimum baseline rather than a safe upper limit for Azure Function containers on AKS.

Whether pod restarts caused by reaching the configured memory limit should be considered expected and acceptable behavior, provided that the Function Service restarts successfully and continues processing messages without functional impact.

1. Whether Microsoft provides official guidance or best practices for memory sizing and capacity planning of Azure Function Services deployed on AKS.

This information is required for production rollout assurance and capacity planning.

Answer 1

Hey @ramprasad.r !

It looks like you're encountering some common behavior with Azure Function Services deployed on AKS, particularly around memory usage and pod restarts. Let’s unpack your questions and concerns.

Expected Memory Growth

• Gradual Memory Growth: It's generally expected that memory usage can gradually increase over time when processing events from Azure Service Bus and Event Hub. As functions process more messages, they might consume more memory. This is typical if there's no memory leak, but continuous or unusual memory growth could indicate a problem.

Memory Limits

• Memory Limits of 2 Gi: A memory limit of 2 Gi can indeed serve as a baseline for many workloads. However, the appropriate limit really depends on the specific memory requirements of your function. If you're consistently hitting that limit, you might want to increase it based on the actual usage patterns you observe with kubectl top pod.

Pod Restarts: It's acceptable in Kubernetes for pods to restart due to exceeding memory limits (OOMKilled). The main concern is whether the restarts impact functionality. If your functions are resuming normal operations post-restart without issues, this behavior can be considered manageable.

Best Practices and Guidance

Microsoft does provide best practices for memory sizing and capacity planning for Azure Function Services on AKS. You may want to consider using tools like the Vertical Pod Autoscaler to help optimize resource requests based on actual usage. Additionally, implementing Pod Disruption Budgets can help minimize impact during pod restarts and upgrades.

Recommendations:

• Monitor Memory Usage: Use kubectl top pod <pod-name> and kubectl describe pod <pod-name> to monitor memory usage and diagnose restarts.
• Consider Increasing Limits: If you observe high memory consumption, consider increasing the memory limit for your function pods.
• Implement Autoscaling: Look into setting up Horizontal Pod Autoscaler and Vertical Pod Autoscaler to ensure your service scales appropriately based on real-time demand.
• Review Logs: Check application logs for any memory leaks or performance issues leading to high memory consumption.

I hope this all helps clarify things! If you have more questions, feel free to ask!

References:

• Troubleshoot OOMKilled in AKS
• Pod stuck in CrashLoopBackOff
• AKS Planned Maintenance
• Best practices for reliability
• Vertical Pod Autoscaler documentation
• Pod Disruption Budgets
• Scaling Azure Functions

Answer 2

The behavior you are observing with Azure Function Services deployed on Azure Kubernetes Service (AKS) regarding gradual memory growth under sustained message processing is not uncommon. Here are some insights based on the context provided:

1. Gradual Memory Growth: It is expected that memory usage may gradually increase as the Azure Functions process messages, especially if there are memory-intensive operations or if the functions are not optimized for memory usage. This behavior can lead to the container reaching its memory limit, resulting in Kubernetes restarting the pod.
2. Memory Limits: Setting a memory limit of 2 Gi can be a suitable baseline for Azure Function containers on AKS, but it should be evaluated based on the specific workload and memory requirements of your functions. Monitoring memory usage over time can help determine if this limit is appropriate or if adjustments are needed.
3. Pod Restarts: Pod restarts due to reaching the configured memory limit can be considered acceptable behavior, provided that the function service resumes processing without functional impact. However, frequent restarts may indicate that the memory limits need to be reassessed or that there may be underlying issues with memory management in the functions.
4. Best Practices: While the context does not provide specific official guidance for memory sizing of Azure Function Services on AKS, it is generally recommended to configure appropriate resource requests and limits for your pods. This includes using tools like kubectl top to monitor memory usage and adjusting limits accordingly to avoid OOMKilled events.

For production rollout assurance and capacity planning, it is advisable to continuously monitor the performance and memory usage of your Azure Function Services and adjust configurations based on observed patterns and resource needs.

---

References:

• Troubleshooting OOMKilled in AKS clusters
• Deployment and cluster reliability best practices for Azure Kubernetes Service (AKS)