Kubernetes POD got stuck not processing any events from RabbitMQ

Dani Kurniawan 0 Reputation points
2023-11-27T01:05:22.7066667+00:00

Hi,

Has anyone here experienced this?

We have a service built using .Net 6, sometimes our service got stuck unable to process messages from RabbitMQ, and restart was required to fix it. Before the service got stuck, we got a lot of errors "System.Net.Sockets.SocketException (0xFFFDFFFE): Unknown socket error"

Configuration :

  • .Net 6
  • Kubernetes in AWS EKS
ASP.NET Core
ASP.NET Core
A set of technologies in the .NET Framework for building web applications and XML web services.
4,190 questions
.NET Runtime
.NET Runtime
.NET: Microsoft Technologies based on the .NET software framework.Runtime: An environment required to run apps that aren't compiled to machine language.
1,125 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. hossein jalilian 3,015 Reputation points
    2024-02-02T21:31:29.36+00:00

    Thanks for posting your question in the Microsoft Q&A forum.

    It could be indicative of various issues. Here are some recommendations to troubleshoot and address the problem: Error Handling and Retry Mechanism: Enhance your service's error handling and implement a robust retry mechanism for RabbitMQ interactions. This can help the service recover from transient issues without requiring a restart.

    1. Graceful Shutdowns: Ensure that your .NET service handles shutdown signals gracefully. This is crucial for Kubernetes to manage the pod lifecycle correctly.
    2. Connection Pooling: Review the configuration related to connection pooling in your .NET service. Ensure that the number of open connections to RabbitMQ is managed efficiently and does not lead to resource exhaustion. Resource Limits: Check if the pod has sufficient resources (CPU, memory) allocated. Resource constraints might lead to unexpected behavior, especially if the service is struggling to handle incoming messages.
    3. Update RabbitMQ Client Library: Ensure that you are using the latest version of the RabbitMQ client library compatible with .
    4. Monitor Kubernetes Events: Leverage Kubernetes events and logs to monitor pod events and status changes. This can provide additional context about what might be causing the pod to become unresponsive.

    Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful

    0 comments