Micro service throws System.Net.Sockets.SocketException in AKS cluster

Bhagavan Basvani 1 Reputation point
2021-06-28T10:39:22.277+00:00

Hi,

We have deployed our (.Net Core - 3.1) micro services in AKS Cluster.
We are experiencing huge number of Socket exceptions like 25K in 24 hours.
What could be the root cause?

  1. Number of IP addresses in the Cluster (using Azure CNI)?
  2. Is it because of micro service code?

Here is what I could find from the Application Insights.
Message: Address already in use Address already in use
Exception Type: System.Net.Sockets.SocketException
Failed method: System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw

Thank you!

Regards,
Bhagavan Basvani

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,999 questions
{count} votes

1 answer

Sort by: Most helpful
  1. SRIJIT-BOSE-MSFT 4,331 Reputation points Microsoft Employee
    2021-06-28T12:46:58.513+00:00

    @Bhagavan Basvani , Thank you for the question. Following is a summary of the error your application has hit.

    WSAEADDRINUSE
    10048

    Address already in use
    Typically, only one usage of each socket address (protocol/IP address/port) is permitted. This error occurs if an application attempts to bind a socket to an IP address/port that has already been used for an existing socket, or a socket that was not closed properly, or one that is still in the process of closing. For server applications that need to bind multiple sockets to the same port number, consider using setsockopt (SO_REUSEADDR). Client applications usually need not call bind at all—connect chooses an unused port automatically. When bind is called with a wildcard address (involving ADDR_ANY), a WSAEADDRINUSE error could be delayed until the specific address is committed. This could happen with a call to another function later, including connect, listen, WSAConnect, or WSAJoinLeaf.

    For more information please refer to this document.

    If this is seen with significantly large outbound traffic generated from the cluster, there is a possibility that there is SNAT port exhaustion. [Reference] To mitigate exhausting SNAT ports please follow this article.

    ----------

    Hope this helps.

    Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.

    0 comments No comments