Micro service throws System.Net.Sockets.SocketException in AKS cluster

Question

Hi,

We have deployed our (.Net Core - 3.1) micro services in AKS Cluster.
We are experiencing huge number of Socket exceptions like 25K in 24 hours.
What could be the root cause?

1. Number of IP addresses in the Cluster (using Azure CNI)?
2. Is it because of micro service code?

Here is what I could find from the Application Insights.
Message: Address already in use Address already in use
Exception Type: System.Net.Sockets.SocketException
Failed method: System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw

Thank you!

Regards,
Bhagavan Basvani

Answer 1

@Bhagavan Basvani , Thank you for the question. Following is a summary of the error your application has hit.

WSAEADDRINUSE
10048

Address already in use
Typically, only one usage of each socket address (protocol/IP address/port) is permitted. This error occurs if an application attempts to bind a socket to an IP address/port that has already been used for an existing socket, or a socket that was not closed properly, or one that is still in the process of closing. For server applications that need to bind multiple sockets to the same port number, consider using setsockopt (SO_REUSEADDR). Client applications usually need not call bind at all—connect chooses an unused port automatically. When bind is called with a wildcard address (involving ADDR_ANY), a WSAEADDRINUSE error could be delayed until the specific address is committed. This could happen with a call to another function later, including connect, listen, WSAConnect, or WSAJoinLeaf.

For more information please refer to this document.

If this is seen with significantly large outbound traffic generated from the cluster, there is a possibility that there is SNAT port exhaustion. [Reference] To mitigate exhausting SNAT ports please follow this article.

----------

Hope this helps.

Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.