Secure SSL Stream.read ...throws System.IO.IOException: The decryption operation failed (System.ComponentModel.Win32Exception: The specified data could not be decrypted)

Question

we are using the SSL Stream Read to receive bytes of the files from the remote machine using TCP port .we have authentication done with X509Certificate2 certifcate

for the file transfer from one machine to another.
In call back of the tcplistener, BeginAcceptTcpClient we are doing this and getting the error in sslstream.read

The decrypt failure io exception

FileTransferInfo info;
using (var tcpClient = serverListener.EndAcceptTcpClient(ar))
{
serverListener.BeginAcceptTcpClient(OnConnect, null);

                // A client has connected. Create the 
                // SslStream using the client's network stream.
                using (var sslStream = new SslStream(
                    tcpClient.GetStream(),
                    true,
                    ValidateClientCertificate,
                    null))
                {
                    sslStream.AuthenticateAsServer(this.serverCertificate, true, SslProtocols.Tls12, false);

                    var bytes = new byte[sizeof(int)];
                    try
                    {
                        sslStream.Read(bytes, 0, sizeof(int));

Please help us it is happening very randomly .. not everytime