While executing custom activity code from ADF we are getting System.Net.WebException: The underlying connection was closed

Siddhesh Chavan 0 Reputation points
2023-11-10T12:32:47.47+00:00

Hello All,

We have a custom activity (C# .Net code) which loads the data from application and create a CSV file which gets placed over the blob location. This Activity is triggered in ADF with the help of batch services.

This code code is working everywhere except in azure batch.

we have tried running the code on vs, via adf everywhere it is working but in batch getting this error


System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.TlsStream.CallProcessAuthentication(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
   at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.ConnectStream.WriteHeaders(Boolean async)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.OData.Client.HttpWebRequestMessage.GetResponse()

Can you please let us know what would be the cause?

Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
302 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,570 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,265 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Amira Bedhiafi 15,446 Reputation points
    2023-11-10T19:19:00.16+00:00

    As a workaround, you may need to set out the Timeout and ReadWriteTimeout of the request to higher values, or set KeepAlive to false.

    Big files uploading (WebException: The connection was closed unexpectedly)

    Check more here :

    https://stackoverflow.com/questions/1459475/system-net-webexception-the-underlying-connection-was-closed-the-connection-wa

    1 person found this answer helpful.

  2. Siddhesh Chavan 0 Reputation points
    2024-03-05T11:26:16.9+00:00

    Hello All,

    post raising the ticket with Microsoft, they informed it that while our code was executing the memory of VM was going till 100% due to which there was nothing to get store.

    post their suggestion to create a new batch pool we were able to establish the connection.
    and we were able to execute the activity.

    Also, we were using sku '2012-r2-datacenter' whch was metioned by the Support team that is in verge of expiration, so we have created with '2022-datacenter'.

    0 comments No comments