Sometime it requires to monitor Service Bus relay connectivity. ConnectionStatusBehavior allows us to monitor the same through Connecting, Online and Offline events. However, it requires to modify the code to handle these events if listener doesn’t have the same implemented already, and most of the times this is the case. However, when there’s situation to monitor the Service Bus connectivity in crisis, there’s no option but to modify the code of relay listener.

Many of us know customers’ concerns when we request them to apply ConnectionStatusBehavior to monitor Service Bus relay connectivity. Their concerns are valid to some extent as we’re requesting them to add code in their application to log monitoring events which requires rebuilding and redeploying relay listener in production L.

Thus, many of us have a demand if there’s an option to apply Service Bus ConnectionStatusBehavior on relay endpoint through configuration which emits connectivity events.

Good news, we’ve created a ConnectionStatusBehavior behavior extension (let’s name it ConnectionStatusBehaviorExtention) which allows to apply ConnectionStatusBehavior on endpoint through configuration. Please find the binaries in attached ConnectionStatusElementExtended.zip file.

ConnectionStatusBehaviorExtention can be applied following below simple steps,

a)      Copy ConnectionStatusElementExtended.dll (in attached zip file) under WCF Relay service installation folder (bin)

b)      Open WCF Relay Service configuration (i.e. Web.config),

                              I.            Add following configuration under <system.serviceModel>\<behaviorExtensions> element to register ConnectionStatusBehaviorExtention,

<addname="connectionStatusBehaviorExtention"type="ConnectionStatusElementExtended.ConnectionStatusExtendedBehaviorElement, ConnectionStatusElementExtended" />

                              II.            Then, add following configuration under endpointBehavior used by relay endpoint  to apply above defined behavior extension. This is the line which applies  our extended ConnectionStatusBehaviorExtension on relay endpoint
<connectionStatusBehaviorExtention /> 

                              III.            I have built ConnectionStatusBehaviorExtention referencing Microsoft.ServiceBus.dll v2.7.00. Considering Microsoft.ServiceBus.dll assembly version changes frequently and assembly is signed, we’d need following configuration to be applied to redirect .net runtime to use Microsoft.ServiceBus.dll used by customer’s WCF Relay Service. Define Microsoft.ServiceBus.dll version used by WCF relay service (listener) under newVersion configuration (highlighted).

<runtime>

    <assemblyBindingxmlns="urn:schemas-microsoft-com:asm.v1">

      <dependentAssembly>

        <assemblyIdentityname="Microsoft.ServiceBus"publicKeyToken="31bf3856ad364e35"culture="neutral" />

        <!-- Assembly versions can be redirected in app, publisher policy, or machine configuration files. -->

        <bindingRedirectoldVersion="2.7.0.0"newVersion="3.0.0.0" />

      </dependentAssembly>

    </assemblyBinding>

</runtime>

For example, based on above configuration, my WCF Relay Service on which ConnectionStatusBehaviorExtention is applied seems to be using Microsoft.ServiceBus.dll v3.0.0.0.
Please note that the version number is to be taken from Microsoft.ServiceBus.dll assembly name, it’s not the file version.

That’s it! With above simple steps ConnectionStatusBehaviorExtention is applied and ready to log the connection event information.

Where’re the logs Written?

======================

a)      ConnectionStatusBehaviorExtention logs the events in Application eventlog with Source: Microsoft.ServiceBus.ConnectStatus. The event ids are between 990-992. You can try filtering the event log to view events based on Source Microsoft.ServiceBus.ConnectStatus or event id range 990 - 992.

b)      I’ve also added tracing feature built on top of System.Diagnostics to allow emit trace events using different TraceListeners (Similar like WCF tracing or System.Net tracing).  The trace source name is same: Microsoft.ServiceBus.ConnectStatus.
Capturing these traces should be extremely useful in situations where we need to correlate the connectivity traces with WCF traces or confirm the callstacks responsible for emitting the events i.e. online or offline.
here’s the sample configuration which can be used to emit events using XmlWriterTraceListener.

    <system.diagnostics>

        <sources>

          <sourcename="Microsoft.ServiceBus.ConnectStatus"switchValue="Information">

            <listeners>

              <addname="ConnectStatusTraceListener">

              </add>

            </listeners>

          </source>

        </sources>

        <sharedListeners>

            <addinitializeData="connection_tracelog.svclog"type="System.Diagnostics.XmlWriterTraceListener"

                name="ConnectStatusTraceListener"traceOutputOptions="Timestamp, ProcessId, ThreadId, Callstack">

            </add>

        </sharedListeners>

        <traceautoflush="true" />

    </system.diagnostics>

Here's a sample WCF Relay listener service configuration which uses extended ConnectionStatusBehavior on relay endpoints (Please see the highlighted text),

<runtime>

    <assemblyBindingxmlns="urn:schemas-microsoft-com:asm.v1">

      <dependentAssembly>

        <assemblyIdentityname="Microsoft.ServiceBus"publicKeyToken="31bf3856ad364e35"culture="en-us" />

        <!-- Assembly versions can be redirected in app, publisher policy, or machine configuration files. -->

        <bindingRedirectoldVersion="2.7.0.0"newVersion="3.0.0.0" />

      </dependentAssembly>

    </assemblyBinding>

  </runtime>

 <system.serviceModel>

    <services>

      <servicebehaviorConfiguration="HelloServiceBus.HelloServiceBus"name="HelloServiceBus.HelloServiceBus">

        <endpointaddress="https://xxxx.servicebus.windows.net/netrelay"behaviorConfiguration="sharedSecretClientCredentials"binding="basicHttpRelayBinding"bindingConfiguration="HttpRelayEndpointConfig"

          contract="HelloServiceBus.IHelloServiceBus"/>

      </service>

    </services>

    <bindings>

       <basicHttpRelayBinding>

        <!--<binding name="HttpRelayEndpointConfig" proxyAddress="sproxy-se.corp.vattenfall.com:8086" useDefaultWebProxy="false" maxReceivedMessageSize="2147483647" maxBufferPoolSize="2147483647" maxBufferSize="2147483647" transferMode="Streamed" closeTimeout="00:05:00" openTimeout="00:05:00" receiveTimeout="00:05:00" sendTimeout="00:05:00">-->

              <bindingname="HttpRelayEndpointConfig">

          <readerQuotasmaxDepth="2147483647"maxStringContentLength="2147483647"maxArrayLength="2147483647"maxBytesPerRead="2147483647"maxNameTableCharCount="2147483647" />

          <securityrelayClientAuthenticationType="RelayAccessToken" />

        </binding>

      </basicHttpRelayBinding>

      </bindings>

    <behaviors>

      <endpointBehaviors>

        <behaviorname="sharedSecretClientCredentials">

          <connectionStatusBehaviorExtention  /> 

          <serviceRegistrySettingsdiscoveryMode="Public" />

          <transportClientEndpointBehavior>

            <tokenProvider>

              <sharedAccessSignaturekeyName="XXX"key="XXX"/>

            </tokenProvider>

          </transportClientEndpointBehavior>

          <!--<ServiceRegistrySettings discoveryMode="Public" />-->

        </behavior>

      </endpointBehaviors>

      ...

      ...

    </behaviors>

    <extensions>

      <!-- In this extension section we are introducing all known service bus extensions. User can remove the ones they don't need. -->

      <behaviorExtensions>

        <addname="connectionStatusBehaviorExtention"type="ConnectionStatusElementExtended.ConnectionStatusExtendedBehaviorElement, ConnectionStatusElementExtended" />

        <addname="transportClientEndpointBehavior"

          type="Microsoft.ServiceBus.Configuration.TransportClientEndpointBehaviorElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>

        <addname="serviceRegistrySettings"

          type="Microsoft.ServiceBus.Configuration.ServiceRegistrySettingsElement, Microsoft.ServiceBus, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>

      </behaviorExtensions>

      ...

      ...

   </extensions>

  </system.serviceModel>

ConnectionStatusElementExtended.zip

I had replied to one of the posts on an online forum sites which had a simple question. However, looking at the upvote count by different users shows that there’re many who had similar question. Thus, I decided to create a blog post on the same,

Question: Why Asp.Net Web Service Application template is missing in Visual Studio 2010?

Solution: ASP.NET Web Service Application project template is not available for .Net framework 4.0, however, available for .Net Framework 3.5.

If it’s a new development targeting .Net framework 4.0, you can chose WCF Service over ASMX web service as ASMX web service is legacy. Please note that you'd need to enable AspNetCompatibilityMode to access HttpContext objects.

If you still want to use classic ASMX web service, choose ASP.NET Empty Web application and then you can add ASMX webservice files to the project.

Hope this helps to save some time!

Regards,

Amit Bhatia  

Recently, I came across a scenario where client application was intermittently failing to connect to a WCF Service with ServiceTooBusyException. The binding had secure conversation enabled.

Of course, the very first plan we followed was to capture powerful WCF traces which were on the spot. We saw traces full with following warnings which confirmed that there’re too many pending security sessions being leaked.

What is Pending security session?

A Security sessions that is established with the server for which service has issued a session token but for which no application messages are sent. For example, client application creates the channel and calls open (channel.Open()). Open call will establish the secure session between client and service. Now, this secure session is added to pending secure session list until client application invokes a WCF operation on this channel.

By default, Pending security session limit is 128. As the pending security session limit was exceeded, WCF service didn’t accept new sessions and client application failed with ServiceTooBusy exception.

Next action for us was to find out why there’s no operation invoked by client and leaving security session as pending security session!

With extensive code review, we found that client application had code for some functionality as follows,

a) New client channel was created and open method was called i.e. Channel.Open().
b) However, the operation was executed if there’s certain conditions met
c) Then, channel.close was called to close the channel

If we look at above code, it looks pretty fine and shouldn’t cause Pending security session to leave open as channel was closed no matter if operation on channel was invoked or not. So, what’s causing so many Pending security sessions in our case?

Researching further, we found that there’s known bug with WCF which doesn’t remove security session from pending security session list if client just opens the channel and closes it. This bug is fixed in .Net Framework 4.5. Upgrading the servers to .Net framework 4.5 helped to resolve the issue.

However, unlike our scenario, it’s also possible to exceed pending security session limit if the service is loaded with burst load. In that case, it requires increasing default limit (128). To increase the default pending security session limit, it requires to convert the standard binding to custom binding and increase pending security session under localServiceSettings element,

<customBinding>

    <bindingname="customSecureBinding">

        <securityauthenticationMode="SecureConversation">

            <localServiceSettings maxPendingSessions="1024" />

        </security>

        <textMessageEncoding />

        <httpTransport />

    </binding>

</customBinding>

Hope this helps to save someone's time. 

Cheers,
Amit Bhatia 

Recently, we came across the scenario where customer had a WCF service built on .Net Framework 4.0. This WCF service has couple of endpoints using TCP bindings with configuration as follows,

    <services>

      <servicename="Service.CalcService">

 <endpointaddress="CalcService" binding="netTcpBinding" bindingConfiguration="nettcp"contract="Service.ICalcService"/>

         <endpointaddress="OrderService" binding="netTcpBinding" contract="Service.IOrderService"/>

        <host>
          <baseAddresses>
            <addbaseAddress="net.tcp://localhost:8082/Service"/>
          </baseAddresses>
        </host>  

      </service>
    </services>

    <bindings>
      <netTcpBinding>
        <bindingname="nettcp"maxConnections="10">
          <securitymode="Transport"/>
        </binding>
      </netTcpBinding>
    </bindings>

Customer had requirement to upgrade (install) their server machines to .Net Framework 4.5. After installing .Net framework 4.5 on machines, customer noticed that WCF service failed to activate with following exception,

Unhandled Exception: System.ServiceModel.AddressAlreadyInUseException: There is already a listener on IP endpoint 0.0.0.0:8080. This could happen if there is another application already listening on this endpoint or if you have multiple service endpoints in your service host with the same IP endpoint but with incompatible binding configurations. ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted.

Cause: Above exception can occur if there’s mismatch in binding configuration for 2 different endpoints using TCP bindings and they cannot share the sample port (see Here). However, looking at the binding configuration applied on endpoints above, they look pretty similar! So, we enabled the WCF diagnostics tracing at verbose level and it indeed showed that transport manager is found incompatible on opening 2^nd endpoint, which means 2 endpoints cannot share same port,

Looking at binding configuration for above 2 endpoints closely again, we see that one of the endpoint has nettcp binding configuration applied which sets parameter maxConnections = 10.

One of the changes in .Net framework 4.5 brings in change in MaxConnection knobs, where using the default bindings, some of the defaults are calculated using OSEnvironmentHelper.ProcessorCount instead of hard coded default values as they were in .Net 4.0.

In this case, when using a named netTcpBinding bindingConfiguration ("nettcp"), the value supplied for the MaxConnections property is 10. (Setting the MaxConnections property on the NetTcpBinding also sets it's TcpTransportBindingElement's MaxPendingConnections property and the TcpTransport's ConnectionPoolSettings.MaxOutboundConnectionsPerEndpoint property to the same value.)

In case of OrderService endpoint, it’s using NetTcpBinding with default values, so the MaxPendingConnections property was calculated by using the following algorithm: return (12 * OSEnvironmentHelper.ProcessorCount);

So, When using a named netTcpBinding bindingConfiguration for CalcService endpoint, the transport's MaxPendingConnections was 10, and for OrderService endpoint, transport's MaxPendingConnections was 12*CPU count. The difference in values for the MaxPendingConnections between these two endpoints causes incompatible transport manager and 2 endpoints cannot share the same port. Same issue could have also been seen if there’s a mismatch in ListenBacklog parameter. Also, in my situation, customer has 2 different endpoints using netTcpBindings. The issue can also happen if there's mex endpoint using maxTcpBinding with default values, which is very common setup.

Application worked in .Net Framework 4, because both bindings resolved to same MaxConnection values i.e. 10. nettcp binding configuration sets MexConnections = 10 and maxTcpBinding didn’t override this value which is by default 10. Thus, there’s no conflict and both endpoint could share the same port.

Solutions: There’re multiple solutions available to resolve the issue and choose per your requirement:

a)      If you want to have a limit on MaxPendingConnections and ListenBacklog on first endpoint, then change the second endpoint to listen on a different base address by specifying the fully qualified URL using different port than used in based address, in endpoint address. For example, to resolve issue in my customer’s case, the changes would look as follows for second endpoint,
<endpointaddress="net.tcp://localhost:8083/Service/OrderService"binding="netTcpBinding"contract="Service.IOrderService"/>
 

b)      Alternatively, use the same binding configuration for Mex endpoint by changing is binding to NetTcpBinding. Again, this option should be followed when you want to have a limit on MaxPendingConnections and ListenBacklog now for both endpoints,
<endpointaddress="OrderService"binding="netTcpBinding"bindingConfiguration="nettcp"contract="Service.IOrderService"/>
 

c)      If you don’t want to have a limit on MaxConnections and ListenBacklog, then simply remove these parameters from binding configuration so that both named NetTcpBinding uses default values for maxConnections and ListenBacklog and there’s no mismatch.

I hope this helps to save someone’s time!

Cheers,

Amit Bhatia