The profile defines two different ways to transmit the password (or password equivalent):

·         wsse:PasswordText: the password is sent in clear-text

·         wsse:PasswordDigest: a digest derived from the password is sent

As documented on MSDN, only the first option is supported by WCF’s built-in UsernameToken. This token is used in scenarios where client credential is set to Username.

The second option, wsse:PasswordDigest, is the subject of this post.

Besides WCF’s predecessor – Web Services Enhancements –, other 3^rd-party SOAP protocol stacks (e.g. Axis) support the digest version, too. This leads to interoperability problems with WCF, and we regularly get Support Requests from our customers to get help on this interoperability issue.

Security tokens are most of the time not easy matter, but probably the UsernameToken is one of the simpler tokens. So after I began writing a custom token that supports the wsse:PasswordDigest, this indeed proved to be doable.

Usual Disclaimer

Sample Code is provided for the purpose of illustration only and is not intended to be used in a production environment. THIS SAMPLE CODE AND ANY RELATED INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE. We grant You a nonexclusive, royalty-free right to use and modify the Sample Code and to reproduce and distribute the object code form of the Sample Code, provided that. You agree: (i) to not use Our name, logo, or trademarks to market Your software product in which the Sample Code is embedded; (ii) to include a valid copyright notice on Your software product in which the Sample Code is embedded; and (iii) to indemnify, hold harmless, and defend Us and Our suppliers from and against any claims or lawsuits, including attorneys’ fees, that arise or result from the use or distribution of the Sample Code.

You can download the sample from here.

Layout of the custom UsernameToken

The custom token builds on the CustomToken sample in the Windows SDK’s WCF Samples section. This lays out the skeleton of a custom token by implementing a simple credit card token. Actually, the MSDN documentation for WCF security classes often showcases parts from this credit card sample, where applicable.

Like this stock sample, our UsernameToken token won’t support most of the advanced concepts, simply because they are not possible with this type of token. One of such would be, for example, to use this token for encryption.

This also means that the implementation’s most interesting part is generating the password digest. The rest of the code is plumbing that makes integration with WCF seamless.

The sample Visual Studio 2010 solution consists of 5 projects:

Here’s a list of the most important classes and their role.

UsernameToken class at the core

This central type deals with cryptography to produce the password digest according to the requirements of the WS-S UsernameToken Profile:

Password_Digest = Base64 ( SHA-1 ( nonce + created + password ) )

This is done in GetPasswordDigestAsBase64 as follows:

        public string GetPasswordDigestAsBase64()

        {

            // generate a cryptographically strong random value

            RandomNumberGenerator rndGenerator = new RNGCryptoServiceProvider();

            rndGenerator.GetBytes(_nonce);

            // get other operands to the right format

            byte[] time = Encoding.UTF8.GetBytes(GetCreatedAsString());

            byte[] pwd = Encoding.UTF8.GetBytes(_usernameInfo.Password);

            byte[] operand = new byte[_nonce.Length + time.Length + pwd.Length];

            Array.Copy(_nonce, operand, _nonce.Length);

            Array.Copy(time, 0, operand, _nonce.Length, time.Length);

            Array.Copy(pwd, 0, operand, _nonce.Length + time.Length, pwd.Length);

            // create the hash

            SHA1 sha1 = SHA1.Create();

            return Convert.ToBase64String(sha1.ComputeHash(operand));

        }

Fortunately, this isn’t too complicated thanks to the rich .NET class library available to us:

1.       Create a cryptographically strong random value, which will be the nonce. This can be used to provide defense against replay-attacks with the combination of a nonce-cache in the service (not part of the sampe).

2.       Take the time the token was created at, and transform it to yyyy-MM-ddTHH:mm:ssZ format. Encode it in UTF8 byte array.
Note that knowing the time is useful if we don’t want the service’s nonce-cache grow unlimited.

3.       Encode in a UTF8 byte array.

4.       Allocate a byte array large enough to hold all of this data, and concatenate them.

5.       Create the SHA-1 hash.

6.       Return the base64 representation.

Validating an incoming token takes simply re-building the hash from the password associated with the username (service must know this), and the time and nonce already stored in the incoming token:

        public bool ValidateToken(string password)

        {

            byte[] pwd = Encoding.UTF8.GetBytes(password);

            byte[] createdBytes = Encoding.UTF8.GetBytes(GetCreatedAsString());

            byte[] operand = new byte[_nonce.Length + createdBytes.Length + pwd.Length];

            Array.Copy(_nonce, operand, _nonce.Length);

            Array.Copy(createdBytes, 0, operand, _nonce.Length, createdBytes.Length);

            Array.Copy(pwd, 0, operand, _nonce.Length + createdBytes.Length, pwd.Length);

            SHA1 sha1 = SHA1.Create();

            string trueDigest = Convert.ToBase64String(sha1.ComputeHash(operand));

            return String.Compare(trueDigest, _usernameInfo.Password) == 0;

        }

Deployment

To test the sample, build it with Visual Studio 2010 (please ignore source control warnings, this solutio.n is part of a Team Foundation Server project), and create a virtual application on local IIS that points to the service. The client is configured to use https://localhost/servicemodelsamples/service.svc.

Interoperability with Axis

In the actual scenario I wrote this custom token for, the Java-based service didn’t like the addressing headers that WCF generated. In particular, the wsa:Action header had mustUnderstand set to true, and we got the error message that “Must Understand check failed for header https://www.w3.org/2005/08/addressing : Action”.

WCF doesn’t allow removing mustUnderstand from the addressing headers (unless you remove it from the serialized message). Therefore, in the sample I simply ask WCF to not add the addressing headers by setting MessageVersion to Soap11.

Security note

WS-S UsernameToken Profile 1.0 warns that digested password should be sent over secure transport.

Obviously, using an unsecure transport doesn’t protect against capturing or modifying the message payload, but I’m not sure why the password would not be protected against attacks. Anyway, I’m not a security expert, so I advise following the recommendation and use a secure transport.

UsernameTokenSample.zip

You can provoke the unexpected behavior with the following code. Note that with Visual C++ 2010, debug build of this produces the unexpected behavior described above (ATL::AtlThrowImpl will be called), while the release build yields the intended one (MyThrowImpl will execute).

#define _ATL_CUSTOM_THROW
#define AtlThrow MyThrowImpl
void __stdcall MyThrowImpl(int i);
#include <stdio.h>
#include <tchar.h>
#include <atlbase.h>
#include <atlstr.h>
#include <atltime.h>

int _tmain(int argc, _TCHAR* argv[])
{
       CTime time(11111, 11, 10, 10, 10, 10);
       return 0;
}

void __stdcall MyThrowImpl(int i)
{
       printf("MyThrowImpl");
}

As far as I can see, the issue is present on .NET 4.0 RTM as well.

The error was indeed thrown by WCF: it was a System.ServiceModel.CommunicationObjectAbortedException: “The communication object, System.ServiceModel.Channels.HttpChannelFactory+HttpRequestChannel, cannot be used for communication because it has been Aborted.” There were two factors that made this error mysterious, right from the beginning:

1.       Issue appeared only in release build, and there it was pretty consistently reproducible. Debug build didn’t exhibit this behavior.

2.       The exception was thrown during the first call through a properly created WCF channel on the client-side. Why would its underlying channel be aborted?

Repro on the rescue

Fortunately, I could reproduce the problem on my machine with customer’s application. At the first glance, it wasn’t obvious what’s happening: in one single (but complex) line of code, among many other things, this particular HttpRequestChannel also got created somewhere deep in WCF, but when preparation was complete to issue the call – still on the same line – the HttpRequestChannel was already aborted.

It was a really long line: first, it called a function that created a WCF Channel via a few function calls, and put it into a templatized wrapper type, implementing IDisposable. Once this type was returned, its member function was called, to obtain the ServiceContract’s interface for the channel. On this interface, an actual method was called with some parameters.

So I wanted to find out who’s aborting the HttpRequestChannel. It was called from, System.ServiceModel.ICommunicationObject.Close, which in turn got called by customer’s templatized wrapper. This was where things got more interesting. Obviously, this call must have happened on a different thread then the one executing that single line of code. But which thread? It wasn’t hard to find out from the call stack: it was the finalizer thread.

Now the picture cleared out. The debugger showed that the wrapper object wasn’t rooted anywhere. So indeed it was a candidate for GC, and because WCF channel creation is a heavy-weight operation, it did induce GC. With debug build, the compiled IL code looks to be more conservative and puts these things into “temporary variables” on the stack.

Lessons learnt

But why wasn’t this object rooted on the stack at least?

Because it was a temporary result of expression evaluation. This makes sense, since only parameters, locals and call addresses are stored on the stack, but no temporary objects (unless we’re in debug build). Still it surprised me that temporaries are not preserved at least until the evaluation of the current expression finishes. C++, for example, guarantees that destructors won’t be called until the expression evaluation completes.

However, managed code is different. I learnt from a developer on the CLR that such temporaries are not guaranteed to remain alive. Most of the time, of course, you don’t need to count with such side-effects. However, be aware that GC is non-deterministic and everything that is not rooted can be eliminated at any time. Knowing this can help tracking down insidious bugs like this.

However, if you have a service that exposes many endpoints or operations (or both), than the WSDL can grow pretty large. Especially when applying security to the service the wsp:Policy parts of the document can became quite large. So large, that under certain circumstances, you won’t be able to consume the resulting XML.

In case of svcutil.exe, this manifests itself with an error message similar to the following:

Metadata contains a reference that cannot be resolved: 'net.tcp://172.22.3.68:55301/mex'.

There is an error in the XML document.

There is an error in the XML document.

The maximum nametable character count quota (16384) has been exceeded while reading XML data. The nametable is a data structure used to store strings encountered during XML processing - long XML documents with non-repeating element names, attribute names and attribute values may trigger this quota. This quota may be increased by changing the MaxNameTableCharCount property on the XmlDictionaryReaderQuotas object used when creating the XML reader.

If the service is defined in the current solution, try building the solution and adding the service reference again.

It could also include the following warning:

Warning: The policy expression was not fully imported because it exceeded the maximum allowable complexity. The import stopped at element element_name.

As the error message suggests, the large metadata document caused the XmlDictionaryReader to hit a certain quota, in this case, MaxNameTableCharCount.

These quotas aren’t there just for nagging us. XML processing is a memory and CPU intensive task, and these quotas provide a built-in protection against misuse, such as DoS attacks with specially crafted XMLs (to give an idea about such XMLs: thousands of nested nodes, extreme long element names, etc.).

So, the hint is to raise the specific quota we’re hitting in XmlDictionaryReaderQuotas. But svcutil.exe is a compiled binary, so how are we going to accomplish this?

We’re lucky again, because svcutil.exe is a WCF application, and the XmlDictionaryReader is actually used by WCF and not svcutil.exe directly. Hence, it’s possible to set the quotas by configuring the binding in svcutil.exe.config. The following example demonstrates this.

<?xml version="1.0" encoding="utf-8"?>

<configuration>

  <system.serviceModel>

    <client>

      <endpoint name="net.tcp"

                binding="netTcpBinding"

                bindingConfiguration="GenericBinding"

                contract="IMetadataExchange" />

      <endpoint name="http"

                binding="wsHttpBinding"

                bindingConfiguration="SecureBinding"

                contract="IMetadataExchange" />

    </client>

    <bindings>

      <netTcpBinding>

        <binding name="GenericBinding"

                 maxBufferPoolSize="2147483647"

                 maxBufferSize="2147483647"

                 maxReceivedMessageSize="2147483647" >

          <readerQuotas maxDepth="2147483647"

                        maxStringContentLength="2147483647"

                        maxArrayLength="2147483647"

                        maxBytesPerRead="2147483647"

                        maxNameTableCharCount="2147483647" />

          <security mode="None"/>

        </binding>

      </netTcpBinding>

      <wsHttpBinding>

        <binding name="SecureBinding"

                 maxBufferPoolSize="2147483647"

                 maxBufferSize="2147483647"

                 maxReceivedMessageSize="2147483647" >

          <readerQuotas maxDepth="2147483647"

                        maxStringContentLength="2147483647"

                        maxArrayLength="2147483647"

                        maxBytesPerRead="2147483647"

                        maxNameTableCharCount="2147483647" />

          <security mode="Message">

            <transport clientCredentialType="Windows" />

          </security>

        </binding>

      </wsHttpBinding>

    </bindings>

  </system.serviceModel>

</configuration>

Note that the configuration must be done per binding, and that we actually configure the endpoints that svcutil.exe uses (these have preconfigured names like ‘http’). Correspondingly, other aspects of the bindings (mainly security) must match with those of the service metadata endpoint.

The pre-requisite is that System.servicemodel.dll should be at least version 3.0.4506.722, corresponding to the hotfix https://support.microsoft.com/kb/951111. Versions 3.5 SP1 (3.0 SP2) of the .NET Framework already contain this.

I was quite busy during the last two months, but today I’m back with a new post.

If you use WCF, you’ve probably already come across this tool, which is mostly marketed as WCF’s troubleshooting companion. However, it’s actually a very useful utility for general .NET troubleshooting. Note that this post does not aim to be a complete guide to WCF tracing, which is documented here. Instead, it gives some background to tracing and shows where Service Trace Viewer really shines.

Tracing on .NET standards

WCF builds on the well established System.Diagnostics namespace to produce traces. This is not just a good design decision to reuse code, but also allows to share common concepts with other types of .NET tracing sources.

The most important concept here is activities. An activity is a container for a series of logically related operations. For example, everything that constitutes sending a message to a server could be grouped into a single activity. If one step itself consists of fair amount of work, it could have its own activity. The great thing about activities is that any trace output will be associated to the current activity (using a thread-static variable containing the ActivityId). This allows you to understand the context of a trace output.
Furthermore, activities are hierarchical – so, another way of looking at them is a logical call stack. This enables us to put trace events into an even broader context.

Another concept is a listener. They act as collectors for trace sources in .NET. Listeners can be either private or shared, but as you’d guess, a shared one has the advantage of collecting trace output from various sources. If you’re planning to use Service Trace Viewer, you’ll need to employ System.Diagnostics.XmlWriterTraceListener, because this is the native format for the tool.

When used together, you can group traces from various sources into a single log file, and have a great overview of what’s happening in the process.
For an example, let’s consider the following tracing configuration:

<?xml version="1.0" encoding="utf-8" ?>

<configuration>

    <system.diagnostics>

        <sources>

            <source name="System.ServiceModel" switchValue="Information,ActivityTracing"

                propagateActivity="true">

                <listeners>

                    <add name="ServiceModelTraceListener"/>

                </listeners>

            </source>

            <source name="System.Net.Sockets" switchValue="Information">

              <listeners>

                <add name="ServiceModelTraceListener"/>

              </listeners>

            </source>

            <source name="System.Net" switchValue="Information">

              <listeners>

                <add name="ServiceModelTraceListener"/>

              </listeners>

            </source>

        </sources>

        <sharedListeners>

            <add initializeData="c:\temp\client_tracelog.svclog"

                type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"

                name="ServiceModelTraceListener" traceOutputOptions="LogicalOperationStack, DateTime, Timestamp, ProcessId, ThreadId, Callstack"/>

        </sharedListeners>

        <trace autoflush="true" />

    </system.diagnostics>

.

.

.

</configuration>

This will trace all socket (System.Net.Sockets), HttpWebRequest, ServicePoint, etc. (System.Net) and WCF (System.ServiceModel) activities at Information level, with the nice context that activities provide (for more informatino on the former trace sources, surf to https://msdn.microsoft.com/en-us/library/ty48b824(VS.80).aspx). This is best viewed with the “Graph” mode within Service Trace Viewer:

This is a breaking change in ATL Server that dates back to Visual Studio 2005. It makes CAtlHttpClient refrain from authentication if the Internet Explorer security zone policy does not allow sending user credentials silently to the remote host, determined by its URL. Following is the identical code snippet from latest CodePlex version, atlhttp.inl @ line 1259:

DWORD dwPolicy=0xFFFF;

hr=spSecurityMgr->ProcessUrlAction(strUrlW.GetString(),

      URLACTION_CREDENTIALS_USE,

      reinterpret_cast<BYTE*>(&dwPolicy),

      sizeof(dwPolicy),

      NULL,

      0,

      PUAF_NOUI,

      NULL);

if (FAILED(hr) || dwPolicy != URLPOLICY_CREDENTIALS_SILENT_LOGON_OK)

{

      return false;

}

This code essentially queries IInternetSecurityManager::ProcessUrlAction for security policy that applies to the URL. This is the same policy system that Internet Explorer uses. If the policy doesn’t allow silent logon using the current user’s credentials, we don’t authenticate.

The solution here is to add the URL to a privileged zone and in Internet Explorer, and enable “User Authentication – Logon – Automatic Logon with current username and password” in custom level for that zone. Note that the zone could be either Intranet or Trusted sites, but the default “Anonymous logon only in Intranet zone” doesn’t appear to be sufficient.

* Beginning with Visual Studio 2008, ATL Server and Tools is not part of the product any more. Instead, it became an open-source project on CodePlex. This also means that there’s no support for ATL Server and Tools versions that you can download from CodePlex. Of course, the previous version of ATL Server and Tools is supported within the timeline of Visual Studio 2005 life cycle.

Shared memory concerns

Besides the obvious performance overhead, managed performance counters also consume memory that is shared among .NET processes running on the machine. This committed memory is backed by the page file, so its size counts to the overall system commit. The memory area is set up per performance counter category, and has a default size of 128KB.

So, the size of this memory area should be kept small. However, when all memory counters are enabled WCF, then – depending on your services – the counters (or more precisely, counter instances) will consume a considerable part of these shared memory areas:

• 15 performance counters for each instance of operation (per endpoint across all services).

• 19 performance counters for each instance of endpoint (across all services).

• 33 performance counters for each service.

This means quite many counter instances for the category “ServiceModelOperation 3.0.0.0”. If you have say 10 operations per endpoint, 10 endpoints in each service and 10 services, you’ll end up with 10 x 10 x 10 x 15 = 15 000 performance counter instances for this category.

But how many bytes does one instance need? Among other factors, this also depends on the name of the counter instances, which tend to be long in WCF (combination of service, endpoint and operation name). According to my measurements, they need approximately 320-350 bytes each.

Summing it up

Writing this back to our formula above, our hypothetical scenario would need 15 000 x 350 = 5 250 000 bytes of shared memory for “ServiceModelOperation 3.0.0.0” alone. Sure enough, in such cases the default 128KB won’t be enough for this category. When this happens, WCF notifies you by writing "System.InvalidOperationException: Custom counters file view is out of memory." to the application event log and in the ServiceModel trace.

The solution is to specify a fileMappingSize suitable for your needs in either machine.config or the registry. I’d recommend the latter, since the registry setting allows doing this per category.

On MSDN, there’s a warning that using Assembly.LoadFrom() has disadvantages. The following could be a good illustration of the problems that may arise.

Not long ago I had a customer who was using Assembly.LoadFrom() to load types into their web service on-demand. Intermittently, he received HTTP 500 errors on the client, which he tracked back to a strange exception thrown in the service:

System.InvalidCastException: [A]<type A> cannot be cast to [B]<type A>. Type A originates from <assembly A> in the context 'LoadNeither' at location <location A>. Type A originates from <assembly A> in the context 'Default' at location <location B>.

As if a given type was not castable to itself. I hope you noticed that the types described above do differ in their load contexts and their containing assembly’s location. This helps investigating such issues, thanks to the improved error message.

Indeed, the problem was that my customer placed his plug-in assemblies into the Bin directory, and therefore these assemblies got copied to the temporary folder that ASP.NET uses to compile the web application (or web service for that matter). This directory happens to be the private path for the application domain the web service is running in. This means that every assembly on this path can be loaded via Assembly.Load().
Since client requests contained types from the plug-in assemblies, at some point, XmlSerializer tried de-serialize the XML. However, XmlSerializer is always using Assembly.Load() to load the required assembly into the default loading context. But types that customer was using in his code got loaded into a different context (LoadNeither in this case). As Suzanne Cook’s excellent post explains, types loaded into different context are never castable to each other.

Needless to say that this problem is not specific to web services or XmlSerializer. I’ve seen it also with .NET Remoting and the binary formatter, and very likely it can appear with other combinations of distributed frameworks and serializers.

This is a quite popular topic on the Internet, and this has a reason: a lot of developers need to integrate legacy components written in Visual Basic 6 with .NET.
COM Interop is or should be one of the easiest ways to bring these two worlds together.

The problem begins when multiple threads, apartments or custom data types float into the picture.

Today I’d like to share an easy way to tackle with data type interop, plus an unexplained discovery I made during a recent case.

The problem

My customer had a .NET method defined as follows:

public void SetData(Data[] param)

{

...

}

‘Data’ was his custom type, also defined in .NET.

When calling this method from Visual Basic 6 through COM interop, we get a run-time error: “Function or interface marked as restricted, or the function uses an Automation type not supported in Visual Basic”.

It’s all about the TLB

When data is passed from client to server or back, chances are that it must be marshaled (i.e. converted). There’re a number of conditions that dictate this, e.g. different representation of data, the need to serialize and transmit data over network, etc.

However, (automated) marshaling cannot happen without some metadata describing the actual data types that are involved. With the advent of .NET, metadata is stored right in the assembly with the code. But in terms of COM, it was stored in a Type Library (TLB). The TLB itself is just a binary representation of the Interface Definition Language (IDL). TLB/IDL is the “common language” that all COM components or clients speak. It’s worth noting that IDL is a powerful language that can express constructs which are not supported by every programming language or runtime. Also keep in mind that a TLB can be included within an EXE or a DLL as a resource.

As for Visual Basic 6, it also relies on the TLB to make the types of the component usable in the client. When doing COM interop, the TLB is generated by the .NET SDK tools tlbexp.exe or regasm.exe.

Therefore, if using a managed type via COM interop from Visual Basic 6 doesn’t work out-of-the-box, then very likely there’s a problem with the TLB/IDL. The problem is not that the TLB is invalid, since then it couldn’t have been compiled at all. Rather, the most likely issue here is that one party (e.g. .NET) expresses a type with a construct that is not supported by the other party (Visual Basic 6). Some constructs do not exist at all, while others are just expressed otherwise. In the latter case, often the difference is not just syntactical, and thus the run-time of Visual Basic 6, for example, won’t be able to handle it.

Thus, we need to look at the TLB to find the problem. Simply look up the type that you’re trying to use, and see how it was defined.

Tool support

Luckily, we don’t need to read TLBs in binary. There’s a tool included with the Windows SDK, called oleview.exe. Just pass the TLB as a command-line parameter and it will show the IDL. It will also allow passing EXE or DLL files, as long as they contain a TLB.

Finding out what’s wrong

In our case, the relevant method looked like:

[id(0x60020020)]
HRESULT SetData([in] SAFEARRAY(_Data*) param);

More often than not, we don’t have an idea about why Visual Basic 6 doesn’t like a given construct. An easy way to tell this is to create a simple ActiveX project in Visual Basic 6 and define a method or type that resembles the construct on the .NET side the most. Then, compile the project, pass the DLL/EXE to oleview.exe and check out how Visual Basic 6 expressed what you wrote in IDL.

We did just the same, and what we saw was:

[id(0x60030000)]
HRESULT SetData([in, out] SAFEARRAY(_Data*)* param);

The difference was an additional level of indirection, highlighted above.

System.Runtime.InteropServices.MarshalAsAttribute comes at the rescue

Since you have close to zero ways to customize things on the Visual Basic 6 size, you need to look at the managed code. Since now you know how the managed type or method should appear to COM consumers, you can change the .NET definition to accommodate this. Fortunately, this attribute allows to change the IDL (and the behavior of the .NET marshaler) without changing the .NET semantics. It goes far beyond the scope of this post to describe what this attribute can do, you can have a look at MSDN.

Sometimes, however, you do need to change .NET semantics, like in our case. Since we need an additional level of indirection, we need to pass the array by reference – thus use the ‘ref’ keyword:

public void SetData(ref Data[] param)

{

...

}

Bonus tip

During this recent support case with my customer, we proceeded up to this point, but still Visual Basic 6 threw run-time errors when using the modified method. After a lot of trial and errors, we discovered that invoking the method as a procedure (i.e. without parentheses) was always successful, while the function-like invoking style (i.e. with parentheses) always failed.

Seasoned Visual Basic 6 developers may be able to explain this, I could not. If anyone has an idea, feel free to add it to comments.