1. All member operator<< defined in std::ostream
2. All global operator<<
3. All operator<< defined in std namespace which are found via ADL (argument dependent lookup)

1. If you know the exact type of the variable you want to serialize, you can call the member operator<< directly. This reduces the time complexity to O(n).
2. If you can group some classes into separate namespaces and define the operator<< in these namespaces, it will reduce the candiate set in 2. This reduces the time complexity to O(n^2/m) if m is the number of namespaces and each namespace contains n/m classes.
3. A more general solution is to use hidden friend. They can only be found via ADL, so they will not participate in the overload resolution of the operator<< for other classes. This also reduces the time complexity to O(n).

These are only a small portion of bugs we have fixed, and there are still large amount of active bugs on which we are working.

If you find any bugs in the compiler, feel free to report them through Microsoft Connect.

This post is a follow up to the previous two posts complaining the System.Uri class. There is an idiom in Chinese which can be translated to ‘Three and Out’. But I hit an issue related to System.Uri the third time :-(.

The corresponding connect bug is here. The repro:

        void Test()
        {
            Uri uri1 = new Uri("https://en.wikipedia.org/wiki/%2751_Dons");
            Uri uri2 = new Uri("https://en.wikipedia.org/wiki/%2738_%E2%80%93_Vienna_Before_the_Fall");

            System.Net.WebClient client = new System.Net.WebClient();

            // https://en.wikipedia.org/wiki/%2751_Dons
            Console.WriteLine("{0}", uri1.AbsoluteUri);
            // https://en.wikipedia.org/wiki/'38_%E2%80%93_Vienna_Before_the_Fall
            Console.WriteLine("{0}", uri2.AbsoluteUri);
            // As you can see, the second Uri unescapes "%27" to "'" while the first doesn't.

            // The second download will throw System.Net.WebException.
            // This is because wiki redirects '38_%E2%80%93_Vienna_Before_the_Fall back to %2738_%E2%80%93_Vienna_Before_the_Fall,
            // which becomes an infinite loop.
            client.DownloadString(uri1);
            client.DownloadString(uri2);
        }

The un-escaping happens inside ‘System.Uri.EnsureParseRemaining’ and its behavior is determined by the following helper ‘System.UriHelper.IsReservedUnreservedOrHash’:

        private const string RFC2396ReservedMarks = @";/?:@&=+$,";
        private const string RFC3986ReservedMarks = @":/?#[]@!$&'()*+,;=";
        private const string RFC2396UnreservedMarks = @"-_.!~*'()";
        private const string RFC3986UnreservedMarks = @"-._~";

        private static unsafe bool IsReservedUnreservedOrHash(char c)
        {
            if (IsUnreserved(c))
            {
                return true;
            }
            if (UriParser.ShouldUseLegacyV2Quirks)
            {
                return ((RFC2396ReservedMarks.IndexOf(c) >= 0) || c == '#');
            }
            return (RFC3986ReservedMarks.IndexOf(c) >= 0);
        }

So it appears to be a breaking change in .Net 4.5 mentioned here. Net 4.5 follows RFC3986 while older versions follow RFC2396 and they have different sets of reserved characters (%27 un-escapes to ‘, which is only reserved in RFC3986).

        private enum UriQuirksVersion {
            // V1 = 1, // RFC 1738 - Not supported
            V2 = 2, // RFC 2396
            V3 = 3, // RFC 3986, 3987
        }

        // Store in a static field to allow for test manipulation and emergency workarounds via reflection.
        // Note this is not placed in the Uri class in order to avoid circular static dependencies.
        private static readonly UriQuirksVersion s_QuirksVersion = 
            (BinaryCompatibility.TargetsAtLeast_Desktop_V4_5
                 // || BinaryCompatibility.TargetsAtLeast_Silverlight_V6
                 // || BinaryCompatibility.TargetsAtLeast_Phone_V8_0
                 ) ? UriQuirksVersion.V3 : UriQuirksVersion.V2;

        internal static bool ShouldUseLegacyV2Quirks {
            get {
                return s_QuirksVersion <= UriQuirksVersion.V2;
            }
        }

However, I believe this is actually a bug because the un-escaping isn’t applied consistently (as shown by the repro above). The un-escaping only happens under the following conditions (if the uri contains non-ASCII unicode and Iri parsing is enabled):

                    if (m_iriParsing && hasUnicode){
                        // In this scenario we need to parse the whole string
                        EnsureParseRemaining();
                    }

The hack below depends on this inconsistency to work, though :-).

Anyway, unfortunately, my code depends on some 4.5 only features and I can’t simply switch it back to the older .Net version. In fact, the component which triggers the issue only uses .Net 2.0.

So I have to hack again (after hours of debugging). BTW, when I was debugging the .Net source, the debugger incorrectly uses my source file instead of downloading the correct one from the source server (both file have the name ‘Uri.cs’). I enable ‘Require source files to exactly match the original version’, so I would expect the debugger to at least tell me that the wrong source file doesn’t match the hash in pdb.

Unlike the previous two hacks, this one impacts all Uri objects because it has to change a static class member. Also, due to the nature of the issue (the Uri object is created by System.Net.WebClient internally, not me), it looks like the only viable option to me.

Here is the code snippet of the hack (setting this property to false will revert to pre-4.5 behavior):

        public static bool AllowIriParsing
        {
            get
            {
                FieldInfo fieldInfoIriParsing = typeof(Uri).GetField("s_IriParsing", BindingFlags.Static | BindingFlags.NonPublic);
                if (fieldInfoIriParsing == null)
                {
                    throw new MissingFieldException("'s_IriParsing' field not found");
                }
                return (bool)fieldInfoIriParsing.GetValue(null);
            }

            set
            {
                FieldInfo fieldInfoIriParsing = typeof(Uri).GetField("s_IriParsing", BindingFlags.Static | BindingFlags.NonPublic);
                if (fieldInfoIriParsing == null)
                {
                    throw new MissingFieldException("'s_IriParsing' field not found");
                }
                fieldInfoIriParsing.SetValue(null, value);
            }
        }

How does the hack work? The trick is that if s_IriParsing is false, m_iriParsing will become null and disable the un-escaping logic. Note that IriParsing is disabled by default prior to .Net 4.5 (see this).

        // Value from config Uri section
        // On by default in .NET 4.5+ and cannot be disabled by config.
        private static volatile bool s_IriParsing = 
            (UriParser.ShouldUseLegacyV2Quirks ? IriParsingElement.EnabledDefaultValue : true);

Ideally, I prefer to alter ‘UriParser.ShouldUseLegacyV2Quirks’ instead. But it is a property and checks against a read-only member we can’t modify even with reflection.

At the end, it says, 'It's fortunate that this is a linker error. Otherwise, you'll waste lots of time in debugging to find out the subtle incompatibility'. Today, in v2, I'd like to share another example which I encountered in real-world code that demonstrates the subtle incompatibility.

Here are the two source files, which are used to generate a dll and an exe:

dll.cpp
__declspec(dllexport) void useenv();

#include <stdio.h>
#include <stdlib.h>

void useenv()
{
    wchar_t *value;
    errno_t err = _wdupenv_s(&value, nullptr, L"ENV");
    if (err == 0)
    {
        wprintf(L"ENV = %s\n", value);
        free(value);
    }
}

exe.cpp
#include <stdlib.h>

__declspec(dllimport) void useenv();

void setenv(const wchar_t *value)
{
    _wputenv_s(L"ENV", value);
}

int main()
{
    setenv(L"1");
    useenv();
    setenv(L"2");
    useenv();
}

Build the program using the following command line:

cl dll.cpp /MDd /LD /nologo
cl exe.cpp /MD /nologo /link dll.lib

What output would you expect? If everything goes as expected, it should output '1' and '2' on two lines.

However, if you try the example above, the actual output are two '1's. Unlike linker errors, this kind of runtime issues is much harder to find and the fix normally involves changes to the build system / project files which are normally non-trivial.

Now, let's take a closer look at the example.

The magic is in the compiler options. The dll is compiled with /MDd (debug CRT) and the exe is compiled with /MD (release CRT). So 'setenv' calls '_wputenv_s' from release CRT, and 'useenv' calls '_wdupenv_s' from debug CRT,

'Why does this matter?', you may ask. This is because CRT maintains many global states internally. Debug CRT and release CRT both have their own copies, changes to one don't affect the other.

In this case, CRT maintains a cache to the wide string version of the environment variables and the cache is created when you access the environment variable for the first time. The complete process can be illustrated as follows:

• The first 'setenv' calls OS API to set the environment variables of the process and they are also cached in the release CRT.
• The first 'useenv' tries to read the environment variables and they are not cached in the debug CRT yet. So it reads them from OS and creates the cache.
• The second 'setenv' calls OS API and also updates the cached environment variables in the release CRT.
• The second 'useenv' reads the environment variables directly from the cache in the debug CRT which are not changed.

So the bug only appears when you call 'useenv' the second time because the cache is created but not correctly updated (I'm not sure whether the behavior of CRT is by-design, but apparently it assumes one CRT in the binary and no direct call to SetEnvironmentString).

It is much more subtle than always fail. Actually it took me quite a while to have a small example from the original code. The first few attempts always output '(null)' instead and you can easily know that sometime goes wrong from that :-)

This is just one of thousands of possibilities how mixing debug and release CRT (or any components compiled with different options) can mess up things. The best practice is to make sure everything is compiled with the same compiler options, but this is not always possible (for example, some libraries are from third party). Then you'd better be very cautious about mixing them in the same binary. Move the incompatible part to a separate dll with an abstract layer (so that there are no assumptions across binary boundary) may help.

These are only a small portion of bugs we have fixed (less than 20% of mine), and there are still large amount of active bugs on which we are working.

If you find any bugs in the compiler, feel free to report them through Microsoft Connect.

It contains the following C++11 features besides those already in VS2012:

• Variadic templates
• Uniform initialization and initializer_lists
• Delegating constructors
• Raw string literals
• Explicit conversion operators
• Default template arguments for function templates

This time, the victim is trailing dot (which is common in wiki pages). See this connect bug for details.

The workaround is similar to the one described in this post. The only difference is that we need 'GenericUriParserOptions.DontCompressPath' instead.

The original article on MSDN can no longer be found. Here is the pdf version on OpenRCE: https://www.openrce.org/articles/files/jangrayhood.pdf.

Overview:

How are classes laid out?
How are data members accessed?
How are member functions called?
What is an adjuster thunk?
What are the costs:
    Of single, multiple, and virtual inheritance?
    Of virtual functions and virtual function calls?
    Of casts to bases, to virtual bases?
    Of exception handling?

In C++98, pass by reference is in general more efficient because it saves extra copies (if the copy is expensive, otherwise the extra indirection may hurt the perf instead). However, after the introduction of move semantics in C++11, the story is now a little different (Note that premature optimization doesn’t buy you anything, so the following is to give you a full picture of the new thought introduced in C++11, see the guidelines at the end and use it when appropriate)

Let’s see the following example:

#include <iostream>
#include <vector>

using namespace std;

struct String {
    String(const wchar_t *) {
        cout << "ctor" << endl;
    }
    String(const String &) {
        cout << "copy ctor" << endl;
    }
    String(String &&) {
        cout << "move ctor" << endl;
    }
};

void f1(String s) {
    vector<String> v;
    v.push_back(std::move(s));
}
void f2(const String &s) {
    vector<String> v;
    v.push_back(s);
}

void g() {
    String s(L"");
    cout << "f1(s)" << endl;
    f1(s);
    cout << "f1(L"")" << endl;
    f1(L"");
    cout << "f2(s)" << endl;
    f2(s);
    cout << "f2(L"")" << endl;
    f2(L"");
}

If you run the code, you will find that 'f1' has one less copy ctor call for rvalue argument and the same  amount of copy ctor call for lvalue argument than 'f2', which means 'f1' is more efficient (assume that move ctor is much cheaper than copy ctor).

The trick is as follows: for lvalue argument, 'f1' has one extra copy to pass the argument because it is by-value, while 'f2' has one extra copy to call push_back. So no difference; for rvalue argument, the compiler has to create a temporary 'String(L“”)' and pass the temporary to 'f1' or 'f2' anyway. Because 'f1' can take advantage of move ctor when the argument is a temporary (which is an rvalue), the costs to pass the argument are the same now for 'f1' and 'f2'.

This means in C++11 we can get better performance by using pass-by-value approach when:

1. The parameter type supports move semantics.
2. The cost of move constructor is much cheaper than the copy constructor (both the time and stack usage).
3. Inside the function, the parameter type will be passed to another function or operation which supports both copy and move.
4. It is common to pass a temporary as the argument.

Update1

I learn today (April 2nd, 2013) that 'return m;' is in general more efficient than 'return std::move(m);'. Both will use move constructor of Matrix, but the latter inhibits NRVO (which means the compiler can't eliminate the call to the move constructor).

They are the same in the above example because 'operator-' returns function parameter. If you return a local variable instead of a function parameter, you can save one call to the move constructor.

The criteria for copy / move constructor elision is in C++11 standard draft n3485.pdf, 12.8 / 31:

    in a return statement in a function with a class return type, when the expression is the name of a non-volatile automatic object (other than a function or catch-clause parameter) with the same cv-unqualified type as the function return type, the copy/move operation can be omitted by constructing the automatic object directly into the function’s return value

The complex two-stage overload resolution to choose move constructor when you return an lvalue object is in 12.8 / 32:

    When the criteria for elision of a copy operation are met or would be met save for the fact that the source object is a function parameter, and the object to be copied is designated by an lvalue, overload resolution to select the constructor for the copy is first performed as if the object were designated by an rvalue. If overload resolution fails, or if the type of the first parameter of the selected constructor is not an rvalue reference to the object’s type (possibly cv-qualified), overload resolution is performed again, considering the object as an lvalue. [ Note: This two-stage overload resolution must be performed regardless of whether copy elision will occur. It determines the constructor to be called if elision is not performed, and the selected constructor must be accessible even if the call is elided. —end note ]

Update2

Here is an example to explain the rules.

Matrix Add(const Matrix &m1, const Matrix &m2)
{
    Matrix m;
    // Calculate m
    return m;
}
Matrix NegateByRef(const Matrix &m)
{
    // Calculate m 
    return m;
}
Matrix NegateByValue(Matrix m)
{
    // Calculate m 
    return m;
}
void NoOpByRef(const Matrix &m)
{
}
void NoOpByValue(Matrix m)
{
}

The following two tables show how many copy ctor (number before the comma) and move ctor (number after the comma) are called in each scenario.
For each scenario, the row is the function we call, the column is the argument to the function. For example, row 'NegateByRef' and column 'm' means 'NegateByRef(m)'.
m, m1 and m2 are known matrices.

1. Rule 1 is the basis of our discussion and we assume that it is always true.
2. From the two tables, you can find that the total numbers of move + copy ctor are the same in each scenario. So if rule 2 is true, it is always better to add a move ctor to the type.
3. You can find that 'NoOpByValue' is always worse than 'NoOpByRef'. This is related to rule3. Note that, rule 3 may be true even if the parameter is read only. That is, whether you modify 'm' in 'Negate' doesn't have impact on the number of ctor called. It is only related to whether you pass the parameter further. In this case, 'return m;' is the operation which supports both copy and move.
4. 'Add' returns rvalue (temporary is an rvalue) and 'm' is an lvalue. You can find that 'NegateByValue' is only better than 'NegateByRef' when rule 4 (the argument is an rvalue) is true.

LinkedIn API allows you to get the details of user’s connection using url like the following:

https://api.linkedin.com/v1/people/url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Flbeebe/connections

However, if you create a url like the above in C# and send the request to LinkedIn, the server will reject the request and complain about the wrong signature.

I hit this kind of issue several times before (due to slightly different escaping rules between the requirement of oAuth 1.0 3.6 / 2 (RFC3986) and Uri.EscapeUriString in .Net 2.0 (RFC2396)). But this time, the root cause is different.

After spending lots of time debugging by myself and searching the web later, I reached the following post: https://connect.microsoft.com/VisualStudio/feedback/details/94109/

So it is clear that .Net framework explicitly alters the input url for ‘security’ reason. But it doesn’t expose the interface to allow the user to change this behavior (even if it is simply a parser flag called ‘GenericUriParserOptions.DontUnescapePathDotsAndSlashes’ which can be set internally).

Well, I’m on my own to hack it. The workaround posted in the bug changes the parser flag for all Uri objects with the same url scheme, so I have to write a more isolated version:

However, I feel guilty for hacking.