I won’t be posting anything new about C# in the near future and the don’t know what shareable halo info i will have. Thanks to all the people who read my blog and kept it entertaining, please feel free to ask me any questions. I will certainly try to answer ( or redirect to my ex-colleagues in C#).

Moving on to Halo, i hope you have seen this ODST trailer if not check it out, ODST has a band of brothers kind of feel and connects at a visceral level.

Here is the sort version of the story, basically the there is no direct translation to IL that would be correct for the operation( an extension method defined on a value type to be current as an instance method and assigned to a delegate).

To try this out one can generated IL like this

IL_0006: nop

IL_0007: ldc.i4.s 10

IL_0009: stloc.0

IL_000a: ldloc.0

IL_000b: box [mscorlib]System.Int32

IL_0010: ldftn int32 Test.Extension::func(int32)

IL_0016: newobj instance void Test.d1::.ctor(object,native int)

here the value of the int passed to the extension method is something like 205XXXXX

The problem is that value type methods (with the exception of the virtuals inherited from System.Object) are typically called with a reference to the value type, not a boxed version of the value type (this allows meaningful mutation of the value type for instance). (When a non-virtual method on a boxed value type is called directly the JIT and runtime actually conspire to route the call through a thunk called an “unboxing stub” which essentially adjusts the this pointer downwards to make it look like a value type reference so we can call the real value type method implementation).

Delegates essentially force you to box any bound parameter you supply (since the .ctor signature takes an Object and internally we store and report this value as an object reference). So straightaway you have a potential semantic problem: by forcing the value type reference into a box you lose any identity (i.e. if a mutating method is called on the boxed value it will mutate the boxed copy, not the original).

But then there’s an additional problem. On the invocation side there are three potential ways a boxed value type argument could be processed:

1. The boxed value is passed in directly. That’s what you want if it’s intended to be the first argument of a static method which is typed as Object or ValueType, or if you’re calling one of the System.Object virtuals on a value type. For instance “static void someType.foo(Object)” or “override String valueType.ToString()”.

2. The boxed value is unboxed and passed in. That’s what you want if calling a static method that with a first argument typed specifically as your value type. For instance “static void someType.foo(int32)”.

3. A reference to the value type stored inside the boxed value is passed in. This is what you want if you’re calling a non-virtual instance method on the value type. E.g. “instance void valueType.Foo()”.

I believe CLR handles cases 1 and 3, but not case 2 (which is the case above). There’s a technical problem with the implementation here, This has to do with the way CLR implements delegates to get the fastest invocation speed possible.

One way I can think to overcome this problem is to create a stub our selfs. That is we could leverage the lambda implementation to create a lambda that calls the static method and then use the lambda as the delegate value.

For anyone Searching the web for this, Hope this helps :)

What’s this post about ?

To begin with i will be talking about debugging dynamic objects.

With C# 4.0 we can instantiate and perform operations on objects  from dynamic languages like iron python, Iron ruby by using the dynamic keyword. In this post i will show you the tools added to  inspection these objects and how one can reuse this for debugging COM objects.

Introduction to Dynamic

Basically the dynamic keyword can be used to define the type of a local, field, return type or parameter. By doing so the user allows the compiler to defers the binding of calls and member accesses to run-time. In order to do this the compiler does some code generation that may or may be interesting to you ( i will briefly touch on this later), but what is sure to pop to ones mind is “ What’s this good for ? ”.

The primary use of this to bridge the gap between the .Net type system and its set of strongly typed languages and other alternate type systems that are in use by the developer to get his/her job one ( think COM, Scripting runtimes , DOM etc). The compiler does this by packaging all the information about the call in a payload and generates a call to the DLR  (dynamic language runtime). This call is executed at runtime where the DLR takes charge of routing the call to the correct provider (IronPython, COM, IronRuby) of the object. This provider  then tries to bind the call and execute it.

What’s the Problem with Debugging this ?

Consider the following code

using System;
using Microsoft.Scripting;//requires reference to Microsoft.Scripting.dll and IronPython.dll
using IronPython.Compiler;
using IronPython.Hosting;

public class C
{
    static void Main(string[] args)
    {
        var sr = Python.CreateEngine();
        // similar overload exists for an file instead of a inline string
        var code = sr.CreateScriptSourceFromString(@"   
class x(object):
 class_val = 5
 def some_method(self): pass
 def meth1(self, foo): 
   return foo
a = x()
a.val = 3
a.Name = 'Sree'", SourceCodeKind.Statements);
        var scope = sr.CreateScope();
        code.Execute(scope);
        dynamic y = scope.GetVariable("a");
        System.Diagnostics.Debugger.Break();

    }
}

Try inspecting y,

since we are talking about dynamic objects by nature they don’t have a strong type that representaion the structure of the object. In this case i am using Iron python, which happens to represent the members in a hash table.

In the case of COM the structure of the object is actually in an alternate type system.

dynamic XlApplication = new Microsoft.Office.Interop.Excel.Application();
dynamic thisWorkbook = XlApplication.Workbooks.Add();
dynamic xlSheet = thisWorkbook.Worksheets.Add(After: thisWorkbook.ActiveSheet);
System.Diagnostics.Debugger.Break();

Try inspecting xlSheet,

In either case the basic debugging tools in VS fail as they have no way of discovering this and only end showing the most basic information, which in almost every case is useless.

In Conclusion

1. Dynamic languages such as IronRuby or IronPython do not have strong types that represent the members they contain
2. COM support is limited to Interop assembly generated and quickly one can fall of the cliff with no type information.
3. The IntelliSense® tool does not show any of the dynamic members ( since it’s driven by the static type system).

debugging can be hampered by an inability to query the state of the object while stepping though code.

So what does the solution look like ?

Dynamic View

In addition to the static view, the C# expression evaluator will add a special node for 

1. Objects implementing IDynamicObject
2. COM objects with out a PIA or strong type (Primary interop Assembly)

named “Dynamic View”. This is similar to the “Results View” node that was added for LINQ debugging, where the user could see the results of the lazy evaluated queries of linq. 

The “Dynamic View” node has the following behavior.

1. Expanding the Dynamic View node will query the IDynamicObject object for the members available for display and get the value for each.

1. The children of the Dynamic View are immutable.
2. Choosing “Add to Watch” for any child of the Dynamic View inserts a new watch that casts the object to dynamic (i.e. ((dynamic)y).Name).
3. The Dynamic View and its children are assumed to be side-effecting, and therefore stepping in the debugger will not automatically re-evaluate the Dynamic View or its children; they are made stale.
4. The Dynamic View and/or its children will need to be manually refreshed by the user.

A “dynamic” format specifier has been added to the expression evaluator. This allows the Dynamic View to be added to the watch by suffixing expressions with “, dynamic”

Pdb support

In response to this implementation, a PDB payload has been added for local variables typed in some way as dynamic. This payload is necessary since there is no metadata corresponding to locals and since there no actual type called dynamic anywhere in the CLR. The compiler must find some way to decorate the locals to infer that its a dynamic local at runtime. The payload includes the name of the local, its slot ID and a boolean array (length<=128) that represents a pre-order traversal of the type. For example, consider:

Dictionary<int, dynamic> dict = new Dictionary<int, dynamic>();

This code will produce a boolean array of false,false,true to indicate the portion of the type that is dynamic.

So remember if you want to debug dynamic locals be sure you generate the debug information(pdb) and have it where you can access it

With regard to declaration (as opposed to inspection), in some embodiments, declaration of dynamic variables within an Immediate window is supported, including complex generic types (e.g. Dictionary<int, List<dynamic>>). But more on this later …

More later … cheers

On Further review there are a few problems with anonymous-types, they all boil down to the fact the names given to these types are not valid C# type names ( so that users don't explicitly use them in code). But while debugging this is exactly the kind of thing that one wants to do, Consider the following cases

Case1:

The anonymous type appears in a cast, like when an anonymous type is returned from a function or when its cast from System.Object to the actual type.

var obj = Func(new{ I = 10, J = "Sree"});
System.Diagnostics.Debugger.Break(); 

private static object Func<T>(T obj)
{
       object o = obj;
       System.Diagnostics.Debugger.Break();
       return obj;
}

At the first breakpoint add to watch any member on o results in

At the second break point adding to watch any member on obj results in

Case 2:

The user wished to create a instance of the anonymous type during a debugging session or when the Anonymous Type is a type argument on a generic type and the user want to evaluate a static member, use it in a cast etc. 

SP1 Changes

The anonymous type names are no longer invalid in the Expression evaluator. Therefor

1. C# Constructs involving Anonymous-types can actually be evaluated at runtime and
2. Instances of Anonymous-types can actually be created during a debugging session.

Which means using the same example as above and trying it with SP1 produces

See how the anonymous type name is no longer hidden and is used in the cast to bind to the members.

As an added bonus while debugging you can now create instances of anonymous types on the fly and use them. Assign them to objects and check for pathological conditions etc

Hope you enjoy discovering and debugging anonymous types now, that they are much more useful in the debugger.

As always do let me know if there are others things you would like to see me improve or implement.

Over the past few months I have been busy closing VS 2008 and working on some fixes for SP1. We have enabled some key debugging scenarios in C# in VS 2008 SP1, they include support for 

1. Range Variables in Queries &
2. Anonymous Types
3. Generic Type arguments

Covering all of them was making the post long so i am going to discuss Range variables this time and continue on with the other 2 in the next post.

Range variables

Range variables are the variables defined and used in a query. Currently they have a limitation in that they are not recognized as "true" local variables when stopped/stepping though a query. And since queries are one of the newest constructs in C#, are a prime candidate for SP1.

Range variables cannot currently be inspected in the watch window or in data tips(hover over them), quick watch and immediate. They can only be seen in the locals window,  but the display shows the internal transparent identifier created for the query. This leaves the variable of interest to the user nested and difficult to understand and use.  This happens as long as the user adds

1. A second from,

int[] ints = new int[] { 1, 2, 3, 45, 56 };

var q = from i in ints
            from k in ints
            where i > k      //add BP here
            select i + k;

2. A join or

3. A let clause to their query,

Seeing these simple examples it is not hard to imagine the debugging state when stopped in queries that use many of these in combination.

Here "Hover over" does not work, can't add anything to watch, the transparent identifier gets larger and the lvl of nesting makes it very hard to find the range variables your looking for.

SP1 Changes

All of these issues are now fixed in SP1 and the user has the full set of debugging tools for him to help inspect and evaluate the current state when stopped at a break-point. using the same example

Conclusion

In concluding this article, As always I am very interested in

1. Any others pain points you have discovered while debugging or
2. Things are are hard to debug.
3. Things that would be nice to inspect etc.

Looking forward to your feedback. Cheers

In the last article I covered the "results view" for lazy evaluated collections like Queries/Enumerable and the use of extension methods in the watch and immediate window. For completeness I will cover stepping, range variables, anonymous types and Xlinq support and this will lead us into the SP1 work for debugging.

Stepping

Stepping has been extended to support the query execution. That is, when the query is being executed one can step though the query created or add break points for the same. This allows the user to actually see the flow of the execution though a query and find out how a particular element was retrieved from the enumerable.

The results view is marked as having side effects ( just like a method call) and is disabled during a step. This mean we don't have unwanted side effects or seeing the contents of a enumerable

Range Variables

The variables that are declared and used in queries can also be inspected when stopped in a query execution ( as shown in stepping). Here the variables in question might be nested under an transparent identifier and if so the top level transparent identifier is show in the locals window and the user can dig in though this object to find the variable he is looking for.

Anonymous Types

Since anonymous type are identified by their structure all instances of anonymous-types give a brief description of their structure (first 5 elements) in the value column. The name of the type is hidden as its an unparseable name in C#. All elements of the anonymous type are un editable( after the immutable dcr)

Xlinq Objects

Objects of type System.Xml.Linq.XNode or any derived type get the built in visualizers ( text, XML and html) and user can see the Xml data, with a few clicks.

This pretty much covers the work done to support debugging in VS 9, however as I will describe soon this leaves a pretty big hole when debugging queries where the range variables are under an transparent identifiers ( e.g. Color, agg, y etc from range variable eg). Secondly since anonymous types are unparseable any construct that contains an anonymous type ( think type argument , casts , ctors) are broken. These are some of the issues i will cover in my next post of VS 9 SP1.

[AttributeUsage(AttributeTargets.Method | AttributeTargets.Class | AttributeTargets.Assembly)]

public sealed class ExtensionAttribute : Attribute { } 

This should be a special note to developers writing their own compilers or doing IL gen. This decoration allows the importer to precisely identify the classes to import when looking for extension methods. Allowing it to omit unnecessary types and their dependencies when looking for extension methods, which can quickly addup.

So if your doing anything custom be sure to inspect the IL to see if other languages can discover and call your extension methods seamlessly

Consider the code bellow

1 binds to Foo(object) and 2 binds to Foo(short). This means that the same method when used as an extension method binds to a different overload than if used as an static method.

That raise the question Why is that ?

This is so because the conversions rules for instance parameter in an Extension methods are

1. Identity conversions
2. Implicit reference conversions
3. Boxing conversions

So we are not considering numeric conversions and this can cause some confusion. 0 is of type int and will not be implicitly converted to long while binding the method as an Extension Method.

Extension methods have different semantics than standard static methods, since they are called by the client as an instance method. So the conversion rules are modified so that they behave as such.

Lets say we define an instance method on int does it mean I can call it on a object of type long ? It is this kind of enforcement that the conversion rules enforce.

uncommenting the perfect match Foo(this int x) and they behave the same ...

C# 3.0 added a few constructs like queries which are delay executed. This means that they are not actually executed until the results of the query are required. Debugging some of them can seem strange since one can't step in to the Query where its created but only where its enumerated, like in a foreach loop. In this article i will show some of the problems that the user can face based on the fact that the Query is delay executed and the debugger tries to be as non-intrusive as possible.

So what does it mean to step over the creation of the query, does it mean the statements that are part of the Query are executed... as you would have guessed No. At this point we have just created delegates that point to code in the Query(the body of the select, where etc), these delegate form the arguments to the extension methods (select , where etc )that form the query.

Consider the following example:

Define a Class Library and add a single extension method to it

using System;
using System.Linq;

public static class Extensions
{
    public static string EM(this UInt32 i)
    {
        return "hello";
    }
}

Now create a console application and add the class lib as a reference. In the console application add the following code ..

using System;
using System.Linq;

namespace lateboundExtensionmethods
{
    class Program
    {
        static void Main(string[] args)
        {
            var q = from i in new uint[] { 1 }
                 select i.EM();                //module containing EM is not loaded

            System.Diagnostics.Debugger.Break();  
            foreach (var v in q)           //module containing EM is loaded
                Console.WriteLine(v);
        }
    }
}

Now run to the breakpoint and try and evaluate i.EM() in the watch window.... we get an error like

What was all that about .... have i not just compiled and ran this code ... What's the debugger smoking ?

The Problem:

Though the problem is not specific to Extension methods. The Query rewrite rules move code in the select, where etc in to a delegate and in order to do that a new static method is generated. Until this Method is Jitted the assemblies referenced in it are not loaded. Since the assembly is not yet loaded in the debugging session the Expression evaluator can't find the methods, if they are added to watch. This leave the user feeling like he has stepped over a code that can't be evaluated in the debugger. once we come to the foreach loop the code in the select is actually need to be stepped into and the Jitter jits this method, this results in all the assemblies referenced in to be loaded.

On the other hand trying something like this, and the extension method in watch will work.

using System;
using System.Linq;

namespace lateboundExtensionmethods
{
    class Program
    {
        static void Main(string[] args)
        {
            uint j = 10;
            System.Diagnostics.Debugger.Break(); //module containing EM is loaded
            j.EM();

        }
    }
}

Workarounds:

So while debugging you code if you are unable to execute an static or extension method defined in a satellite assembly check the modules window to see if the assembly is loaded, if not for the debugging session you might want to add something like typeof(className) to you'r code. Where className is the class defined in the satellite assembly.

C# 3.0 introduces many new constructs and opens entirely new ways of thinking and developing code. In this article I will talk about the new debugging features that make it easy to see the running code and better understand it. In my experience one spends as much time writing the initial code as debugging it later. In fact stepping though the code and inspecting variables and expressions is one of the best ways to understand it. So its not just the original writer but many more who will eventually debug it. To this end there are many new things that you will notice when you start debugging in a C# 3.0 project, I will enumerate few of them in this part of the article.

Results View

The primary aim of the results view is to provide a array like view for Enumerable objects. This allows the user to see the items that are the results of the Query. The Query/Enumeration might be on a local collection(Linq/Xlinq/iterators) or from a remote store(Dlinq/customProviders). The idea here is not to change/hide the object's default structure while adding a simple way to inspect the result. This is quite different from debugger proxies which hide the actual view of the object under a "raw" node and show the users the custom view. The rational being that there are many useful properties/fields on the users query and we wanted to add this functionality without loosing all that information. This brings up 2 interesting questions.

1. When is this applied?

The Results View is applied to an object/struct if

1. It does not have a Debugger type proxy 
2. Does not implement IList or ICollection (they already have items view)  
3. It does implement IEnumerable or IEnumerable<T>.

2. What does the View look like?

1. For a object whose compile time type is an interface of type IEnumerable, IEnumerable<T>, IQueryable or IQueryable<T>, we hide the derived node and show the runtime type’s  members directly under the object  .
2. A new node called “Results View” is added to the Expression which when expanded will enumerate the Enumerable.    
3. The Results View's value column warns the user that expanding will change the state of the object being inspected.
4. A new format specifier is added to represent the view and can be accessed by typing object/expression, results.

E.g. for query like

int[] array  = new int[]{ 23, 3,54, 8, 10, 39, 87, 3, 7};
var q = from i in array.AsQueryable()
        let y = i * i
        let z = y * y
        select new { y, i, z };

System.Diagnostics.Debugger.Break();

here we can see the Results View Node is added as a child of the Query q. Though q has a static type of IQueryable<SomeAnontype>, we directly show the members of the actual runtime type. The value column has a important message for the user, stating that expanding this view will enumerate the object.If the base of the expression in watch is also an enumerable the results view is added to it. If a field/property which implements IEnumerable is hidden then the results view inherits the hidden property from its parent.

Finally we can also access the results view by using the results specifier “, results”.

There are some exceptions to the rule, Sytem.String is excluded from having this view though it implements IEnumerable.

Extension Methods

Extension methods can now be used when calling an instance method in the watch and the immediate window. This is very useful if you want to  see

1. Which extension method will get called if the statement was placed here in the code.
2. The results of a Extension method not present in the code being executed.
3. Limit the number of elements returned by a query, before expanding its results.
4. Most importantly since Queries boil down to extension methods, we can use the watch/immediate like scratch pad to try out some Ad-hoc Query like behavior.  

Consider the following code

    public class Program
    {
        static void Main(string[] args)
        {
            int[] array = new int[] { 1, 23, 45, 67, 12 };      // simple in memory array

            Func<Func<int, bool>, IEnumerable<int>> where = array.Where<int>;  //curry an Extension method

            System.Func<int, int> f = Program.identity;
            System.Diagnostics.Debugger.Break();
        }

        public static bool Eval(int i){ return i > 5; }

        public static bool Eval1(int val) { return val < 50; }

        public static int identity(int x) { return x; }
    }

Trim the Enumerable before inspecting it ( very useful if this call will be remoted to the provider of the extension method)

Calling an Extension methods with a method (predicate) as argument

Using Curried delegate from the code

Chaining extension methods to archive Ad-hoc query like execution

This assumes that the assembly contain the extension method is already loaded by an earlier runtime call.

All in all using these 2 features in combination allows users to do some powerful analysis on a  program while it executes. Though not as powerfully as a full fledged Query Analyzer with lambdas, it does allows the user to do very interesting stuff. I am interested in knowing potential pain points that you come across, things that are in your way when debugging etc.

In the coming article i will cover Stepping, Anonymous-types, Range-variables and Xlinq Support.