Module Module1

    Sub Main()

        Dim arr() As Integer = {1, 2, 3, 4}

        Dim q = From n In arr Select num = n, square = n ^ 2

        For Each record In q

            Console.WriteLine(record)

        Next

    End Sub

End Module

The output:

{ num = 1, square = 1 }

{ num = 2, square = 4 }

{ num = 3, square = 9 }

{ num = 4, square = 16 }

The values that are combined into an anonymous type do not need to be defined in the same operator. As long as you have more than one iteration variable at the end of the query, they will be parts of composite element of resulting sequence. Explicit Select-ing is not required. Example:

Module Module1

    Sub Main()

        Dim arr1() As Integer = {1, 2, 3, 4}

        Dim arr2() As String = {"a", "b", "c"}

        ' this query defines control variable n

        ' then (after filtering) it adds variable k

        ' and does more filtering

        ' at the end we have both n and k variables visible

        Dim q = From n In arr1 _

                Where n > 1 _

                From k In arr2 _

                Where (k <> "b")

        For Each record In q

            Console.WriteLine(record)

        Next

    End Sub

End Module

Output:

{ n = 2, k = a }

{ n = 2, k = c }

{ n = 3, k = a }

{ n = 3, k = c }

{ n = 4, k = a }

{ n = 4, k = c }

It is surprisingly trivial to do with queries. It takes one line of code -

Module Module1

    Sub Main()

        'create a list of lists

        Dim list As New List(Of List(Of Integer))

        For i = 0 To 5

            Dim inner_list As New List(Of Integer)

            For j = 0 To 3

                inner_list.Add(j)

            Next

            list.Add(inner_list)

        Next

        '======

        ' get flattened IEnumerable

        '=====

        Dim q = From i In list, j In i Select j

        'print out the sequence

        For Each j In q

            Console.Write(j)

        Next

        Console.WriteLine()

        'can make it into an array if needed

        Dim arr() As Integer = q.ToArray

        'print out the array

        For Each j In arr

            Console.Write(j)

        Next

        Console.WriteLine()

    End Sub

End Module

This post is a part of a series of posts about query design. For the previous post see: https://blogs.msdn.com/vladimirsadov/archive/2007/06/07/query-pattern-composability-query-design-continued.aspx

As you might have seen queries have a lot more operators than just Select. Here we will take a look how we can add support for Where to our queryable type.

It is surprisingly easy to implement Where. Compiler will use any function that is named Where, can be applied to the source collection and takes a delegate function that takes an element of the collection and returns a Boolean result. 

I will not do delayed execution tricks in this sample for simplicity (do not want to implement all that IEnumerable support again), however it is fairly trivial to generalize this sample so that both Select and Where are delay-executed.

Class MyGenericQueryable(Of T)

    Public elements As IEnumerable(Of T)

    Public Function [Select](Of S)(ByVal projection As Func(Of T, S)) As MyGenericQueryable(Of S)

        Dim new_col As New MyGenericQueryable(Of S)

        Dim new_elements As New List(Of S)

        new_col.elements = new_elements

        For Each element In elements

            new_elements.Add(projection(element))

        Next

        Return new_col

    End Function

    Public Function Where(ByVal predicate As Func(Of T, Boolean)) As MyGenericQueryable(Of T)

        Dim new_col As New MyGenericQueryable(Of T)

        Dim new_elements As New List(Of T)

        new_col.elements = new_elements

        For Each element In elements

            If predicate(element) Then

                new_elements.Add(element)

            End If

        Next

        Return new_col

    End Function

End Class

Module Module1

    Sub Main()

        Dim arr As Integer() = New Integer() {1, 2, 3}

        Dim col As New MyGenericQueryable(Of Integer)

        col.elements = arr

        Dim q = From i In col Where i > 1 Select i

        For Each t In q.elements

            Console.WriteLine(t)

        Next

    End Sub

Here is another important example of a queryable type. There are two differences from the previous examples:

It uses System.Func delegate type that is defined in System.Core.dll . There is nothing special in using this type. VB compiler will accept any delegate type that matches the queryable pattern. However using predefined delegate type reduce the amount of code to type.

Another very important difference is that the Select does not create another collection with projected elements. Instead it creates an object that “knows” how to produce projected elements when asked for them. There are many advantages in doing that. First of all we may not need all the elements so it makes sense to delay computation until the elements are needed. Secondly for a big source collection, the resulting collection will take much less space. This is especially important if you compose a query with multiple projections applied in sequence.

Basically instead of creating another collection for every Select in a composed query we will build a pipeline of enumerable types that represent projection delegates. When another element is requested from the query, by delegating the request through the chain of these enumerable types, we will get the next element from the source collection and every projection in the pipeline will be applied to the element.

This concept is commonly known as a delayed execution. It does complicate the debugging somewhat as there seem to be too many levels of indirection. However the benefits far outweigh that.

Here is the sample
(it is not as big as it seems as most parts are just required to implement IEnumerable pattern):

Class EnumerableQueryable(Of T)

    Implements IEnumerable(Of T)

    Private elements As IEnumerable(Of T)

    Sub New(ByVal elements As IEnumerable(Of T))

        Me.elements = elements

    End Sub

    Public Function [Select](Of S)(ByVal selector As Func(Of T, S)) As EnumerableQueryable(Of S)

        Dim new_enum = New Enumerable4Select(Of T, S)(Me, selector)

        Dim new_col As New EnumerableQueryable(Of S)(new_enum)

        Return new_col

    End Function

    Public Function GetEnumerator() As System.Collections.Generic.IEnumerator(Of T) Implements System.Collections.Generic.IEnumerable(Of T).GetEnumerator

        Return elements.GetEnumerator

    End Function

    Public Function GetEnumerator1() As System.Collections.IEnumerator Implements System.Collections.IEnumerable.GetEnumerator

        Return GetEnumerator()

    End Function

End Class

Class Enumerable4Select(Of T, S)

    Implements IEnumerable(Of S)

    Private collection As IEnumerable(Of T)

    Private selector As Func(Of T, S)

    Public Sub New(ByVal coll As IEnumerable(Of T), ByVal func As Func(Of T, S))

        collection = coll

        selector = func

    End Sub

    Public Function GetEnumerator() As System.Collections.Generic.IEnumerator(Of S) Implements System.Collections.Generic.IEnumerable(Of S).GetEnumerator

        Return New Enumerator4Select(collection.GetEnumerator(), selector)

    End Function

    Public Function GetEnumerator1() As System.Collections.IEnumerator Implements System.Collections.IEnumerable.GetEnumerator

        Return GetEnumerator()

    End Function

    Class Enumerator4Select

        Implements IEnumerator(Of S)

        Private collection As IEnumerator(Of T)

        Private selector As Func(Of T, S)

        Public Sub New(ByVal coll As IEnumerator(Of T), ByVal func As Func(Of T, S))

            collection = coll

            selector = func

        End Sub

        Public ReadOnly Property Current() As S Implements System.Collections.Generic.IEnumerator(Of S).Current

            Get

                Return selector(collection.Current)

            End Get

        End Property

        Public ReadOnly Property Current1() As Object Implements System.Collections.IEnumerator.Current

            Get

                Return Current

            End Get

        End Property

        Public Function MoveNext() As Boolean Implements System.Collections.IEnumerator.MoveNext

            Return collection.MoveNext()

        End Function

        Public Sub Reset() Implements System.Collections.IEnumerator.Reset

            collection.Reset()

        End Sub

        Private disposedValue As Boolean = False        ' To detect redundant calls

        ' IDisposable

        Protected Overridable Sub Dispose(ByVal disposing As Boolean)

            If Not Me.disposedValue Then

                If disposing Then

                    ' TODO: free managed resources when explicitly called

                End If

                ' TODO: free shared unmanaged resources

            End If

            Me.disposedValue = True

        End Sub

#Region " IDisposable Support "

        ' This code added by Visual Basic to correctly implement the disposable pattern.

        Public Sub Dispose() Implements IDisposable.Dispose

            ' Do not change this code.  Put cleanup code in Dispose(ByVal disposing As Boolean) above.

            Dispose(True)

            GC.SuppressFinalize(Me)

        End Sub

#End Region

    End Class

End Class

Here is some code to test the above sample:

Module Module1

    Sub Main()

        Dim col As New EnumerableQueryable(Of Integer)(New Integer() {1, 2, 3})

        Dim q = From i In col Select i_sqr = i * i Select i_sqr + 1

        For Each i In q

            Console.WriteLine(i)

        Next

    End Sub

End Module

Here is one more example of a queryable type. While weird and fairly useless it illustrates that queryable type does not have to be a collection. This one represents a single scalar value.

Public Delegate Function SelectorDelegate(Of T, S)(ByVal arg As T) As S

Class ScalarQueryable(Of T)

    Public element As T

    Public Function [Select](Of S)(ByVal projection As SelectorDelegate(Of T, S)) As ScalarQueryable(Of S)

        Dim new_col As New ScalarQueryable(Of S)

        Dim new_element As S

        new_col.element = new_element

        new_col.element = projection(element)

        Return new_col

    End Function

End Class

Module Module1

    Sub Main()

        Dim col As New ScalarQueryable(Of Integer)

        col.element = 1

        'note that the the result of the query is

        'another ScalarQueryable that contains a string

        Dim q = From i In col Select i_sqr = i * i

        Console.WriteLine(q.element)

    End Sub

End Module

Here is another example of a queryable collection. In this case we have a collection that is generic so it can be used with elements of different types. What is more interesting is that MyGenericQueryable can be used with query projections that change type of the element.

Public Delegate Function SelectorDelegate(Of T, S)(ByVal arg As T) As S

Class MyGenericQueryable(Of T)

    Public elements() As T

    Public Function [Select](Of S)(ByVal projection As SelectorDelegate(Of T, S)) As MyGenericQueryable(Of S)

        Dim new_col As New MyGenericQueryable(Of S)

        Dim new_elements(elements.Length - 1) As S

        new_col.elements = new_elements

        For I As Integer = 0 To elements.Length - 1

            new_col.elements(I) = projection(elements(I))

        Next

        Return new_col

    End Function

End Class

Module Module1

    Sub Main()

        Dim arr As Integer() = New Integer() {1, 2, 3}

        Dim col As New MyGenericQueryable(Of Integer)

        col.elements = arr

        'note that the projection produces an element of a different type

        'while col contains integer elements, q contains strings

        Dim q = From i In col Select i_string = i.ToString

        For Each t In q.elements

            Console.WriteLine(t)

        Next

    End Sub

End Module

Here I want to summarize what exactly the minimum requirements are for a type to be considered queryable with some explanations why we have such rules.

Quaryable type requirements:

1.       There must be a method called Select that can be called with the collection instance. Note that this can be either an instance method or an extension method. As long as you can make a call like col_instance.Select(..) it is acceptable as a select operator.

Why: by having a proper Select method a type indicates that it can be used in query expressions. It is also the method that represents “Select” in the query syntax.

Dim result = From el in col_inst Select el.ToString
translates into:
Dim result = col_inst.Select(...)

2.       Select must take exactly one argument. The argument must be of a function delegate type that takes one argument.

Why: The argument must be a delegate because compiler will generate a function that takes one argument and will pass this function to the Select method in form of a delegate.

3.       While the selector delegate can be (and often is) a generic type, the delegate’s argument type (let’s call it T) must be completely inferable from the collection type.

Why:T must be known because it indicates the type of collection elements. Since the collection is completely in charge of iterating itself we need some way to know what type its elements have. In the above example T would be the type of el. Knowing the type of iteration variable compiler can verify that projection expression (el.ToString in the above example) makes sense. T is also used to provide intellisense when you type “el.” .

Example of very simple queryable type:

Public Delegate Function SelectorDelegate(ByVal arg As Integer) As Integer

Class MyQueryable

    Public elements() As Integer

    Public Function [Select](ByVal projection As SelectorDelegate) As MyQueryable

        Dim new_col As New MyQueryable

        new_col.elements = elements.Clone

        For I As Integer = 0 To elements.Length - 1

            new_col.elements(I) = projection(elements(I))

        Next

        Return new_col

    End Function

End Class

You can test this queryable type with code like this:

Module Module1

    Sub Main()

        Dim arr As Integer() = New Integer() {1, 2, 3}

        Dim col As New MyQueryable()

        col.elements = arr

        Dim q = From I In col Select square = I * I

        For Each t In q.elements

            Console.WriteLine(t)

        Next

    End Sub

End Module

Output:

                1,4,9

Note that result of Select in our example is MyCollection that is a queryable type. This is not a requirement of queryable pattern. Select could return any type it wants. However returning a queryable type is very convenient as you can “compose” query by chaining Select operators:

This is a valid expression in VB. The second Select will be applied to the result of the first select

Dim q1 = From I In col Select square = I * I Select sq_2 = square * 2

If you replace the query in the previous example with the new query, you will get output:

                2,8,18

So far we have followed design decisions for the language integrated queries to the point where user specifies the query in the following form.

Dim num_squares = From cur_num In numbers Select result = cur_num * cur_num

Compiler transforms this into code that iterates over the source collection (numbers in this case), applies user specified expression to every element and produces the result.

If you think about universal applicability of this pattern you may notice that there are different kinds of collections. How can compiler know how to iterate over those? One possible solution is to handle only known kinds of collections (like those that implement IEnumerable(of T) ). We certainly know how to iterate such collections, but requiring that all queryable collections implement IEnumerable(of T) seems to be overly restrictive. Also it seems that producing the result should also be type-specific as we would expect the result of the query to be of the same kind as the source collection. For example if we are making a query on some kind of Set type or a Data Table we would expect to get back another Set  or Data Table.

One of the object oriented design principles tells that these kinds of type specific operations should be handled polymorphic ally by the types themselves. In this case the queryable types could all derive from a common ancestor and override functions that we need to do a query. That would certainly solve the problem, but it would still require all queryable types to share some common ancestor which is too restrictive and does not work for most of the types that already exist. We really want to be able to do queries on unrelated types to make queries truly extensible.

A good example of solving similar problem is the “For Each” operator. In addition to IEnumerable types “For Each” can handle types that implement “collection pattern”. The pattern requires that type has “GetIterator” function with appropriate signature and compiler goes from there by using this function. It seems to be a good idea to have some kind of queryable pattern that compiler can recognize and use for query translation and that is exactly how VB queries are implemented:

When compiling a query VB compiler looks for a function called “Select” with a signature that can accept a representation of the user expression. It is the “Select” function that does all the work of iterating over the source collection, applying the expression and collecting results into the result collection.

Dim num_squares = From cur_num In numbers Select result = cur_num * cur_num

Becomes:

Dim num_squares = numbers.Select(AddressOf VB_ CompilerGeneratedFunction)

' =====================================

' synthetic function that performs actions specified in user expression

'======================================

Function CompilerGeneratedFunction(ByVal cur_num As Integer) As Integer

        Dim result = cur_num * cur_num

        Return result

End Function

Now that we have a well defined way to detect and use queryable collections in queries, we just need to figure how to add these missing “Select” functions to various types. And surely enough in Orcas there is a mechanism to do this – it is called “Extension methods”.  Without going into details it makes possible to declare a shared method in such a way that it could be called as an instance function of its first argument. Example:

User defines:

 Shared Function Select(target as MyCollection, selector as SomeDelegate) as MyCollection

The function can be used as an instance function of MyCollection:

Dim Col1 as New MyCollection

‘ this should work as long as selector_del is of type SomeDelegate

Col1.Select(selector_del)     

To make this even better one of the part of the LINQ technology that will ship with VB9 are extensive libraries of extension methods that do all kind of query operations for common types such as IEnumerable(of T), arrays or Xml data. These implementations (also called query operators) are also very type agnostic through usage of generics so they should handle most if not all the scenarios that query user may ever require.

The result is that when we combine pattern matching for queryable types in the compiler and standard implementations of this pattern in LINQ libraries, it becomes a complete solution that allows users to apply language integrated queries to common collection types right away.

The pattern matching mechanism is also extremely flexible and extensible. If somebody wants to make some type queryable or to override an already provided implementation with a more specific one, it is very easy to do. One can simply add a Select function to the type. In fact through usage of Extension Methods the types do not really need to be modified to become queryable as you can provide Select that is an extension method.

In the previous posts we have designed a very handy function that can iterate any IEnumerable(of T) and apply given expression to all elements, but the disadvantages are obvious – more code to type. We tried to reduce the amount of actual code that user writes, but now we require that:

1)      User code is placed into a function.

2)      Instance of a delegate representing user code needs to be created and passed into the function.

3)      User needs to specify type arguments for the delegate

The good part is that these actions are extremely mechanical. They are so mechanical, that they can be done by the compiler itself. We just need a new language construct that will take only essential information from the user. The essential information that is needed here is the source collection and the expression that is applied to elements. In addition to this we need to specify the name for the current element so that we could refer to it inside the expression.

Dim result = <KEYWORD> current_number <SEPARATOR> numbers <SEPARATOR> expression

Interestingly enough this is very similar to SELECT statement in SQL. This is not a coincidence though as the SQL query basically performs the same action of projecting elements of one table into another using user specified expression.

SELECT expression FROM current_number IN numbers

So it should not be surprising that VB syntax for query expressions uses same keywords as in SQL. In fact the initial syntax for VB queries was almost identical to SQL. However it was discovered that there are many advantages such as better intellisense when iteration variable (current_number) is already declared by the time when user starts typing the expression. As a result the syntax was revised and From clause was brought to the beginning of the query expression.

Example:

' query that gets number names

Dim num_names = From cur_num In numbers Select result = cur_num.ToString

The code clearly describes what we want – we want to get all elements in “numbers” collection and generate a sequence of “result” values that are produced by applying “ToString()” function to each element. Based on the query, compiler can extract the expression that we specified in Select clause and generate a synthetic function that performs this action.

' translation of the query

Dim num_names = IterateAndApply(numbers, AddressOf CompilerGeneratedFunction)

' =====================================

' synthetic function that is automatically generated by the compiler

' it performs the actions specified in Select clause

'====================================== 

Function CompilerGeneratedFunction(ByVal cur_num As Integer) As String

        Dim result = cur_num.ToString

        Return result

End Function

This is much better. It takes only one line to produce a sequence of number names. The rest of the code is either compiler generated or in a library. In order to fully appreciate the convenience, let’s consider another example. What if instead of strings we need to produce a sequence of squares?

Code:

Dim num_squares = From cur_num In numbers Select result = cur_num * cur_num

This is it!!  This is all the code that you need to write!!!

Compiler can take care of generating a function that will perform the Select expression (also called projection). The rest of what is needed (iterating the source collection, applying the projection and forming the result sequence) did not really change and the same library functions can be reused.

No more duplication of code. No more off-by-one bugs and out-of-range exceptions. All we need is to make sure that the projection does what we wanted. Reusability rules!!!

Lets’ now consider the implementation of Apply function. In theory we could somehow interpret the string inside the Apply function, but having code in a form of string still causes problems. Consider the following example:

        Dim number_names As New List(Of Integer)

        number_names = IterateAndApply(numbers, _

                        "squares.Add(current_number.ToString())")

Take a look at the expression. While the code seems fine, our Apply function expects that the resulting type of the expression matches the type of the collection elements. This particular expression takes Integer and returns String.

We can add another type argument to Apply function to represent the return type of the expression:

Function Apply(Of T, S)(ByVal item As T, ByVal code As String) As S

However this just replaces one problem with another. Someone somewhere must specify S. Generic type argument T can be inferred from the function arguments (in particular from “item”). This is not the case for S. Since “code” is just a string, there is no way of knowing the type of the results of this expression. In fact there is no way to know at compile time that the expressions returns anything or even runs.

Consider this example

        Dim number_names As New List(Of Integer)

        number_names = IterateAndApply(numbers, "BLAH BLAH BLAH")

This is not going to run, but compiler has no idea.

Clearly a string is a very bad representation of user code. Besides the problems with interpreting it provides no information on what this code returns or whether this code will work.

Fortunately there is a way to write a piece of code and describe its arguments and return type. For that we need to put the code in a separate function and pass a reference to that function into the IterateAndApply as a delegate:

   Dim numbers As New List(Of Integer)

   numbers.Add(42)

   Dim number_names As New List(Of Integer)

   Dim MyCodeRef As Func(Of Integer, String) = AddressOf MyCode

   number_names = IterateAndApply(numbers, MyCodeRef)

   Function MyCode(ByVal current_number As Integer) As String

        Return current_number.ToString

   End Function

   ' =====================================

   ' implementation of IterateAndApply

   '======================================

   'delegate type to represent user expressions

   ' T is the type of argument

   ' S is the type of the result

   Delegate Function Func(Of T, S)(ByVal arg As T) As S

   ' source - the collection that is being iterated

   ' code   - reference to user supplied code to apply to each element

   ' returns a collection of results when user's code is applied to

   ' all lements.

   Function IterateAndApply(Of T, S)(ByVal source As IEnumerable(Of T), _

                    ByVal code As Func(Of T, S) _
                    ) As IEnumerable(Of S)

        Dim result As New List(Of S)

        'iterate the source

        For Each item In source

            'apply the code that user has provided to each element

            result.Add(Apply(item, code))

        Next

        Return result

   End Function

   'function that applies given code to the given object and returns
   'the result

   Function Apply(Of T, S)(ByVal item As T, ByVal code As Func(Of T, S)) As S

        Return code.Invoke(item)

   End Function

The code above solves a lot of problems. First of all there is no need for “magic” to apply the user code. In fact Apply function becomes so trivial, there is no need to even have it. The inner loop of IterateAd Apply could simply look like this:

For Each item In source

            'apply the code that user has provided to each element

            result.Add(code.Invoke(item))

      Next

Secondly, because user code is contained in a regular function and expressed as regular code, existing checks in the compiler can verify the syntax, type safety and everything else that can be verified at compile time.