As part of my work with the SQL Server Modeling CTP, I developed an end-to-end sample application which demonstrate model driven content based routing.  The RouterManager makes it simple to configure new WCF .NET 4 Routing Service by creating a domain specific language (DSL).  Routing configurations are stored in the SQL Modeling CTP’s Repository database.  The Entity Framework is used to read the router model data to configure the router. 

You can download the RouterManager sample on Code Gallery.  And read more about the details of the sample on our new team blog.

From “Oslo” to SQL Server Modeling

On DSLs and a few other things…

I personally am very excited about this announcement, as we now have a ship vehicle, schedule and a clear path to v1.  As you will hear and see at PDC09 there has been a lot of work on the technology since the May CTP.

Kraig Brockschmidt posted on the Model Citizens blog that we will be releasing a new version of the modeling CTP in conjunction with PDC.  I have been working with the new bits and created a cool end to end example that will also be released.   But I will post more details about this project next week.

The “data & modeling keynote” with Don Box and Chris Anderson

Data Programming and Modeling for the Microsoft .NET Developer

Astoria and EF talks

ADO.NET Data Services- What’s new with the RESTful data services framework

Evolving ADO.NET Entity Framework in .NET 4 and Beyond

“Oslo” talks

Building Data-Driven Applications Using Microsoft Project Code Name "Quadrant" and Microsoft Project Code Name "M"

Microsoft Project Code Name “Repository”: Using Metadata to Drive Application Design, Development, and Management

Microsoft Project Code Name “M”- The Data and Modeling Language

Be sure to keep checking the PDC website as new sessions are added daily.

First we need to the model and model data into the Repository.  To do this we execute the mx command.  The actual command will look like this:

mx install Employee.mx -d:Repository

This basically tells the mx command to install the Employee.mx values and our Employee model (which if you remember when we compiled this in Part II, we had to pass in a reference to our compiled model EmployeeModel.mx)  into the Repository database.  Now that our model is in the Repository lets take a look at it using “Quadrant”.

Oslo “Quadrant” is a visual tool for interacting with models and model data is included as part of the “Oslo” May CTP.  Quadrant itself is a great example of a model driven application.  We will make use of this fact a little later to customize the view of our model.  Let’s use Quadrant to take a look at our Employee model.  Launch Quadrant by clicking on the icon in the “Microsoft Codename Oslo Quadrant” folder in the Start Menu.

When Quadrant loads up, the Repository is displayed in a Tree Master/Detail view.  Expand Catalog and you will see our EmployeeInfo domain holding our Employees model.  Click on the Employees model. Notice it displays a list of the first names of the employees that we specified in our DSL input text which is now in the Repository.  Remember this as we will come back to it shortly.

Now drag the Employees model onto the workspace.  Notice Quadrant displays our model values in a Table view.  Let’s change this to a Master/Detail view to make it easier to see.  In the upper right of our Employees view, click on where it says Table.  A list of different viewers will be shown in the dropdown.  Select Master/Detail.  A picture of the view setting is shown below (I placed a red oval around the Master/Detail setting you should select):

You will see Quadrant now shows the list of first names on the left and when you click on a name, it shows the detail on the right (in a property view). Notice, the Table view was tagged in the dropdown as the Default view.  Let’s make the Master/Detail view the Default view.  From the main menu select View|Set Default View.  Master/Detail will now be the default view for our model.  If we close the Employees model and re-drag it from the Catalog, it should come up in our new default view.

When Quadrant displays our Employees model data, it shows the FirstName field.  Quadrant looks at a model and grabs the first field it finds that contains “name” as the default field to display.  Remember previously I said Quadrant was a great example of a model driven application.  Well, we can change this behavior for our Employees model by updating one of Quadrant’s models, telling Quadrant what field we want it to display by default.  This is done by adding instance data for the ViewerHintCatalogTypes in Quadrant’s own model Repository. 

We can use M to insert this data.  Let’s have Quadrant display an Employee’s Number instead of FirstName as the default display value.  Below is what the M value looks like for our ViewerHintCatalogTypes.  We tell Quadrant that our host is the localhost – “.”, our the Catalog where our Employees model is stored is called “Repository”, the schema name for our model is call “EmployeeInfo”, the type we want to modify the display for is “Employees” and that when it displays the Employees model, it should show the “Number” field.  Put the below M in a file called EmployeeConfiguration.m:

module Quadrant.SchemaExtensions
{
   import Quadrant.SchemaExtensions;

   ViewerHintCatalogTypes
   {
       {
         Host => ".",
         CatalogName => "Repository",
         SchemaName => "EmployeeInfo",
         TypeOrAssociationName => "Employees",
         ViewerHint =>
         {
            DescriptionProperty => "Number",
            GroupingProperty => "",
            SpatialProperty => ""
          }        
      }
   }
}

Because we are inserting values into one of Quadrant’s models, we included an import of Quadrant’s SchemaExtensions model.  Now we need to compile our EmployeeConfiguration M.  We will need to pass the M compiler a reference to Quadrants models on the command line.  Quadrant’s models are compiled into a file called Quadrant.mx which can be found in the Oslo install directory.  The command to compile our EmployeeConfiguration is shown below:

m EmployeeConfiguration.m /r:"C:\Program Files\Microsoft Oslo\1.0\bin\QuadrantModels.mx"

The next step is to install our configuration values for our model into Quadrant’s repository.  To do this we first need to find out what Quadrants repository is named.  You can do this by loading up SQL Server Manager and looking at the databases or executing the command below from a command line:

sqlcmd -s ./SQLSERVER -Q "select name from sys.databases"

On my machine, Quadrant’s repository is contained in a database called Quadrant.3.0.1803.10.DKAUFMAN-OSLO which is a combination of the name Quadrant, the Oslo version and the name of my computer.  Yours should be similar with a different computer name at the end.  We will use the mx command to install the configuration file, telling it to use the Quadrant repository and passing a reference to Quadrant’s compiled models.  The command looks like this:

mx install EmployeeConfiguration.mx -d:Quadrant.3.0.1803.10.DKAUFMAN-OSLO.Administrator /r:"C:\Program Files\Microsoft Oslo\1.0\bin\QuadrantModels.mx"

Our customization is now installed. Close Quadrant and re-start it.  Notice now in the Repository Tree view when you click on our Employees model, a list of Employee Number is displayed on the right instead of FirstName.  Also, in our default model view for our Employees model, Employee Number is now shown on the left hand side in the Master/Detail and clicking on the number shows the employee info on the right.  Below is a screenshot of Quadrant showing the visual customization for our Employees model:

That’s it for our tour of the “Oslo” May CTP.  Through these posts, we saw how easy it was to build a model, create a DSL for getting data into our model, install our model into the Oslo Repository, display our model visually using Quadrant and customizing the view of our model.  You can download the Oslo artifacts for this example here.  I included batch files to build, clean and install the example. Also, included is a batch file to re-build the default Oslo repository (you might have to edit directories or the name of the Quadrant repository in the batch files to match your install).

Now it’s your turn.  Install the latest Oslo CTP and take it for a spin.  And please let me know about the applications you build as you experiment with this exciting new technology.

To re-cap, in Part I, we created a module called EmployeeInfo that defined an entity called Employee which has the following definition:

type Employee{
  FirstName: Text#50;
  LastName: Text#50;
  Number:Text#10;
  id : Integer32 => AutoNumber();   
} where identity(id);

EmployeeInfo also defined and extent called Employees which contains zero or more Employee instances.

Now we will use M language support to create a very simple DSL called EmployeeGrammar which describes an Employee.  Below is an example of the EmployeeGrammar:

My name is Edward Brown, my employee number is 364321
My name is Joe Smith, my employee number is 342343

The above input text is placed in a text file called Employee.txt. 

Now lets create the language used to parse the input text for our language. A grammar is a set of rules which determine if a sequence of characters conform to a language.  The grammar parser will parse the input text based on the rules of the grammar and build a syntax tree which is a hierarchy of data that was parsed.  The syntax tree can be used in a number of ways in your application including storing it (e.g. in the Repository or as XML) or by iterating through the tree in memory at runtime directly in your application.  For this example we will be using the M tool chain to generate the syntax tree as a set of model values and import it into the Oslo Repository.

We can build our grammar in the Oslo Intellipad tool.   For our grammar, first we need to specify the interleave for the language which indicates the rule for which values should be used as whitespaces.  For our language, whitespaces will be defined as spaces, tabs, carriage returns/linefeed and commas.  The interleave command looks like this:

interleave Whitespace = ' ' | '\t' | '\n' | '\r' | ',';

Next we need to define tokens which are used to designate rules that defines the language.  The language parser will try to match the tokens in the input text.  The first token we define will be for the start text for our language which is “My name is”.  The token statement looks like this:

token TkStart = "My name is";

We then define a token for name which can be any letter, lower or uppercase.  A name can be made up of one or more letters.  The definition for the name token looks like this:

token TkName = ("A".."Z" | "a".."z")+;

In the above token definition, the “+” is what indicates that the token is made up of one or more characters.  That takes care of tokens for the name part of the DSL but we still need to specify the tokens for the employee number.  The tokens for employee number section looks like this:

token TkNumStart = "my employee number is";  
token TkEmpNum = ("0".."9")+;

Now that we have the tokens we need to define the syntax of how the tokens should be used.  M language below is the syntax statement that tells how the tokens can be used to parse the input text.  All M language files must have a Main syntax statement which is the starting point for the grammar rules.  Our will look like this:

syntax Main =  EmployeeData*;

The above indicates that the EmployeeData rule should be used for the grammar which means there needs to be a syntax statement for EmployeeData and that there will be one or more lines of the input textto be processed.   

Next we need to define the syntax for EmployeeData which will consist of the start token, a token for first name, a token for last name, a token for the start of the employee num and finally the token for the employee number.  The EmployeeData syntax statement looks like this:

syntax EmployeeData =  TkStart TkName TkName TkNumStart TkEmpNum;

The complete text for our language so far is shown in Intellipad below.  Again we use the module name EmployeeInfo.  Our language will be called EmployeeLanguage. 

Save the grammar in a file called EmployeeGrammar.mg.  Now we can test it.  To do this we put Intellipad in MGrammar mode and load up our input text in the Employee.txt file we created above.  With the EmployeeGrammar.mg file loaded in Intellipad, press <CTRL> <SHIFT> T.  Intellipad should prompt you to select the input text file and then it will go into Tree Preview Mode.  The input text will be shown on the left side, the grammar in the middle and the generated syntax tree will be on the right side.  At the bottom will be a window that will display parse errors.  Below is a screen shot of Intellipad in Tree Preview Mode:

If you look at the generated syntax tree on the far right for our sample language input text you will notice that it seems rather ambiguous.  It also contains extra text such as the text of the static tokens. To fix this, we will use projections.  Projections specify how the values should be generated from the language input text should look.  The projection operator “=>” is used and immediately followed by the pattern we want to use in place of the default tree structure.

Let start by cleaning up the EmployeeData syntax.  You can reference the values of the tokens by prefacing the token with a <name>:.  In this example we will reference the first name of TkName with f, the last name with l and the employee number as n.  We will then use those references in the projection for instance indicated the first name should be output in the syntax tree as with the text “FirstName =” and the value of f by writing  FirstName => f. The full definition for the EmployeeData syntax should look like this:

syntax EmployeeData =  TkStart f:TkName l:TkName TkNumStart n:TkEmpNum =>{FirstName => f, LastName => l, Number =>n};

Now let’s add a projection to the Main syntax.  The values from the EmployeeData rule will be referenced in the production by the identifier e.  We need to make sure the projection generates the name of our extent Employees from the our previously defined model and we will use the valuesof grammar keyword to remove an extra set of brackets that was being generated in the default projection.  The Main syntax should look like this:

syntax Main =  e:EmployeeData* => Employees{valuesof(e)};

The complete text of our language is shown below:

module EmployeeInfo{
language EmployeeLanguage{
   syntax Main =  e:EmployeeData* => Employees{valuesof(e)};
   syntax EmployeeData =  TkStart f:TkName l:TkName TkNumStart n:TkEmpNum =>{FirstName => f, LastName => l, Number =>n};
   token TkStart = "My name is";
   token TkName = ("A".."Z" | "a".."z")+;
   token TkNumStart = "my employee number is";
   token TkEmpNum = ("0".."9")+;
   interleave Whitespace = ' ' | '\t' | '\n' | '\r' | ',';
  }
}

The generated syntax tree using the added projections looks much cleaner and now maps to our schema definition of Employee.  Here is a screenshot of Intellipad with the new syntax tree displayed:

Now let’s use the M tool chain to compile our grammar and generate M values of our input text.  We compile the grammar using the m.exe command, passing it our language file EmployeeGrammar.mg.  The result will be  an mx file called EmployeeGrammar.mx.  We then use the mgx.exe command to generate M values for our input text.  We pass the mgx command a the name of our input text file, use the /r switch to reference the EmployeeGrammar.mx file we generated previously and use the /m switch to tell it that the module generated with the values should be EmployeeInfo to be consistent with the modules we defined previous for our model schema.  The full command will looks like this:

mgx Employee.txt /r:EmployeeGrammar.mx /m:EmployeeInfo

This generates the following M values in a file called Employee.m:

module EmployeeInfo {
    Employees {
         {
            FirstName => "Edward",
            LastName => "Brown",
            Number => "364321"
        },
         {
            FirstName => "Joe",
            LastName => "Smith",
            Number => "342343"
        }
    }
}

We can then compile the newly generated M values from our input text using the m.exe command, which will generate a file called Employee.mx.  To compile the newly generated M values from our input text we need to reference the Employee model schema which we compiled previously which is in the EmployeeModel.mx file.   The command looks like this:

m Employee.m /r:EmployeeModel.mx

And here is a screenshot of the commands executing:

That’s it for Part II.  As you can see, creating new domain specific languages is very easy using the Oslo M tools.  In Part III we will install our Employee domain model and values into the “Oslo” Repository and then customize the visual view of our model using “Quadrant”.

For this example we’ll create a very simple domain model using MSchema that will contain information for a fictional company’s employee information.  The DSL will take a simple input text that will be parsed and stored as model instance data.  Lastly we will display and customize the view of the Employee model in Quadrant.

The posts will be split as follows:

• Part I – Creating the Model with MSchema
• Part II – Building a DSL with MGrammar
• Part III – Viewing the Model in Quadrant

The Module will be called EmployeeInfo and the schema will contain the definition of an entity called Employee.  An employee will have Text fields called FirstName, LastName and Number where Number is the fictional employee’s employee number .  The schema for Employee looks like this:

type Employee{
  FirstName: Text#50;
  LastName: Text#50;
  Number:Text#10;
  id : Integer32 => AutoNumber();   
} where identity(id);

Also we include a field call id which is an integer which will contain a  number auto generated by “Oslo” that will be used as a unique identity field for each Employee. 

Note: In the Oslo May CTP, the use of “=” has been deprecated.  Instead “=>” is used in MSchema and MGrammar.  You can read more about the May CTP language changes in the Oslo May CTP Release Notes and Shawn Wildermuth's blog post “M Language Changes Coming in Next CTP”

Now that we have Employee defined, we still need an extent to hold a list of employees.  The line below will define and extent called Employees that contains zero or more Employee records:

Employees:Employee*;

That’s it, our simple Employee model is done.  Below is the full MSchema file for the Employee model. I used Oslo’s Intellipad editor to create the definition and saved it to an M file called EmployeeModel.m.  Here is a picture of the Employee model Intellipad:

The final step will be to compile the Employee model.  We do this using the Domain Compiler - m.exe that is part of the “Oslo” SDK from the May CTP and pass it the name of our M file – EmployeeModel.m.  This will generate a file called EmployeeModel.mx which contains our compiled model.  Below is a screenshot of the compilation step:

That’s it for this installment, we now have our compiled Domain Model.  In Part II we will create a DSL that converts some easy to understand text into instance data for our Employee Model. 

An example input file to my grammar for an Orchestration looks like this:

Orchestration test
  Receive rcvMsg
    where ReportToAnalyst = True
  end

  If "x>2" then
    Transform trnsfrmMsg1 end
  else
    Transform trnsfrmMsg2 end
  end
 
  Send sndMsg end
end

The above creates an Orchestration that receives a message, makes a decision which message transform to run and then sends the message.  Also notice some shapes can contain other shapes and you can set parameters on the Shapes (eg. "where ReportToAnalyst = True"). The syntax for the DSL is very easy to understand, most people could figure out what the process was trying to accomplish simply by looking at the input file.

Next, I need to be able to parse the new syntax.  I did this using the “Oslo” SDK and creating a grammar.  Below is a screenshot of the “Oslo” Intellipad editor in tree mode.  On the left is my input text.  The center pane is my MGrammar and the right pane is the parse tree of the input file:

Now that we can parse the input using MGrammar, I needed a way to generate the ODX file.  To do this, I built a commandline compiler in C# that uses Oslo’s  System.Dataflow library that walks the parse tree and generates the correct XML for the ODX file.  The compiler creates an in memory object for each shape and sets the correct properties and relationships between the shapes.  Each shape object knows how to render itself in XML and also knows it’s child shapes.  Below is the output from running the compiler (for debug purposes, I display the Orchestration shapes and their child relationships):

Now that I have the Orchestration file, I can load up the ODX in BizTalk and check out what was generated.  I used an empty BizTalk project and “Add Existing Item…” to add the Orchestration to the project and load it up:

Notice it generated the Receive, Decision, Construct/Transform and Send shapes all based on our original input.  The Orchestration still needs to be completed by the developer (for example message types have not been set) but you can see how “Oslo” makes it so even a non-programmer could describe a process flow and have that generated into something executable.

MGrammar makes it easy to create and modify the language.  For instance, I want to make the syntax even more non-programmer friendly.  One simple way could be to change the “end” token to a period.  The syntax to create a shape would change from:

Receive msg1 end

to

Receive msg1.

The key is that I can make that modification to the language without changing the parse tree.  Using the Projections functionality of MGrammar, I can make changes and additions to my language without the need to make major changes (if any) to my compiler.

As you can see, one of the benefits of the Oslo technology is that it can make software development and customization much more approachable.  I talk with many ISVs who are excited  about using “Oslo” as a way to help their end users and system integrators adapt and configure the ISV's out of the box software to run in customer environments.

I encourage you to download the Microsoft "Oslo" SDK - January 2009 CTP and try it out yourself.  And check out the great content on the Oslo Developer Center to help you get up to speed.  Let me know your experiences and thoughts on how you might use “Oslo” with your software.

Doug Purdy and Don Box a few weeks ago posted additional information on Oslo.  Both Doug and Don will be speaking on Oslo at PDC and I encourage you to see their talks.

These technologies coupled with the enhancements we are making to WF and WCF for .Net 4.0 has really excited the ISVs I am engaged with.  The work we are doing holds the promise of making the ISV solutions much easier to develop, customize, deploy and maintain.  These technologies have the opportunity to be truly transformational.

But if you want to find out more in the near term, you'll just have to go to PDC. Registration is still open.  The PDC folks have thoughtfully even posted a sample letter to your boss about why you can't miss PDC (if I haven't given you enough ammunition already).

• New web service registry capabilities with support for UDDI (Universal Description Discovery and Integration) version 3.0
• Enhanced service enablement of applications (through new and enhanced adapters for LOB applications, databases, and legacy/host systems)
• Enhanced service enablement of “edge” devices through BizTalk RFID Mobile
• Enhanced interoperability and connectivity support for B2B protocols (like SWIFT, EDI, etc)
• SOA patterns and best practices guidance to assist our customer’s implementations 

BizTalk 2006 R3 will also add support for our latest wave of technology releases including Windows Server 2008, .NET Framework 3.5, Visual Studio 2008 and SQL Server 2008.  If you are interested in this new version, the BizTalk 2006 R3 TAP Program is still accepting nominations.

If you are interested in this program, please visit our Connect site to sign-up.

TAP Sign-up Instructions:

• Please log on to Connect
• Under Categories select Server
• Look for BizTalk TAP Programs and select the "apply" link next to BizTalk Server 2006 R3
• Download and complete the BizTalk Server 2006 R3 TAP Nomination form and click the select button

There is a lot of confusion about what is "Secure FTP".  Most people don't realize that there are primarily two different ways of securing FTP. 

• FTPS or FTP over SSL uses the SSL/TLS layer to encrypt the FTP traffic.  It is similar to HTTPS which web browsers use to secure HTTP.
  
• SFTP or FTP over SSH tunnels FTP sessions over an SSH connection.

So next time you're asked about secure FTP, be sure to find out which one they are referring to.