Problem 1: The TenantId

If this value is wrong, you’ll get the above error. So where do you get the tenantId from? I neglected to leave this out of my original post. So where does it come from and why did I leave it out of my original post? Let me reverse the questions. In my original post I created an application from scratch and enabled AAD integration which created a LocalDb with all of the data populated for me. That’s the easy way – what if you’re enabling an existing application (I did say I was going to get to that in a later post but I’ve been busy)? The TenantID can actually be found quite easily – you just have to know what to look for. In this case, the TenantID can be found by going to the Azure Portal (manage.windowsazure.com – this can’t be accessed from the Preview Portal just yet), clicking on the Active Directory link, clicking the directory that you are using (Default Directory in my case), clicking Applications and finally clicking the name of the application that you’ve integrated with AAD. Finally, at the bottom of the page click View Endpoints (on the icon bar at the bottom). That will display the dialog shown here:

In this dialog, focus on the WS-FEDERATION SIGN-ON ENDPOINT which looks like this in my case:

https://login.windows.net/67c3a3d5-b264-4328-ade6-7a7b81136117/wsfed

The guid in this string is the TenantID and this is what goes in the single row in the Tenants table in the database (refer to the original post).

That’s one problem taken care of.

Problem 2: The IssuingAuthorityKeys

This one was a little more interesting and I have to say it is one of those things that is just not documented well anywhere. I came across it in a forum post as I was researching the problem. The original forum post is here (but it leaves out a couple of steps): https://social.msdn.microsoft.com/Forums/en-US/8f2b7807-8d49-4190-978c-535c91f440c4/no-valid-key-mapping-found-for-securitytokensystemidentitymodeltokensx509securitytoken?forum=WindowsAzureAD. In this post there is a link to a documentation page here: https://msdn.microsoft.com/en-us/library/azure/dn641920.aspx. This documentation is excellent but without the forum post I never would have found it – and as I said it leaves out a couple of steps. However, I urge everyone to read it because it has topics that cover why you might get the above error even though the application worked previously. Another item to note is that the suggestions for automatically handling key rollover for VS 2013 applications don’t make sense – my application was created with VS 2013 and it didn’t get the rollover logic. Read the section on VS 2012 and update your application accordingly so this problem won’t happen in the future.

The key rollover problem occurs because Microsoft frequently creates new certificates to help mitigate any security issues and deal with the fact that certificates expire. But that leaves applications which depend on AAD integration in the lurch. So, how do you get the new key (or keys in this situation)? In the MSDN article referenced (in the section on manually updating your keys) above it tells you to navigate to the above dialog and copy the URL for the Federation Metadata Document and paste it into the address bar of your browser. It then tells you to find the x509Certificate element, copy the value of this element and save it as a .cer file (paste it into notepad and save the resulting file as key1.cer for example). After you do that you can double-click the file and view the details. On the Details tab it will give you the Thumbprint for the certificate and you’re good to go, right? Wrong – this is where the article gets it wrong. You need to do this for each certificate in the response. In this case there are now two keys that need to be entered into the IssuingAuthorityKeys table.

In Conclusion

One thing I haven’t figure out yet (more experimentation necessary) is that after making the changes to the code to automatically update the keys, it puts those thumbprints in the web.config file. I do not know if the database is needed at all after this. I will experiment some more and come back and update this post with additional information.

Hope this helps!

In this post we’ll look at three things – configuring a server from scratch, deploying the required PowerShell DSC modules and using variables in the PowerShell DSC scripts.

Note

In the Windows Management Framework 5.0 which is currently in preview, the process for deploying DSC modules will (or can) change. More information on WMF 5.0 and the new PowerShell Gallery can be found here. With WMF 5.0 you can execute the PowerShell command Install-Module –Name xxxx and the module will be installed directly from the gallery. The method being provided here can be used as an alternate if you don’t want to do this (also, I really wanted to show a clean installation without having to pre-install WMF 5.0).

Deploying PowerShell DSC Modules as part of the Release and configuring the server from scratch

Let’s start with the basics – a clean machine. I loaded up a new machine in Azure and did absolutely nothing with it except to add the http endpoint so I could check to see if the website actually installed successfully. The name of the server is, appropriately I think, “AppFromScratch”.

Next, I’m going to continue with the BlogDemo example from the previous post so if you haven’t read that, it’s probably good to skim it.

I’ll quickly go through the setup for this walkthrough.

1. Create a new build called “BlogDemo From Scratch”
2. In the RM client I linked the new environment and server
3. I created a new path with only a Development environment pointed to the clean server

Next, I’ll go through the changes in the solution – specifically the DeploymentScripts project. At the end of this the project will look like that in Figure 1.

Figure 1 – Solution at the end of this section of the walkthrough

The first thing to do is to add the xWebAdministration PowerShell to source code. This is kind of a pain in the butt and I’m working on a better solution. The reason it’s a pain is because it requires you to go into every folder and mark each file as “Copy if newer”. But stay tuned for a more streamlined approach to be presented in a later blog. I’ll come back and update this post with a link.

The next thing to do is to add a new PowerShell script file called “CopyDscScript.ps1”. This script looks like the following:

  1: configuration FileCopy 
  2: {
  3:   node localhost
  4:   {
  5:     File CopyDSCModules
  6:     {
  7:       Ensure           = "Present"
  8:       Type             = "Directory"
  9:       Recurse          = $true
 10:       SourcePath       = join-path $applicationPath "DSCModules"
 11:       DestinationPath  = "C:\program files\WindowsPowerShell\Modules" 
 12:     }
 13:   }
 14: }
 15: 
 16: FileCopy

This is a pretty basic DSC configuration – it just copies the DSCModules folder from the output location to one of the PowerShell module locations on the server (there are ways to get this value but in many cases they return multiple locations so it’s just hard-coded here as a safe location that’s always available).

Next, add a new deployment script called “DeploymentScript2.ps1” which looks like the following:

  1: configuration FullSetup 
  2: {
  3:   Import-DscResource -Module xWebAdministration
  4: 
  5:   node localhost
  6:   {
  7:     WindowsFeature IIS
  8:     {
  9:        Ensure          = "Present"
 10:        Name            = "Web-Server"
 11:     }
 12: 
 13:     WindowsFeature ASPNet45
 14:     {
 15:        Ensure          = "Present"
 16:        Name            = "Web-Asp-Net45"
 17:        DependsOn       = "[WindowsFeature]IIS"
 18:     }
 19: 
 20:     File CopyDeploymentBits
 21:     {
 22:       Ensure          = "Present"
 23:       Type            = "Directory"
 24:       Recurse         = $true
 25:       SourcePath      = join-path $applicationPath "_PublishedWebsites"
 26:       DestinationPath = "C:\inetpub\wwwroot"
 27:       DependsOn       = "[WindowsFeature]ASPNet45"
 28:     }
 29: 
 30:     xWebsite DefaultSite
 31:     {
 32:        Ensure          = "Present"
 33:        Name            = "Default Web Site"
 34:        State           = "Stopped"
 35:        PhysicalPath    = "C:\inetpub\wwwroot"
 36:        DependsOn       = "[File]CopyDeploymentBits"
 37:     }
 38: 
 39:     xWebsite NewWebsite
 40:     {
 41:        Ensure          = "Present"
 42:        Name            = "BlogDemo"
 43:        State           = "Started"
 44:        PhysicalPath    = "c:\inetpub\wwwroot\BlogDemo"
 45:        DependsOn       = "[xWebsite]DefaultSite"
 46:     }
 47:   }
 48: }
 49: 
 50: FullSetup

The only thing different in this listing than the DeploymentScript1 from the previous blog post is that (well, besides the fact that I changed the name of the configuration to “FullSetup” since it more accurately describes the script) I added the installation of the IIS server and ASP.NET 4.5 as noted between lines 7 and 18.

Save all of this and check it into source control.

Next up is the release template. I created a new template called BlogDemo from Scratch which is shown in Figure 2.

Figure 2 – New Release Template

In this template there are two “Deploy Using PS/DSC” actions which I’ve renamed to illustrate their purpose. The configuration of both is identical except that I’m calling a different script. In the first one I’m deploying the DSC Module(s) and in the second I’m performing the server setup and deploying the application. You can of course break these up however you wish.

The beauty of DSC is that if IIS is already installed or ASP.NET 4.5 is already installed, those steps are skipped. This means that the second time I deploy to this server it will go much faster. In addition this servers as a communication point with Operations. Hence the new industry buzzword of “DevOps”. Anybody from Operations can see that this requires a web server with ASP.NET 4.5 installed.

Using variables in the PowerShell script

Up to this point everything has been hard-coded. As this is a fairly simple application there isn’t a huge need to parameterize the file but in any reasonably complex environment this is almost a requirement. In this section I’ll show you how to parameterize the physical path of the website because while it might be fine to deploy the web application c:\inetpub\wwwroot\BlogEngine it isn’t likely that Operations wants you to deploy your website their for numerous reasons. By parameterizing this value it can be changed per environment as needed.

There are two ways to do this. You can add the variable to the component (New Component 1 in this case) or you can add it directly into the action on the release design surface. The reason for the different approaches is that by adding the variable to the component it is available to all releases which use that component. Adding the variable to the release template makes it available only to that release. In this case I’ll add it to the component.

On the Configure Apps tab, select the Component page and edit New Component 1. On the Configuration Variables page add a new variable called physicalPath as shown in Figure 3.

Figure 3 – New configuration variable

Save and close the component and jump over to the release template.

Expand the activity that deploys the application and click the green + under Custom Configuration. The physicalPath variable will be available for you in the drop down for the first path. Select it and then add the path in the second column as shown in Figure 4.

Figure 4 – Configuration variable used as part of a release template

To add a variable in the release template without adding it to the component you would just type the name in the first column here but otherwise everything would be the same.

Next, go back and edit the DeploymentScript2 as follows:

  1: configuration FullSetup 
  2: {
  3:   Import-DscResource -Module xWebAdministration
  4: 
  5:   node localhost
  6:   {
  7:     WindowsFeature IIS
  8:     {
  9:         Ensure          = "Present"
 10:         Name            = "Web-Server"
 11:     }
 12: 
 13:     WindowsFeature ASPNet45
 14:     {
 15:         Ensure          = "Present"
 16:         Name            = "Web-Asp-Net45"
 17:         DependsOn       = "[WindowsFeature]IIS"
 18:     }
 19: 
 20:     File CopyDeploymentBits
 21:     {
 22:         Ensure          = "Present"
 23:         Type            = "Directory"
 24:         Recurse         = $true
 25:         SourcePath      = join-path $applicationPath "_PublishedWebsites"
 26:         DestinationPath = $physicalPath
 27:         DependsOn       = "[WindowsFeature]ASPNet45"
 28:     }
 29: 
 30:     xWebsite DefaultSite
 31:     {
 32:         Ensure          = "Present"
 33:         Name            = "Default Web Site"
 34:         State           = "Stopped"
 35:         PhysicalPath    = "C:\inetpub\wwwroot"
 36:         DependsOn       = "[File]CopyDeploymentBits"
 37:     }
 38: 
 39:     xWebsite NewWebsite
 40:     {
 41:         Ensure          = "Present"
 42:         Name            = "BlogDemo"
 43:         State           = "Started"
 44:         PhysicalPath    = join-path $physicalPath "BlogDemo"
 45:         DependsOn       = "[xWebsite]DefaultSite"
 46:     }
 47:   }
 48: }
 49: 
 50: FullSetup

The specific changes are on lines 26 and 44. The reason line 35 wasn’t changed is because that’s the physical path of the default website and we don’t have any control over that (it’s also a different location than where we might put our website).

Check the change in and when the new release kicks off the website will have been updated successfully.

Hopefully this post has given you some more insight into what you can do with DSC and RM. In the next post I’ll expand this to a more typical scenario which will provide for deploying a database as part of the release. After that, well, we’ll see as there are a lot of places to go with this…

Overview

For those who haven’t used or heard of Release Management, it is a part of Microsoft’s DevOps strategy to help enable the three ways made popular by the book The Phoenix Project. There are other parts such as the automated testing, automated builds and App Insights. Depending on what technologies you are using there may be other technologies that play into this story but these are the four core ones. In this post I’ll be talking specifically about RM and a little bit about team build since I have tied them together (Team Build is not a requirement to use RM).

At its core, RM can turn on environments, configure them through various means (we’ll look at using Desired State Configuration or DSC in this post), deploy your application across as many servers as needed and then execute a series of tests. It allows for a gated approval process which is controlled and auditable.

In this series of posts I will walk you through using RM with DSC and Visual Studio Online (hint, if you know how to use it on-premises you know how to use it in the cloud). We’ll start small and simple at first and work our way to more complex, more real-world examples such as multi-tier deployments.

Connecting to Visual Studio Online

In a large part it’s pretty underwhelming because it just works – which is the way it should be. The only pre-requisite for this is that you have a Visual Studio Online (VSO) account and that there is a Team Project already created. When RM launches the first time you’ll be asked to enter a URL. Simply enter your VSO URL (in my case jefflevinson.visualstudio.com) and click OK as shown in Figure 1 (note the URL is different because this is the fully qualified URL but you can just enter the shortened version).

Figure 1 – Configuring the RM Service

The first thing you’ll want to do is to avoid a specific bug I discovered. To do this, go to the Administration > Manage Users screen. Change the “Admin” account to your name as shown in Figure 2 (Admin was changed to Jeff Levinson).

Figure 2 – RM Users

**Warning**

There is currently a known issue here – DO NOT change the e-mail address of the Release Management Service no matter what. You will lock yourself out of the RM Service and will have to open a ticket with the VSO service which may take a little while to fix. The team is currently aware of this issue.

Connecting to Azure Environments

The next step is to set up your Azure environments for each stage of the deployment process. There is a definite structure you need to follow to get the most out of this. Each environment needs to be hosted in a separate cloud service. Whether or not they are in the same subscription is immaterial. For the purpose of this demonstration I have created three virtual machines in three different cloud services (you can have more than one machine in a cloud service but make sure each cloud service contains one environment). The names, for this series, are MyApplication1Dev, MyApplication1Test and MyApplication1 (which represents the production environment). The virtual machine names are all the same – MyApplication1 which is permitted because they are in different cloud service environments. This is obviously an optimal situation if you are running tests from the local servers. If you are running tests from another machine, consider putting a client machine in each cloud service – if you access these machines external of the environments you have to use the fully qualified name which means you have to parameterize the URL’s (i.e. you have to access them with http://myapplication1dev.cloudapp.net).

One note, for our purposes (and probably yours as well) is that I have opened up port 80 for each of the VM’s in order to be able to see the changes on the website in the VM without having to log into the VM.

Once you have the Azure virtual machines created you can go to town in Release Management.

To being, go to the Configure Paths tab, Environments page (Figure 3).

Figure 3 – Environments page

Click New vNext: Azure in the upper right which will bring up the new Azure Environments page as shown in Figure 4.

Figure 4 – New Environment page

Click Clink Azure Environment in the upper right. Select the subscription to use, select the cloud service and click Link (Figure 5).

Figure 5 – Linking an existing cloud service

You can add as many subscriptions as you like from the Administration > Manage Azure page. And this requires a small aside so it’s easier to do. The easiest way to get the required information is to grab the publishing profile for your azure subscription. To do this, go to https://manage.windowsazure.com/publishsettings/index?client=powershell. This will trigger a file download which will allow you to save your publishing profile. This file contains all of the information you need to link RM to your subscription. This screen is shown in Figure 6.

Figure 6 – New Azure Subscription link page

The name can be arbitrary. The Subscription ID is included in the file and the Management Certificate Key is the whole string contained in the ManagementCertificate element of the publishing profile. The storage account is whatever storage account you want to use from Azure.

Okay, back to linking environments… After an environment is linked, click Link Azure Servers in the lower left of the window. Select one or more servers from the given cloud service and click OK. The resulting screen will look like that in Figure 7.

Figure 7 – A linked environment with a single linked server

After linking however many cloud services you want, you’ll have an Environments screen that looks like those in Figure 8.

Figure 8 – Environments linked to RM

After doing this, click the Servers tab to review the cloud services and servers which are now linked to Release Management.

Configuring the Release Path

The next step is to configure the release path. To do that we need to specify which environments will be supported. Go to the Administration tab > Manage Pick Lists and select the Stage Type entry. I have added the Dev, Test and Production entries as shown in Figure 9.

Figure 9 – Stage Types

Once this has done we can now configure the actual release path which involves mapping the stage types to the environments. To do this, head to the Configure Paths > vNext Release Paths page and click New. Enter the stages as needed. For each stage, click Add, select the stage label and then select the environment. The end result of this operation is shown in Figure 10.

Figure 10 – A fully configured release path

The only absolutely critical thing about this is that the first stage (Development in this case) Acceptance Step must be set to Automated. The reason for this is that if it is manual and you trigger it from a build, the build will fail if it is not automated.

Once the release path is configured you’re ready to start building out the actual release process.

Creating the Application

We’ll create the ASP.NET MVC application next and structure it so that it is optimal for our purposes.

To begin with, create a new MVC project and change the authentication to “No Authentication”. In addition, uncheck the Host in Cloud checkbox (note that currently RM doesn’t support Azure Websites but it will in the future and you can make it work with Azure PowerShell scripting today but that’s a story for another day). I have opted to add unit tests to my project as is a best practice but that’s entirely up to you. The generated project structure looks like that in Figure 11.

Figure 11 – New MVC project

Note also that this is in version control in VSO which is required for this series of walkthrough’s. It is absolutely not a requirement to use RM in VSO – you can build code in any system and have RM deploy it but I’m not covering that in this series of posts.

Next, add a new Console Application project to the solution and call it “DeploymentScripts”. This is a quick and dirty way to get extra files deployed on build. There are various other options to accomplish this but this is by far the simplest. Delete the Program.cs file and change the output to Class Library. This project will contain any deployment scripts used for our application. The next step is to add in our DSC file. However, we’re going to edit it in PowerShell ISE as opposed to Visual Studio because there is no IntelliSense for Visual Studio at this time and I’ll show you a simple trick to figure out what functionality is available to you.

Open PowerShell ISE and copy and paste this script as follows:

  1: configuration FileCopy 
  2: {
  3:   Import-DscResource -Module xWebAdministration
  4: 
  5:   node localhost
  6:   {
  7:     File CopyDeploymentBits
  8:     {
  9:       Ensure           = "Present"
 10:       Type             = "Directory"
 11:       Recurse          = $true
 12:       SourcePath       = join-path $applicationPath "_PublishedWebsites"
 13:       DestinationPath  = "C:\inetpub\wwwroot" 
 14:     }
 15: 
 16:     xWebsite DefaultSite
 17:     {
 18:        Ensure          = "Present"
 19:        Name            = "Default Web Site"
 20:        State           = "Stopped"
 21:        PhysicalPath    = "C:\inetpub\wwwroot"
 22:        DependsOn       = "[File]CopyDeploymentBits"
 23:     }
 24: 
 25:     xWebsite NewWebsite
 26:     {
 27:        Ensure          = "Present"
 28:        Name            = "BlogDemo"
 29:        State           = "Started"
 30:        PhysicalPath    = "c:\inetpub\wwwroot\BlogDemo"
 31:        DependsOn       = "[xWebsite]DefaultSite"
 32:     }
 33:   }
 34: }
 35: FileCopy

DSC has many parts. More information on DSC can be found by looking at the ALM Rangers guidance which was just released (I am on this ALM Rangers team).

The first thing is, “how do I know what to put in here – what are my options?” I’m glad you asked! The components used here are either the built-in File module or the xWebAdministration module available for download and noted above. The PowerShell DSC gallery has very good documentation and source code! In order to use these modules they need to be imported, hence the Import-Dsc statement in Line 3.

Once you have this statement you have to set which node (server) to apply this configuration to (i.e. you can apply a configuration to more than one server at a time but for clarity sake we will not do that and will configure each server separately. Localhost here refers to whatever server this DSC is deployed.

So, what does “File” do? On line 7 put your cursor on the word “File” and press Ctrl + spacebar. You end up with the tooltip shown in Figure 12 (the CopyDeploymentBits statement is a label that identifies the section as you can have multiple configuration sections).

Figure 12 – File DSC tooltip

In many cases these values are straightforward. The benefit of this list is it provides you with the allowed array values. For this walkthrough the values are easy to understand.  We want to ensure that a directory (Type) is present (Ensure). The source of the files is $applicationPath (this is a built-in parameter passed by RM which points to C:\Windows\DtlDownload\[Component Name]) plus the _publishedWebsites folder. The destination is the deployment path (note that this can be completely arbitrary but again this is for simplicity sake) and we want to take all of the contents recursively.

This file is so clear that it can be passed from a development team to an operations team and there should be a common understanding of what is happening.

The next section stops the default website since our plan is to deploy our website to port 80 which the default website runs on. This is fairly straightforward but we’ve added another line down at the bottom: DependsOn. This tells DSC not to execute this section until the CopyDeploymentBits section has executed successfully. In this way we can chain a series of actions together to run in a specific order.

Finally the last section creates a new website and starts it but only if the default website was first stopped (there are options to deploy it to another port).

The last line actually triggers the DSC to create a Managed Object Framework file which causes the configuration to be executed.

Save this file as DeploymentScript1.ps1 and add it to the DeploymentScripts project in Visual Studio (and make sure that the properties on the file are such that the copy output is “Copy if Newer” so it is deployed as part of our build) and check it in.

Create the build

The build is very straightforward. Unlike previous versions of Release Management, the Release Management Service executes deployments based on triggers from the build service itself, not the build template so no special templates are needed now.

Create a new build using any template (I’m using the Default Template with no changes to the build). Because I have the solution currently opened in Visual Studio it is automatically pulling the correct solution. Save the build definition as BlogDemo OnDemand (you could trigger this as a CI build if desired). Save the build definition and jump back into the Release Management client.

Configuring the Release Process

The first part of configuring a release process which is going to use DSC and VSO is to tell the Release Management tool where the build is located. To do this we’re going to add the most basic component. Head over to the Configure Apps > Components page and click New vNext. Without making any other changes, select Builds with Application on the Sources tab as shown in Figure 11 and put a single backslash in the [Build Drop Location] field (also shown in Figure 13).

Note that it is critical you leave the name as “New Component 1”. If you’ll remember from the DSC script, this is part of the path that our application is copied to as part of the deployment.

Figure 13 – Build location setting

Save and close the new component. Now it’s time to configure the actual release process. Go to the Configure Apps > vNext Release Templates page and click New. This opens the properties page for the designer as shown in Figure 14.

Figure 14 – Release Template properties page

Rename the release process, select the release path that you created earlier and select the team project and build definition. Then check the “Can Trigger a release” checkbox and click Create.

You’ll start out with a blank slate with tabs defined for the number of environments you created in your release process as shown in Figure 15.

Figure 15 – Blank release process

The first thing to do is add the component you created previously. Right-click the Components node and select Add and then choose New Component 1. Because we are using PowerShell DSC drag a Deploy Using PS/DSC action to the Deployment Sequence and click the double arrow in the upper right to expand the activity.

Select the server from the ServerName field. Enter the UserName and Password of an account which has administrative access to the server. Select New Component 1 from the ComponentName dropdown. Finally, in the PSScriptPath field, enter DeploymentScript1.ps1.

To copy the configuration to the Test and Production environments, right-click the deployment tab (shown in Figure 16) and select Copy Deployment Sequence. Then right-click the Test and eventually Production tabs and select Paste Deployment Sequence. Change the ServerName field to the correct server and you’re done!

Figure 16 – Copying the deployment sequence

At this point you can trigger a build in Visual Studio and watch the application deploy.

In the next post in the series I’ll look at configuring a completely clean server and using more variables so as to avoid hard-coding entries in the DSC file.

To start with, you have a mobile service in Azure that looks like the following:

Figure 1: Mobile Service Dashboard

Clicking the highlighted link takes you to this page:

Figure 2: Mobile Service entry page

Clicking the Try it out button brings you to the username/password dialog. Looking back at the mobile services dashboard you’ll see the Manage Keys button at the bottom of the screen as shown in Figure 3.

Figure 3: Manage Keys

Click the button and then copy either the primary or secondary key. Now, click the Try me button and when the username/password dialog is displayed, paste the key into the password field (leave the username field blank) and click OK. And voila you get to the service as shown in Figure 4.

Figure 4: The working mobile service

In virtually all Ranger projects, experts from Microsoft, Microsoft Most Valuable Professionals and various other experts in the industry are brought in to help with a project (it is truly remarkable as we all work on these projects in our spare time). We engage with the product team(s) as needed when we need technical guidance and someone from the product team is almost always chosen as the Product Owner. This was different. Desired State Configuration (DSC) is relatively new. The guidance out there is still in development. Many blog posts, books and articles have been written about it but a lot of the details and plumbing were either missing or incomplete. While PowerShell is ubiquitous when it comes to managing Windows Servers and even Azure, its application with DSC is very new.

But Rangers aren’t shy – a bunch of us with PowerShell experience, ALM and Release Management and Operations experience committed to work on the project even though we knew relatively little about DSC. However, we’re a committed bunch when we put our minds to it – I’m proud to be a part of a group who’s vocabulary does not include the word “quit”. We all had to learn a lot and fast. The Product Team, lead by Keith Bankston as our PO spent time educating us, pointing us to all of the material available and really helping us over every bump we ran into – and we ran into a lot of them. We spent several weeks ramping up on the technology and then we started asking questions.

One thing I can’t emphasize enough is that while DSC isn’t particularly hard – that’s one of the beauties of it – the process has to be understood and in many ways that is way, way harder than the technology which we all picked up relatively easily. But from a process perspective we started asking questions and we came up with some really good answers so that you, the consumer of DSC, don’t have to think about quite as much as we did.

Is DSC absolute?

When we think of the word “State” in computer science we think of an absolute. Here is how it is defined by Wikipedia:

“…the state of a digital logic circuit or computer program is a technical term for all of the stored information, at a given instant in time, to which the circuit or program has access.”

There’s no room for fudging here. However, that’s not quite how “State” is defined when it comes to DSC because there are two instances of use in DSC.

Instance 1: A resource reserved for a single use exclusively

In this scenario, the individual configuring the system is the only individual needing a resource to be in a given state. An example of this is a server that is dedicated to support only a single application.

Instance 2: A shared resource

In this scenario one or more individuals may need the system configured for their own use but they are doing it on the same system. An example of this is a web server hosting two different applications.

The Ranger guidance talks about minimum and absolute configurations. But the question of minimum configuration led to a question that I asked during the development of the guidance – what happens when there are two different configuration files which target a single server? The answer: this situation is not possible. A single target machine can only pull its configuration (or have a configuration pushed to it) by a single machine. On the source machine, only a single configuration can apply to a target machine. This leads to the second important item.

Version control your configuration scripts

We mention this quite prominently in the guidance. However, the paragraph above describes a scenario where multiple people are accessing the same configuration file and possibly making changes to the same configuration settings. My first thought was, “well, how do you stop someone from changing your configuration?” Keith had the best answer that I’ve ever heard: “The people making changes to the configuration have to talk with each other!” Well, how often does that happen? It’s going to have to happen – and even better, the configuration is also a very easy way to communicate with the operations team.

Take advantage of DSC as means to communicate with other people using a server and the operations team. And maybe the operations team is the group that owns the configuration and helps facilitate communication with all teams using a given server.

I hope you enjoy the guidance and I’m happy to have been a part of this team – it was great, great people, great technology and a great environment.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Web.Http;
using Microsoft.WindowsAzure.Mobile.Service;
using System.Xml;
using System.Web.UI.WebControls;
using System.IO;
using System.Text;
using System.Web.Http.Description;
namespace DataService.Controllers
{
    public static class XmlTypeExtensions
    {
        private static readonly Dictionary<Type, string> TypeMappings = new Dictionary<Type, string>()     
     {                 
        {typeof(string), "string" },
        {typeof(bool), "boolean" },
        {typeof(decimal), "decimal" },
        {typeof(float), "float" },
        {typeof(double), "double" },
        {typeof(int), "int" },
        {typeof(DateTime), "datetime" },
        {typeof(Guid), "guid" }
     };
        public static string ToXmlTypeString(this Type type)
        {
            string typeString;
            if (TypeMappings.TryGetValue(type, out typeString))
            {
                return typeString;
            }
            return null;
        }
    }
    [AuthorizeLevel(Microsoft.WindowsAzure.Mobile.Service.Security.AuthorizationLevel.Anonymous)]
    public class WadlController : ApiController
    {
        public ApiServices Services { get; set; }
        // GET api/Wadl
        public HttpResponseMessage Get()
        {
            StringBuilder sb = new StringBuilder();
            using (XmlWriter xw = XmlWriter.Create(new StringWriter(sb)))
            {
                xw.WriteStartElement("application", "http://wadl.dev.java.net/2009/02");
                xw.WriteStartElement("resources", "http://wadl.dev.java.net/2009/02");
                xw.WriteAttributeString("base", "https://" + Services.Settings.Name + ".azure-mobile.net");
                var apiDescriptions = Configuration.Services.GetApiExplorer().ApiDescriptions;
                foreach (System.Web.Http.Description.ApiDescription api in apiDescriptions)
                {
                    xw.WriteStartElement("resource", "http://wadl.dev.java.net/2009/02");
                    xw.WriteAttributeString("path", api.RelativePath);
                    xw.WriteStartElement("doc", "http://wadl.dev.java.net/2009/02");
                    xw.WriteAttributeString("title", api.Documentation);
                    
                    xw.WriteEndElement();
                    xw.WriteStartElement("method", "http://wadl.dev.java.net/2009/02");
                    xw.WriteAttributeString("name", api.HttpMethod.Method);
                    xw.WriteStartElement("request", "http://wadl.dev.java.net/2009/02");
                    foreach (var param in api.ParameterDescriptions)
                    {
                        xw.WriteStartElement("param", "http://wadl.dev.java.net/2009/02");
                        xw.WriteAttributeString("name", param.Name);
                        xw.WriteAttributeString("style", "template");
                        xw.WriteStartAttribute("type");
                        xw.WriteQualifiedName(param.ParameterDescriptor.ParameterType.ToXmlTypeString() ?? "string", "http://www.w3.org/2001/XMLSchema");
                        xw.WriteEndAttribute();
                        xw.WriteAttributeString("required", "true");
                        xw.WriteEndElement();
                    }
                    var model = Configuration.GetHelpPageApiModel(api.GetFriendlyId());
                    
                    xw.WriteEndElement(); // request
                    xw.WriteEndElement(); // method
                    xw.WriteEndElement(); //resource
                }
                xw.WriteEndElement(); //resources
                xw.WriteEndElement(); //application
            }
            HttpResponseMessage response = Request.CreateResponse();
            response.Content = new StringContent(sb.ToString(), UTF8Encoding.Unicode, "text/xml");
            return response;
        }
    }
}

Add this controller to your Web API application and your done. At this point you can import your API into API Management.

CORS stands for Cross-Origin-Resource-Sharing (detailed information can be found at the W3C here). I’ll give it to you in a nutshell here. It is a security mechanism to ensure that data designed to be used for a domain stays in that domain and can’t be used outside of it. In other words it’s a security mechanism. In addition, whether or not you are allowed to access the data is accomplished by a response, not request, header. That means that you cannot make a request to a server for it to serve up data that is protected against CORS. It is the source server that must allow a CORS request.

So, back to the story. I built a small website that retrieved the data from API Management and continued to get a CORS error. I tried everything on the client I could to make it work. I finally found the W3C documentation which pointed me to the server. I checked API Management and I couldn’t find anything on CORS… until I found the Policy tab.

Clicking the Policies tab on the left hand side (highlighted) brings us to this screen:

On the Policies tab, select the Product, API and specific call. You’ll notice on the right side the CORS policy (as well as others that I would urge you to explore). Put your cursor after the <inbound> tag and above the <base/> tag then click the CORS policy (note that this is an inbound request because the client makes a request of the server and the server sends the response saying yes you can have it). There are some other default CORS options but the way I configured it was to allow requests from anywhere because it was simply a demonstration API.

This should get you past your CORS problem.

Creating the basic MVC application

• Open Visual Studio and select File > New Project
• Select Web > ASP.NET Web Application

• Rename it something unique and click OK
• Click Change Authentication

• Select Organizational Accounts
• Enter your Azure domain name
• You can either leave the Access level at Single Sign On or change it as desired – in this case I changed it to Single Sign On, Read directory data
• Expand the More Options and make sure the App ID URI is filled in appropriately – if it is, don’t make any changes to it
  • If it is not filled in, enter “https://[domain as listed in the domain field]/[application name]

• Click OK
• Make sure Host in the cloud is checked and Website is selected in the drop-down
• Click OK

• On the Configure Azure Website dialog, leave the defaults as is (or change the server location if necessary) and click OK
  • Note that I’m walking through the default settings – this isn’t sufficient as you’ll see later on

Very shortly the website will be provisioned (but not published).

Everything’s done – like every good developer we want to run what we just created to get that sense of satisfaction so let’s do that.

Running the default web application

Let’s see how well this works:

• Press F5

• A certificate warning page is displayed because by default it puts SSL in place and it uses a self-signed certificate. No problem here, click Continue to this website (not recommended).
• Next the Sign in to Azure Active Directory page is displayed – so far so good!

• Click the user to sign in as (if you don’t have an AD user (versus your Microsoft credentials) skip this part – we’ll add some credentials in a little bit).

• As a side note, if you do enter your Microsoft credentials you’ll get this little gem (we’ll come back and address this in a bit):

• Enter your username and password and click Sign in

• Lo and behold we have access to our application

Sweet! Now let’s publish it…

• Right-click the project name in Solution Explorer and select Publish
• Without making any changes (nothing you can do can change the outcome at this point) click Publish
• After the application publishes it will automatically launch. Crossing our fingers at this point, closing our eyes and praying a little bit we open them up and…

DOH!

This isn’t particularly helpful so let’s modify the web.config to get some additional information. Adding the section indicated and re-running it (which by the way you don’t have to republish to do this – you can simply edit the web.config on the server by looking in Server Explorer as shown here:

Double-click the web.config file to open it and you’ll note that it says [Remote] in the document tab. After adding the following code:

<customErrors Mode="Off" />

Save the web.config, right-click on the website in Solution Explorer and click Browse. Now we get the following back (instead of a screenshot of an entire stack trace, I’m just going to show the first line which is the only important line here):

[SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 52 - Unable to locate a Local Database Runtime installation. Verify that SQL Server Express is properly installed and that the Local Database Runtime feature is enabled.)]

Um, huh? What database? We didn’t add a database and the default MVC application doesn’t have anything that uses a database. Let’s check out the web.config and see what’s accessing a database.

Looking at the web.config we see the following entry:

  <connectionStrings>
    <add name="DefaultConnection" connectionString="Data Source=(LocalDb)\v11.0;AttachDbFilename=|DataDirectory|\aspnet-ADIntegrationDemo-20140909062518.mdf;Initial Catalog=aspnet-ADIntegrationDemo-20140909062518;Integrated Security=True" providerName="System.Data.SqlClient" />
  </connectionStrings>

What’s this about? Well, if you’ve written MVC applications you’ll know this is where VS stores data related to ASP.Net related authentication. Well, this is the same type of situation but different.

What the application is trying to do is to grab two specific pieces of information so it can authenticate the client using SAML assertions against Azure AD. The problem is that once you deploy it to the web, this database is not accessible anymore. So how do you fix it and what goes in the database.

Fixing the problem

In order to fix the problem we have to look at the local database. To do that, do the following:

• In Server Explorer, expand the Data Connections node and you should already see a connection to the application’s database as shown here:

• If it doesn’t exist, add it as follows:
  • Right-click Data Connections and select Add Connection
  • Change the data source to Microsoft SQL Server Database File
  • Browse to the App_Data folder and select the mdf file there
  • Click OK
• Expanding the table nodes we find a very simple schema
• The important tables are the IssuingAuthorityKeys and the Tenants table. Right-click each table and select Show Data

Each table has a single column and a single row. These tables need to be moved to a database accessible to the ASP.Net MVC application running in Azure. For our purposes we’ll use a SQL Database. For simplicities sake I’ve created the SQL statements needed to be executed against a SQL database. Replace the two insert statements with the values found in each of the respective tables as shown (for example):

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[__MigrationHistory] (
    [MigrationId]    NVARCHAR (150)  NOT NULL PRIMARY KEY,
    [ContextKey]     NVARCHAR (300)  NOT NULL,
    [Model]          VARBINARY (MAX) NOT NULL,
    [ProductVersion] NVARCHAR (32)   NOT NULL
);
GO
CREATE TABLE [dbo].[IssuingAuthorityKeys] (
    [Id] NVARCHAR (128) PRIMARY KEY NOT NULL
);
GO
CREATE TABLE [dbo].[Tenants] (
    [Id] NVARCHAR (128) NOT NULL PRIMARY KEY
);
GO
INSERT INTO [dbo].[IssuingAuthorityKeys]
           ([Id])
     VALUES
           ('92B88C3DD981BF1EBCB244FCFA63C007706C79E0')
GO
INSERT INTO [dbo].Tenants
           ([Id])
     VALUES
           ('67c3a3d5-b264-4328-ade6-7a7b81136117')
GO

• If you don’t have an existing SQL database, create one in the Azure Management Portal or Server Explorer
• Select the SQL Databases node and highlight the appropriate database (in my example I’m using a customer_demonstrator database) and click Manage at the bottom of the page

• Provide the login credentials for the database and click Log On

• Click New Query in the resulting window

• Paste the script in the sql window and click run (after replacing the issuing authority key and tenants key)

Now we have a table that holds the appropriate data needed by the application. Let’s try publishing the application once again. This time however we’re going to make a change to the connection string (make sure to grab the connection string for the database while you’re in the Azure portal).

• On the Publish Web page, select the Settings tab:

• Change the connection string using the connection string copied from the server (remember to put your password in)

Now, I’ll save you the suspense – in our default configuration if we publish right now it will fail – there’s one last problem. By default SSL is not set up yet the configuration by default requires SSL (you’ll get the following error message: ID1059: Cannot authenticate the user because the URL scheme is not https and requireSsl is set to true in the configuration, therefore the authentication cookie will not be sent. Change the URL scheme to https or set requireSsl to false on the cookieHandler element in configuration.)

• Click Close on the Publish Web dialog
• Open the web.config file and edit the following like:

<cookieHandler requireSsl="true" />

• Change “true” to “false” – now publish the application

Doh!!! We are just not having a lot of luck with this. What’s the problem now? Looking at the URL we can see that the callback for the logon takes us back to https://localhost:4430 only we aren’t running on our local machine anymore – we want to go to http://adintegrationdemo.azurewebsites.com. Let’s take a look at what was done in Azure under the covers. The fix is simple but it’s important to understand the end-to-end flow.

Azure Active Directory

Bet you thought we weren’t going to make it here…

When you first sign up for Azure, a default active directory instance is created for you. Navigating to the Azure Management Portal it looks like this:

Clicking the Default Directory takes you to this page:

Users can be added on the Users page (and obviously Groups on the groups page) but we’re going to focus on the Applications page. Clicking that we see the following (you’ll have only one application in there – the demo that we just put together most likely):

Drill into the appropriate application and click Configure. Update the Sign In URL and the Reply URL to the correct URL on Azure (like http://ADIntegrationDemo.azurewebsites.net. Click Save.

In a new browser window, enter the URL. And lo and behold we have a working application!

Now, back to the problem we ran into very early on. For some reason (no, I don’t know why) when you try to log on with a Microsoft account the first time it returns null. If you try logging on a second time it works fine. I haven’t been able to puzzle through this but a simple change to the _LoginPartial.cshtml page – it doesn’t handle nulls at all. At the following to the top of the _LoginPartial page:

@{
    var user = "Null User";
    if (!String.IsNullOrEmpty(User.Identity.Name))
    {
        user = User.Identity.Name;
    }
}

And update the user action link to the following:

@Html.ActionLink(user, "UserProfile", "Home", routeValues: null, htmlAttributes: null)

Republish the application and now it will handle any situation.

Enjoy!

I spoke at VS Live! Redmond a couple of weeks ago and Jason Zander, a Corporate Vice President in the Azure group did just an amazing demo of the new Event Hub feature and I wanted to duplicate it from the ground up. Here’s the basic setup. The demo consists of a web client that sends a message (and the browser being used) to an event hub. The event hub passes it on to a queue (this is mostly because it makes the demo much more fun) and then a client reads out of the queue. The best part is that it happens almost instantaneously.

The main thing to note is that Queues are really designed to be read from sequentially (although you can certainly peek at any message and Event Hubs can have a message processed by many systems at once. The differences between Queues, Topics and Event Hubs can be found here: http://msdn.microsoft.com/en-us/library/azure/dn789972.aspx. This example doesn’t show these differences but instead shows the basic techniques of writing and reading to/from hubs and queues. Because Event Hubs are so new, the documentation is extremely limited right now (it’s still in Preview right now in fact).

With all of this said, hopefully this will be helpful and you’ll be able to adapt it for your own purposes… Before beginning I would also like to call out Jeff Wilcox for helping provide a couple of key pieces/ideas for the solution.

Solution Overview

The solution consists of an Azure Website which allows users to enter a message. The message is passed to an Event Hub. An Azure Cloud Service Worker processes the Event Hub and passes the message to an Azure Queue. Finally a Wpf application reads the Azure Queue and displays an aggregation of the data to the end user. Conceptually this is similar to a messaging app where 500 people send one person a message at the same time. In more practical terms thing of an Internet of Things (IoT) where thousands of devices send messages every second and that data has to be processed and stored and displayed. So there is a potential real word application for this type of structure.

Creating the Event Hub

In the Azure management portal select New > Service Bus > Event Hub as follows:

Provide a name (DemoApp in this case):

If you don’t already have an existing service bus namespace, it will create one for you (it is automatically appended with “-ns”). This is a mistake that I made originally – my Event Hub and my Queue are in two different namespaces. This is not required – I just made it more complicated for myself.

Next, provide a partition count and days for retention:

The partition count is a value between 1 and 32. Click the check mark and the hub will be created. After the hub is created, make sure to click the Connection Information at the bottom of the hub page to grab the string to allow the client applications to send data to and read data from the hub.

The Web Client – Writing to the Hub

Before attempting to use an Event Hub you need to add a reference to the right package. To do this, right-click the solution and select Manage NuGet Packages… Search for the Microsoft Azure Service Bus Event Hub – EventProcessorHost. Once you add that you just need to update or add a key with the Event Hub connection data to the web.config file. The page that takes the message is very simple and looks like the following:

This is a standard MVC application which I added an EventHubs page to. The code in the EventHub.cshtml page is as follows:

@{
    ViewBag.Title = "Event Hubs";
}
<p style="padding-top:20px">
    Enter a message to send: <input type="text" id="message" name="eventMessage" /><input type="button" id="sendMessageButton" value="Send Message" />
</p>
<script type="text/javascript">
    $("#sendMessageButton").click(function() {
        $.getJSON(getHomeUrl() + "home/SendMessage/?" + $.param({ eventMessage: $("#message").val(), browserInfo: $.browser.name }), "", function () {
            $("#message").val("");
        })
    })
</script>

The main work on this page is done by the AJAX call which sends the data to the server. There’s nothing related to the Event Hubs in this page – it’s just standard jQuery.

The Home controller gets this call and does the real work. The main piece is the SendMessage function (be sure to including using Microsoft.ServiceBus.Messaging).

public JsonResult SendMessage(string eventMessage, string browserInfo)
{
  //Put all of the data into a single class
  EventHubData ehd = new EventHubData() { Message = eventMessage, BrowserInfo = browserInfo };
  //serialize it
  var serializedString = JsonConvert.SerializeObject(ehd);
  //Create the Event Data object so the data and partition are known
  EventData data = new EventData(Encoding.Unicode.GetBytes(serializedString))
  {
    PartitionKey = "0"
  };
  //Get a reference to the event hub
  EventHubClient client = EventHubClient.CreateFromConnectionString(ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"], "DemoApp");
  //Send the data
  client.SendAsync(data);
  //Return to the client
  return Json("", JsonRequestBehavior.AllowGet);
}

There is a single class which backs the data and it simply encapsulate the information to send in the message so it can be easily deserialized on the back end. Next the object is serialized and an EventData object is created. The PartitionKey can be whatever you want to call the partition – there are a potential of 32 partitions and you can set this value when you create your event hub – they are referenced by ordinal number so I just lacked incredible creativity by naming it “0”. The next line gets the reference to the event hub in azure and then finally the message is send asynchronously.

That’s all there is to it. As you start sending messages you’ll see the following type of view in Azure (note that this doesn’t update instantaneously):

So, we can now add data to the event hub! Obviously that in itself isn’t particularly useful so without further ado, moving on!

Creating the Queue

This is where you can do it better than I did it. Select the DemoApp-ns namespace from the Azure Management Portal and click the Queues tab.

Click Create a new Queue. Provide a queue name and select an appropriate region.

Click the next arrow and set any additional settings needed then click the check mark.

These settings are discussed more in this great tutorial on Queues: http://social.technet.microsoft.com/wiki/contents/articles/2173.azure-and-sql-database-tutorials-tutorial-4-using-windows-azure-worker-role-and-windows-azure-queue-service.aspx.

Remember to grab your connection string!

Now here’s the tricky thing – having a queue isn’t enough. A queue has to store its data somewhere. That somewhere is in a storage service. And not only in a storage service – but in a container in a storage service. Creating this is trivial but if you aren’t aware of it (like I wasn’t because I tend to “do” before I “read” – isn’t that how it is with everyone in the tech industry?) it’s hard to figure it. To create a storage account, go to the storage management page and select New. Create the storage service then click the Containers tab.

Select new at the bottom and provide a name and click the check mark – you’re done! For this you’re going to need a bit more information to connect to it. On the dashboard page for the storage account you’ll see the following:

Grab not only the URL for Queues but also click the Manage Access Keys at the bottom and grab the storage account name and the primary access key as you’ll need these to write to the queue.

Cloud Service Worker – Reading from the Hub

At this point we have data in the hub, now we need to process it. For this we’ll create a cloud service which constantly monitors the hub and processes any data that is received.

This took me a while to figure out as it isn’t obvious – you really do need to do it the way it is described in this post – I spent hours trying to work around what I thought was an overly involved process. It turns out that the separation of concerns really is the smartest way to manage this.

The key to reading from an Event Hub is the IEventProcessor interface. After you create a new Cloud Service worker role, add a reference to the event hub (as in the above sample – and also add a reference to the Azure Service Bus because you’ll need that to access the queue). Then add the following class:

  1: using Microsoft.ServiceBus.Messaging;
  2: using Microsoft.WindowsAzure.Storage;
  3: using Microsoft.WindowsAzure.Storage.Queue;
  4: using System;
  5: using System.Collections.Generic;
  6: using System.Diagnostics;
  7: using System.Linq;
  8: using System.Text;
  9: using System.Threading.Tasks;
 10: using Newtonsoft.Json.Linq;
 11: using Microsoft.ServiceBus;
 12: 
 13: namespace WorkerRole1
 14: {
 15:     public class EventProcessor : IEventProcessor
 16:     {
 17:         PartitionContext partitionContext;
 18:         Stopwatch checkpointStopWatch;
 19: 
 20:         QueueClient queue;
 21: 
 22:         public EventProcessor()
 23:         {
 24:             queue = QueueClient.CreateFromConnectionString("[your connection string here]","[your queue name here]");
 25:         }
 26: 
 27:         public async Task CloseAsync(PartitionContext context, CloseReason reason)
 28:         {
 29:             Console.WriteLine(string.Format("Processor Shuting Down.  Partition '{0}', Reason: '{1}'.", this.partitionContext.Lease.PartitionId, reason.ToString()));
 30:             if (reason == CloseReason.Shutdown)
 31:             {
 32:                 await context.CheckpointAsync();
 33:             }
 34:         }
 35: 
 36:         public Task OpenAsync(PartitionContext context)
 37:         {
 38:             Console.WriteLine(string.Format("SimpleEventProcessor initialize.  Partition: '{0}', Offset: '{1}'", context.Lease.PartitionId, context.Lease.Offset));
 39:             this.partitionContext = context;
 40:             this.checkpointStopWatch = new Stopwatch();
 41:             this.checkpointStopWatch.Start();
 42:             return Task.FromResult<object>(null);
 43:         }
 44: 
 45:         public async Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
 46:         {
 47:             try
 48:             {
 49:                 foreach (EventData eventData in messages)
 50:                 {
 51:                     Console.WriteLine("Processing event hub data...");
 52: 
 53:                     string key = eventData.PartitionKey;
 54: 
 55:                     string data = System.Text.Encoding.Unicode.GetString(eventData.GetBytes());
 56:                     try
 57:                     {
 58:                         var json = JObject.Parse(data);
 59:                         string text = json["Message"].ToString();
 60:                         string agent = json["BrowserInfo"].ToString();
 61: 
 62:                         if (queue != null)
 63:                         {
 64:                             
 65:                             await queue.SendAsync(new BrokeredMessage((agent + "##" + text)));
 66: 
 67:                             Trace.TraceInformation("Added to queue: " + agent);
 68:                         }
 69:                     }
 70:                     catch(Exception exx)
 71:                     {
 72:                         Console.WriteLine(exx.Message);
 73:                     }
 74: 
 75:                     Console.WriteLine(string.Format("Message received.  Partition: '{0}', Device: '{1}'",
 76:                         this.partitionContext.Lease.PartitionId, key));
 77:                 }
 78: 
 79:                 //Call checkpoint every 5 minutes, so that worker can resume processing from the 1 minutes back if it restarts.
 80:                 // Doing every ONE MINUTE now.
 81:                 if (this.checkpointStopWatch.Elapsed > TimeSpan.FromMinutes(1))
 82:                 {
 83:                     await context.CheckpointAsync();
 84:                     lock (this)
 85:                     {
 86:                         this.checkpointStopWatch.Reset();
 87:                     }
 88:                 }
 89:             }
 90:             catch (Exception exp)
 91:             {
 92:                 Console.WriteLine("Error in processing: " + exp.Message);
 93:             }
 94:         }
 95:     }
 96: }
 97: 

Much of this is boilerplate code so I’ll cover the parts that require the most explanation as it relates to event hubs and queues.

All Console statements are for debugging purposes only – I left them in because you may need them. Being that this is an Azure worker role, these can be made visible in many different ways including using AppInsights.

Line 24 creates the connection to the queue where messages will be passed from the Event Hub. Obviously it’s better to read this from the web.config file. No, I don’t know why I did it this way – it was late :)

Line 45-77, the ProcessEventsAsync method does the bulk of the work and it monitors the event hub (the next code block shows where the connection to this comes from). The EventData contains the message from the client. The first task is to get the data out of the message (Line 55) and the next task is to deserialize it – in this case converting it from JSON to variables we can use (Lines 58-60). Finally it is written to the Queue using a BrokeredMessage object (Line 62-68).

This is all that’s required to read data out of the event hub and place it into the queue. The worker role class manages the lifetime of this object and passes in the necessary configuration information. This class is shown in its entirety here – but most of it is boilerplate – I only changed the OnStart method.

  1: using System;
  2: using System.Collections.Generic;
  3: using System.Diagnostics;
  4: using System.Linq;
  5: using System.Net;
  6: using System.Threading;
  7: using System.Threading.Tasks;
  8: using Microsoft.WindowsAzure;
  9: using Microsoft.WindowsAzure.Diagnostics;
 10: using Microsoft.WindowsAzure.ServiceRuntime;
 11: using Microsoft.WindowsAzure.Storage;
 12: using Microsoft.ServiceBus.Messaging;
 13: 
 14: namespace WorkerRole1
 15: {
 16:     public class WorkerRole : RoleEntryPoint
 17:     {
 18:         private readonly CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();
 19:         private readonly ManualResetEvent runCompleteEvent = new ManualResetEvent(false);
 20: 
 21:         public override void Run()
 22:         {
 23:             Trace.TraceInformation("WorkerRole1 is running");
 24: 
 25:             try
 26:             {
 27:                 this.RunAsync(this.cancellationTokenSource.Token).Wait();
 28:             }
 29:             finally
 30:             {
 31:                 this.runCompleteEvent.Set();
 32:             }
 33:         }
 34: 
 35:         public override bool OnStart()
 36:         {
 37:             // Set the maximum number of concurrent connections
 38:             ServicePointManager.DefaultConnectionLimit = 12;
 39: 
 40:             // For information on handling configuration changes
 41:             // see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
 42: 
 43:             bool result = base.OnStart();
 44: 
 45:             Trace.TraceInformation("WorkerRole1 has been started");
 46: 
 47:             string storage = "DefaultEndpointsProtocol=https;AccountName=[your account name here];AccountKey=[your key here";
 48:             string serviceBus = "Endpoint=sb://[your namespace here].servicebus.windows.net/;SharedAccessKeyName=[your key name here];SharedAccessKey=[your key here]";
 49:             string eventHubName = "demoapp";
 50: 
 51:             EventHubClient client = EventHubClient.CreateFromConnectionString(serviceBus, eventHubName);
 52:             Trace.TraceInformation("Consumer group is: " + client.GetDefaultConsumerGroup().GroupName);
 53: 
 54:             _host = new EventProcessorHost("singleworker", eventHubName, client.GetDefaultConsumerGroup().GroupName, serviceBus, storage);
 55: 
 56:             Trace.TraceInformation("Created event processor host...");
 57: 
 58:             return result;
 59:         }
 60: 
 61:         private EventProcessorHost _host;
 62: 
 63:         public override void OnStop()
 64:         {
 65:             Trace.TraceInformation("WorkerRole1 is stopping");
 66: 
 67:             _host.UnregisterEventProcessorAsync().Wait();
 68: 
 69:             this.cancellationTokenSource.Cancel();
 70:             this.runCompleteEvent.WaitOne();
 71: 
 72:             base.OnStop();
 73: 
 74:             Trace.TraceInformation("WorkerRole1 has stopped");
 75:         }
 76: 
 77:         private async Task RunAsync(CancellationToken cancellationToken)
 78:         {
 79:             await _host.RegisterEventProcessorAsync<EventProcessor>();
 80: 
 81:             // TODO: Replace the following with your own logic.
 82:             while (!cancellationToken.IsCancellationRequested)
 83:             {
 84:                 //Trace.TraceInformation("Working");
 85:                 await Task.Delay(1000);
 86:             }
 87:         }
 88:     }
 89: }
 90: 

Lines 47-49 set the account information and identifying names for the storage, service bus and queue which are required by the EventProcessor class. Line 51 creates the EventHub client and line 54 starts the process. Everything else is there to manage the lifetime of the EventProcessor object. This can be published to Azure and it will start reading data from the EventHub immediately upon deployment. The queue dashboard should start to look like the following:

WPF Client – Reading from the Queue

Finally we get to reading data from the queue. It’s important to note that you can do this a million different ways. You can use Node.js or SignalR to do real-time web page applications, pass it to other applications for additional processing (which you can do from the event hubs as well) or virtually anything else you want. You can read it with an Android or iOS app, put it into a tabular model for use with PowerView or hundreds of other applications.

I chose WPF because it showed a mix of different technologies and I’m not as comfortable with Node or SignalR (I’m still learning both of them).

The key to this application is a class called QueueReader – everything is used for the presentation of data.

  1: using Microsoft.ServiceBus.Messaging;
  2: using System;
  3: using System.Collections.Generic;
  4: using System.Linq;
  5: using System.Text;
  6: using System.Threading;
  7: using System.Threading.Tasks;
  8: using System.Windows;
  9: 
 10: namespace WpfQueueReader
 11: {
 12:     class QueueReader
 13:     {
 14:         QueueClient _queue;
 15:         Thread _thread;
 16:         QueueData _persistentData = new QueueData();
 17: 
 18:         public delegate void MessageReceivedEventHandler(object sender, MessageEventArgs e);
 19:         public event MessageReceivedEventHandler MessageReceived;
 20: 
 21:         public void MonitorQueue()
 22:         {
 23:             _queue = QueueClient.CreateFromConnectionString("[your connection string here]", "[your queue name here]");
 24: 
 25:             BrokeredMessage message = null;
 26:             while (true)
 27:             {
 28:                 try
 29:                 {
 30:                     //receive messages from Queue
 31:                     message = _queue.Receive(TimeSpan.FromSeconds(5));
 32:                     if (message != null)
 33:                     {
 34:                         string m = message.GetBody<string>();
 35:                         string[] values = m.Split(new string[] { "##" }, 2, StringSplitOptions.None);
 36: 
 37:                         _persistentData.Message = values[1];
 38:                         _persistentData.Browsers.Add(values[0]);
 39: 
 40:                         Application.Current.Dispatcher.BeginInvoke(new ThreadStart(() => OnMessageReceived(_persistentData)), null);
 41: 
 42:                         // Further custom message processing could go here...
 43:                         message.Complete();
 44:                     }
 45:                     else
 46:                     {
 47:                         Thread.Sleep(TimeSpan.FromSeconds(5));
 48:                     }
 49:                 }
 50:                 catch(OperationCanceledException)
 51:                 {
 52:                     _queue.Close();
 53:                     
 54:                 }
 55:             }
 56:         }
 57: 
 58:         public void EndMonitoring()
 59:         {
 60:             _queue.Close();
 61:         }
 62: 
 63:         protected void OnMessageReceived(QueueData data)
 64:         {
 65:             if (MessageReceived != null)
 66:             {
 67:                 MessageReceived(this, new MessageEventArgs(data));
 68:             }
 69:         }
 70: 
 71:         public void StartTask()
 72:         {
 73:             ThreadStart threadStart = new ThreadStart(MonitorQueue);
 74:             _thread = new Thread(threadStart);
 75:             _thread.Start();
 76:         }
 77: 
 78:         public void EndTask()
 79:         {
 80:             _thread.Abort();
 81:         }
 82:     }
 83: }

Most of the functionality in this class deals with the fact that it’s running on a background thread. The main portion of this class is the MonitorQueue method. Remember to replace the connection string with your own connection information.

The MonitorQueue method does the following:

1. Creates a connection to the queue
2. Starts an endless loop to check the queue (note that this is a cheap way of doing it and I didn’t want to go crazy just for demo purposes – an excellent article on a better way to do this can be found here: http://www.developerfusion.com/article/120619/advanced-scenarios-with-windows-azure-queues/)
3. Check the queue
4. If there is a message,
1. Get the data from the message
2. Add it to a class
3. Raise an event to the UI thread with the data
4. Check the queue again
5. If there isn’t a message,
1. Go back to sleep for five seconds before checking the queue again

In the UI, the entire class for the single window looks like the following:

  1: using System;
  2: using System.Collections.Generic;
  3: using System.Linq;
  4: using System.Text;
  5: using System.Threading.Tasks;
  6: using System.Windows;
  7: using System.Windows.Controls;
  8: using System.Windows.Data;
  9: using System.Windows.Documents;
 10: using System.Windows.Input;
 11: using System.Windows.Media;
 12: using System.Windows.Media.Imaging;
 13: using System.Windows.Navigation;
 14: using System.Windows.Shapes;
 15: 
 16: namespace WpfQueueReader
 17: {
 18:     /// <summary>
 19:     /// Interaction logic for MainWindow.xaml
 20:     /// </summary>
 21:     public partial class MainWindow : Window
 22:     {
 23:         QueueReader _reader = new QueueReader();
 24: 
 25:         public MainWindow()
 26:         {
 27:             InitializeComponent();
 28:             _reader.MessageReceived += _reader_MessageReceived;
 29:             _reader.StartTask();
 30:         }
 31: 
 32:         void _reader_MessageReceived(object sender, MessageEventArgs e)
 33:         {
 34:             TextBlock block1 = new TextBlock();
 35:             
 36:             block1.Text = e.Data.Message;
 37: 
 38:             spMessages.Children.Add(block1);
 39:             spBrowsers.DataContext = null;
 40:             spBrowsers.DataContext = e.Data.Browsers;
 41:         }
 42: 
 43:         private void btnExit_Click(object sender, RoutedEventArgs e)
 44:         {
 45:             _reader.EndMonitoring();
 46:             _reader.EndTask();
 47:             this.Close();
 48:         }
 49: 
 50:         private void Window_Closed(object sender, EventArgs e)
 51:         {
 52:             _reader.EndMonitoring();
 53:             _reader.EndTask();
 54:         }
 55:     }
 56: }
 57: 

The constructor here (Lines 25-30) start the background process executing and wires up the event to the UI. The _reader_MessageRecieved method gets the data and simply appends values and re-binds the a collection of browsers to display. The end result of all of this looks like the following:

In a real time fashion, data is transmitted from the browser, to Azure and back to the WpfQueueReader.

Now, set your imaginations to work because this has real world applications! And it’s never been this easy to build a scalable, distributed application.

1. Create a new C# (or VB) Unit Test Project
2. From the SQL Server 2014 (or SQL Server 2012 – but for SQL Server 2012 this is all built into Visual Studio) feature pack download page (http://www.microsoft.com/en-us/download/details.aspx?id=42295) select Download and select the 32 or 64 bit version of ADOMD.NET
3. Add a reference to the Microsoft.AnalysisServices.AdomdClient.dll (located in C:\Program Files\Microsoft.NET\ADOMD.NET\120)
4. Add a reference to System.Data
5. Import the namespace Microsoft.AnalysisServices.Adomd

You’re ready to go. The following is a simple unit test again the TFS Analysis cube:

  1: using System;
  2: using Microsoft.VisualStudio.TestTools.UnitTesting;
  3: using Microsoft.AnalysisServices.AdomdClient;
  4: using System.Diagnostics;
  5: 
  6: namespace dwUnitTests
  7: {
  8:     [TestClass]
  9:     public class UnitTest1
 10:     {
 11:         [TestMethod]
 12:         public void TestMethod1()
 13:         {
 14:             //Create the connection to the cube using integrated authentication
 15:             AdomdConnection connection = new AdomdConnection("Data Source=Olympia;Initial Catalog=Tfs_Analysis");
 16:             //Create the command and initialize it
 17:             AdomdCommand command = new AdomdCommand();
 18:                         
 19:             //Provide the MDX to retrieve data from the cube
 20:             command.CommandText = "SELECT NON EMPTY { [Measures].[Work Item Count] } ON COLUMNS, " 
 21:                 + "NON EMPTY { ([Work Item].[System_WorkItemType].[System_WorkItemType].ALLMEMBERS ) } ON ROWS "
 22:                 + "FROM [Team System]";
 23: 
 24:             //Set the connection for the command
 25:             command.Connection = connection;
 26: 
 27:             //Open a connection to the cube
 28:             connection.Open();
 29: 
 30:             //Retrieve the data
 31:             CellSet cs = command.ExecuteCellSet();
 32: 
 33:             //Close the connection
 34:             connection.Close();
 35: 
 36:             //Validate the data
 37:             Assert.IsTrue(cs.Axes[1].Positions[0].Members[0].Name == "[Work Item].[System_WorkItemType].&[Bug]");
 38:             Assert.IsTrue((int)cs.Cells[0].Value == 3);
 39:             Assert.IsTrue(cs.Axes[1].Positions[1].Members[0].Name == "[Work Item].[System_WorkItemType].&[Product Backlog Item]");
 40:             Assert.IsTrue((int)cs.Cells[1].Value == 1);
 41:             Assert.IsTrue(cs.Axes[1].Positions[2].Members[0].Name == "[Work Item].[System_WorkItemType].&[Task]");
 42:             Assert.IsTrue((int)cs.Cells[2].Value == 4);
 43:             Assert.IsTrue(cs.Axes[1].Positions[3].Members[0].Name == "[Work Item].[System_WorkItemType].&[Test Case]");
 44:             Assert.IsTrue((int)cs.Cells[3].Value == 2);
 45:         }
 46:     }
 47: }
 48: 

A couple of things to make life easier. I don’t write MDX that often and I can never remember the syntax when I have to. I had though that the most difficult part of this would be creating the select statement (line 20). It turns out you can do it without any work at all. To generate the MDX statement:

• Open SQL Server Management Studio and connect to your cube
• Browse the cube and create the dataset that you want returned as shown in Figure 1

Figure 1 – Browse view of an Analysis Services cube

• Next, click the Design button on the right side of the toolbar to unselect it which results in the view shown in Figure 2

Figure 2 – MDX View to provide the result set shown in Figure 1

• At this point, just copy the MDX, remove the formatting options (including and everything following the “Cell Properties”) and you’re done!

A word of caution on the Assert statements. I am using Axes and Positions to get the values here because I am using ExecuteCellSet to retrieve the data. There are other methods of retrieving the data so be careful when constructing the assert statements. It took me a bit of trial and error to be able to drill through the object model in debug mode to figure out what I should be testing.

And that’s it – you can now perform quality checks on the results of a cube to ensure that the cube is processing correctly. This probably does not apply to very many people but for those that it does apply to, using proper software development techniques for cube development is every bit as important as using these techniques elsewhere. Maybe more so because end users rely on data from cubes to drive business decisions – if this data is wrong…