In this post I’ll explain how you can use AngularJS together with ASP.NET to create Angular apps together with ASP.NET Web APIs. For this post I used the Community Editions of Visual Studio 2015 with MVC 4 and .NET Framework 4.6. I used the latest production version of Angular 1. I also used TypeScript and Bootstrap. There are many great tutorials on the web for each of these technologies. Together Bootstrap, Angular and TypeScript (the ‘BAT’ stack) complement each other to create a great client-side stack for building modern web applications.

The role of ASP.NET is to host our server-side logic. There are two important parts to this: Web APIs are used to expose REST endpoints which are backed by .NET code. Additionally we’ll leverage ASP.NET’s rich identity framework to implement security. ASP.NET’s identity framework supports many powerful features including social login, two-factor authentication and active directory support.

This tutorial assumes you’re already familiar with the basics of ASP.NET, AngularJS and Bootstrap. If you want to learn more about any of these topics. there are lots of other good tutorials on the net.

At the end of this tutorial you’ll have a skeleton Angular application backed with a simple Web API and security enabled so that users of the app will be presented a login experience. You can find the source code for this project on GitHub: https://github.com/andreasderuiter/bat-mvc.

Something to understand before we get started

In this tutorial we will not implement a login page within the Angular app itself. Instead we’ll create the boilerplate for an AngularJS app which will integrate with the existing login pages which are part of the standard MVC project template.

At first, this approach may seem non-optimal because the whole point of a SPA application is to embed all functionality in a single page. In my perspective however, the goal is to create a fluid and responsive user experience and SPA is just a means to an end. The fact that a user will temporarily be redirected to a login page, which will add 1 second of delay once per user session, is a small price to pay for the benefits we get from using this approach:

• The solution is very simple and easy to understand
• No extra code means no bugs and minimal cost of maintenance
• The solution fully supports all of the features of MVCs authentication system
• If you have a traditional non-SPA web application which you want to migrate to Angular, this offers a smoother migration path than rewriting all of this in Angular at once.

Nonetheless, if you wish to implement authentication within your Angular app, here are some resources:

• http://www.codeproject.com/Articles/806029/Getting-started-with-AngularJS-and-ASP-NET-MVC-Par. This tutorial shows how you can implement authentication in an  easy to understand way.
• https://bitbucket.org/david.antaramian/so-21662778-spa-authentication-example/overview. Another tutorial.
• https://github.com/JustMaier/MVC5-SPA-Angular. This is a component which you can include as a script library in your app. This library works with MVC5 (not MVC4 which we’re using in this tutorial)

If you’re not sure which way to go, my advice would be to follow this tutorial first so you quickly have a working solution and can then focus on the actual core of your app. There’s nothing that will prohibit you from adding authentication plumbing directly into your Angular code at a later point in time.

Step 1: Setting up the initial project

In Visual Studio, create a new ASP.NET MVC4 application with MVC forms and Web API enabled, and configured for Individual Users Accounts.

Test the default app to ensure it runs fine, and the login link in the top right of the page works. If there’s no login link in the upper right corner of the home page, you probably selected Web API instead of MVC as the project type. In that case, go back and recreate the project.

Since we want to keep things simple for now, we'll only use nuget as our package manager and rely on the standard build process. We won’t use bower, npm, gulp, grant or any of the other modern web development tools.

Next we'll use the nuget Package Manager Console (Tools menu) to get the latest stable versions of the following components by entering the following commands:

Install-Package jquery
Install-Package bootstrap
Install-Package angularjs
Install-Package jquery.TypeScript.DefinitelyTyped
Install-Package angularjs.TypeScript.DefinitelyTyped
Install-Package bootstrap.TypeScript.DefinitelyTyped

In my case, the following versions were installed: jquery.2.1.4; bootstrap.3.3.6; angularjs.1.4.8; jquery.TypeScript.DefinitelyTyped 2.3.3; angularjs.TypeScript.DefinitelyTyped.4.4.0; bootstrap.TypeScript.DefinitelyTyped 0.1.7.

In addition to jquery, AngularJS and BootStrap, we loaded the API definitions for those JavaScript libraries which enables Visual Studio to provide IntelliSense as we start building our Angular app using TypeScript.

Now make sure you have the latest stable version of the following tools installed (via Extensions and Updates in the Visual Studio Tools menu).

• TypeScript for Visual Studio. At this time the most recent stable version is 1.7.6.
• SideWaffle Template Pack. This will install TypeScript item templates for various Angular components, such as directives and controllers, which we will user later during this tutorial.

Step 2 - creating "Hello World"

The next step is to create a simple Hello World app using AngularJS and TypeScript. To keep things as simple as possible, we’ll simply use the About page (/Home/About) to host our Angular app. 

Create a folder called "App" under the Scripts folder. This is where we'll put the app code.

The HTML page will need to load all the required script files, including the libraries we installed through nuget and all the scripts which will eventually belong to our SPA app. This is a pain if you do it manually, so we'll automate this by adjusting BundleConfig.cs in the App_Start folder by adding the following lines.

bundles.Add(new ScriptBundle("~/bundles/angular").Include("~/Scripts/angular.js"));
bundles.Add(new ScriptBundle("~/bundles/app").IncludeDirectory("~/Scripts/App", "*.js", true));

On GitHub: https://github.com/andreasderuiter/bat-mvc/blob/master/bat-mvc/App_Start/BundleConfig.cs

If you wish to use additional Angular components, such as for routing and animations, you can include these in the first bundle statement. The second statement takes all the .js files under our App directory and includes them as well. Note that we'll be writing the code in TypeScript files which have a .ts extension, and TypeScript will transpile them into .js files in the same folder.

Next we'll adjust the _Layout.cshtml in the Views/Shared folder to ensure our bundles are actually loaded in the HTML page. By doing this in the _Layout.cshtml file, any view/page in our MVC project will be able to contain Angular apps.

Add the following lines in the bottom of _Layout.cshtml where the other bundles are also loaded:

@Scripts.Render("~/bundles/angular")
@Scripts.Render("~/bundles/app")

On GitHub: https://github.com/andreasderuiter/bat-mvc/blob/master/bat-mvc/Views/Shared/_Layout.cshtml

Now we're ready to create the SPA app itself! First, open About.cshtml in the Views/Home folder and remove everything in it and replace it as follows:

On GitHub: https://github.com/andreasderuiter/bat-mvc/blob/master/bat-mvc/Views/Home/About.cshtml

Now we'll create the AngularJS app itself, which initially consists of an Angular module file and an Angular Directive which implements the <my-app> tag.

In the Visual Studio solution explorer, right-click the Scripts/App folder and select Add Item. Create a new file called app.ts using the AngularJs TypeScript Module template under the TypeScript/SideWaffle section.

The default code loads ngAnimate and ngRoute, but we're not using those for now and we'll comment them out to avoid errors. (If you want to use these modules later, you'll need to first install them through the package manager, then update the BundleConfig.cs file to ensure they are loaded on the page and finally add the modules back in app.ts.)

On GitHub: https://github.com/andreasderuiter/bat-mvc/blob/master/bat-mvc/Scripts/App/app.ts

Next we add a file called MyAppDirective.ts to the Scripts/App folder, using the AngularJs TypeScript Directive item template. We'll make two modifications: add a template line for the code we wish to generate, and on the last line we'll rename the directive to "myApp", which is how we will wire it up to the "<my-app>" tag in the HTML file we edited before.

The file should look like this: myapp.ts. Later in this tutorial we’ll make more changes to the file.

Now run the app to test that it works. When you navigate to the About page you should see this:

Step 3: adding a Web API

In the Visual Studio solution explorer, add a folder under the Controllers folder called "api". This is where our API controllers will live. Right-click the folder to add an item and add a file called SampleController.cs using the Web API Controller Class (v2.1) template.

The default code already has a sample Get method which returns an array of two string values “value1” and “value2”. This will suffice and we won’t make any further changes to the Web API: SampleController.cs

Check that the controller works by starting the app and then navigate to the http://localhost:<port>/api/sample (keep the port number which is already used for your app). If you're using Internet Explorer, you'll be prompted to download a file. If you open Chrome and navigate to the same URL, you'll see the following:

So now we know our new Web API is working!

Step 4: letting our Angular app consume the Web API

Create an HTML file in the Scripts/App folder called my-app.html with the following contents:

On GitHub: https://github.com/andreasderuiter/bat-mvc/blob/master/bat-mvc/Scripts/App/my-app.html

Create a new file under /Scripts/App called MyAppController which you create using the "AngularJs TypeScript Controller using 'Controller As' " item template, and adjust the code as follows: MyAppController.cs.

The controller class has a method called GetValues() which retrieves the data from the Web API REST service and stores it in a variable called “values”. The GetValues() method is called in the constructor, so the data will be loaded as soon as the page loads. The values variable is bound to the HTML template we added before so that the data from the web service is shown on the page.

Now we’ll adjust the MyAppDirective.ts file which we created in step 2 by copying the contents of the following file on GitHub:

On GitHub: https://github.com/andreasderuiter/bat-mvc/blob/master/bat-mvc/Scripts/App/MyAppDirective.ts

The MyAppDirective.ts file contains all the plumbing to link the my-app.html template with the controller and map it to the <my-app> tag.

Now run the app and click the About link to verify that the app works:

Adjust the URL on line 21 of MyAppController.ts file to test what happens if the call fails:

As you see the Angular app returns an error message in red when it fails retrieving the data from the Web API.

Change the URL back to the original value.

Step 5: Securing the Web API

One way to secure the app is by decorating the About() method in the HomeController class with the [Authorize] attribute. This method would prevent the About page to be loaded unless the user is logged in. Keep in mind that you also need to secure the Web API controllers otherwise a hacker could still call the Web API directly.

In our app we'll leave the About() method unsecured, but we'll secure the Web API so that users can't access the data, even if they can initially load the About page. Doing this is very simple: just add the [Authorize] attribute to the Web API controller class or method which needs to be secured:

On GitHub: https://github.com/andreasderuiter/bat-mvc/blob/master/bat-mvc/Controllers/api/SampleController.cs

We just secured the entire Web API controller class, so let's see what happens when you run the app:

The good news is that we don’t see the data. Security works!

The bad news is that we don't get redirected to the Login page. Even worse, we don't get an error message, which is strange because we built the app such that it displays an error message whenever it fails to get data from the web service. At the very least, we would expect an error message like “Access denied”.

The reason for this behavior is that by default ASP.NET transparently returns the login page instead of a HTTP 401 status code when a user is not authenticated. For a traditional web application which directly renders the HTML it receives from the server this is fine, but in our case the REST service is the one getting back the HTML and it only knows how to parse a JSON response, therefore we don’t see anything.

Fortunately we can modify this default behavior in ASP.NET by removing a single line of code in the Startup.Auth.cs file in the App_Start folder.

Find the following line (probably at line 28)…

                LoginPath = new PathString("/Account/Login"), 

… and remove it or comment it out.

On GitHub: https://github.com/andreasderuiter/bat-mvc/blob/master/bat-mvc/App_Start/Startup.Auth.cs

Now, run our Web application to see what happens when we navigate to the About page:

Yay! We now have an error message!

Step 6: Redirecting to the ASP.NET Login page

Adjust the catch part of getValues() in MyAppController.ts to check for a 401 status code and redirect to the login page:

On GitHub: https://github.com/andreasderuiter/bat-mvc/blob/master/bat-mvc/Scripts/App/MyAppController.ts

Note how we also pass the return URL so that the login page knows where to return back to after the user successfully logged in. You’ll need to add this code to every REST call you make to the Web API in your app (in this boilerplate app this, this is the only place).

That's it! Now, when the Angular app gets a 401 message back, it will redirect the user to the /Account/Login page. Once the user logged in, he or she is redirected back to the About page and subsequent Web API calls return data for as long as the session lasts.

If you implement routing in your Angular application, you can even pass a deep link as returnurl so that the login page will redirect back to a certain “location” in your Angular app.

Conclusion

As you see, ASP.NET works very nicely together with Angular apps. Enabling the solution to leverage all the powerful security functionality in ASP.NET just requires a couple of simple tweaks. I hope you found this useful!

With regard to your statement " With staging tables, transformations are implemented as database views"  Are you suggesting that your Dimension and Fact "tables" will be views or do you mean the query that populates the Dimension and Fact tables will be based on a view?

That’s a good question, and let me explain what I’ve found most practical.

Initially, when I create a new dimension or fact table, I simply create it as a view. Doing so allows me to quickly develop, test and debug the transformation in SSMS. If performance is not an issue, I might even deploy it this way into production.

In practice, once the transformation seems to work well, what I usually do is rename the view and create a stored procedure which creates a table using the original view. Because the new table has the same name as the original view, nothing breaks. Here’s what the stored procedure might look like:

   1: CREATE PROCEDURE [dbo].[sp_AccountFact]

   2: AS

   3: BEGIN

   4:     SET NOCOUNT ON;

   5:  

   6:  

   7:     SELECT * 

   8:     INTO #MyTempTable

   9:     FROM dbo.[Account Fact View]

  10:     

  11:     BEGIN TRANSACTION MY_TRANS

  12:  

  13:     IF  EXISTS (

  14:             SELECT [name]

  15:             FROM [sys].[tables]

  16:             WHERE name='Account Fact'

  17:     )

  18:     BEGIN

  19:         DROP TABLE tabular.[Account Fact]

  20:     END;

  21:  

  22:     SELECT * 

  23:     INTO tabular.[Account Fact]

  24:     FROM #MyTempTable

  25:  

  26:     COMMIT TRANSACTION MY_TRANS 

  27:  

  28:     CREATE UNIQUE NONCLUSTERED INDEX [Account key] ON [tabular].[Account Fact]

  29:     (

  30:         [Account key] ASC

  31:     )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]

  32:  

  33:     CREATE COLUMNSTORE INDEX [NonClusteredColumnStoreIndex-20130801-145157]

  34:     ON [tabular].[Account Fact](

  35:         -- Add columns here

  36:     );

  37:  

  38: END

In this example, the stored procedure [dbo].[sp_AccountFact] does the heavy lifting of recreating the [tabular].[Account Fact] table. You would call this stored procedure during the transform phase of your ETL process.

The view with the transformation logic (which was previously called [tabular].[Account Fact]) is now called [dbo].[Account Fact View]. The content of this view is first copied into a temporary table using the SELECT INTO statement. On line 11 we start a transaction so that the entire update is atomic. In lines 13-20 the existing table is deleted if it exists, and then in line 22 the table is recreated from the temporary table we created before, again using the SELECT INTO statement. Then the transaction is committed. Finally we recreate the indexes. Note that the second index in this example is a column store index which greatly improves the performance of most ad hoc queries.

The transaction ensures that clients performing queries on the data warehouse get valid results while the ETL is running. However, those clients might get blocked until the transaction completes. Therefore it’s important to minimize the duration of the transaction. This is why we first copy the view into a temporary table before the transaction starts. We also recreate the indexes outside the transaction for this reason.

Another design decision is whether to drop the table and recreate it – as I do here – or to just truncate the table. If we would just truncate the table, the stored procedure would become even simpler because we don’t need to check if the table already exists before dropping/truncating it, and we don't need to recreate the indexes. On the other hand, a benefit of using the drop-and-recreate method is that we only need to maintain the table’s schema in one place, i.e. in the view. In the truncate scenario we would need to update the schema of the table every time we modified the view. A second benefit is that it simplifies the usage of column store indexes. In SQL Server 2012, you cannot modify the contents of a table which has a column store index. (I believe this constraint no longer exists in SQL Server 2014, but I have not yet verified that myself).

Note: another good practice is to use schemas. In this data warehouse, the dimensional tables are in a schema called “tabular”. I also typically create a separate schema for each source system and its staging tables. The dbo schema has “internal” objects which are important for the ETL but are not visible to normal users. Normal users of the data warehouse only need read access to the tabular schema to perform the OLAP queries of their desire.

First, add an Execute R Script item to the model.

Then, adjust the code as follows:

   1: # Map 1-based optional input ports to variables

   2: dataset1 <- maml.mapInputPort(1) # class: data.frame

   3:  

   4: colnames(dataset1) <- gsub("Scored Probabilities for Class ","",colnames(dataset1))

   5: colnames(dataset1) <- gsub("\"","",colnames(dataset1))

   6:  

   7: # Select data.frame to be sent to the output Dataset port

   8: maml.mapOutputPort("dataset1");

The trick is to use the gsub function in R to replace a text with another text, and use that on the column names. In the example above I did two replacements, but normally you would need to do only one.

The reason I wanted to do this is that my experiment has a multi-class classification model with a few thousand classes, which results in a column to be added to the data for each of the classes while it is scored. This is exactly what I needed, however, Azure ML inconveniently names each of the added columns Scored Probabilities for Class “<class name>”. The most annoying thing about this is that the column name include the double quotes, and the SQL transformation module which I added downstream choked on that. Hence I needed to rename a couple thousand columns. In the code above, the second gsub call removes the double-quote characters. The result of this code is that each scored column is simply named after the class.

There is however a potential issue with this. Even though the script doesn’t do much, the entire data set needs to be loaded into the dataset1 variable, and later the same data is passed back to the Azure ML runtime. When the dataset is large, this can take a lot of time.

As I was experimenting with Azure ML, I noticed a module called “Execute Python Script”.

I am pretty sure I haven’t seen this before and I also couldn't find any official announcement about it from Microsoft. (I also noticed a new “Deprecated” section, and under “Data Input and Output” I saw an “Enter Data” module which I can’t remember seeing before).

Anyway, this means you can now easily add Python scripts to Azure ML which is great because Python and R are the two most commonly used languages for machine learning in the industry, and both can now be used in the cloud with Azure ML.

Like with the Execute R Script module, the Python module accepts three inputs, two of which are for data sets.

The version of Python which Azure ML currently uses seems to be 2.7.7 and not the more recent 3.4. This is not necessarily a bad thing because versions 2 and 3 of Python are incompatible and not all of the libraries are available for Python 3. (I did not find a setting in Azure ML which allows you to select the version of Python you wish to use.)

The Python script runs in a sandbox which has 93 packages pre-installed, which I listed below. Pre-installed packages include scikit-learn – which is a set of machine learning algorithms - and  scikit-image, which includes powerful routines for image processing.

You can also use other packages via a zip file which you provide to the third input. The comments in the Python template script in Azure ML say:

If a zip file is connected to the third input port is connected, it is unzipped under ".\Script Bundle". This directory is added  to sys.path. Therefore, if your zip file contains a Python file mymodule.py you can import it using:
import mymodule

I have not tested the ability to load other packages with Python this way. When I tested the equivalent functionality for R a while ago, it did not work very smooth. Keep in mind that Azure ML is still in preview and some rough edges are to be expected.

Below is the list of packages currently pre-installed with Azure ML:

'argcomplete==0.6.7'

'astropy==0.3.2'

'atom==0.3.7'

'backports.ssl-match-hostname==3.4.0.2'

'beautifulsoup4==4.3.1'

'binstar==0.5.3'

'bitarray==0.8.1'

'blaze==0.5'

'blz==0.6.2'

'bokeh==0.4.4'

'boto==2.28.0'

'casuarius==1.1'

'cdecimal==2.3'

'chaco==4.4.1'

'colorama==0.2.7'

'conda-build==1.3.5'

'conda==3.5.5'

'configobj==5.0.5'

'cubes==0.10.2'

'cython==0.20.1'

'datashape==0.2'

'docutils==0.11'

'enable==4.3.0'

'enaml==0.9.1'

'flask==0.10.1'

'future==0.12.1'

'gevent-websocket==0.9.3'

'gevent==1.0.1'

'greenlet==0.4.2'

'grin==1.2.1'

'h5py==2.3.0'

'ipython==2.1.0'

'itsdangerous==0.24'

'jdcal==1.0'

'jinja2==2.7.2'

'kiwisolver==0.1.2'

'llvmpy==0.12.6'

'lxml==3.3.5'

'markupsafe==0.18'

'matplotlib==1.3.1'

'menuinst==1.0.3'

'mock==1.0.1'

'multipledispatch==0.4.3'

'networkx==1.8.1'

'nltk==2.0.4'

'nose==1.3.3'

'numba==0.13.2'

'numexpr==2.3.1'

'numpy==1.8.1'

'openpyxl==1.8.5'

'pandas==0.14.0'

'patsy==0.2.1'

'pep8==1.5.6'

'pil==1.1.7'

'ply==3.4'

'psutil==2.1.1'

'py==1.4.20'

'pycosat==0.6.1'

'pycparser==2.10'

'pycrypto==2.6.1'

'pyface==4.4.0'

'pyflakes==0.8.1'

'pygments==1.6'

'pyparsing==2.0.1'

'pyreadline==2.0'

'pytest==2.5.2'

'python-dateutil==1.5'

'pytz==2014.3'

'pywin32==218.4'

'pyyaml==3.11'

'pyzmq==14.3.1'

'requests==2.3.0'

'rope==0.9.4'

'runipy==0.1.0'

'scikit-image==0.10.0'

'scikit-learn==0.14.1'

'scipy==0.14.0'

'six==1.6.1'

'sphinx==1.2.2'

'spyder==2.3.0rc'

'sqlalchemy==0.9.4'

'statsmodels==0.5.0'

'sympy==0.7.5'

'tables==3.1.1'

'tornado==3.2.1'

'traits==4.4.0'

'traitsui==4.4.0'

'ujson==1.33'

'werkzeug==0.9.6'

'wsgiref==0.1.2'

'xlrd==0.9.3'

'xlsxwriter==0.5.5'

'xlwings==0.1.0'

'xlwt==0.7.5'

This final article will cover the threshold setting, and how to find the optimal value for it. As you will learn, this requires a good understanding of error cost, that is, the cost of inaccurate predictions.

The first article laid the foundation by covering several statistical measures: accuracy, precision, recall and F1 score. These measures require a solid understanding of the two types of prediction errors which we also covered: false positives and false negatives.

Part 1: Performance measures in Azure ML: Accuracy, Precision, Recall and F1 Score

The second article discussed the ROC curve and the related AUC measure. We also looked at another graph in Azure ML called the Precision/Recall curve.

Part 2: Using ROC plots and the AUC measure in Azure ML

If you have not yet done so, I encourage reading the other articles before first.

At the time of this writing, Azure ML is still in preview and most of the documentation you find on the internet appears to be written by data scientists for data scientists. This blog is focused on people who want to learn Azure ML, but do not have a PhD in data science.

Getting started

When you build a machine learning solution, you’ll normally first try and compare multiple models. As we discussed in part 2, you'd normally use the AUC measure to determine the best performing model. Once you made your selection, you can remove all the other models and optimize the selected one. (In case you want to re-evaluate the models in the future with a new data set, it might be better to just create a new experiment).

In binary models you can influence the behavior using the threshold slider. If you’re not sure how to do this and what the effect is of changing the threshold, please go back and review parts 1 and 2 of this series.

The threshold lets you change the balance of two types of prediction errors: false positives and false negatives. The cost associated with one type of error can be much higher than the cost of the other. If that’s the case, it makes sense to err on the side of safety which means you will accept a (much) higher number of cheap errors to reduce the number of expensive errors.

Therefore, in order to determine the optimal value for threshold, you first need to determine the cost of false positives and false negatives. I’ll use an example to explain this and provide a spreadsheet which helps you do the calculation.

An example

We’ll use the same data set as in the previous posts, but instead of predicting if someone is above 50 years old, we’ll act as if this is data from a quality test in a factory. The data is from a test device which tests whether a given product is thought to be faulty (positive) or not (negative). The testing data has been labeled with an additional piece of data: whether the product was in fact defective after further inspection. This allows us to train the model and to validate its performance. Based on the labeled data, we know that about 24% of the devices are in fact defective.

Now you need to determine the cost of false positives and false negatives. This will require some research and if possible you may want to include an accountant from your company to do this. Keep in mind that you need to include opportunity cost and that some of the cost, such as customer dissatisfaction and reputation damage, is difficult to quantify.

False Positive: these are devices which were tested as defective, when in fact they are not. This type of error leads to rework, and let’s assume that the cost associated with this unnecessary re-work is $10 per wrong prediction.

False Negative: these are defective devices which were shipped to customers because the defects were not detected. The cost of these types of errors are much higher due to extra handling, customer dissatisfaction and reputation damage. Let’s assume that the cost of this type of error is $100 per wrong prediction.

Based on the cost per error and the frequency that each type of error occurs, we can calculate the average error cost per prediction. The optimal threshold is the one with the lowest average error cost per prediction. The math is fairly straightforward, and the following spreadsheet shows the average error cost per prediction in the right-most column and the related threshold setting in the left-most column. The optimal threshold is 0.10 which has a cost of $2.70.

The table shows various performance measures for different threshold settings. There are a lot of things we can conclude from this table:

• The default threshold of 0.5 would have resulted in an error cost of $6.50 per prediction, much higher than the optimal cost of $2.70.
• Precision and recall are – when used independently – useless to determine the right threshold setting because they are conflicting forces. If you only focus on one of them, you’ll end up with a threshold of either 0 or 1.
• The optimal accuracy is obtained with a threshold value of 0.55. The accuracy at that point is 90.0%. As we concluded in part 1, accuracy is not a very good measure when the class distribution is uneven, and it does not take the error cost into account. This table confirms that. If we would use this threshold, we’d incur 2.7x too much cost.
• The F1 Score is the weighted average between Precision and Recall. If the cost of false positives and false negatives were roughly equal, this would be a pretty good measure. However, that’s not the case. If we would have used the F1 score to determine the optimal threshold, we’d incur almost 2x to much cost.

Here is how the costs vary by threshold, displayed in a graph:

As you can see, the optimal threshold is where the grey line reaches its lowest point.

Sample spreadsheet

You can download the spreadsheet which was used to generate the table and chart here:

1207.UAC and ROC Aalyses.xlsx

To use this spreadsheet and analyze data from a different data set, you need to replace the data in the Data worksheet:

• Column A (Actual) has the actual (labeled) value for each observation. In the spreadsheet the cells in this column contain either “Defective” or “Not defective”, but this could be anything as long as there are only two different values.
• Column B (Scored Probabilities) has the scores of the scored data set. Every score is a real number between 0 and 1.
• Column C (Pos) and D (Neg) have a simple formula which set to value to 0 or 1, depending on the value in column A. You’ll probably need to adjust these formulas to reflect the data in column A.

When you want to load data from another data set, it’s important that this data is already scored. Here’s how you would download the scored data for the sample experiment “Sample 5: Train, Test, Evaluate for Binary Classification: Adult Dataset”.

• Log into your Azure ML workspace.
• In the Experiments screen, click on the Samples link in the top of the screen and then click on the sample called “Sample 5: Train, Test, Evaluate for Binary Classification: Adult Dataset” .
• Click the Save As button in the bottom of the screen and save the experiment under a different name. This will enable us to edit this sample.
• Add a “Convert to CSV” item to the workspace and connect its input to the output of the Score Model item which you wish to analyze (see screenshot below)
• Run the model
• After the model has completed running, right-click the output of the Convert to CSV item and select Download.

Putting it together

This was the last of a series of three articles on using performance measures and plots in Azure ML. We discovered that for model selection, the ROC plot and AUC measure are the best and easiest to use. Once the best model has been selected, you can optimize its performance by adjusting the threshold, as we discussed in this last article.

I hope you found this useful. Please leave comments or send me email if you have any further suggestions.

This first article lays the foundation by covering several statistical measures: accuracy, precision, recall and F1 score, These measures require a solid understanding of the two types of prediction errors which we will also cover: false positives and false negatives.

In the second article we’ll discuss the ROC curve and the related AUC measure. We’ll also look at another graph in Azure ML called the Precision/Recall curve.

Part 2: Using ROC plots and the AUC measure in Azure ML

The final article will cover the threshold setting, and how to find the optimal value for it. As you will learn, this requires a good understanding of the cost of inaccurate predictions.

Part 3: Setting the threshold of a binary learning model in Azure ML

At the time of this writing, Azure ML is still in preview and most of the documentation you find on the internet appears to be written by data scientists for data scientists. This blog is focused on people who want to learn Azure ML, but do not have a PhD in data science.

Getting started

When you build experiments in Azure ML, you need to score the used learning models and evaluate their performance to understand how well they perform and which model actually works best. Ideally you would want just one number which tells you how good a model is, but Azure ML provides 9 metrics plus a couple of charts. If you’re new to Azure ML and you’re not already a data scientist, this can be a bit intimidating.

You can usually rely on the AUC measure to select which model works best for a given data set. We will cover this further in the second article of this series. After you select a model. you will need to find the optimal threshold value for it, which, as we will see in the third article, you cannot do using the AUC measure. This first article will set the foundation with regard to statistical measures in Azure ML and set you up for the other articles.

The reason Azure ML provides multiple performance measures is that data sets differ in two key ways:

• The distribution of positives and negatives, that is, the ratio between positive values and negative values in the data. Data scientists call this the class distribution. More often than not the class distribution of positive and negative values is not 50/50.

• The cost of wrong observations. When data scientists say “observation”, they mean the predicted value. It is important to realize there are two types of errors – false positives and false negatives - which often have a different associated cost. For example, the cost of not administering a drug to someone who needs it might be different than the cost of administering the drug to someone who does not need it.

Binary models just predict one of two values. The meaning of these values could be anything, depending on the data. For example, whether a tested device is defective, if a person is over 50 years old, or whether a certain stock will increase in value. To generalize this, statisticians and data scientists use the terms “positive” and “negative”, rather than true/false or 0/1 as a software developer would do. We will follow this convention.

The decision of which value to assign the “positive” label is not arbitrary. You should assign “positive” to the value that drives your decision. For example, if you are going to run a marketing campaign targeted towards people over 50 years old, and your model will predict if someone is over 50 years old, you should assign “>=50 years” as the positive value. Or, if you build a model to detect defective devices so you can rework them before shipping to the customer. you should define “is defective” as the positive value. If you don't do this right, many statistical measures, including precision and recall, will make little sense.

Accessing performance metrics in Azure ML

Azure ML has a large number of sample models. One you can use to follow along in called “Sample 5: Train, Test, Evaluate for Binary Classification: Adult Dataset”.

To view the performance metrics, right-click on the output port of the Evaluate Model item and select Visualize.

After you click the Visualize button, you’ll initially see the ROC plot. Scroll down to see the performance metrics:

Above, all measures except AUC can be calculated based on the left-most four numbers. I created a spreadsheet which allows you to experiment with these numbers. You can also use the spreadsheet to review the formulas for each of the metrics. Here’s what the spreadsheet looks like:

You can download the spreadsheet here: 5141.Binary Model Performance Metric Calculator.xlsx

Understanding false positives and false negatives

The performance of a trained model is determined by how good the observations (predictions) reflect the actual events. When you train a (supervised) model, you need a “labeled” data set which includes the actual values which you’re trying to predict. You set aside a subset of the labeled data so that, after you trained the model, you can evaluate the model against labeled data which was not used to train the model. By comparing the predicted values against the labeled ones, you end up with four bins:

True positives and true negatives are the observations which were correctly predicted, and therefore shown in green.

The terms “false positive” and “false negative” can be confusing. False negatives are observations where the actual event was positive. The way to think about it is that the terms refer to the observations and not the actual events. So if the term starts with “false”, the actual value is the opposite of the word that follows it.

Accuracy

Accuracy is perhaps the most intuitive performance measure. It is simply the ratio of correctly predicted observations.

In the screenshot you saw before, the two green cells are the correctly predicted observations: 2894+11750=14,644. The total number of events is 16,281, hence the accuracy is 14,644/16,281=0.899, or approximately 90%.

Using accuracy is only good for symmetric data sets where the class distribution is 50/50 and the cost of false positives and false negatives are roughly the same. It can be attractive at first because it’s intuitively easy to understand, however. you should not rely on it too much because most data sets are far from symmetric.

Example: you are building a model which predicts whether a device is defective. The class distribution is such that 6 in 1000 devices is truly defective (positive). A model which simply returns “negative” – i.e. not defective – all the time gets it right 99.4% of the time and therefore has an accuracy of 0.994, when in fact it never correctly identifies a defective device!

Precision

Precision looks at the ratio of correct positive observations.

The formula is True Positives / (True Positives + False Positives).

Note that the denominator is the count of all positive predictions, including positive observations of events which were in fact negative.

Extending the example we used before to explain the accuracy, let's assume we improved the model and it now predicts 8 out of 1000 devices being faulty. If only 5 of those 8 are truly defective the precision is 5/8 = 0.625.

Recall

Recall is also known as sensitivity or true positive rate. It’s the ratio of correctly predicted positive events.

Recall is calculated as True Positives / (True Positives + False Negatives).

Note that the denominator is the count of all positive events, regardless whether they were correctly predicted by the model.

Using the same example as before, we already knew that 6 out of 1000 devices are truly defective. The model correctly predicted 5 of them. (It also predicted 3 incorrectly, but for the recall measure that’s not important). The recall therefore is 5/6 = 0.833.

Comparing Precision and Recall

To better understand the difference between precision and recall, here is a screenshot with all the stats:

Both precision and recall work well if there’s an uneven class distribution as is often case. They both focus on the performance of positives rather than negatives, which is why it’s important to correctly assign the “positive” predicate to the value of most interrest.

The precision measure shows what percentage of positive predictions where correct, whereas recall measures what percentage of positive events were correctly predicted. To put it in a different way: precision is a measure of how good predictions are with regard to false positives, whereas recall is measures how good the predictions are with regard to false negatives. Whichever type of error is more important – or costs more - is the one that should receive most attention.

As we will further explore in the third article, you can influence precision and recall by changing the threshold of the model. Based on the cost of false positives and false negatives, you can calculate the optimal threshold value.

F1 Score

The F1 Score is the weighted average of Precision and Recall. Therefore, this score takes both false positives and false negatives into account. Intuitively it is not as easy to understand as accuracy, but F1 is usually more useful than accuracy, especially if you have an uneven class distribution. It works best if false positives and false negatives have similar cost. If the cost of false positives and false negatives are very different, it’s better to look at both Precision and Recall.

The formula for F1 Score is 2*(Recall * Precision)/(Recall + Precision)

Putting it together

The table below shows you which of the measures described in this article probably best to focus on, depending on the class distribution and the cost of false positives and false negatives:

In addition to the measures mentioned in this post, you might encounter others in literature such as “negative recall”. With the basic understanding of this article you should be able to figure out what they mean and when they are to be used. If you correctly assign the value that is of most interest as “positive”, you usually won't need those other measures anyway.

I hope you found this useful. Please leave comments or send me an email if you think I missed something important or if you have any other questions or feedback about this topic.

Continue reading:

Part 2: Using ROC plots and the AUC measure in Azure ML

Part 3: Setting the threshold of a binary learning model in Azure ML

The first article laid the foundation by covering several statistical measures: accuracy, precision, recall and F1 score. These measures require a solid understanding of the two types of prediction errors which we also covered: false positives and false negatives. If you have not yet done so, I encourage reading that article first:

Part 1: Performance measures in Azure ML: Accuracy, Precision, Recall and F1 Score

In this second article we’ll discuss the ROC curve and the related AUC measure. We’ll also look at another graph in Azure ML called the Precision/Recall curve.

The final article will cover the threshold setting, and how to find the optimal value for it. As you will learn, this requires a good understanding of the cost of inaccurate predictions.

Part 3: Setting the threshold of a binary learning model in Azure ML

At the time of this writing, Azure ML is still in preview and most of the documentation you find on the internet appears to be written by data scientists for data scientists. This blog is focused on people who want to learn Azure ML, but do not have a PhD in data science.

Getting started

If you’re new to Azure ML, you’ll probably soon wonder what the term “AUC” means and how to read ROC plots. Both of these are used as a way to measure how well a binary learning model is performing. You can use this to compare multiple learning models and decide which one works best for a given data set.

This is what a ROC plot in Azure ML looks like:

If you want to follow along yourself, you can get to the screen above as follows:

• Log into your Azure ML workspace.
• In the Experiments screen, click on the Samples link in the top of the screen and then click on the sample called “Sample 5: Train, Test, Evaluate for Binary Classification: Adult Dataset” .
• In the design window, right-click on the output of the Evaluate item and select Visualize.

Before we dig deeper into the ROC plot, we need to take a closer look at the threshold setting.

Threshold and scoring values

When you scroll down in the screen with the ROC plot, you’ll see something like this:

The statistics you see here are for the blue curve which represents the left input of the Evaluate item in the experiment. Notice that when you drag the Threshold slider, all the numbers, except for AUC, change.

To understand why this is the case, you need to know that the learning model doesn’t just predict positive or negative as an outcome, but it generates a score which is a real number between 0 and 1, and then uses the threshold setting to decide if the prediction is a “positive” or a “negative”. By default, the threshold is set to 0.5 which means that all scoring values above that value are interpreted as positive, and everything below as negative. Therefore, the threshold impacts the predictions which the model makes and thus the related statistics.

While 0.5 is a good default to start with, you will eventually want to tweak the threshold to get the best results. This is the topic of the third article in this series.

The ROC plot

In the ROC plot which was shown before, two learning models are compared. The blue curve represents the model which feeds the left input of the Evaluate item, the red one represents the right input pin. (In this context performance does not relate to the speed but to how well the model predicts the desired outcome).

The blue and red lines in the plot connect the lower-left corner with the upper-right corner of the graph. Each position on the lines represents how the model performs in two different dimensions for a certain threshold setting. Unfortunately the ROC plot does not clearly show which point on the lines represents the current threshold value (perhaps the Azure ML Dev team can note this as a feature request:-), however the lower-left corner (0,0) represents a threshold of 1 whereas the upper-right corner (1,1) represents a threshold of 0. The two dimensions are the false positive rate on the horizontal axis and the true positive rate on the vertical axis.

In part 1 I explained how there are two different types of errors, false positives and false negatives. The two dimensions in the graph basically reflect this.

By changing the threshold, you decrease the frequency of one type of error at the expense of increasing the other type. The position on the curve where the slope is 1 is where they are in balance such that, if you would move the threshold slider a little bit, the number of one type of errors would increase by the same percentage as the other would decrease..

Usually when you see two models compared in a graph as the one above, the lines only intersect in the corners. The line which comes closest to the upper-left corner – in this case the blue line - provides the best predictions. For every threshold value it scores better than the other model for both false positives and false negatives. The further a curve is to the middle, the worse is its performance. Theoretically the worst possible model is one which produces random predictions with the same distribution as the class distribution. Such a model has a ROC curve which approximates a straight diagonal line from (0,0) to (1,1).

You might wonder if curves can also occur under the diagonal line. The answer is that if the ROC plot for a learning model would consistently perform under the diagonal, you could simply use the opposite value of what the model predicts as your actual prediction, and the result would then perform better than a random prediction.

The ROC plot was invented in WWII for the analyses for radar signals (see the article about Receiver operating characteristic on Wikipedia) and is commonly used by data scientists. However, I don't find it extremely insightful because the way I think of it, the ROC plot is about striking the right balance between the two types of errors. While the horizontal axis represents the ratio of false positives, the vertical axis shows the ratio of true positives, which only indirectly implies the ratio of false negatives. In other words, on the horizontal axis a higher value implies worse performance, while on the vertical axis a higher value implies a better performance.

Precision/Recall plot

Azure ML also includes another graph called for the precision/recall, which essentially displays the same information as the ROC plot, but in a more intuitive way.

To understand how this graph relates to false positives and false negatives, you need to remember that Precision and Recall are calculated as follows:

Recall = True Positives / (True Positives + False Negatives)

Precision = True Positives / (True Positives + False Positives)

In this curve, the “sweet spot” for the ideal model is in the upper right corner.

AUC

AUC stands for “area under curve”, and as it's name implies, it refers to the amount of area under the ROC curve, which theoretically is a value between 0 and 1. As I explained, the worst possible curve in practice is a diagonal line, hence the AUC should never be lower than 0.5 (for large data sets) .

Using the AUC metric you can quickly compare multiple learning models. Remember that the ROC curves of two models usually don’t cross each other, hence when comparing two models, the one with a higher AUC will be the better one regardless of the threshold setting. Compared to the statistical measures of accuracy, precision, recall and F1 score, AUC’s independence of threshold makes it uniquely qualified for model selection.

On the other hand, unlike accuracy, precision, recall and F1 score, AUC does not tell us what performance to expect from the model for a given threshold setting, nor can it be used to determine the optimal value for threshold. In that regard it doesn't take away the need for the other statistical measures.

Putting it together

The ROC plot and the AUC are very useful for comparing and selecting the best machine learning model for a given data set. A model with an AUC score near 1, and where the ROC curve comes close to the upper left corner, has a very good performance. A model with a score near 0.5 will have a curve near to the diagonal and its performance is hardly better than a random predictor.

After selecting the best model, the next step will be to configure the threshold and the other model configuration settings. The Precision/Recall plot can be helpful for understanding the trade-off between false positives and false negatives. In the third article we’ll look closer at the threshold value and learn now to calculate the optimal value for it.

Continue reading:

Part 3: Setting the threshold of a binary learning model in Azure ML

As I started experimenting with R scripts, I frequently encountered scripts which ran well outside of Azure ML – in RStudio – but failed inside Azure ML with the following error:

Error 1000: RPackage library exception: Failed to convert RObject to DataSet

As an Azure ML and R-newbie it took me some time  to get to the bottom of this.  The error occurs when you pass data back from R back to Azure ML. If you run into this error, try commenting out the line which calls the maml.mapOutputPort function, which is usually the last line in the R script. Now run the experiment again. If the error no longer occurs, this confirms that our suspicion. Of course, without calling maml.mapOutputPort you won’t get any data back to Azure ML so you need to add that line back and fix the actual problem.

I encountered at least three situations which can cause this error:

1. Limited date/time support in Azure ML
2. Duplicate column names
3. You have a bug in your code which does not surface in RStudio

I’ll describe them in more detail below

Limited date/time support in Azure ML

Because Azure ML does not  natively support learning models to deal time series data, this is an important scenario where R is very useful. For example, if you want to build a model which predicts the gold price based on historic gold prices, you need to resort to R which has powerful time series analyses and forecasting capabilities. R has several classes which deal with dates and times, including Date, POSIXct and POSIXlt. One of the problems according to Mike Lanzetta is that Azure ML does not yet support parsing of POSIXlt values in the result set. I have not yet been able to verify this myself, but I guess it's a good idea to stay away from POSIXlt dates and use Date and POSIXct instead.

Another problem is that date and time values don’t always seem to convert well between different types of variables. I could only reproduce this with date/time values which were passed to R through an input port in Azure ML. Therefore I suspect there may be an issue in the way Azure ML initializes date/time values going into the R script component.

For example, let’s assume that your data contains two columns, one called Date which is a DateTime field and gets passed as vector of POSIXct into the script, and the other called Value which is vector of numeric values (see Loading daily gold prices into Azure ML for a quick way to implement this data source).

Let’s say we want to perform time series analyses using the zoo class.  We’ll leave away all the code that does actual time analyses and just focus on the conversions. The R script is shown below:

When we create the zoo time series variable called z in line 5, we provide the prices and the dates. The dates variable is a vector of POSIXct. In line 6 we load the dates from the z variable into a new vector called t, using the time() function. You’d expect t to be a vector of POSIXct since that was the type of the original dates vector which we used before, and when you run the code in RStudio, this is the case. However, when running this code in Azure ML, t becomes a vector of numeric. Somehow the time() function no longer sees this data as POSIXct and instead returns the numeric which represents the number of seconds since 1/1/1970.

Note: this sample does not generate the Error 1000 error message, but it does lead to a state where this could easily happen if you change the code a bit. However I no longer have the version of the code which generated that error message.

The work-around I found was to add two extra lines of code after line 3. The updated script below now also works in Azure ML:

Know a better solution, let me know!

Duplicate column names

Another case I encountered which caused Error 1000 was when multiple columns have the same name. It’s easy to understand that Azure ML does not like duplicate column names and throws an error.

You have a bug in your code which does not surface in RStudio

Consider the following R script:

This R script does not require any input data. It creates a vector called test and then two variables called v1 and v2 which will hold the classes of all elements in the vector. Then it concatenates v1 and v2 using the rbind() function and returns that to the output port. If you run this in RStudio (except for the last line) and then inspect the data.set variable, all looks as expected and this is the result:

        X1      X2      X3
v1 numeric numeric numeric
v2 numeric numeric numeric

However, when you run this in Azure ML, you’ll get “Error 1000: RPackage library exception: Failed to convert RObject to DataSet”.

The code has a bug. The rbind() function is intended to concatenate vector and matrix variables, but the output of the lapply() function is a list variable which can hold heterogeneous objects. Azure ML does not like heterogeneous objects because you can’t map the values to columns with a specific type. (I would have liked R to throw an error when I used rbind(), I’m not sure why it didn't do so)

One solution is to cast the output of lapply() to a text string, as follows:

Now it also works in Azure ML!

Want to load daily gold prices into Azure ML? Here’s one way to do that:

• Create a new experiment, or open a new one
• Add a “reader” object onto the canvas
• Select the Reader object and set the following properties:
  • Data source = Web URL via HTTP
  • URL = https://www.quandl.com/api/v1/datasets/BUNDESBANK/BBK01_WT5511.csv
  • Data format = CSV, and check “CSV or TSV has header row”

The properties for the Reader object now looks like this:

Now run your experiment, and after it completes, check the results by clicking Visualize on the output port:

CREATE TABLE [dbo].[ImportTable]( 
    [ID] [int] NULL, 
    [Name] [nvarchar](100) NULL, 
    [Country] [nvarchar](100) NULL 
 ) ON [PRIMARY]

I called this method of tracking history “snapshot-based versioning”. It does not rely on database triggers or on CDC. While those features might be useful for other scenarios, I’ve found them too limiting for data warehouse scenarios. For example, these features don’t work well if your ETL process refreshes the data by truncating the existing table and then reloading it.

Snapshot-based versioning instead relies on TSQL’s MERGE statement. The MERGE statement is supported in SQL Server 2008 or higher. A limitation is that you cannot track changes to fields of type image, ntext and text. This is usually not a blocker because in real-life scenarios most tables have a “modified” field that is updated whenever a change to the record is changed.

The idea is to have a second table called H_ImportTable which has the same fields as the original ImportTable. This second table, which I’ll refer to as the “history table”, has a versioned copy of the data in H_ImportTable. In other words, it works like a log which appends a row for any change to ImportTable. Obviously, if the table in the external database has changed multiple times between two ETL runs, this will appear only as one change in H_ImportTable, hence the name “snapshot-based versioning”.

You can create H_ImportTable as follows:

CREATE TABLE [dbo].[H_ImportTable] ( 
    [_ACTION] [char](1) NOT NULL, 
    [_KEY] [int] IDENTITY(1,1) NOT NULL, 
    [_EFF_FROM] [datetime] NOT NULL, 
    [_EFF_TO] [datetime] NOT NULL, 
    [ID] [int] NULL, 
    [Name] [nvarchar](100) NULL, 
    [Country] [nvarchar](100) NULL 
 )

• _ACTION indicates whether a row has been added (‘A’), changed (‘C’), deleted (‘D’) or undeleted (‘U’).
• _KEY is a unique row identifier. This field is only useful in specific scenarios
• _EFF_FROM indicates the date/time from which this row is effective
• _EFF_TO indicates the date/time until which this row is active. In case the row is effective now, this field is set to 9999-12-31.

After refreshing ImportTable, the ETL process should update H_ImportTable using the following SQL statement:

INSERT INTO [dbo].[H_ImportTable] 
 SELECT _ACTION, SYSDATETIME() AS _EFF_FROM, '9999-12-31' AS _EFF_TO,[ID],[Name],[Country] 
 FROM 
 ( 
  MERGE [dbo].[H_ImportTable] AS T 
  USING [dbo].[ImportTable] AS S 
  ON (S.[ID]=T.[ID] AND T._EFF_TO='9999-12-31') 
  WHEN NOT MATCHED BY TARGET THEN 
    INSERT(_ACTION,_EFF_FROM,_EFF_TO,[ID],[Name],[Country]) 
    VALUES('A',SYSDATETIME(),'9999-12-31',[S].[ID],[S].[Name],[S].[Country]) 
  WHEN MATCHED AND T._EFF_TO='9999-12-31' AND (T._ACTION='D' OR ([S].[Name]<>[T].[Name] OR [S].[Country]<>[T].[Country])) THEN 
    UPDATE SET T._EFF_TO=SYSDATETIME() 
  WHEN NOT MATCHED BY SOURCE AND T._EFF_TO='9999-12-31' AND T._ACTION<>'D' THEN 
    UPDATE SET T._EFF_TO=SYSDATETIME() 
  OUTPUT $Action Action_Out 
         ,CASE WHEN S.[ID] IS NULL THEN 'D' WHEN Inserted._ACTION='D' THEN 'U' ELSE 'C' END AS _ACTION 
         ,ISNULL(S.[ID],Deleted.[ID]) AS ID,[S].[Name],[S].[Country] 
 ) AS MERGE_OUT 
 WHERE MERGE_OUT.Action_Out = 'UPDATE' 

You can adjust this SQL code to work with the tables in your data warehouse. To do so, adjust all the parts that are underlined. If you need to do this for many tables, making these changes manually is laborious and error-prone, This is why I wrote the program “Table Merge Scripts” which you can download here. This program generates the SQL code based on an existing table in the database. You can see the UI below:

First, edit the connection string by entering the correct server name and database name. Then click Connect.

Next select the table and the field that is the primary key.

The first button in the bottom half of the screen copies the SQL code to create the history table to the clipboard. The new table will have the same name as the selected source table, prefixed with “H_”. The code also creates indexes for _ACTION, _EFF_FROM, _EFF_TO, _KEY and the field which you designated as the primary key in the source table.

The next button copies the SQL code to update the history table to the clipboard. This consists of a MERGE command similar to the example shown above.

The final button copies the same SQL code as the second table, but wrapped in a stored procedure.

Good luck!