However when it comes to the world of Machine Learning for examle  there are several technologies even from Microsoft:

• Azure Machine Learning with support for Python & R
• R Server 
• CNTK (Cognitive Tool Kit)
• Infer.net
• F#

Now imagine for a moment you wanted to be a good blogger – would a certification in WordPress, Word or the Open Live Writer this is written in be of help? Possibly, but only if you have grasped the principles of good blogging like understanding your audience (that’s a post for another day!).  I would argue you need the theory and then get technically checked out on your tools of choice. 

So I used spare time in the holiday season to mug up theory and techniques with the Microsoft Professional Program for Data Science rather than watch movie re-runs. The theory in this program comes from some excellent lecturers at American Universities such as  Columbia , MIT, Duke etc).  From what I have seen so far (4 passed one in progress) they are all high quality – both well constructed and well presented. It is also practical, and yes it is based on Microsoft technologies, but those technologies are there as in any course, to underpin the theory rather than being the subject being studied.  A good example is the use of Azure Machine Learning in the DAT203x module (which also the best I have seen on this topic) where AML is simply used as a fast way to work through examples rather than writing unnecessary code – so not relevant to the learning objective. 

The complete program requires completion of 10 modules (there are electives like options to do either R or Python) and each one of which will take a few weeks to get through especially if you want to learn this stuff and not just pass the course.  The final module is a Data Science Challenge Project is there to prove to yourself and Microsoft that you really know what you are doing.

The knowledge check questions throughout the modules I have seen  (except the final project), are in my opinion, just there to ensure you have done the exercises and not fast forwarded through all the videos. They will not, in my opinion, stress test your knowledge like a Microsoft certification.  So yes you could skim through the professional program and possibly fudge your way through the final project or you could do what I do -  reverse engineer some of the examples, read the recommended additional material and make copious notes like this ..

That you will get the intended value out of every module and properly ground you in the approach.  After all if you don’t enjoy doing this stuff why would you study it?

This also means I will continue to do exams to compliment courses like this both to keep up with the new stuff¹ and to keep the old brain ticking over especially as you can do them at home via a web cam (details are here).

¹ by new stuff I don’t mean re-certifying which you can now do on the Microsoft Virtual Academy, I mean new exams on new technologies like Azure Machine Learning

Disclaimer: This post is one of a series about data science from someone who started out in BI and now wishes to get into that exciting field so if that’s you read on if not click away now

Over Fitting

If we go back to the example about delegates in my last post  not turning up for community technical events then if the model we build can predict at a level of 99.99% accuracy then that’s probably because one of the features  (other columns in the dataset) used to make the prediction is directly correlated to the label (the answer). In this scenario we might have a column for which size of t-shirt they were given at the event and if the column has null in it they didn’t get one.  T-shirts are very popular at events so the only reason for a null in that column would be that they weren’t there and so this column matches the attendance very well but of course is useless because can’t itself be predicted before the event.  This is the sort of thing that is referred to over fitting in data science

Confusion Matrix

Without the t-shirt column the accuracy of our model might drop to 94% i.e. 94 times out of a hundred we can say whether a delegate turned up or not against whether they actually did or not. That  might seem a pretty good result but if we have an attendance rate of 80% i.e. 80 out of a hundred delegates turn up then if all we do is predict everyone will turn up will get us an accuracy of 80% anyway.  So what is more interesting to data scientists is the confusion matrix, a set of analysis performed by comparing the models predictions against a set of data it hasn’t seen before where the out come is known and can be compared to the prediction: 

I have taken this example from the Azure Machine Learning workshop Amy and I have developed and here we can see the true positives/true negatives and false positives/negatives. If this was an evaluation of event attendance and the positive label of 1 indicates that a delegate didn’t attend/ show up and a negative label of 0 means they did then we can infer the following from a test data set of 480,731 delegates across a variety of historical events:

• true positives = the model correctly predicted the non-attendees = 67,419
• true negatives = the model correctly predicted those who did actually attend = 375,695
• false positives = those the model predicted would not attend but did = 6,409
• false negatives = those the model predicted would attend but did not = 31,208

The definitions  for the other numbers are as follows:

All this is based on a threshold of 0.5 meaning that the label is set to 1 when the scored probability is > 0.5 and 0 otherwise. In other word if there is more than a 50% chance of a delegate turning up then score them as predicted to turn up. In Azure ML we can move the threshold higher or lower to affect these scores e.g. only score them as predicted to turn up if the scored probability is say to 0.8 (80%) then we get this for our confusion matrix:.

so while the number sum of the true answers (positive & negative)  goes up from 433,348 to 443,114 and the false positive drops as well there are now another 15,229 false negative cases (where the model thought a delegate would not turn up but they did.

So how we use this result and move the threshold is just as important as the accuracy of the numbers.  For example if we want to give every delegate a t-shirt (assume for a moment they are all one size) how many should we order?  A simple but effective approach would be to add up the scored probabilities for each individual to estimate the number of no shows and subtract that from the number of registrations for the event.  so imagine if we have four delegates have registered each with a score 0.25 (25%) of NOT attending from this model adding this up gives 1 which we subtract from the 4 registration and only order 3 shirts because across all four of them it is extremely likely (but not certain) that only three will attend.  I say not certain because the  prediction might not be very good for small events but as the registrations for an event get larger then the accuracy improves.  So if I download the scores from the model I have been using and sum the scored probabilities across the 480,731 examples in the sample  I get 98,816 as my predicted no shows which compares very well with the 98,617 actual no shows.   

A good way of think about this is that statistics is analysis of the behaviour of large groups of thinks (called the total population) based on the behaviour of a sample of that population.  We don’t know for sure that this is the case but we can work out the probabilities involved to make a bridge between what the sample behaviour is and apply that to the whole. 

Finally if you want to follow along you’ll need nothing more than a modern browser and spreadsheet tool that can work with .csv files and can handle 400,000 rows of data and I’ll write this up in the next post.

Note: I can’t share the actual data for the events model as the data and model aren’t mine.

Disclaimer: This post is one of a series about data science from someone who started out in BI and now wishes to get into that exciting field so if that’s you read on if not click away now

Firstly how does data science apply to this problem?  if we had access to some data about events where we have the attendance details including who registered but didn’t show up  then we could possibly make inference about what might affect attendance and use these characteristics to make predictions about drop out rates at future events. 

In the world of statistics and surveys we are using a sample (the historical data) to infer the behaviour of a wider population ( people who come to IT community events).  The important thing here is that I have good data and in this case good might mean that there aren’t invisible (confounding) variables affecting attendance e.g. road accidents and transport strikes might mean some people who wanted to come found it too hard to travel and went home.  Also some of my friends in the SQL community reckon that if the weather is good then drop outs will be higher so if I don’t take sunshine into account than that may also affect the accuracy of any model as well. However while modelling these is fine  we would have trouble predicting travel disruptions and predicting sunshine is hard as well.

Assuming we have a viable model it’s then up to us as humans on how we use it:

• We could overbook the event by the amount of predicted dropouts.
• While it would be hard to ban no shows from future events we could target potential no shows with e-mails to encourage them to come or at least let us know if they can’t make it.  We can then see if this works by only doing this e-mail targeting for some events.

So that’s the basic idea and to get started we need data but which data i.e. what factors might affect non attendance and is that data available?

Let’s look at the factors for the event itself:

• Time of day – is it an evening or day event
• Day of the week – At Microsoft we reckon that you our audience won’t be as likely to come to events on Mondays or Fridays and then there are weekend events.
• Location? Is that a factor or are there factors about the venue such as parking and being near a train station?
• What technology is being covered?
• Does the headline speaker matter. For example at AzureCraft we had ScottGu over and certainly this leads to more registrations but was the dropout rate lower?
• and as I said earlier was it sunny!

Then what about the delegate information? 

• I don’t think someone’s name or address is important, however their proximity to the venue might be. So perhaps  we could get still get some location insights  but anonymise the address by working out the distance to the event or better yet the journey time to the event and plug that in. 
• Whether what marketeers call the vertical they work in, matters e.g. finance, public sector( local central , NHS etc.) , retail and so on might be interesting to look at.
• What about the role of the delegate e.g. developer, dba , infrastructure specialist, architect etc. 
• Are they regular followers
• Whether they attended or not

To put this last statement into data science speak this is supervised learning - the model being built is based on example data where the answer is in the data we are using which in this case the attendance of a registered delegate.  There are only two values in the answers “attended”  or “did not attend” so  this is known as a two class or binary classification problem.  There are several completely different techniques for modelling this and choosing the right one is partly about understanding what success is and partly by knowing certain characteristics about the data. 

Success in Machine learning might mean going for accuracy at the expense of time to compute or the other way around. some techniques/algorithms like neural networks take time to train, are resource intensive but are more generally more accurate. It’s also important to understand what data we are dealing with and there are three things to be aware of

Features & Labels

You may have heard the term feature engineering and that is the business of having the right attributes/ columns/ variables to use to predict an outcome.  For this example the features are the list above of things we know about the event and delegate. In supervised learning we then use these features to predict a label which  in our case the column/variable with whether the delegate turned up or not. 

Data Types

I have already mentioned that the label in this case is one of two categories  (attended or did not attend) and so the label holds categorical data.  There are other features described above that are also categorical like the role the delegate has, the location of the event and anything  else that is a  string.  However numbers can also be categorical – the staff ID at Microsoft, my mobile phone number as these are just lookup codes to identify me or some object – In other words we can’t apply maths to them e.g. average min max etc.   On the other hand  quantitative data are numbers where we can do this like my age, the journey time for a delegate to get to an event.  Finally there is also ordinal data like education level where BSc, MSc and PhD are levels where one is more advanced than the others.  Sometimes we can make quantitative data ordinal by creating buckets and putting values in them like age ranges you often see in surveys (under 18, 18-25, 25-40 40+ etc.)

Data Understanding & Visualisation

Then we need to understand how  the data is distributed compared to the thing I am looking to predict and across the other features.  Data Scientists often want to plot data to do this like this box plot showing how inferences can be made about the speed of light form the famous Michelson Morley experiment..

Here the box shows the middle 2 quartiles of data split by the median the “T” and upside down T shows the minimum and maximum and the dots in column 1 & 3 show excluded outliers from this analysis 

Plots like this and building up layers of charts are one of the great strengths of R or Pandas in Python as well as the rich ecosystem of statistical algorithms they have.  They inform the feature selection and engineering as well as evaluation of how well the model is performing. 

Data Wrangling

The whole process of understanding the data and feature engineering is known as data wrangling which  If like me you are from a BI background you may think that Excel and technologies like Integration Services in SQL Server are the tools for this but data wrangling goes beyond this as it’s as much about understanding the data as transforming it. So I have seen people using Azure Machine Learning just as a data wrangling tool as it puts a public web front end at scale over their Python & R scripts and to be able to share their notebooks containing their work.  My point is that not only do need to manipulate the data we need to use statistical techniques to analyse it like from simple plotting of logs to look for outliers to the common techniques like Chi Squared, Pearson, Spearman, Kendall Tau etc. depending on the problem space.

Hopefully that’s given you some ideas on how all this works in a simple scenario.  However that simple scenario with different features could be used to look at patients not turning up for appointments with their GP, truancy in children and that holy grail of marketeers customer churn. If you want to know more then there is a degree course for this and currently my favourite part of that is module 4 on statistics from Columbia University. They are free to evaluate and $49 per course if you need a certificate.

So what do you need to get started? – My Top Six would be:

Curiosity

Are you the sort of person that is always challenging why things are the way they are in your organisation instead of taking matters on trust?  Are you keen on experimenting with new technologies? Are you interested in the art of the possible? If the answer is yes then data science might be for you if you are also..

Evidence Based

I got a great e-mail after a presentation I gave from a lady called Charisma from Ghana – “In god we trust everyone else bring data”. Data scientists need to be scientists by using their creativity and curiosity to come up with hypotheses and then prove them.

Statistics.

Yes you do need to understand the basics of statistics, but with the amazing tooling out there and the great online resources for learning like EdX and Coursera you can acquire the principles and then quickly apply these to your problem domain.

Data Skills

Hopefully you are reading this because you are some sort of data professional e.g. from a BI or data background.  You know about joins, you get data quality and worry about missing data, cartesian products, and how annoying working with dates can be.

Business knowledge

.  Hopefully you understanding the domain you are working in and what you do is closely aligned to that.  This is your sanity check that means that when you find a cause an effect from a piece of analysis it does actually make sense and is not merely a couple of random statistics that have the same curve e.g. Per capita cheese consumption is not really related to deaths by getting entangled in the bed sheets as per Spurious Correlations by Tyler Vigen:

Ethics

Just because we can doesn’t mean we should.  In the work I do I am trying to make a difference so my data science dojo is a hack with a charity trying to improve a given situation be that environmental issues, saving lives or at least improving the quality of it.  I also mention this because computer don’t have ethics per se any more than a new born baby does they are imbued with behaviour patterns by other humans.  My litmus test is would my reputation improve  if my use of analytics and big data was made public. 

Is data science is for you?

I mention all this because demand for data scientists is outstripping supply and while many people like my are working with academia to drive and promote the next generation of data scientists we have a huge hole today.  Now I am not going to suggest you can just mug up on some of this stuff and put data scientist on your CV. Rather I am suggesting their are places in the data science world for people like me who have come from a data/BI background as many of our skills are transferrable.  For example our knowledge of the business and data skills are still very valuable.

However we are likely to be light on statistics and even if we studied it to some extent we have forgotten a lot of it and what matter now is applying those statistical theories to our data.  For example which algorithm should I use to select which attributes of a patient and their doctors appointment are most likely to influence their attendance at a doctors appointment?  In Azure Machine Learning (MAML) if you use the filter based feature selection module to do this you are presented with seven options such as Chi Squared, Spearman, Pearson, Fisher, Kendall but which to use?  The answer is a post in itself and the good thing about data science is that there lots of resources on line.  The bad thing about the online content is that while much of it is written in English it is English-Stats not English-GB so can be very hard to decode as so much prior knowledge of stats is needed.  On thing I use a lot is  this article on which algorithm to use for what in MAML and even if you are in R or Python or some other technology this is still useful.

So a question – Do you want to get into data science?

To test this hypothesis for yourself – I would encourage you to look at the Microsoft data science degree course on EdX.  This can be done for free to see if you like this stuff, but if you like what you are seeing then you can actually get a degree.  To do that you have to pay for a verified certificate for each of the ten modules you must do and there are options at various stages..

The last module is an actual project where you apply what you have learnt, and on a side note my good friend Amy Nicholson (@amykatenicho) has just made the module on Developing Intelligent Applications so watch out for tough questions on this!

So let’s do some science, see if the course helps and feedback any comments you have as you learn.

Note: none of these degree course is going to get deep on the stats I have ben talking about but there are courses out there for that too.  My own journey has been greatly helped by Prof Andy Field and his book on Learning statistics using R.  He’s also all over YouTube..

You have been warned!

Here’s the problem we all forget to turn stuff off or kill it when we don’t need it which can be because it can be a hassle to recreate.  So in my simple world I want to make sure that expensive resources like VMs and my Azure SQL Data Warehouse aren’t burning costs (as I have mentioned before). The answer to this is hidden away in Azure - Azure Automation.

It pretty much does what it says it does, but frankly this used to be a bit of a pain partly because of the disjoint between the kind of PowerShell around at the time which is based on Azure Service Management and what we have now with Azure Resource Management.  There have also been changes to how subscriptions can be controlled and this comers into the mix too.

So what do you need to do manage these services with a firm hand?

First I am going to assume you have access to an Azure subscription.  All you need to do is track down Azure Automation and set your account up. 

This is essentially a container for all of the assets needed to control resources in a subscription, so that they can run when we aren’t around to click on the portal.  When we create one of these are asked for the usual setting for any Azure service and we also get the option to create a run as account. 

Once we have our account we can see that it is essentially a container for the resources we need to manage services:

On the left blade we can see my runbooks which are the scripts themselves and then I have expanded the assets to see the other stuff I need to make my runbooks work :

Schedules to determine when to run jobs for example my nightly schedule runs every night at 10:30 and will do so for ever..

Modules are the PowerShell modules I can use just like I would have set up on any machine..

Connections which is that Run as account I created when I setup the account..

Credentials to authenticate against what we are trying to manage and I have one of those to connect to my MSDN subscription

Certificates to enable trust between our script and what we are managing

Variables that can be consumed and updated in our scripts for example here I have a variable ResourceGroup of type string..

So let’s see write and publish a run book and put it into production.  For this worked example I am going to do nothing more than pause my Azure SQL Data Warehouse every night as it has cost serious money in the past because I left it on.  To create a Runbook click on the Runbooks icon.  Here you can see what I have done already and if I click on the Browse Gallery icon there is a long list  of  examples from Microsoft and the wider community to help us get started..

If I filter on warehouse there is a script to pause and restart it based on the parameter it is passed.

To use that script i could setup credentials but I wanted to save time and use the built in run as account for this so I tweaked the sample script as follows..

It’s just a matter of clicking on it to import it , naming it and then editing, it all form the portal (by clicking on the edit button) ..

however to use this script I am going to have to create a credential where I just want to use the run as account. Not a problem other example in the gallery do this so all I need to do is substitute this in place of line 3-13..

$connectionName = "AzureRunAsConnection"
$servicePrincipalConnection=Get-AutomationConnection -Name $connectionName        
         "Logging in to Azure..."
         Add-AzureRmAccount `
             -ServicePrincipal `
             -TenantId $servicePrincipalConnection.TenantId `
             -ApplicationId $servicePrincipalConnection.ApplicationId `
             -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint

Notice that in sharing this I haven’t compromised my subscription where in my old style PowerShell passwords and credentials were embedded all over the place which is not a good story for production
The next cool thing is we can test this by clicking on the Test Pane in the editor blade in the portal, and if the script need parameters we’ll be prompted for those..

Now we have tested it can be published (another button on the editor. Once that’s done we can setup a schedule e.g. I have a nightly (10:30 every night) schedule that never expires.

If you have a job that needs parameter like this script to stop all VMs for the specified Resource Group and whether to shutdown the VMs then we can set these as part of this process..

The blades for Azure automation also show us how the overnight runs are doing as per the screenshot at the top of this post  and hopefully yours will be nice and green too .

So Azure Automation makes managing service a little bit easier and is something us data professionals need to know.  However for a deeper dive into this world you’ll want to track my good friend Marcus Robinson who is really hot on this.

I am in the middle of preparing for some more events the next big one being a day of machine learning (ML) at SQL Saturday Exeter.  The organisers asked me to make a short video to explain what I’ll be covering and that got me to thinking about what my advice would be having worked with the tool for a good year.

What’s the question?  Unlike many aspects of BI and data mining we are not on a fishing expedition, and although ML can just be use to explore data with its built in tools or via your own Python / R scripts that is not what is was designed for.  The typical use cases for ML - predictive maintenance, customer churn, sentiment analysis are all about answering very specific questions when will a part fail, which customers am I about to loose, was that tweet being nice about me or not?

Does the answer lead to action? At recent briefing on ethics in medical data one of the speaker remarked that a significant factor in patients not completing their treatment was their credit card rating , the lower it was the more likely they would drop out.  The problem is that it’s not easy to see what to do about that (no suggestions please), whereas in more typical customer churn scenarios, we might setup some sort of offer to retain those at risk. 

For me this is why ML has not quite taken in many organisation as expected based on the massive interest in it – It wows everyone but then it becomes of a question of applying it in organisations. Having said that I think a massive area is process automation and checking e.g. in contracts and financial approvals to detect fraud and spot mistakes.

ML is not perfect.  We are very used to humans making mistakes indeed I would argue unless you made a mistake today you haven’t learnt anything. Learning machines also make mistakes - they can’t perfectly recognise speech or images for example. However they are starting to equal or out perform us in some of these areas.  My point here is to have a process in place to handle this. Say we are in the business of approving  loan applications -  If ML is not confident of the outcome it will have a mid-range confidence value (say 0.33 to 0.66), where a lower score will automatically be rejected and a higher one accepted.  For the mid range values we might then refer it to a human to decide.  Similarly if ML can’t decode the postcode on a letter then maybe a human can- but then again in image recognition ML has ben proved to be better at humans when it comes to recognising objects and faces.

ML needs to be retrained.  An example is the way to explain this – an algorithm for basket recommendation maybe superb but when new products appear there are no patterns to follow and even if they are these may not be typical behaviour.  So for production you’ll need a process to do this retraining and ideally this should be programmatic rather than interactive.  This is not any harder than consuming an API as you would to make a prediction. However in this case it’s a batch call with your training set as the argument.

Algorithms are about numbers not industries.  The patient not following though on treatment, the student dropping out of a course are just like the customer churn long used in retail and the same algorithms will work equally well in these other areas as all ML is doing is comparing correlations of factors against what we are trying to predict.  Similarly anomaly detection works for bank fraud and for predictive maintenance

Not your normal neural network.  Buried in the neural network module in ML is ability to do deep learning using a special scripting language net# (nothing to do with .Net really). It is more for the hardened data scientist and if you want to know more check this session at last year’s Cortana Analytics Workshop.

ChChChanges.  ML is continually evolving and this is important for two reasons, one check back regularly to see what has been added, and two if you have feedback then share it to drive features you feel you need.  A good example of this is the much deeper support for Python in ML.  However I am more interested in seeing more algorithms e.g. time series analysis and more OpenCV modules to handle images, both of which will mean we should be writing less of our own code anyway.

Do Something.  The only way to begin to see the potential of ML in your organisation is to try it. Yes you need a question, yes you need data, and yes you do need to be scientific in your approach to the problem.  However you don’t need to be a full on data scientist, you just need their curiosity, added to you data wrangling skills and business knowledge.  The Cortana Analytics Gallery is awash with interesting business based scenarios and tutorials to help you, so don’t just sit there…

Fortunately, we are allowed to make mistakes at work the trick is not to make the same one twice!!

In this case the mistake was that I made was to leave this running and at this point it’s important to understand that while an ADW server looks a bit like an ordinary Azure DB server it’s actually a very different thing (apart from the cost that is).  In Azure DB Server you can quickly scale up and down a server but you can’t completely shut down the server and leave your database parked there for later. 

However not only can you pause ADW, you can do this safe in the knowledge that your data will still be there for later when you want to resume your work.  I should also mention that of course ADW scales massively more than ADB.  In the screenshot below I have two databases running on my DeepFatDW Azure database server

where TechNetDW is a data warehouse and DeepFat is an “ordinary” database which cannot be paused. ( but can be scaled up and down to a certain extent).

I can pause and restart this ADW and I could have done this in PowerShell, for example

Suspend-AzureSqlDatabase`

–ResourceGroupName "ResourceGroup11"`

–ServerName "DeepFatDW" `

–DatabaseName "TechNetDW"

Plus I can also dial up/down the compute power against the database, without affecting the data itself, so I could have a truly massive DW with only minimal compute against it until I realise I need more power 

Set-AzureSQLDatabase

-DatabaseName "TechNetDW"`

-ServerName "DeepfatDW.database.windows.net" `

-ServiceObjective "DW100"

and I could go further and embed this in Azure Automation to schedule a scale down or pause overnight.

So ADW is a great example of how services should work in the cloud, where scale down to save costs is as important as scale up, while data is protected, and you are in control. 

About a decade a go I went on a Microsoft BI boot camp which was also  attended by Microsoft staff specifically Matthew Stephen and Rob Gray who inspired me to work towards joining Microsoft myself.  Two years later I got in and haven’t looked back and I still track Matt on LinkedIn, so I was very interested in his article on Data Lakes.  I won’t repeat that here other than to summarise it briefly for those that don’t like to click on hyperlinks.

Data lakes are large heterogeneous repositories of data the idea being that essentially you can quickly store anything without worrying about it’s storage.  Contrast this with a carefully curated data warehouse which has been specifically designed to be a high quality consistent store of business critical information.  Of course this means that Gb per Gb  data /.nbvcx much more expensive to build and maintain but this is because every bit of data in them is directly useful and applicable to the business.

On the other hand a data lake will contain vast amounts of less obviously useful data for example while the data warehouse might contain all the transactions from the online store, the data lake would store all of the logs from the site running the store.  The former is immediately valuable in showing sales trends the later seems to be largely useless.  However if storage is nearly free (and it largely is if data is stored in the cloud) we can afford to keep everything on the off chance it might be useful.  But we couldn’t justify processing the log data into our warehouse as this would be expensive in terms of compute and the human effort to make this work.  Not only that we aren’t sure what we need from this other kind of data or what it will be used for. Instead we just catalogue it and hang on to it until we realise we need it.

Let’s take an example – the company on-line store.  The actual sales transactions get written into the data warehouse, but we decide it might be useful to store the web logs from the site as well, so they get thrown into the data lake, typically in a folder/container with a file for each time slice/ web server and the process is just a simple file copy from the web server.  A couple of weeks later there seems to be something odd happening with the site and the cyber security firm we engage ask to see our logs and so we can provide them access to our data lake (we archive off the logs form the actual site to keep the servers nice and lean).  They use their tools to analyse this and advise on a mitigation strategy.  A few months later analysis of the data warehouse shows that the number of new customers being acquired is dropping despite discounts and promotions being in place.  The suspicion is that the culprit might be the new customer sign up experience and a design agency is called in to conduct click stream analysis over the stored web logs to confirm this.

A couple of points come out of this.

• The Data Warehouse is still needed for the reporting & analytics that we have always done.
• The use of the logs wasn’t known when the data was stored and this data might not have been used at all. so trying to second guess how to store it at the time of arrival would be at best difficult and could be pointless.
• What is important is to catalogue what is stored in the lake and its structure as well as how it relates to other data be that in the data lake or in the data warehouse. 

So in my opinion the term data lake is not a great description of what it does (how about a data museum?) and without understanding what is in there it is not a lot of use. So a key companion to the Azure Data Lake is the new  Azure Data Catalog. 

This new freemium service does is a bit like Master Data Services in SQL Server, but because it’s on line on Azure  it’s accessible and available by default. However it also leverages and requires  Azure Active Directory ¹ so that catalogs are protected while still enabling teams inside an organisation to collaborate.

If this is used as intended then not only do the data professionals record what is going into the data lake AND the data warehouse, everyone shares their knowledge about how data is used and how to join the dots to do specific analytics.

So I would suggest two things start having a look at the principles of master data management in general and think about how this might play out in your organisation by having a look at the Azure Data Catalog.

^1. This means that if you have an MSDN subscription linked to a personal account you may not be able to sign up to try this service however you can setup Azure Active Directory inside your subscription and then make a domain account a member of your subscription (a topic for another post).

A different kind confusion seems to surround another icon of science fiction Cortana. Originally a fictional AI  that was embedded in the Halo franchise of XBox games, Cortana is the name and inspiration of your digital assistant on phones and in Windows 10.  That sort of makes sense as she was a very smart advisor to the Master Chief in those games, and now she can help us with our daily lives, if we want her to.

However the real point of confusion seems to be the announcement of the Cortana Analytics Suite, specifically what is it and how does that relate the assistant on Windows 10 devices and phones?

From a services perspective Cortana Analytics Suite represents the collection of data tools to do analytics that exist in Microsoft’s cloud services, Azure and Office 365. So it includes Power BI , Machine Learning, Data Lake, Data Catalog and Data Factory, amongst others as well as the digital assistant, Cortana..

This makes it a superset of all of these capabilities much in the same way as Office 365 covers traditional tools like Excel and Word as well as newer ones like Sway.  However today this is just a collective term in that you can’t buy Cortana per se, so is this just the marketing folks trying to create impact and build on what is a cool brand?

Not really, as that set of services can be used by any developer to create intelligence in application that also mimic Cortana in that game.  For example you could develop applications to monitor your health and your team’s health through wearables like the Microsoft Band, recommend courses of action based on behaviour to make your web sites appear more personal and relevant to predict what action to take before systems fail and if allowed to carry out that preventive maintenance,

Is Cortana Analytics Suite relevant?  I have already done a series of posts on these services and how they can be wired up to create solutions around predictive analytics, big data, near real time analysis and “traditional BI” – I just didn’t know at the time this would all be joined up in this way. What this means for me is that I am betting my future on Cortana.