Developing Analytics Solutions with the Data & AI Global Practice https://docs.microsoft.com/archive/blogs/data_insights_global_practice/feed.xml Tue, 16 Apr 2019 04:58:05 GMT en-US hourly 1 Can a machine learning product be trusted? https://docs.microsoft.com/archive/blogs/data_insights_global_practice/can-a-machine-learning-product-be-trusted Tue, 13 Nov 2018 21:17:09 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=1275 Author: Michele Usuelli, Lead Data Scientist, Microsoft Enterprise Services The data scientist develops products based on machine learning models, having as a result an actionable answer. In other words, a machine is advising an end user about some actions to perform. Why would we trust the advice of a machine? Before designing a machine learning product, it's crucial to keep in mind its application. This article covers the key aspects to take into account to have a reliable product. Specifically, we will look into predictive models, aiming to foresee future events. To be more concrete, we will be utilizing three real-life examples
  • Predictive maintenance: the engines of one or more machines break down frequently. The predictive model foresees whether there will be an engine failure in the near future.
  • Credit scoring: a bank provides its customers with loans. Having a new customer requesting a loan, the bank estimates the likelihood of having a default.
  • Sales forecast: a retailer is interested in predicting the sales volume of one or more items.
Collecting lessons learned from data science projects, common key aspects are the following.
  • Interpretability: capability of explaining the rationale behind a machine learning model.
  • Measurability: assessment of the performance and accuracy of the model.
  • Actionability: influence on the end user to perform a specific action.
Each aspect is covered by the related section.

Interpretability

Predictive models foresee the breakdown of an engine, the credit score of a customer or the sales volume of a new product. In either case, how do we know if we can trust the outcome of a model? Also, can we check if it complies to rules and abides by ethical principles? To enable that, the first step is ensuring that the model is explicable. Let's look into our examples We might predict that an engine will break down because it's running slower than usual. A bank customer will likely default because of a high credit card debt. The sales volume will raise because of a commercial promotion. These example are close to reality, but often the set of useful information is wider. Still aiming to explain the rationale behind its outcome, a machine learning model combines and utilizes more useful information. Machine learning models are based on data sources, so  tracking all the information being used is crucial. In this way, we can easily spot ethical issues and legislation incompliances. The next step is to identify which specific information is being used by the model and what is being discarded. Ultimately, applying the predictive model on new data, the information being used is a small highly-relevant subset. Depending on the technique, the methodology might be different. It's usually straightforward to explain tree-based models such as the decision forest, as shown by my article. Likewise, linear models can be easily interpreted. However, some techniques, e.g. neural networks, are too complex and obscure. Even in those cases there are workarounds such as LIME and SHAP, showing what would have changed if the data differed.

Measurability

Let's assume that a predictive model is already explainable. The thought process to define its outcome is fully transparent and we can track all the information being used. What will happen applying the model in a real-life situation? Will it be able to foresee future events accurately? If we predict engine breakdowns, we want to know how many of them we can predict and the number of false positives. The same applies to credit risk. If we are predicting the sales volume, a metric could be the average error, whichever way the "average" is defined. Being able to assess the reliability and accuracy of the outcome of a model is a key to be trustworthy. Clearly, it's impossible to know how a model will perform in the future. However, it's possible to know how the same methodology would have performed in the past. Predicting something that is already known is trivial. Therefore, a common pitfall it to "cheat", either voluntarily or involuntarily. To prevent that, the key is to replicate the real life application. Some important aspects are the following.
  • Separate training and test set entirely.
  • Take into account the time: in most cases, we are predicting and event in time. A common pitfall is to predict the past utilizing future information.
In addition, the outcome of the testing should reflect the broader context. Key aspects are the following.
  • Business focus: in addition to measuring the accuracy, are we also translating it into measurable business KPIs?
  • Benchmark: if we weren't utilizing a predictive model, by how much would the forecast be worse?
My colleague Ajay published a series of two articles, showing how to measure the performance of classification and regression models.

Actionability

Last but not least, even if we can explain and measure the outcome of a machine learning product, to have a tangible impact it will trigger an action. In the predictive maintenance scenario, the action is to either send an engineer to check the machine or to provide confidence that the machine doesn't need any maintenance. In the loan use-case, the action is to either approve or reject the loan. Predicting sales helps optimizing the production volume. Prior to start experimenting and building the product, having in mind the end action helps defining the direction of the project. After having delivered the product, keeping track of the actions and feedback allows to improve the product further. Actionability applies to both interpretability and measurability.
  • Interpretability: knowing the root causes allows to define an action to fix them. Not all the root causes are necessarily actionable. Knowing what can be changed is an enabler to define an action. For instance, in the predictive maintenance scenario, the action is to fix the root cause leading to a failure. If the root cause is having the last maintenance too far back in time, the solution is to send an engineer to maintain the machine.
  • Measurability: the estimated error is as important as the prediction. Knowing the actions that can be taken enables the definition of an accuracy metric accordingly. In the sales forecast use-case, the accuracy metric might be related to the way the forecasting will be consumed.
Actionability is the starting point and the end point of most of the data science projects.

Conclusions

This article covered interpretability, measurability and actionability, key aspects of each data science project. One remark is about other important topics, such as ethics, privacy and compliance, that will be enabled by these.]]> How to deliver a data science project https://docs.microsoft.com/archive/blogs/data_insights_global_practice/1055 Tue, 04 Sep 2018 23:18:06 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=1055
  • Who to work with and how: "Keep the key people involved".
  • How to build the product: "Iteratively build the product".
  • What are the dependencies and limitations: "Proactively highlight dependencies and limitations".
    • The assumptions are that the project already has a measurable business impact and a clear scope, and that the data is available and relevant.

    Keep the key people involved

    With "people" we are referring to specific roles in a project. There might be more than one person covering the same role whereas the same person could cover more than one role. There are several people involved in a data science project and they belong to either of these two groups:
    • Delivery team: people actively building the product. To keep the article simple and focused, we only mention the data scientist. Other aspects of the project, such as the IT infrastructure work, are equally important, but they're not covered for the sake of simplicity.
    • Everybody else who is involved: who is related either to the context or to the outcome of the project. The key people are the stakeholder, the domain expert and the end user.
    Depending on the project, the team structure might differ. However, there is always someone sponsoring the project, someone providing domain expertise and someone ultimately utilizing the product. The same person could cover more than one role, but it's easier to think in terms of people to structure how the data scientist captures different perspectives. Let's have a closer look at the three key people:
    • The stakeholder is ultimately impacted by the outcome of the project. Therefore, the data scientist should get constant feedback from the stakeholder to make sure that the project is heading towards the right direction and to tune the project objective in the most impactful way. Also, to keep the stakeholder engaged in the project, it's usually a good practice to provide them with quick results driving some value or at least showing useful insights.
    • The subject-matter expert has domain knowledge related to the context of the problem and the meaning of the data. Therefore, the input of the expert should be the starting point of any data science project. Also, the data scientist will be able to discover new insights that will benefit the expert, and about which the expert will be able to provide useful advices. The checkpoints between the data scientist and the expert should be frequent.
    • The end user is the person ultimately utilizing the product. To design a solution for success, a key aspect is to be aware of its implication. Therefore, the data scientist needs the input and feedback from the end user as well.
    The key questions are:
    • Do we have a stakeholder, an expert and a user actively involved in the project?
    • How often is the delivery team communicating with them?

    Iteratively build the product

    We identified three key people the data scientist interacts with. This section highlights:
    • What inputs the data scientist gets
    • What outcomes the data scientist gives
    Some relevant aspects are:
    • Stakeholder management:
      • What questions are addressed by data science? What is the business impact if the questions remain unanswered?
      • Is there any quantifiable value or is the project mostly driven by hype?
      • Is data science the most appropriate methodology to answer the questions?
    • Leverage the expert knowledge:
      • How much business expertise is the model capturing?
      • Do the results validate any prior business expertise? Do they highlight new areas that were previously unknown?
      • Is there anything contradictory in the results?
    • Provide the end user with transparent results:
      • Is it possible to explain the overall flow of the solution in simple terms?
      • Can the end user trace the data sources and how they are being used?
      • If we weren't using a machine learning model, how would the solution design differ?
    The following chart summarizes the interactions The information required to deliver the project often depends on the findings from the data. Typically, at the beginning of a project, the data scientist is not fully aware of all the required information. Therefore, it's crucial to keep all the three players engaged throughout the project to iteratively get more input and feedback. The product is built in small steps and in an agile manner. Having regular checkpoints with the key people ensures that the data scientist has the information needed to define the next steps.

    Proactively highlight dependencies and limitations

    To scope a project for success, it's crucial to proactively highlight dependencies and risks. Besides the input from the key players, data science projects require some technical input that can be summarized in these three areas:
    • Data: the sources of information being used by the model.
    • Techniques: the algorithms that discover patterns from the data.
    • Technology: the technical infrastructure to store the data and run the algorithms.
    Each area has a strong impact on the goodness and performance of the solution. Some key questions are:
    • Data
      • Is the data consistent and complete? Is it possible to match the data sources?
      • Does the data capture most of the relevant information? What is missing?
      • Do we have enough history of data?
    • Techniques
      • Are the techniques explainable?
      • Are the techniques simple and generic?
      • Do the techniques perform consistently if applied on another dataset?
    • Technology
      • Can all the data be stored and analysed?
      • How fast can the solution be built and applied?
      • Can the data volume be handled?
    Data, techniques and technology define the boundaries of the performance. To have a reliable and trusted solution, it's crucial to highlight the boundaries since the very beginning of the project and to regularly track them.

    Conclusions

    The chart below summarizes the article.
    This article hasn't covered all the aspects. Other important topics are the IT infrastructure work and the attributes of a successful data science solution.
    ]]>         Measuring Model Goodness - Part 2 https://docs.microsoft.com/archive/blogs/data_insights_global_practice/measuring-model-goodness-part-2 Tue, 04 Sep 2018 17:29:57 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=995 Author: Dr. Ajay Thampi, Lead Data Scientist

    Measurability is an important aspect of the Team Data Science Process (TDSP) as it quantifies how good the machine learning model is for the business and helps gain acceptance from the key stakeholders. In part 1 of this series, we defined a template for measuring model goodness specifically looking at classification-type problems. This is summarised in the table below.

    Classification

    Quantify Key Business Metrics

    Once the business context is understood, quantify the key business metrics as follows:

    • The value and cost of predicting true positives/negatives and false positives/negatives, respectively. For a binary classification problem, this can be represented as a 2D value-cost matrix.
    • The percentage of cases that can be automated by the model without requiring human intervention.

    Define Reasonable Baselines

    By doing some exploratory data analysis, define reasonable baselines as follows:

    • Random classifier: A model that determines the classes randomly.
    • Majority-class classifier: A model that picks the majority class always. This is relevant when the classes are highly imbalanced like in the case of fraud/spam detection.
    • Current deployed classifier: This can be a model/human that the customer has currently deployed to perform the classification task.

    Build and Evaluate Model

    Using historical data, train a classifier and tune hyperparameters using cross-validation. Then evaluate the model on a hold-out test set using the following metrics:

    • True Positives
    • True Negatives
    • False Positives
    • False Negatives
    • Precision
    • Recall

    The first 4 metrics can be visualised using a confusion matrix and ROC curve. The last 2 metrics can be visualised using a PR curve.

    Translate Model Performance to Business Metrics

    Finally, translate the model performance metrics to the key business metrics as follows:

    • Expected value: Can be computed by using the confusion matrix and the value-cost matrix.
    • % cases to manually review: Can be computed using the precision, recall and actual positive class rate.
    In this post, we will look at regression which is another broad class of machine learning problems.

    Regression

    Regression is the process of estimating a quantitative response. It is a supervised machine learning technique that given a new observation, predicts a real-valued output. This is done based on training data that contains observations whose real-valued outputs are known. In this post, I will look at a specific example of regression which is to predict the dollar value of a home. I will be using the open Boston house prices dataset  which consists of 506 examples, 13 numeric/categoric features such as per capita crime rate, average number of rooms per dwelling, distance to employment centres, etc. and 1 real-valued target representing the median value of a home in 1000s of dollars. More details on the dataset can be found in the Jupyter notebook that I created for this topic on Github.
    [caption id="attachment_1005" align="aligncenter" width="500"] Photo by Adrian Scottow on Flickr[/caption]
    The business need is to predict the home value (or a real-valued output in the general case) with as low an error as possible and to also be able to see how confident we are of our predictions. These needs can be quantified using the following metrics:
    • Error metrics:
      • Mean/Median Absolute Error (MAE)
      • Mean/Median Absolute Percentage Error (MAPE)
      • Root Mean Squared Error (RMSE)
    • x% Confidence Interval (CI) or Coefficient of Variation (CoV) for each of our estimates
    The 3 metrics that are typically used to quantify the estimation error are MAE, MAPE and RMSE. It is important to understand the differences between them so that the appropriate metric(s) can be used for the problem at hand. Both MAE and RMSE are in the same units of the target variable and are easy to understand in that regard. For the housing price problem, these errors are expressed in dollars and can easily be interpreted. The magnitude of the error, however, cannot be easily understood using these two metrics. For example, an error of $1000 may seem small at first glance but if the actual house price is $2000 then the error is not small in relation to that. This is why MAPE is useful to understand these relative differences as the error is expressed in terms of % error. It is also worth highlighting a difference between MAE and RMSE. RMSE has a tendency to give a higher weight to larger errors - so if larger errors need to be expressed more prominently, RMSE may be the more desirable metric. The M prefix in MAE and MAPE stands for either mean or median and they are both useful in understanding the skew in the error distributions. The mean error would be affected more by outliers than the median. The CI and CoV metrics can be used to quantify how confident we are in our estimates. By choosing an appropriate threshold for the CoV, we can determine the proportion of cases that we can predict with reasonably high certainty and therefore can potentially automate those. There are of course other ways of quantifying uncertainty [1, 2, 3] but in this post, we will only cover CI and CoV. We now need to define reasonable baselines to compare our models to.
    • Baseline 1: Overall mean (or median)
      • For the housing price example, this is the overall mean value of all houses in Boston.
    • Baseline 2: Mean (or median) of a particular group
      • For the housing price example, we can define the group as houses with a certain number of rooms. The predicted value will then be the mean value of the house with those number of rooms. We need to make sure that we have enough samples of data in each of the groups. If there aren't sufficient samples, then we should fall back to the overall mean.
    • Baseline 3: Current deployed model
      • Since we do not have access to this model for our example, we will drop this baseline.
    The next step is to train a regression model. For this example, I used a pipeline consisting of a standard scaler, PCA to potentially reduce the dimensionality of the feature vector and a gradient boosting regressor to finally predict the value of the house. The quantile loss function was used so that we can predict the median value (alpha = 0.5) and the lower and upper values of the confidence interval (defined by alpha of 0.05 and 0.95 respectively). The code used to build the model can be found here. Once the models are trained, we then need to evaluate them. We consider the following evaluation metrics:
    • Absolute Error (AE): This is defined as | y - ŷ |, where y is the actual value and ŷ is the estimated value
    • Absolute Percentage Error (APE): This is defined as | y - ŷ | / y * 100
    • Squared Error (SE): This is defined as (y - ŷ)2
    We can now translate the above metrics into the final business metrics as follows:
    • Mean/Median AE: Mean/median of the absolute errors over all examples in the test set
    • Mean/Median APE: Mean/median of the absolute % errors over all examples in the test set
    • RMSE: Square root of the mean squared error over all examples in the test set
    The above errors are visualised in the figure below. Observations:
    • The Quantile Regression (Q-Regression) model does much better than the baselines achieving the lowest values for all error metrics. The mean and median APEs for the model are 10% and 7% respectively.
    • The mean and median error metrics show that the distribution of the errors is positively skewed. This is also reflected in the RMSE metric where RMSE is greater than MAE showing the effect of the larger errors.
    In order to get a better understanding of the distribution of the errors, it is useful to plot the cumulative distribution of the APE. This is shown below.  It can be seen that for the Q-Regression model, 90% of the cases achieve an error of less than 20%. For baselines 1 and 2, we only get roughly 60% and 50% of the cases respectively, with less than 20% error. Finally, in order to determine which cases we can accurately estimate and in turn automate, we need to look at the confidence interval (CI) and the coefficient of variation (CoV). The 95% CI is estimated using the Q-Regression model by setting the quantile parameter alpha to 0.05 and 0.95. This gives us a rough measure of the spread of the estimate value. We can then compute a rough measure of the CoV by dividing the spread by the median predicted value. We expect to see a higher CoV for an estimate that we are highly uncertain of. Ideally, the CoV will also be correlated with the error metric but that need not always be the case. By setting a threshold on the CoV, we can determine how many cases we can predict with reasonably high certainty and thereby, automate. In this example, a CoV threshold of 0.625 is chosen. This results in automating roughly 50% of the cases with reasonably high certainty in our estimate value. The figure below shows the box plots of the APE metric for all cases and the 'automatable' cases.

    Observations:

    • The 50% cases identified as 'automatable' have much lower errors than all the cases.
      • Mean APE drops from 10% to 6%
      • Median APE drops from 7% to 5%
      • Maximum APE drops from 92% to 17%
    This concludes the two-part series on measuring model goodness. We've looked at measurability through the lens of the business and defined a template for the entire process for two broad classes of machine learning problems - classification and regression. This is summarised in the table below. For the sake of completeness, the classification column is copied from the table above.

    Classification

    Regression

    Quantify Key Business Metrics
      • The value and cost of predicting true positives/negatives and false positives/negatives, respectively. For a binary classification problem, this can be represented as a 2D value-cost matrix.
      • The percentage of cases that can be automated by the model without requiring human intervention.
      • Error metrics, where a lower value is preferred:
        • Mean/median absolute error (MAE)
        • Mean/median absolute % error (MAPE)
        • Root mean squared error (RMSE)
      • Measure of confidence or certainty for the predictions. In this post, we looked at confidence interval (CI) and coefficient of variation (CoV).

    Define Reasonable Baselines
      • Random classifier: A model that determines the classes randomly.
      • Majority-class classifier: A model that picks the majority class always. This is relevant when the classes are highly imbalanced like in the case of fraud/spam detection.
      • Current deployed classifier: This can be a model/human that the customer has currently deployed to perform the classification task.
      • Overall mean: This is the overall mean value of the target variable.
      • Mean value by group: This is the mean value of the target variable for a particular group. Need to make sure that there are enough samples for the groups of interest. If not, then fall back to the overall mean.
      • Current deployed model: This can be a model/human that the customer has currently deployed to perform the predictions.

    Build and Evaluate Model

    Using historical data, train a classifier and tune hyperparameters using cross-validation. Then evaluate the model on a hold-out test set using the following metrics:

    • True Positives
    • True Negatives
    • False Positives
    • False Negatives
    • Precision
    • Recall

    The first 4 metrics can be visualised using a confusion matrix and ROC curve. The last 2 metrics can be visualised using a PR curve.

    Using historical data, train a classifier and tune hyperparameters using cross-validation. Then evaluate the model on a hold-out test set using the following metrics:

    • Absolute Error (AE)
    • Absolute % Error (APE)
    • Squared Error (SE)

    The distributions of these metrics can be visualised using a PDF or CDF.

    Translate Model Performance to Business Metrics
      • Expected value: Can be computed by using the confusion matrix and the value-cost matrix.
      • % cases to manually review: Can be computed using the precision, recall and actual positive class rate.
      • Error metrics: Can be easily computed by taking the mean and median of the above metrics.
      • Confidence measure: Can estimate the CI for each estimate using quantile regression. A rough measure of CoV can be computed using the CI and the median predicted value. By setting a CoV threshold, we can determine the % of cases that can be 'automatable'.
    Hope you enjoyed this series. Thanks for reading!
    ]]>         Measuring Model Goodness - Part 1 https://docs.microsoft.com/archive/blogs/data_insights_global_practice/measuring-model-goodness-part-1 Wed, 22 Aug 2018 16:30:43 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=785 Author: Dr. Ajay Thampi, Lead Data Scientist Data and AI are transforming businesses worldwide from finance, manufacturing and retail to healthcare, telecommunications and education. At the core of this transformation is the ability to convert raw data into information and useful, actionable insights. This is where data science and machine learning come in. Building a good and reliable machine learning system requires us to be methodical by:
    • understanding the business needs
    • acquiring and processing the relevant data
    • accurately formulating the problem
    • building the model using the right machine learning algorithm
    • evaluating the model, and
    • validating the performance in the real world before finally deploying it
    This entire process has been documented as the Team Data Science Process (TDSP) at Microsoft, captured in the figure below. This post, as part of a two-part series, is focused on measuring model goodness, specifically looking at quantifying business value and converting typical machine learning performance metrics (like precision, recall, RMSE, etc.) to business metrics. This is typically how models are validated and accepted in the real world. The relevant stages of the process are highlighted in red in the figure above. Measuring model goodness also involves comparing the model performance to reasonable baselines, which will also be covered in this post. To illustrate all of this, two broad classes of machine learning problems will be looked at:
    • Classification: Covered in Part 1
    • Regression: To be covered in Part 2

    Classification

    Classification is the process of predicting qualitative or categorical responses [source]. It is a supervised machine learning technique that classifies a new observation to a set of discrete categories. This is done based on training data that contains observations whose categories are known. Classification problems can be binary in which there are two target categories or multi-class in which there are more than two mutually exclusive target categories. In this post, I will look at the binary classification problem taking breast cancer detection as a concrete example. I will be using the open breast cancer Wisconsin dataset for model building and evaluation. The techniques discussed below can easily be extended to other binary and multi-class classification problems. All the source code used to build and evaluate the models can be found on Github here. Let’s first understand the business context and the data. Breast cancer is the most common cancer in women and is the main cause of death from cancer among women [source]. Early detection of the cancer type — whether it is benign or malignant — can help save lives by picking the appropriate treatment strategy. In this dataset, there are 569 cases in total out of which 357 are benign (62.6%) and 212 are malignant (37.4%). There are 30 features in total, computed from a digitised image of a biopsy of a breast mass. These features describe the characteristics of the nuclei like radius, texture, perimeter, smoothness, etc. The detection of the type of cancer is currently being done by a radiologist which is time consuming and the business need is to speed up the diagnosis process so that the appropriate treatment can be started early. Before diving too deep into the modelling stage, we need to understand the value/cost of making the right/wrong prediction. This information can be represented using a value-cost matrix as shown below: The rows in the matrix represent the actual class and the columns represent the predicted class. The first row/column represents in the negative class (benign in this case) and the second row/column represents the positive class (malignant in this case). Here's what each of the elements in the matrix represents:
    • Row 0, Column 0: Value of predicting the negative class correctly
    • Row 0, Column 1: Cost of predicting the positive class wrongly
    • Row 1, Column 0: Cost of predicting the negative class wrongly
    • Row 1, Column 1: Value of predicting the positive class correctly
    For breast cancer detection, we can define the value-cost matrix as follows:
    • Detecting benign and malignant cases correctly are given equal and positive value. Although the malignant type is the worse situation to be in for the patient, the goal is to diagnose early, start treatment and cure both types of cancer. So, from a treatment perspective, detecting both cases accurately have equal value.
    • The cost of flagging a malignant case as benign (false negative) is much worse than flagging a benign case as malignant (false positive). Therefore, the false negative is given a cost of -4 and the false positive is given a cost of -2.
    Another business need is to automate as much of the process as possible. One way of doing this is to use the machine learning model as a filter, to automate the detection of simpler benign cases and only flag the possible malignant cases for the radiologist to review. This is shown in the diagram below. We can quantify this business need by looking at the % of cases to review manually (denoted as x in the figure). Ideally, we want to automate everything without requiring the human in the loop. This means that the model should have 100% accuracy and x should be equal to the actual proportion of positive classes. After the business needs are properly quantified, the next step is to define reasonable baselines to compare our models to.
    • Baseline 1: Random classifier that decides randomly if the cancer type is benign / malignant
    • Baseline 2: Majority-class classifier that picks the majority class all the time. In this case, the majority class is benign. This strategy would make much more sense if the classes were heavily imbalanced.
    We could add a third baseline which is the performance of the radiologist or any other model that the business has currently deployed. In this example though, this information is not known and so this baseline is dropped. Now comes the fun part of building the model. I will gloss over a lot of the detail here, as an entire post may be necessary to go through the exploratory analysis, modelling techniques and best practices. I used a pipeline consisting of a feature scaler, PCA for feature reduction and finally a random forest (RF) classifier. 5-fold cross validation and grid search were done to determine the optimum hyperparameters. The entire source code can be found here. Once the models are trained, the next step is to evaluate them. There are various performance metrics that can be used but, in this post, I will go over the following 6 metrics as they can very easily be explained, visualised and translated into business metrics
    • True Positives (TP) / True Positive Rate (TPR): Number of correct positive predictions / Probability of predicting positive given that the actual class is positive
    • False Negatives (FN) / False Negative Rate (FNR): Number of wrong negative predictions / Probability of predicting negative given that the actual class is positive
    • True Negatives (TN) / True Negative Rate (TNR): Number of correct negative predictions / Probability of predicting negative given that the actual class is negative
    • False Positives (FP) / False Positive Rate (FPR): Number of wrong positive predictions / Probability of predicting positive given that the actual class is negative
    • Precision (P): Proportion of predicted positives that are correct
    • Recall (R): Proportion of actual positives captured
    All these metrics are inter-related. The first four can be easily visualised using a ROC curve and a confusion matrix. The last two can be visualised using a precision-recall (PR) curve. Let's first visualise the performance of the model and the baselines using the ROC and PR curves. Both plots show the performance of the models using various thresholds to determine the positive and negative classes. As can be seen, the RF classifier outperforms the baselines in every respect. Once the appropriate threshold is picked, we can plot the normalised confusion matrices as shown below. These matrices show the conditional probabilities of predicted values given the actual value. We can see that the baselines do quite poorly especially in predicting the positive class — the false negative rates are quite high. The RF classifier, on the other hand, seem to get much more of the positive and negative class predictions right achieving a TPR of 93% and TNR of 97%. Now that we’ve established that the new RF model outperforms the baselines, we still need to make sure that it meets our business objectives. i.e. high positive business value and fewer cases to review manually. We therefore need to translate the above performance metrics into the following:
    • Total expected value/cost of the model
    • % of cases that need to be reviewed manually
    The first business metric can be computed by taking the dot product of the flattened confusion matrix (normalised by the total number of samples, S) with the flattened value-cost matrix. This is shown below. Note that S is a scalar equal to TN + FP + FN + TP. The expected value is essentially the weighted average of the values/costs, where the weights are the probabilities of each of the four outcomes [source]. The second business metric can be computed using precision and recall as follows. The positive class rate is known from the data used to evaluate the model, i.e. the test set. Based on the business needs, we then need to decide how much of those positive classes we must accurately determine, i.e. the recall. If we want to detect all the positive cases, the target recall should be 100%. We can then find the corresponding precision (and threshold) from the PR curve. For the ideal model with 100% precision and recall, the proportion of positive cases to review would be equal to the actual positive class rate. This means we could, in theory, achieve 100% automation without a human in the loop. The following two plots show the final evaluation of the model goodness in terms of the two key business metrics. Observations:
    • Both baseline models have quite large costs because of the high false negatives. The random forest classifier has a good positive value by getting a lot of the positive and negative cases right.
    • For the case of breast cancer detection, we want to capture all the positive/malignant cases, i.e. 100% recall. Looking at 50% or 75% recall does not make sense in this example because of the high cost of not treating the malignant cases. For other binary classification problems like fraud detection for instance, a lesser recall may be acceptable so long as the model can flag the high cost-saving fraudulent cases. Regardless, we can see that the RF classifier outperforms the baselines on the automation front as well. To capture 100% of the malignant cases, the radiologist would only need to review about 50% of all cases, i.e. 13% additional false positives, whereas he/she would have to review almost all the cases using the baseline models.
    In conclusion, the first part of this series has looked at measuring model goodness through the lens of the business, specifically looking at classification-type problems. For further details on this subject, the books below are great references. In the second and final part of this series, I will cover regression which requires looking at slightly different metrics. Further Reading ]]>         Demystifying decision forests https://docs.microsoft.com/archive/blogs/data_insights_global_practice/demystifying-decision-forests Fri, 11 Aug 2017 14:22:03 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=685 By Michele Usuelli, Lead Data Scientist

     

    This article doesn't require a data science background, but just some basic understanding of predictive analytics. Besides that, all the concepts are explained from scratch, including a popular algorithm called the "decision forest". Throughout the article you won't see any fancy or advanced machine learning algorithm, but by the end you will have more clarity about how decision forests work.

    Dealing with predictive analytics, decision forests are popular techniques as they usually produce satisfying results requiring a little data preparation. Decision forests, like other machine learning models, are often perceived as obscure in the way they operate. This leads to the misconception that artificial intelligence consists in building a machine doing some magic on the data.

    The scope of this article is to demystify this and to prove that decision forests operate in a simple transparent way. Also, their way of operating is somehow similar to the human approach to decision-making.

    To demonstrate this, we use as an example a mock use-case based on simulated data. Even if the solution makes sense from a logical standpoint, its assumptions are not necessarily related to reality. A customer of a bank requests a mortgage and the bank must decide whether to give them a loan or not. How does the decision-making process work?

    The bank can use some information related to the customer such as their credit score, age, income, excluding everything that could lead to discrimination or legal issues. Starting from historical data about other customers, the bank knows who couldn't pay off the mortgage. If it's possible to map out the profile of insolvent customers, the bank can come up with a way of identifying them.

    The bank has already put together some information about their previous customers into a nice structured table. Also, to make everything easier, around half of the customers failed. Assuming that we are in the utopian scenario of having the data nice and clean, this is the table.

    ## creditScore houseAge yearsEmploy ccDebt year default id
    ## 1 708 21 5 4190 2002 0 26472
    ## 2 649 20 3 2151 2003 0 37088
    ## 3 725 23 3 3603 2005 0 57076
    ## 4 785 24 4 5534 2009 0 90648
    ## 5 649 14 6 5243 2002 0 20110
    ## 6 734 21 0 3441 2008 0 89639

    The columns are

    • Credit score: score measuring the "reliability" of the customer, assessed by a third party
    • House age: age of the house they're purchasing
    • Years of employment
    • Debt on the credit card
    • Year in which they purchased the house
    • Failure: 1 if the customer didn't pay off the loan, 0 otherwise

    Out of the five personal details, that we can call "features", which is the most relevant? After having identified it, we can divide the customers into two groups based on the most relevant feature. Then, for each of the two groups, we repeat the process of finding the most relevant feature and we split the customers into two sub-groups. A popular algorithm, called decision tree, works exactly in this way. In the end, the tree divides the customers into eights groups and we know the percentage of customers experiencing a default per group. We can visualise it through the following chart.

     

    At each split of the tree, the first number is a score representing the portion of customer failing. Since 49% of the customer had a default, the initial score is 0.49. Also, the colour of the boxes gets darker as the score gets higher. The second number represents the percentage of customers belonging to the group. At the beginning, it's 100%. Then, at each level, it shows the proportion of customers into each split.

    Not surprisingly, the most relevant indicator is the debt on the credit card. 6.6% of the customers having a debt below 6755 experienced a failure, opposed to 82% of the customers having a debt equal to or above 6755.

    The tree visualization shows in a simple way how the model works. Having a new customer, the bank can estimate its propensity of not paying off the debt. How does it work? Not having any information about a customer, its probability of failing is the percentage of failures in the past. In this case, it's 49%. Then, knowing the credit card debt, we go down to the second level of the tree and the score can either increase or decrease, depending on the impact of credit card debt. The impact of knowing the credit card debt is:

    • if below ccDebt < 6755, the score becomes 6.6%, so it gets decreased by 42.4%
    • if above ccDebt <= 6755, the score becomes 82%, so it gets increased by 33%

    Summarising, this is the impact:

    • ccDebt < threshold: -0.424
    • ccDebt <= threshold: +0.33

    In the same way, we can measure the impact of being above or below the threshold of other features. If the same feature is being used more than once, we can simply sum up the effects. Depending on the customer, the path on the tree is different, so the features being used and their impact changes. To keep track of that, we can sum up the effects separately for each customer and visualise them through a decision-making heatmap. Each row corresponds to a customer and each column correspond to being above or below a threshold for a given features (e.g., ccDebt < any threshold at some point in the tree). The colour is blue if the score decreases and red if it increases, and the less transparent it is, the higher the impact.

    Measuring the impact of each feature, the bank can explain the decision-making process to assign each score, given more transparency and clarity to its customers.

    As shown above, a decision tree operates in a way that is like the human problem solving. Coming back to the original topic, we want to demystify decision forests. As the name suggests, a decision forest is a combination of decision trees.

    A hundred people can access some information of the bank, but each person is aware of a part of the customers and of a part of their features. If we ask each of them to build a decision tree, we'll have 100 different trees. Applying the trees on a new customer, each of them will assign a different score. To come up with a unique answer, we can compute the average score. That's exactly how a decision forest works: we build decision trees on a random subset of the information, then we combine them. This methodology, called "ensembling", usually improves the performance of the solution.

    Since there might be several trees in the forest, it'll be time-consuming to have a look at each tree separately. However, it's still possible to measure the impact of each feature across all the trees. Combining them, we can build a decision-making heatmap taking account of everything. Therefore, the bank can explain the decision-making process of the entire forest.

    The methodology shows by the article gives transparency and clarity about the way random forests operate. A similar approach is applicable in other contexts and it allows to give more clarity about how machine learning operates.

     

    ]]>         What is the role of a data scientist? https://docs.microsoft.com/archive/blogs/data_insights_global_practice/what-is-the-role-of-a-data-scientist Thu, 23 Feb 2017 15:48:55 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=665 Michele Usuelli, Lead Data Scientist Data Science has been around for decades, but it recently increased in popularity among companies. Although the tools and techniques existed already, there are some changes. Digital technologies generate more data that can drive new advanced analytics use-cases. Also, there are more success stories show-casing the value in data, making companies more keen to invest resources into new solutions. Because of the hype, phrases like "Data Science" and "Big Data" became buzzwords. However, their meaning is loosely defined and it's not entirely clear to many businesses. From a practical perspective, what is data science? Is the role of the data scientist the same as the statistician, or are there new challenges? The scope of this article is to understand what the role of a data scientist consists in. Depending on the company and on the specific role, there are lots of differences, so it's challenging to come out with a universal profile. However, it's still possible to understand the role from a high-level perspective. Breaking down the core responsibilities of a data scientist, we can have a better understanding of the related skillset. The starting point of a data science project is "why". Why does the company need an advanced analytics solution? Why is it willing to allocate a budget? Why will the project have an impact? To address these queries, the data scientist is capable of understanding the business context, brainstorm solutions, and identify what's valuable. The related skills are business acumen, experience in applying a solution in a specific industry, and soft skills like stakeholder management and listening. Being knowledgeable about the field is definitely useful although not mandatory, since data scientists can interact with subject matter experts to get the information they need. Therefore, the core skill is being capable of collecting business information and defining advanced analytics use-cases accordingly.Also, the data scientist should be able to present the solution and its value, and good presentation skills help in that. After having defined the target, the next question is "what". What do we need to do to solve the challenge? What techniques can we use? What are the main steps from the current situation to the final solution? To address these queries, the data scientist should be able to define the logical steps to build an end-to-end solution. The required knowledge is about statistics, data processing, machine learning techniques, model validation. However, being knowledgeable about the separate steps is not enough as the data scientist needs to be capable of designing an end-to-end solution that every time is different depending on the data and on the target. The main challenges are to
    • prepare the data: blend the original data sources, structure the data in the required format
    • apply a machine learning model: define the machine learning model addressing the business challenge
    • validate the model: according to the context, define a meaningful way to measure the success of the model
    Each step requires thinking outside the box and using common sense, in addition to some knowledge about the techniques.Knowing what are the logical step of an advanced analytics solution doesn't imply being able to build it. The final question is "how". How can we implement the solution? How can we put the data together? How can we prototype and deploy the solution? This part is more technical and the skillset is diverse. The main areas are
    • dealing with data challenges: the data can be incomplete, have a large volume, have a challenging structure (e.g., text, images). The data scientist should be able to identify and use the tools required by the data. For example, some tools included into the Hadoop ecosystem became popular recently.
    • coding: although there are pre-built high-level tools, every data science solution is unique and it will involve some coding. Being able to use specific languages like R and Python is useful, but not always necessary. The core skill is knowing how to code, no matter the language. In presence of large amount of data, it's also useful to be able to design parallel algorithms to scale across large data volumes.
    • organising the data into databases: knowing how to store and organise the data, and extract the relevant information. This part is performed using SQL/NoSQL databases although managing them can be out of the scope of a data scientist.
    • prototyping: quickly build a good-enough solution that works.
    • deploying: put the solution into production.
      The technical skills depend a lot on the context, so there is more diversity in the "how" area. The data science process requires a broad expertise and the data scientist can’t go very deeply into each component of the solution. That’s especially true for data scientist consultants, given that they join new projects where the customer has already a deep knowledge about the context and the tools. To design and build the solution, the data scientist needs to interact with
    • Customer stakeholders: to define the scope of the solution and to measure its success, the data scientist should have a conversation with the stakeholders.
    • Customer subject-matter experts (SMEs): data science often consists in improving an existing solution using advanced analytics methodologies, so the starting point is to understand the current solution and use it as a starting point. Also, the data scientist needs to know how to interpret the data. To get some help on that, the data scientist needs to work close to the SMEs.
    • Academic researchers: although data scientists are experts of machine learning techniques, their knowledge is not as deep as academic researchers. Also, data scientists are focused on bringing value to projects, so they often don’t have enough time to develop complex algorithms. The most common way to work together with academic researches is to use tools developed by open-source community. A good example is CRAN providing with its R 10000+ packages, providing the most cutting edge statistical tools and machine learning techniques. Also, in larger projects there might be some researchers working together on the machine learning part of the solution.
    • Solution architects: the data scientist defines and builds advanced analytics models that are utilised by the solution. Usually an architect designs the overall solution, taking into consideration the business context and the technological infrastructure.
    • Data engineers: the “how” part of the engagement is usually the most time consuming and it required a broad range of skills. Data engineers help building and deploying the solution, designing the pipeline translating the data into actions and insights. In some engagement, the data scientists build the prototype and the data engineers put it into production.
    • Software SMEs: if the solution integrates other technologies, there might be other people involved, especially solution architects.
    This article shows what's common across most of the data scientists and aims to provide more clarity about the role. Depending on the specific case, the skillset can be more detailed and it varies a lot depending on the industry, seniority, level, team. Also, in larger teams there will be people specialised in different aspects of the solution, so it's less important to have a person having the full skillset.    ]]>         Scaling up Scikit-Learns Random Projection using Apache Spark https://docs.microsoft.com/archive/blogs/data_insights_global_practice/scaling-up-scikit-learns-random-projection-using-apache-spark Thu, 16 Feb 2017 18:07:32 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=646 By Sashi Dareddy, Lead Data Scientist What is Random Projection (RP)? Random Projection is a mathematical technique to reduce the dimensionality of a problem much like Singular Value Decomposition (SVD) or Principal Component Analysis (PCA) but only simpler & computationally faster. [Throughout this article, I will use Random Projection and Sparse Random Projection interchangeably.] It is particularly useful when:
    • Calculating PCA/SVD can be very prohibitive in terms of both running time & memory/hardware constraints.
    • Working with extremely large dimensional sparse datasets with both large n (>> 100million rows/samples) & large m (>> 10 million columns/features) in a distributed setting.
    • When you require sparsity in the low-dimensional projected data
    Why can't we use Scikit-Learn? You can use Scikit-Learn if your dataset is small enough to fit in a single machine's memory. But, when it comes to the actual projection you will find that it doesn't scale well beyond a single machine's resources. Why can't we use Apache Spark? As at version 2.1.0 of Apache Spark MLLib, the following dimensionality reduction techniques are available:
    • Singular Value Decomposition (SVD)
    • Principal Component Analysis (PCA)
    There has been at least one attempt to implement Random Projection in Apache Spark MLLib but those efforts don't appear to have made it through to the latest release. In this article, I will present a recipe to perform Random Projection using PySpark. It brings the scalability of Apache Spark to the Random Projection implementation in Scikit-Learn. As a bonus, you can extend the idea presented in this article to perform general sparse matrix by sparse matrix multiplication (as long as one of the sparse matrix is small enough to fit in memory) resulting in another sparse matrix. Further reading:
    • 4.5. Random Projection - particularly, The Johnson-Lindenstrauss lemma and Sparse random projection sections
    So, how do we apply Random Projection? There are two main steps in projecting a n x m matrix into a low dimensional space using Random Projection:
    1. Generating a m x p Projection Matrix with a pre-specified sparsity factor - this is where we will leverage Scikit-Learns implementation of Sparse Random Projection and generate the projection matrix.
    2. Matrix multiplication - wherein we multiply an n x m input dataset with an m x p Projection Matrix yielding a new n x p matrix (which is said to be projected into a lower dimension) - we will leverage Scipy & Spark to deliver the large scale sparse matrix by sparse matrix multiplication resulting in another sparse matrix in a lower dimension
    The Setup Here, I will show you how to apply Random Projection by way of an example. Data We will be working with KDD2012 datasets available in LibSVM format. The training dataset has 119,705,032 rows and 54,686,452 features - we will apply Random Projection and transform this dataset into one which has 119,705,032 rows and ~4k features. In order to demonstrate the scalability of my approach, I'm going to artificially increase the size of the training set by a factor of 9 - giving us just over 1 Billion rows. Note, even the transformed dataset needs to be in sparse format otherwise the dataset could take up ~16TB in dense format. Compute Resources I will be using a 6-node (90-core, 300G RAM) Spark HDInsight Cluster from Microsoft Azure Microsoft is giving away £150 in free credit to experiment in Azure. I will also be using Azure Storage, particularly the Blob Service to store my input/output datasets. It roughly takes 1ms per row (single core performance) including I/O to process the data. To project a dataset with 1 billion rows and 54 million columns into a new dataset with 1 billion rows and 4096 features took me about 3.75 hours on the above-mentioned cluster. Memory Consideration Your spark executors would need enough RAM to hold 2 x size of a Spark DataFrame partition plus a projection matrix (this is usually a few 100 MBs). If you are running low on memory, then you can repartition your Spark DataFrame to hold smaller chunks of data. Code walkthrough: There are two versions of random projection code depending on whether you want to run the code on a single machine or on a cluster. Refer to ./code/localmode/ or ./code/clustermode/ depending on your requirement. But here I will discuss the cluster mode code. In [1]: First, let's get the required imports out of the way. Of particular note is the johnson_lindenstrauss_min_dim function - this function, given an input matrix, returns the no. of dimensions the projected space should have. In my case I'm going to fix the no. of dimensions at 4096. # imports import logging import os import numpy as np import math import scipy.sparse as ssp from pyspark.sql import functions as f from sklearn.random_projection import johnson_lindenstrauss_min_dim, SparseRandomProjection from pyspark.sql import SparkSession from pyspark.sql.types import * from pyspark.ml.linalg import Vectors, VectorUDT from sklearn.externals import joblib In [2]: Below we will generate a Sparse Random Projection Matrix which will be a Scipy's CSR matrix which needs to be saved to disk to transform future data. Note: the dimensionality of the reduced dimension space is independent of the no. of features (in our case 54 million) - it only depends on the no. of rows (in our case 1 Billion) according to the The Johnson-Lindenstrauss lemma. # generating the random projection matrix dummy_X_rows = train.count() # 1,077,345,288 rows dummy_X_cols = train.first()["features"].size # 54,686,452 features # we only use the shape-info of this dummy matrix dummy_X = ssp.csr_matrix((dummy_X_rows, dummy_X_cols), dtype=np.float32) # find the optimal (conservative estimate) no. of dimensions required according to the # johnson_lindenstrauss_min_dim function # rproj_ndim = johnson_lindenstrauss_min_dim(dummy_X_rows, eps=0.2) # returns 4250 rproj_ndim = 4096 # using a fixed value instead of johnson_lindenstrauss_min_dim suggestion logging.info("Fitting Sparse Random Projection to have %d columns after projection " % rproj_ndim) srp = SparseRandomProjection(n_components=rproj_ndim, random_state=123) srp.fit(dummy_X) logging.info("Saving the projection matrix to disk... ") joblib.dump(srp, os.path.join("/tmp","srp.pkl")) In [3]: Here we will define a Python function which takes a whole Spark DataFrame partition and a projection matrix to return projected data. In a nutshell, this function converts a whole Spark DataFrame partition into a Scipy CSR matrix and then simply multiplies it with our projection matrix. def random_project_mappartitions_function(rdd_row_iterator, local_csr_matrix): """ This function is intended to be used in a <df>.rdd.mapPartitions(lambda rdd_map: random_project_mapf\ (rdd_map, local_rnd_mat)) setting. :param rdd_row_iterator: a list of n-dim sparse vectors :param local_csr_matrix: the projection matrix - should have dimensions: n x p :return: a list of p-dim sparse-vectors - same length as input """ keys = [] # this will be a list of single row sparsevectors transformed into scipy csr matrix features_single_row_matrix_list = [] PROJECT_DIM_SIZE = local_csr_matrix.shape[1] for row in rdd_row_iterator:    # capture keys    if "label" in row:       keys.append((row["id"], row["label"]))    else:       keys.append((row["id"]))     # work on values:     feature_dim_size = row["features"].size # feature dimensionality before projection     col_indices = row["features"].indices     row_indices = [0] * len(col_indices) # defaulting to 0 as we are creating single row matrix     data = row["features"].values.astype(np.float32)     feature_mat = ssp.coo_matrix((data, (row_indices, col_indices)),     shape=(1, feature_dim_size)).tocsr()     features_single_row_matrix_list.append(feature_mat)  # vstacking single row matrices into one large sparse matrix  features_matrix = ssp.vstack(features_single_row_matrix_list)  del features_single_row_matrix_list  projected_features = features_matrix.dot(local_csr_matrix)  del features_matrix, local_csr_matrix  projected_features_list = (Vectors.sparse(PROJECT_DIM_SIZE, zip(i.indices, i.data))                             for i in projected_features)  if "label" in row:     return zip((i[0] for i in keys), (i[1] for i in keys), projected_features_list)  else:     return zip((i[0] for i in keys), projected_features_list) In [4]: and here is where the main action takes place – the projected output is saved to a parquet file. if __name__ == '__main__':   N_FEATURES=54686452   # these are the no. of features in the dataset we are going to use in this app   logging.basicConfig(format='%(asctime)s %(levelname)s:%(message)s', level=logging.INFO)    # fire up a spark session   spark = SparkSession \   .builder \   .appName("PySpark Random Projection Demo") \   .getOrCreate()   sc = spark.sparkContext   DATA_DIR = 'wasb://kdd12-blob@sashistorage.blob.core.windows.net/'   # Azure Storage blob   train = spark.read.format("libsvm").load(DATA_DIR, numFeatures=N_FEATURES)\   .withColumn("id", f.monotonically_increasing_id())   print(train.show())   train_schema = StructType(   [StructField("id", LongType(), False)   ,StructField("label", FloatType(), False)  , StructField("features", VectorUDT(), False)   ] )  # generating the random projection matrix  dummy_X_rows = train.count()  dummy_X_cols = train.first()["features"].size  dummy_X = ssp.csr_matrix((dummy_X_rows, dummy_X_cols), dtype=np.float32) # the shape-only of this dummy  # find the optimal (conservative estimate) no. of dimensions required according to the  # johnson_lindenstrauss_min_dim function  # rproj_ndim = johnson_lindenstrauss_min_dim(dummy_X_rows, eps=0.2) # returns 4250  rproj_ndim = 4096  logging.info("Fitting Sparse Random Projection to have %d columns after projection " % rproj_ndim)  srp = SparseRandomProjection(n_components=rproj_ndim, random_state=123)  srp.fit(dummy_X)  logging.info("Saving the projection matrix to disk... ")  joblib.dump(srp, os.path.join("/tmp","srp.pkl"))  local_rnd_mat = srp.components_.T.astype(np.float32)   # broadcast the local_rnd_mat so it is available on all nodes   local_rnd_mat_bc_var = sc.broadcast(local_rnd_mat)   logging.info("Applying random projection to rdd map partitions")   train_projected_df = train.rdd\   .mapPartitions(lambda rdd_map_partition:      random_project_mappartitions_function(rdd_map_partition,local_rnd_mat_bc_var.value))\   .toDF(train_schema)   logging.info("Writing projected data to disk...")   train_projected_df\   .write\   .mode("overwrite")\   .parquet(DATA_DIR+"/train_features_random_projected.parquet/")  logging.info("Sample rows from training set before projection...")  print(train.show())  logging.info("Sample rows from training set after projection...")  print(spark.read.parquet(DATA_DIR+"/train_features_random_projected.parquet/").show())  spark.stop() In [5]: Finally, here is how we submit the PySpark app to the cluster.  #!/usr/bin/env bash  echo "Submitting PySpark app..."  spark-submit \  --master yarn \  --executor-memory 3G \  --driver-memory 6G \  --num-executors 85 \  --executor-cores 1 \  code/clustermode/randomProjection.py  echo "Exporting Projection Matrix to Azure Storage..."  zip /tmp/srp.pkl.zip /tmp/srp.pkl*  export AZURE_STORAGE_ACCOUNT=<storage account name>  export AZURE_STORAGE_ACCESS_KEY=<storage account access key>  azure storage blob upload --container modelling-outputs --file /tmp/srp.pkl.zip For the full code refer to https://github.com/SashiDareddy/RandomProjection ]]>         Scaling a recommender system across large data volumes https://docs.microsoft.com/archive/blogs/data_insights_global_practice/scaling-a-recommender-system-across-large-data-volumes Mon, 08 Aug 2016 17:35:28 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=575 By Michele Usuelli, Data Scientist Consultant

    Building a recommendation engine in presence of large data volume

    E-commerce businesses can suggest new products to their customers. How do they choose the products to recommend? The companies collect data about the purchases of their customers. Starting from the purchase history, they can identify items that have been taken by the same customers and customers that bought the same items. Given a new customer, the company can recommend him/her items similar to the ones that he/she purchased already, or items that have been purchased by similar customers. The recommendations can also take account of other data sources like personal information, items descriptions, customer ratings and reviews. Large companies like Amazon and eBay already built sophisticated recommendation engines tailored to their environment. What if a smaller company wants to build a smaller recommendation engine from scratch? Even if the complexity is not the same as huge companies, the volume of data to process may be large. In this article, we see how to build simple recommender systems that are scalable across large data volumes. To show the concepts avoiding unnecessary complications, in this article we are using the information about purchases only. Due to the complexity of the problem and to the variety of contexts, this article describes it from a generic high-level perspective.

    Choosing the environment

    In this article, the technology that we use is SQL Server 2016 with R Services. The reason is that we can store large SQL tables and process them using advanced analytics techniques provided by R. The use-case consists in recommending movies to the users. Given a set of movies and users, the table describes what movies each user has watched. To avoid further complications, the table doesn't include user ratings. Its structure is:
    • A row for each movie
    • A column for each user
    • Cells equal to 1 if the user watched the movie and to 0 otherwise
    The data has already been imported into SQL 2016 into a table called movies. To analyse the data, we use the R environment and we define a SQL table object called sql_movies that allows us to access the data. The function creating the object is RxSqlServerData and it needs a connection string for the database and the name of the table. We also define another SQL table object, called sql_clust_users, referring to a table that doesn't exist yet. We'll create and use this table from R.

    connection_string <- "Driver=SQL Server;Server=.;Database=movies;Uid=usernam;password"

    sql_movies <- RxSqlServerData(connectionString = connection_string, table ="movies") sql_clust_users <- RxSqlServerData(connectionString = connection_string, table ="sql_clust_users") Let's have a quick look at the data.

    dim(sql_movies)

    ## [1] 11503 865

    rxGetVarInfo(sql_movies)[1:4]

    ## $MovieId ## Type: integer, Low/High: (91, 3630218) ## ## $U9928 ## Type: numeric, Low/High: (0.0000, 1.0000) ## ## $U15152 ## Type: numeric, Low/High: (0.0000, 1.0000) ## ## $U7850 ## Type: numeric, Low/High: (0.0000, 1.0000)

    head(sql_movies)[, 1:8]

    ## MovieId U9928 U15152 U7850 U9065 U8166 U5251 U10249 ## 1 368688 0 0 1 0 0 0 0 ## 2 368709 0 0 0 0 0 0 0 ## 3 368794 0 0 0 0 0 0 0 ## 4 368891 0 0 0 0 1 0 0 ## 5 368909 0 0 1 0 0 0 0 ## 6 368933 0 0 0 0 0 0 0

    The table has 865 columns. The first column contains the Movie ID and each other column corresponds to a user. The values in the user columns are equal to 1 if the user watched the movie and to 0 otherwise. To build a recommendation engine, we can use the R package recommenderlab. However, this package needs all the data to be stored into the RAM, so it can deal with relatively small data volumes. Therefore, to process a large dataset we need to reduce its volume. In this example, the data volume is not really huge. However, the solution is designed in such a way that it's applicable on a larger dataset.

    Reducing the data volume

    To produce a small table, we can either reduce the number of rows or the number of columns. In this chapter we are reducing both. A common challenge is that there are many users, i.e. columns. A solution consists in identifying small groups of similar users and defining a new table having a column for each group of users. The steps are
    1. Define a formula describing what columns we are dealing with. We include all the columns apart from the first
    2. Measure the similarity between users using the function rxCor
    3. Starting from the similarity, define a matrix containing the distance between the users
    4. Starting from the distance matrix, identify groups of similar users applying hierarchical clustering
    5. Define a new table which columns correspond to the clusters. The value of each column is equal to one if at least one user of the cluster watched the movie
    The new table contains a row for each movie and a column for each group of users. This is the related code to detect 100 clusters of users.

    # define the formula cols_movies <- names(sql_movies)[-1] string_users <- paste(cols_movies, collapse = " + ") formula_users <- formula(paste("~", string_users))

    # build the hierachical clustering model cor_users <- rxCor(formula_users, sql_movies, rowsPerRead = 5e02)

    ## Rows Read: 11503, Total Rows Processed: 11503, Total Chunk Time: 3.424 seconds ## Computation time: 7.211 seconds.

    dist_users <- as.dist(1 - abs(cor_users)) model_hc <- hclust(dist_users, method = "complete") which_cluster <- cutree(model_hc, k = 100) var_clusters <- sort(unique(which_cluster))

    # group the users into clusters clusterCols <- function(list_in) { # define a field for each cluster list_out <- lapply(var_clusters, function(this_cluster) { these_cols <- names(which_cluster[which_cluster == this_cluster]) df_cols <- data.frame(list_in[these_cols]) rowSums(df_cols) > 0 }) names(list_out) <- paste0("clust", var_clusters) # for each movie, count the number of clusers having watched it df_views <- data.frame(list_out) list_out$n_views <- rowSums(df_views) # add the movie ID list_out$MovieId <- list_in$MovieId list_out } rxDataStep( inData = sql_movies, outFile = sql_clust_users, overwrite = TRUE, transformFunc = clusterCols, transformObjects = list(var_clusters = var_clusters, which_cluster = which_cluster))

    ## Warning in rxLinkTransformComponents(transforms = transforms, transformFunc = transformFunc, : ## There are 'transformObjects' that have the same name as an ## object in the evaluation environment, which could cause ## misleading results. Try renaming the following objects in the ## 'transformObjects' list: 'var_clusters', 'which_cluster'.

    ## Rows Read: 11503, Total Rows Processed: 11503, Total Chunk Time: 2.088 seconds

    dim(sql_clust_users)

    ## [1] 11503 102

    The other approach to decrease the size is to reduce the number of rows. Applying a clustering algorithm, we can identify groups of similar items and define a table with a row for each group of items. For this purpose, we can use the function rxKmeans. The starting point is sql_clust_users and the steps are
    1. Build a formula defining the variables to include into the clustering model
    2. Identify 20 clusters of movies using rxKmeans
    3. Extract the centers of the clusters. They contain averages that in this context express the percentage of users that watched each movie
    4. Starting from the centers, define the table df_movies which values are 1 if the percentage is above 10% and 0 otherwise
    This is the related code.

    cols_km <- names(sql_clust_users) feat_km <- paste(cols_km, collapse = " + ") formula_km <- formula(paste("~", feat_km)) model_km <- rxKmeans(formula_km, sql_clust_users, numClusters = 20, reportProgress = 0) df_movies <- ifelse(model_km$centers[, -1] > 0.1, 1, 0) df_movies[1:10, 1:6]

    ## clust1 clust2 clust3 clust4 clust5 clust6 ## 1 1 0 0 1 1 1 ## 2 0 0 0 0 1 0 ## 3 1 0 1 1 1 1 ## 4 0 0 0 0 0 0 ## 5 0 0 0 0 1 1 ## 6 1 0 1 0 1 1 ## 7 1 0 0 1 1 1 ## 8 0 0 0 1 1 1 ## 9 0 0 0 0 1 1 ## 10 1 0 0 1 1 1

    The table df_movies is small-sized and it contains a column for each cluster of users and a row for each cluster of movies. The value of each cell is equal to 1 if the related cluster of users watched at least 10% of the related cluster of movies, and 0 otherwise.

    Building the engine

    In the previous step we produced a small table stored into the R environment. Starting from that, we can build a recommender system using the package recommenderlab. The steps are
    1. Starting from df_movies, build, test and validate a recommendation engine. There are a few options in the recommenderlab package, so a good approach is to test and evaluate a few of them, and to pick the best-performing
    2. Given a new user, measure the distance between each cluster center and he/she. Then, identify the closest center and associate the user to the related cluster
    3. Identify the cluster of movies to recommend to the cluster related to the new user
    4. Out of the recommended cluster of movies, choose one or more movies to recommend to the user

    Conclusions

    Using the approach described by this article, it's possible to apply a recommender system on a large data volume. To extend this solution, it's possible to use a matrix containing user ratings instead of just 0s a 1s. To include some information about the users and/or movies, it's possible to summarise it for each cluster. Other adjustments can follow a similar methodology.]]>         Analysing data in SQL Server 16, combining R and SQL https://docs.microsoft.com/archive/blogs/data_insights_global_practice/analysing-data-in-sql-server-16-combining-r-and-sql Mon, 01 Aug 2016 22:43:14 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=535 By Michele Usuelli, Data Scientist Consultant

    Overview

    R is the most popular programming language for statistics and machine learning, and SQL is the lingua franca for the data manipulation. Dealing with an advanced analytics scenario, we need to pre-process the data and to build machine learning models. A good solution consists in using each tool for the purpose it's designed for: the standard data preparation using a tool supporting SQL, the custom data preparation and the machine learning models using R.

    SQL Server 2016 has the option to include an extension: Microsoft R Services. It's based on MRS (Microsoft R Server), a tool designed by Revolution Analytics to scale R across large data volumes. In SQL Server 2016, R Services provides a direct interface between MRS and the SQL databases. In this way, it's possible to analyse the SQL tables and to create new tables using the MRS tools. In addition, the R package RODBC allows to run SQL queries from R.

    This article shows a simple example integrating R and SQL. To follow all the steps, some prior knowledge about the basic R functions is required.

    Setting-up the environment

    In this article, we are using a SQL Server 2016 VM. To set-up the environment, the steps are

    1. Log-in to Azure and set-up a SQL Server machine with R Services. To install R Services, refer to this: https://msdn.microsoft.com/en-us/library/mt696069.aspx
    2. Connect to SQL Server: from the Windows task bar, open SQL Server Management Studio and connect to it using your credentials.
    3. Create a new database: right-click on Databases and choose New Database….

    4. Give a name to the new database, e.g. iristest. Then, click on Add and OK.

    5. Open the query editor: right-click on the new database and choose New Query.

    6. Define the credentials to authenticate to the database: in the editor, paste and run the following query. You should replace the parameter testiris with the name of your database, and username and password with new credentials to access this database.

    USE [testiris]
    GO
    CREATE LOGIN [username] WITH PASSWORD='password', CHECK_EXPIRATION=OFF,
    CHECK_POLICY=OFF;
    CREATE USER [username] FOR LOGIN [username] WITH DEFAULT_SCHEMA=[db_datareader]
    ALTER ROLE [db_datareader] ADD MEMBER [username]
    ALTER ROLE [db_ddladmin] ADD MEMBER [username]

    1. Build a new RStudio or Visual Studio project and create a new R script. See https://www.rstudio.com/products/RStudio/
    2. Define a list with the parameters to connect to the database. They should be the same as in the previous SQL query.

    sql_par <- list(
    Driver = "SQL Server",
    Server = ".",
    Database = "testiris",
    Uid = "username",
    Pwd = "password"
    )

    1. Starting from the list of parameters, define a connection string for the SQL database. For this purpose, you can use the function paste.

    sql_connect <- paste(names(sql_par), unlist(sql_par),
    sep = "=", collapse = ";")

    1. Define a MRS data source object containing the information to access the iristest database. The inputs are the table name, e.g. iris, and the connection string.

    table_iris <- "iris"
    sql_iris <- RxSqlServerData(connectionString = sql_connect,
    table = table_iris)

    1. Import the data into a SQL table. For this purpose, we can use rxDataStep. Its inputs are the local dataframe iris, the data source object sql_iris, and overwrite = TRUE, specifying that we are overwriting the table if existing already. To avoid issues in the SQL queries, we also re-name the columns of iris, replacing the . with an *_*.

    names(iris) <- gsub("[.]", "_", names(iris))
    rxDataStep(iris, sql_iris, overwrite = TRUE)

    ## Rows Read: 150, Total Rows Processed: 150, Total Chunk Time: Less than .001 seconds

    1. Install and load the package RODBC and initialise it using the connection string. We define the object channel containing the information to access the database.

    library(RODBC)
    channel <- RODBC::odbcDriverConnect(sql_connect)

    Exploring the data

    To explore the data, we can either use MRS or SQL. In MRS, there is the option to quickly access the metadata using rxGetVarInfo.

    rxGetVarInfo(sql_iris)

    ## Var 1: Sepal_Length, Type: numeric, Low/High: (4.3000, 7.9000)
    ## Var 2: Sepal_Width, Type: numeric, Low/High: (2.0000, 4.4000)
    ## Var 3: Petal_Length, Type: numeric, Low/High: (1.0000, 6.9000)
    ## Var 4: Petal_Width, Type: numeric, Low/High: (0.1000, 2.5000)
    ## Var 5: Species
    ## 3 factor levels: setosa versicolor virginica

    The output contains the name and type of each field. Also, it's possible to compute a quick summary of the table using rxSummary. The inputs are

    summary_iris <- rxSummary(~ ., sql_iris)

    ## Rows Read: 150, Total Rows Processed: 150, Total Chunk Time: Less than .001 seconds
    ## Computation time: 0.005 seconds.

    summary_iris$sDataFrame

    ## Name Mean StdDev Min Max ValidObs MissingObs
    ## 1 Sepal_Length 5.843333 0.8280661 4.3 7.9 150 0
    ## 2 Sepal_Width 3.057333 0.4358663 2.0 4.4 150 0
    ## 3 Petal_Length 3.758000 1.7652982 1.0 6.9 150 0
    ## 4 Petal_Width 1.199333 0.7622377 0.1 2.5 150 0
    ## 5 Species NA NA NA NA 150 0

    This summary contains some basic statistics like the mean and the standard deviation.

    Using RODBC, we can run any SQL query. The inputs are the connection object channel, defined previously, and the query string. Let's see a simple example.

    RODBC::sqlQuery(channel, "select count(*) from iris")

    As expected, the output is the number of records.

    R contains good string manipulation tools that allow us to build complex queries and to write extensions. To show the approach, we perform a simple group by operation. The steps are

    1. Define the skeleton of the query, leaving the column names as %s parameters
    2. Define the parameters
    3. Define the query inclusive of its parameters using sprintf
    4. Run the query

    The R code is

    query_sql <- "select %s, avg(%s) as avg_sl from iris group by %s"
    col_group <- "Species"
    col_value <- "Sepal_Length"
    query_count <- sprintf(query_sql, col_group, col_value, col_group)
    df_avg_value <- RODBC::sqlQuery(channel, query_count)
    df_avg_value

    ##
    ## Attaching package: 'dplyr'

    ## The following objects are masked from 'package:stats':
    ##
    ## filter, lag

    ## The following objects are masked from 'package:base':
    ##
    ## intersect, setdiff, setequal, union

    ## Species avg_sl
    ## 1 setosa 5.006
    ## 2 versicolor 5.936
    ## 3 virginica 6.588

    Using a similar approach, we can define more complex queries. Also, if we need to run similar queries many times, it's possible to build R functions that build the query depending on some given parameters.

    Processing the data

    To process the data, it's possible to use both MRS and SQL. With MRS, we can process the data using the function rxDataStep. This function allows to build custom data processing operations. A simple example consists in defining a new column Sepal.Size with simple maths operations. The steps are

    1. Define a new SQL data source object
    2. Define the SQL table using RxSqlServerData. Its arguments are the input and output data object, and overwrite = TRUE, similarly to before. In addition, we define the new column using the transforms input.
    3. Have a look at the new table using rxGetVarInfo

    This is the code:

    table_iris_2 <- "iris2"
    sql_iris_2 <- RxSqlServerData(connectionString = sql_connect,
    table = table_iris_2)
    rxDataStep(inData = sql_iris,
    outFile = sql_iris_2,
    transforms = list(Sepal.Size = Sepal.Length * Sepal.Width),
    overwrite = TRUE)
    rxGetVarInfo(sql_iris_2)

    ## Rows Read: 150, Total Rows Processed: 150, Total Chunk Time: Less than .001 seconds

    ## Var 1: Sepal_Length, Type: numeric, Low/High: (4.3000, 7.9000)
    ## Var 2: Sepal_Width, Type: numeric, Low/High: (2.0000, 4.4000)
    ## Var 3: Petal_Length, Type: numeric, Low/High: (1.0000, 6.9000)
    ## Var 4: Petal_Width, Type: numeric, Low/High: (0.1000, 2.5000)
    ## Var 5: Species
    ## 3 factor levels: setosa versicolor virginica
    ## Var 6: Sepal_Size, Type: numeric, Low/High: (10.0000, 30.0200)

    As expected, we have a new column: Sepal_Size.

    Using the same approach, it's possible to build complex custom operations, making use of the open-source R functions.

    Using RODBC, it's possible to run an SQL query defining a new table. For instance, we can join two tables. Using an approach similar to the previous SQL query, we build the related string and execute it using RODBC.

    table_avg_value <- "avg_value"
    sql_avg_value <- RxSqlServerData(connectionString = sql_connect,
    table = table_avg_value)

    rxDataStep(df_avg_value, sql_avg_value, overwrite = TRUE)
    query_join <- "select Sepal_Length, Species from iris
    left join avg_value
    on iris.Species = avg_value.Species"
    df_join <- RODBC::sqlQuery(channel, query_join)
    df_join <- df_join[, c("Sepal_Length", "Species", "avg_sl")]
    head(df_join)

    ## Joining by: "Species"

    ## Sepal_Length Species avg_sl
    ## 1 5.1 setosa 5.006
    ## 2 4.9 setosa 5.006
    ## 3 4.7 setosa 5.006
    ## 4 4.6 setosa 5.006
    ## 5 5.0 setosa 5.006
    ## 6 5.4 setosa 5.006

    After having processed the data, it's possible to build machine learning models starting from the SQL tables. In this way, we don't need to pull the data in-memory and we can deal with a large data volume. For instance, to build a linear regression, we can use rxLinMod. Similarly to rxSummary, the arguments are formula, defining the variables to include, and data, defining the data source. After having built the model, we can explore is using summary, similarly to open-source R.

    model_lm <- rxLinMod(formula = Petal_Length ~ Sepal_Length + Petal_Width,
    data = sql_iris)

    ## Rows Read: 150, Total Rows Processed: 150, Total Chunk Time: Less than .001 seconds
    ## Computation time: 0.000 seconds.

    summary(model_lm)

    ## Call:
    ## rxLinMod(formula = Petal_Length ~ Sepal_Length + Petal_Width,
    ## data = sql_iris)
    ##
    ## Linear Regression Results for: Petal_Length ~ Sepal_Length +
    ## Petal_Width
    ## Data: sql_iris (RxXdfData Data Source)
    ## File name: iris.xdf
    ## Dependent variable(s): Petal_Length
    ## Total independent variables: 3
    ## Number of valid observations: 150
    ## Number of missing observations: 0
    ##
    ## Coefficients:
    ## Estimate Std. Error t value Pr(>|t|)
    ## (Intercept) -1.50714 0.33696 -4.473 1.54e-05 ***
    ## Sepal_Length 0.54226 0.06934 7.820 9.41e-13 ***
    ## Petal_Width 1.74810 0.07533 23.205 2.22e-16 ***
    ## ---
    ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    ##
    ## Residual standard error: 0.4032 on 147 degrees of freedom
    ## Multiple R-squared: 0.9485
    ## Adjusted R-squared: 0.9478
    ## F-statistic: 1354 on 2 and 147 DF, p-value: < 2.2e-16
    ## Condition number: 9.9855

    The output is the same as using open-source R function lm. The difference is that we can apply this function on a large table without needing to pull it in-memory.

    Conclusions

    SQL and MRS are designed for different purposes. Having them into the same environment, it's possible to use each of them in the most proper context: SQL to prepare the data, MRS to build advanced machine learning models. Also, the string manipulation tools provided by R allow to build SQL queries, making it possible to write code extensions. Another option is to include R code within the SQL queries although we haven't dealt with that in this article.

    ]]>         Validating a model in R Services using the k-fold https://docs.microsoft.com/archive/blogs/data_insights_global_practice/validating-a-model-in-r-services-using-the-k-fold-4 Fri, 08 Jul 2016 21:52:44 GMT https://blogs.msdn.microsoft.com/data_insights_global_practice/?p=425 By Michele Usuelli, Data Scientist Consultant

    Why k-fold

    Predictive modelling consists in predicting a future outcome based on the data. Starting from data which outcome is already known, the predictive models detect patterns that had an impact on the outcome. Then, in presence of data which outcome is unknown, the model looks for the same patterns and predicts the outcome on their base. Since there are several business scenarios where a prediction will drive a lot of value, predictive modelling is among the most popular machine learning scenarios. Before deploying a predictive model, it's essential to estimate how accurate it will be. To measure the accuracy of a model, a common approach is the cross-validation and it's based on testing the model on the data which outcome is already known. The measures of the accuracy are based on comparing the prediction with the actual value. In order to simulate a real-life scenario, cross-validation consists in splitting the data into two parts: the training set and the test set. The model learns the patterns from the training set and predicts the outcome of the test set. Splitting the data in two parts as described, the estimation of the accuracy is based on the prediction of the test set only. To extend this approach to the entire data-set, we can split the data multiple times, in such a way that we perform a prediction on each record at least once. This approach is called k-fold and these are the steps:
    1. Split the data into k sets
    2. For each set, build a predictive model using the remaining data and predict the outcome on the set
    3. Validate the model comparing the real outcomes with the predictions
    In this article we see how to implement it using SQL Server 16 with Microsoft R Services.

    A use-case

    To show the k-fold approach, we deal with a business-like scenario. Some customers of a bank have mortgages and the bank wants to know which people are the most likely to have a default. The bank collected some data about its customers. Each record corresponds to a customer and each column to an attribute. The last column is 1 if the customer defaulted and 0 otherwise. This is a sample of the data
    creditScore houseAge yearsEmploy ccDebt year default

    691

    16

    9

    6725

    2000

    0

    691

    4

    4

    5077

    2000

    0

    743

    18

    3

    3080

    2000

    0

    728

    22

    1

    4345

    2000

    0

    745

    17

    3

    2969

    2000

    0

    539

    15

    3

    4588

    2000

    0
    Starting from data about new customers, the target is to predict whether the last column will be 0 or 1.

    Facing the use-case using SQL Server 16

    In this article, we use a SQL Server 16 VM in the Azure cloud with Microsoft R Services and RStudio. To organize the R code, we use a RStudio project. The data is initially stored into a xdf (eXternal Data Frame, the Microsoft R Services proprietary file-system). To predict the default, we use a popular model: the logistic regression. This article is high-level in the sense that we're going through the logic of the steps, without diving into the technical details of the code syntax. The steps to build the k-fold cross-validation are
    1. Define the data sources and the parameters
      • Define a xdf specifying where the original data is stored. The data is contained into a sample data directory. To define the data source, we use the function rxReadXdf.

        mortDefaultSmall <- rxReadXdf(file = file.path(rxGetOption("sampleDataDir"),

        "mortDefaultSmall.xdf"))

      • Define a local xdf where we want to build a copy of the original data. The file will be stored within the RStudio project.

        xdf_in <- RxXdfData("mort.xdf")

      • Define the number of folds of the algorithm.

        n_folds <- 5

      • Define the outcome that we want to predict.

        target_outcome <- "default"

      • Define the fields that we want to use to predict the target outcome. In this case, the predictors are the first 5 columns of mortDefaultSmall.

        predictors <- names(mortDefaultSmall)[1:5]

      • Import the data and define the fold column: we define a local copy of the mortgages data, with the addition of a fold column, containing a random number between 1 and 5, that assigns each record to one of the 5 sets. Also, we define a prediction column, containing the predicted score of having a credit default. The default value of the prediction is -1. For these purposes, we use the function rxDataStep.

      rxDataStep(mortDefaultSmall, xdf_in,

      transformObjects = list(n_folds = n_folds),

      transforms = list(fold = sample(1:n_folds, .rxNumRows, TRUE),

      predicted = rep(-1., .rxNumRows)),

      overwrite = TRUE)

    2. Define the features to include into the predictive model: to build the prediction model, we need to define a formula object describing which column is the outcome we want to predict and which columns are the predictors. The syntax is outcome ~ predictor1 + predictor2. To build the string, we use the function paste. Then, to define a formula object, we use the function formula.

      string_predictors <- paste(predictors, collapse = " + ")

      formula_predict <- formula(paste(target_outcome, string_predictors, sep = " ~ "))

      formula_predict

      default ~ creditScore + houseAge + yearsEmploy + ccDebt + year

    3. For each fold, build and apply the model as per described before
      We run a for loop defining a prediction for each set. for(this_fold in 1:n_folds) {
      • Build a predictive model on the base of the remaining data. For this purpose, we use rxLogit and we specify that the model is based on the data which fold is different from this_fold. To build a logistic regression model, we use rxLogit.

        model_logit <- rxLogit(formula = formula_predict,

        data = xdf_in,

        transformObjects = list(this_fold = this_fold),

        rowSelection = fold != this_fold)

      • Predict the default on the base of the predictive model. For technical reasons, the easiest way is to predict it on the entire dataset and to select the data about the fold later. We call this column prediction_fold. The function predicting the score is rxPredict.

        rxPredict(model_logit, xdf_in,

        predVarNames = "prediction_fold",

        overwrite = TRUE)

      • For each record of the fold, set the predicted column equal to prediction_fold. For this purpose, we use the function rxDataStep.

        rxDataStep(xdf_in, xdf_in,

        transformObjects = list(this_fold = this_fold),

        transforms = list(

        predicted = ifelse(fold == this_fold, prediction_fold, predicted)),

        overwrite = TRUE)

      }

    4. Measure the accuracy of the model: to evaluate the performance, we use a popular metric called AUC. For this purpose, we use the functions rxRoc and rxAuc.

      roc <- rxRoc(target_outcome, "predicted", xdf_in)

      auc <- rxAuc(roc)

      auc

      [1] 0.9328864

    The AUC score measures the predictive power of the model and it can be used to compare different models and optimize their parameters.

    Conclusions

    Using SQL Server with Microsoft R Server, we have been able to set-up the K-fold cross-validation. The code is written in such a way that it can be scaled across large data volumes.]]>