During cross-validation, Analysis Services divides the data in a mining structure into multiple cross-sections and then iteratively tests the structure and any associated mining models. Based on this analysis, it outputs a set of standard accuracy measures for the structure and each model.
The report contains some basic information about the number of folds in the data and the amount of data in each fold, and a set of general metrics that describe data distribution. By comparing the general metrics for each cross-section, you can assess the reliability of the structure or model.
Analysis Services also displays a set of detailed measures for mining models. These measures depend on the model type and on the type of attribute that is being analyzed: for example, whether it is discrete or continuous.
This section provides a list of the measures that are contained in the Cross-Validation report, and what they mean. For details on how each measure is calculated, see Cross-Validation Formulas.
List of Measures in the Cross-Validation Report
The following table lists the measures that appear in the cross-validation report. The measures are grouped by test type, which is provided in the left-hand column of the following table. The right-hand column lists the name of the measure as it appears in the report, and provides a brief explanation of what it means.
Test Type
Measures and Descriptions
Clustering
Measures that apply to clustering models
Case likelihood
This measure usually indicates how likely it is that a case belongs to a particular cluster.
For cross-validation, the scores are summed and then divided by the number of cases, so here the score is an average case likelihood.
Classification
Measures that apply to classification models
True Positive
True Negative
False Positive
False Positive
Count of rows or values in the partition where the predicted state matches the target state, and the predict probability is greater than the specified threshold.
Cases that have missing values for the target attribute are excluded, meaning the counts of all values might not add up.
Pass/Fail
Count of rows or values in the partition where the predicted state matches the target state, and where the predict probability value is greater than 0.
Likelihood
Likelihood measures apply to multiple model types.
Lift
The ratio of the actual prediction probability to the marginal probability in the test cases. Rows that have missing values for the target attribute are excluded.
This measure generally shows how much the probability of the target outcome improves when the model is used.
Root Mean Square Error
Square root of the mean error for all partition cases, divided by the number of cases in the partition, excluding rows that have missing values for the target attribute.
RMSE is a popular estimator for predictive models. The score averages the residuals for each case to yield a single indicator of model error.
Log score
The logarithm of the actual probability for each case, summed, and then divided by the number of rows in the input dataset, excluding rows that have missing values for the target attribute.
Because probability is represented as a decimal fraction, log scores are always negative numbers. A number closer to 0 is a better score. Whereas raw scores can have very irregular or skewed distributions, a log score is similar to a percentage.
Estimation
Measures that apply only to estimation models, which predict a continuous numeric attribute.
Root Mean Square Error
Average error when the predicted value is compared to the actual value.
RMSE is a popular estimator for predictive models. The score averages the residuals for each case to yield a single indicator of model error.
Mean Absolute Error
Average error when predicted values are compared to actual values, calculated as the mean of the absolute sum of errors.
Mean absolute error is useful for understanding how close overall the predictions were to actual values. A smaller score means predictions were more accurate.
Log Score
The logarithm of the actual probability for each case, summed, and then divided by the number of rows in the input dataset, excluding rows that have missing values for the target attribute.
Because probability is represented as a decimal fraction, log scores are always negative numbers. A number closer to 0 is a better score. Whereas raw scores can have very irregular or skewed distributions, a log score is similar to a percentage.
Aggregates
Aggregate measures provide an indication of the variance in the results for each partition.
Mean
Average of the partition values for a particular measure.
Standard Deviation
Average of the deviation from the mean for a specific measure, across all the partitions in a model.
For cross-validation, a higher value for this score implies substantial variation between the folds.