Pipeline Class
Implementation of a pipeline.
 Inheritance

builtins.objectPipeline
Constructor
Pipeline(steps=None, model=None, random_state=None)
Parameters
 steps
the list of operator or (name, operator) tuples that are chained in the appropriate order.
 model
the path to the model file (".zip") if want to load a model directly from file (such as a trained model from ML.NET).
 random_state
the integer used as the random seed.
Remarks
The Pipeline class assembles a pipeline of transforms, followed optionally by a trainer. The transforms need to implement fit() and transform() methods. The final trainer only needs to implement the fit() method.
The Pipeline class only accepts trainers and transforms implemented in this package.
The data sources for the methods may be a list, numpy.array, scipy.sparse_csr, pandas.DataFrame or a FileDataStream.
By default, the first transform will take all columns as input ( i.e. will transform all columns), unless specific columns are requested (see Columns for how to specify columns to transform). The output column of the first transform is passed as the input column into the second transform for processing by default, unless the second transform requests a different column to operate on.
The final trainer (if one exists) can select which columns to use for feature, labels, weights etc. See Roles for more details on how to select these.
Methods
append 
Extends the pipeline with a new transform/learner at the end.
Note that a fitted pipeline cannot be modified.
Example: Example: 
clone 
Clones the pipeline and returns it in a nontrained state
if the trained model was stored in a file on disk.
You can clone the trained pipeline by running:

combine_models 
Combine the models of multiple pipelines, transforms and/or predictors in to a single model. The models are combined in the order they are seen. 
decision_function 
Apply transforms and generate decision values 
fit 
Fit the pipeline. 
fit_transform 
If a pipeline only has transforms, returns transformed data as a pandas dataframe 
get_feature_contributions 
Calculates observation level feature contributions. Returns dataframe with raw data, predictions, and feature contributiuons for each prediction. Feature contributions are not supported for transforms, so make sure that the last step in a pipeline is a model. Feature contriutions are supported for the following models:

get_fit_info 
Returns information about the pipeline. Example: 
get_output_columns 
Returns the output list of columns for the fitted model. :return: list . 
get_params 
Returns pipeline parameters 
insert 
Inserts a transform/learner into the pipeline. Example: Example: 
load_model 
Load model from file. The model can be generated from ML.NET in .zip format. For more details, please refer to load/save model 
permutation_feature_importance 
Permutation feature importance (PFI) is a technique to determine the global importance of features in a trained machine learning model. PFI is a simple yet powerful technique motivated by Breiman in section 10 of his Random Forests paper (Machine Learning, 2001). The advantage of the PFI method is that it is model agnostic  it works with any model that can be evaluated  and it can use any dataset, not just the training set, to compute feature importance metrics. PFI works by taking a labeled dataset, choosing a feature, and permuting the values for that feature across all the examples, so that each example now has a random value for the feature and the original values for all other features. The evaluation metric (e.g. NDCG) is then calculated for this modified dataset, and the change in the evaluation metric from the original dataset is computed. The larger the change in the evaluation metric, the more important the feature is to the model, i.e. the most important features are those that the model is most sensitive to. PFI works by performing this permutation analysis across all the features of a model, one after another. Note that for increasing metrics (e.g. AUC, accuracy, RSquared, NDCG), the most important features will be those with the highest negative mean change in the metric. Conversely, for decreasing metrics (e.g. Mean Squared Error, Log loss), the most important features will be those with the highest positive mean change in the metric. PFI is supported for binary classifiers, classifiers, regressors, and rankers. The mean changes and statndard errors of the means are evaluated for the following metrics are evaluated for PFI:
Reference 
predict 
Predict based on the input data 
predict_proba 
Apply transforms and predict probabilities 
save_model 
Save model to file. For more details, please refer to load/save model 
score 
Return performance metrics for the corresponding problem 
set_params 
Set parameters to the pipeline. 
summary 
Return summary for fitted model. 
test 
Return both predictions and performance metrics. For more details please refer to Metrics. 
transform 
Apply transforms 
append
Extends the pipeline with a new transform/learner at the end.
Note that a fitted pipeline cannot be modified.
Example: pipe.append(FastLinearRegressor())
.
Example: pipe.append(("learner", FastLinearRegressor()))
.
append(step)
Parameters
 step
the transform/learner to append
clone
Clones the pipeline and returns it in a nontrained state
if the trained model was stored in a file on disk.
You can clone the trained pipeline by running:
Pipeline(**pipe.get_params())
.
clone()
combine_models
Combine the models of multiple pipelines, transforms and/or predictors in to a single model. The models are combined in the order they are seen.
combine_models(*items, **params)
Parameters
 items
the fitted pipelines, transforms and/or predictors which contain the models to join.
 contains_predictor
Set to True if the last item contains or is a predictor. Set to False if items only contains transforms. The default is True.
Returns
A new Pipeline which is backed by a model that is the combination of all the models passed in through items.
decision_function
Apply transforms and generate decision values
decision_function(X, verbose=0, **params)
Parameters
Returns
array, shape=(n_samples,) if n_classes == 2 else ( n_samples, n_classes)
fit
Fit the pipeline.
fit(X, y=None, verbose=1, **params)
Parameters
 y
{arraylike [n_samples]}
fit_transform
If a pipeline only has transforms, returns transformed data as a pandas dataframe
fit_transform(X, y=None, verbose=0, as_binary_data_stream=False, **params)
Parameters
 y
{arraylike [n_samples]}
 params
Additional arguments.
If as_csr=True
and as_binary_data_stream=False
then
return the transformed data in CSR (sparse matrix) format.
If as_binary_data_stream
is also true then that
parameter takes precedence over as_csr
and the output will
be an IDV file.
Returns
Returns a pandas DataFrame if no other output format
is specified. See as_binary_data_stream
and as_csr
for other available output formats.
get_feature_contributions
Calculates observation level feature contributions. Returns dataframe with raw data, predictions, and feature contributiuons for each prediction. Feature contributions are not supported for transforms, so make sure that the last step in a pipeline is a model. Feature contriutions are supported for the following models:
Regression:
OrdinaryLeastSquaresRegressor
FastLinearRegressor
OnlineGradientDescentRegressor
PoissonRegressionRegressor
GamRegressor
LightGbmRegressor
FastTreesRegressor
FastForestRegressor
FastTreesTweedieRegressor
Binary Classification:
AveragedPerceptronBinaryClassifier
LinearSvmBinaryClassifier
LogisticRegressionBinaryClassifier
FastLinearBinaryClassifier
SgdBinaryClassifier
SymSgdBinaryClassifier
GamBinaryClassifier
FastForestBinaryClassifier
FastTreesBinaryClassifier
LightGbmBinaryClassifier
Ranking:
 LightGbmRanker
get_feature_contributions(X, top=10, bottom=10, verbose=0, as_binary_data_stream=False, **params)
Parameters
 top
the number of positive contributions with highest magnitude to report.
 bottom
The number of negative contributions with highest magnitude to report.
Returns
dataframe containing the raw data, predicted label, score, probabilities, and feature contributions.
get_fit_info
Returns information about the pipeline.
Example: pipe.get_fit_info(X,Y)
.
get_fit_info(X, y=None, **params)
Parameters
 y
{arraylike [n_samples]}
Returns
tuple (list of dictonaries, list of entrypoints), both lists do not necessarily have the same length
Remarks
The first list is the list of operators the user defines. In that case, the result is a list of dictionaries with keys operator, name, inputs, outputs, type, current_schema. The last is the schema after the transform or a learner is applied.
The second list is what nimbusml internally uses. The number of entrypoint may be different from the list of operators. This information is mostly used by contributors.
get_output_columns
Returns the output list of columns for the fitted model. :return: list .
get_output_columns(verbose=0, **params)
Parameters
 verbose
get_params
Returns pipeline parameters
get_params(deep=False)
Parameters
 deep
boolean, optional If True, will return the parameters for this pipeline and contained subobjects that are estimators.
insert
Inserts a transform/learner into the pipeline.
Example: pipe.insert(1, FastLinearRegressor())
.
Example: pipe.insert(1, ("learner", FastLinearRegressor()))
.
insert(pos, step)
Parameters
 pos
position to insert, should be integers
 step
the transform/learner to insert
load_model
Load model from file. The model can be generated from ML.NET in .zip format. For more details, please refer to load/save model
load_model(src)
Parameters
 dst
source filename to be loaded
permutation_feature_importance
Permutation feature importance (PFI) is a technique to determine the global importance of features in a trained machine learning model. PFI is a simple yet powerful technique motivated by Breiman in section 10 of his Random Forests paper (Machine Learning, 2001). The advantage of the PFI method is that it is model agnostic  it works with any model that can be evaluated  and it can use any dataset, not just the training set, to compute feature importance metrics.
PFI works by taking a labeled dataset, choosing a feature, and permuting the values for that feature across all the examples, so that each example now has a random value for the feature and the original values for all other features. The evaluation metric (e.g. NDCG) is then calculated for this modified dataset, and the change in the evaluation metric from the original dataset is computed. The larger the change in the evaluation metric, the more important the feature is to the model, i.e. the most important features are those that the model is most sensitive to. PFI works by performing this permutation analysis across all the features of a model, one after another.
Note that for increasing metrics (e.g. AUC, accuracy, RSquared, NDCG), the most important features will be those with the highest negative mean change in the metric. Conversely, for decreasing metrics (e.g. Mean Squared Error, Log loss), the most important features will be those with the highest positive mean change in the metric.
PFI is supported for binary classifiers, classifiers, regressors, and rankers.
The mean changes and statndard errors of the means are evaluated for the following metrics are evaluated for PFI:
Binary Classification:
Area under ROC curve (AUC)
Accuracy
Positive precision
Positive recall
Negative precision
Negative recall
F1 score
Area under PrecisionRecall curve (AUPRC)
Multiclass classification:
Macro accuracy
Micro accuracy
Log loss
Log loss reduction
Top k accuracy
Perclass log loss
Regression:
Mean absolute error (MAE)
Mean squared error (MSE)
Root mean squared error (RMSE)
Loss function
RSquared
Ranking
Discounted cumulative gains (DCG) @1, @2, and @3
Normalized discounted cumulative gains (NDCG) @1, @2, and @3
Reference
Breiman, L. Random Forests. Machine Learning (2001) 45: 5.
permutation_feature_importance(X, number_of_examples=None, permutation_count=1, filter_zero_weight_features=False, verbose=0, as_binary_data_stream=False, **params)
Parameters
 number_of_examples
Limit the number of examples to evaluate on.
'None'
means all examples in the dataset are used.
 permutation_count
The number of permutations to perform.
Returns
dataframe containing the mean change in evaluation metrics and standard error of the mean for each feature. Features with the largest change in a metric are the most important in the model with respect to that metric.
predict
Predict based on the input data
predict(X, verbose=0, as_binary_data_stream=False, **params)
Parameters
predict_proba
Apply transforms and predict probabilities
predict_proba(X, verbose=0, **params)
Parameters
Returns
array, shape = [n_samples, n_classes]
save_model
Save model to file. For more details, please refer to load/save model
save_model(dst)
Parameters
 dst
filename to be saved with
score
Return performance metrics for the corresponding problem
score(X, y, evaltype='auto', group_id=None, weight=None, verbose=0, **params)
Parameters
 y
{arraylike [n_samples]}
 evaltype
the evaluation type for the problem, can be { 'binary', 'multiclass', 'regression', 'cluster', 'anomaly', 'ranking'}. The default is 'auto'. If model is loaded using the load_model() method, evaltype cannot be 'auto', and therefore must be explicitly specified.
set_params
Set parameters to the pipeline.
set_params(**params)
summary
Return summary for fitted model.
summary(verbose=0, **params)
Examples
###############################################################################
# Pipeline
import numpy as np
import pandas as pd
from nimbusml import Pipeline, FileDataStream
from nimbusml.linear_model import FastLinearRegressor
from nimbusml.preprocessing.normalization import MeanVarianceScaler
X = np.array([[1, 2.0], [2, 4], [3, 0.7]])
Y = np.array([2, 3, 1.5])
df = pd.DataFrame(dict(y=Y, x1=X[:, 0], x2=X[:, 1]))
pipe = Pipeline([
MeanVarianceScaler(),
FastLinearRegressor()
])
# fit with pandas dataframe
pipe.fit(X, Y)
# Fit with FileDataStream
df.to_csv('data.csv', index=False)
ds = FileDataStream.read_csv('data.csv', sep=',')
pipe = Pipeline([
MeanVarianceScaler(),
FastLinearRegressor()
])
pipe.fit(ds, 'y')
print(pipe.summary())
# Bias Weights.x1 Weights.x2
# 0 1.032946 0.111758 1.210791
test
Return both predictions and performance metrics. For more details please refer to Metrics.
test(X, y=None, evaltype='auto', group_id=None, weight=None, verbose=0, output_scores=False, as_binary_data_stream=False, **params)
Parameters
 y
{arraylike [n_samples]}
 evaltype
the evaluation type for the problem, can be { 'binary', 'multiclass', 'regression', 'cluster', 'anomaly', 'ranking'}. The default is 'auto'. If model is loaded using the load_model() method, evaltype cannot be 'auto', and therefore must be explicitly specified.
 group_id
the column name for group_id for ranking problem
 weight
the column name for the weight column for each sample
 output_scores
if set to True will return raw scores, otherwise None in the returned tuple.
Returns
tuple (dataframe of evaluation metrics, dataframe of
scores). If scores are
required, set >><<output_scores
=True, otherwise None is
returned by default.
transform
Apply transforms
transform(X, verbose=0, as_binary_data_stream=False, **params)
Parameters
 y
{arraylike [n_samples]}
 params
Additional arguments.
If as_csr=True
and as_binary_data_stream=False
then
return the transformed data in CSR (sparse matrix) format.
If as_binary_data_stream
is also true then that
parameter takes precedence over as_csr
and the output will
be an IDV file.
Returns
Returns a pandas DataFrame if no other output format
is specified. See as_binary_data_stream
and as_csr
for other available output formats.