Schema
Introduction to Schema
The NimbusML data framework relies on a schema to understand the column names and mix of column
types in the dataset, which may originate from any of the supported Data Sources. It is
automatically inferred when a nimbusml.FileDataStream or nimbusml.DataSchema is created.
Transforms have the ability to operate on subsets of columns in the dataset, as well as alter the
resulting output schema, which effects other transforms downstream. For users, it would be very useful to
understand how NimbusML processes the data in a pipeline for debugging purposes or training the model with nimbusml.FileDataStream.
The schema comes with two formats for its representation, (1) object representation and (2) string format. After generating a nimbusml.FileDataStream, users can view the
object representation of the schema by using repr() function:
from nimbusml import FileDataStream
import numpy as np
import pandas as pd
data = pd.DataFrame(dict(real = [0.1, 2.2], text = ['word','class'], y = [1,3]))
data.to_csv('data.csv', index = False, header = True)
ds = FileDataStream.read_csv('data.csv', collapse = True,
numeric_dtype = np.float32, sep = ',')
print(repr(ds.schema))
#DataSchema([DataColumn(name='real', type='R4', pos=0),
# DataColumn(name='text', type='TX', pos=1),
# DataColumn(name='y', type='R4', pos=2)],
# header=True,
# sep=',')
The name, type, position of the columns are shown as well as the information about if the data has a header or what
the seperation of the columns is. It is always useful for users to examine the Schema of a nimbusml.FileDataStream before training
the model.
As can be seen in the above example, the arguments for the nimbusml.FileDataStream.read_csv() are used
to modify the Schema of the generated nimbusml.FileDataStream. More details about how
to modify the Schema is presented in DataSchema Alterations. All the arguments
discussed in this section are also applicable for nimbusml.FileDataStream.
The schema string format looks like a series of entries shown below:
col=<name>:<type>:<position> [options]
where
col= is specified for every column in the dataset,
name is the name of the column,
position is the 0-based index (or index range) of the column(s),
type is one of the column-types. When the position is a range (i.e. start_index-end_index), the column is of VectorDataViewType.
options
header= [+-] : Specifies if there is a header present in the text file
sep= [delimiter] : the delimiter for the columns
For instance,
schema = 'sep=, col=Features:R4:0-2 col=Label:R4:3 col=Text:TX:4 header+'
schema = 'sep=tab col=Sentiment:BL:1 col=SentimentSource:TX:2 col=SentimentText:TX:3 col=rownum:R4:4 header=+'
schema = 'sep=, col=Features:R4:0-4 col=UniqueCarrier:TX:5 col=Origin:TX:6 col=Dest:TX:7 col=Label:BL:9 header=+'
The first example indicates that the data is seperated by ,, the first three columns (with index ranging from 0 to 2) are named Features and with type R4, i.e. single precision floating-point.
The fourth column (with index 3) is named Label and with type R4. The fifth column (with index 4) is named Text and with type TX. The data has a header.
DataSchema Class
The nimbusml.DataSchema class can be used to automatically infer the schema from the different data sources.
Example of Schema for List
Lists are the simplest source of data. The schema inferred below shows that the values are treated as a single column with name Unknown of type TX, starting at index 0. The header=+ indicates that there is a header row in the data.
import numpy as np
from pandas import DataFrame
from nimbusml import DataSchema
list = [[1.0, 1.0, 2.0], [3.0, 5.0, 6.0]]
schema = DataSchema.read_schema(list)
print(repr(schema))
#DataSchema([DataColumn(name='c0', type='R8', pos=0),
# DataColumn(name='c1', type='R8', pos=1),
# DataColumn(name='c2',type='R8', pos=2)],
# header=True)
print(schema)
#col=c0:R8:0 col=c1:R8:1 col=c2:R8:2 header=+
Example of Schema for numpy.array
The DataSchema class infers that there is a header row in the dataset, and there are 3 columns, all of type R4 with index range of 0 to 2. When the type is changed from float32 to int16, the schema changes accoringly.
arr = np.array(list).astype(np.float32)
schema = DataSchema.read_schema(arr)
print(repr(schema))
#DataSchema([DataColumn(name='Data', type='R4', pos=(0, 1, 2))],
# header=True)
print(schema)
#col=Data:R4:0-2 header=+
arr = np.array(list).astype(np.int16)
schema = DataSchema.read_schema(arr)
print(repr(schema))
#DataSchema([DataColumn(name='Data', type='I2', pos=(0, 1, 2))],
# header=True)
print(schema)
#col=Data:I2:0-2 header=+
Example of Schema for pandas.DataFrame
The DataSchema class infers that there is a header row in the dataset, and there are 3 columns, all of types R8, I8 and TX, with column names X1, X2 and X3.
df = DataFrame(dict(X1=[0.1, 0.2], X2=[1, 2], X3=["a", "b"]))
schema = DataSchema.read_schema(df)
print(repr(schema))
#DataSchema([DataColumn(name='X1', type='R8', pos=0),
# DataColumn(name='X2', type='I8', pos=1),
# DataColumn(name='X3',type='TX', pos=2)],
# header=True)
print (schema)
#col=X1:R8:0 col=X2:I8:1 col=X3:TX:2 header=+
Example of Schema for a File
The transforms and trainers in NimbusML support various Data Sources as inputs.
When the data is in a pandas.DataFrame, the schema is inferred automatically from the
dtype of the columns.
When the data is in a file, the schema will be inferred when creating a nimbusml.FileDataStream using read_csv() or
using nimbusml.DataSchema.read_schema(). [Update when methods are included in API].
Example (from file):
from nimbusml import DataSchema
from pandas import DataFrame
from collections import OrderedDict
data = DataFrame(OrderedDict(real1=[0.1, 0.2], real2=[0.1, 0.2], integer=[1, 2], text=["a", "b"]))
# write dataframe to file
data.to_csv('data.txt', index=False)
# infer schema directly from file
schema = DataSchema.read_schema('data.txt')
print(repr(schema))
#DataSchema([DataColumn(name='real1', type='R8', pos=0),
# DataColumn(name='real2', type='R8', pos=1),
# DataColumn(name='integer', type='I8', pos=2),
# DataColumn(name='text', type='TX', pos=3)], header=True)
print(schema)
#col=real1:R8:0 col=real2:R8:1 col=integer:I8:2 col=text:TX:3 header=+
DataSchema Alterations
Merge Consecutive Columns of Same Type
Data may consist of numerous columns of the same type, and often it’s convenient to group them
under a single name. The nimbusml.DataSchema provides the collapse
argument to shorten the schema representation by grouping homongenous types.
Example:
schema = DataSchema.read_schema('data.txt', collapse=True)
print(repr(schema))
#DataSchema([DataColumn(name='real1', type='R8', pos=(0, 1)),
# DataColumn(name='integer', type='I8', pos=2),
# DataColumn(name='text', type='TX', pos=3)], header=True)
print(schema)
#col=real1:R8:0-1 col=integer:I8:2 col=text:TX:3 header=+
We see that columns real and real2 are merged into a single one col=real1:R8:0-1. It is not a
real anymore but a vector of two floats. Every learner uses features encoded as a vector of
features. Every transform in a pipeline would convert text, categories, floats into feature vectors. It is faster to do that
at loading time. The parameter collapse=True forces the function to merge consecutive columns
with the same type into vectors.
Merge All Columns of Same Type
If collapse == 'all', it merges all columns of the same type unless specified in argument names. Let’s see an example:
Example
from nimbusml.datasets import get_dataset
from pandas import read_csv
path = get_dataset("infert").as_filepath()
df = read_csv(path)
print(df.head(n=2))
Output:
row_num education age parity induced case spontaneous stratum pooled.stratum
0 1 0-5yrs 26 6 1 1 2 1 3
1 2 0-5yrs 42 1 1 1 0 2 1
case is the target, eveything else must be features if numeric. We want to merge every column into Features except row_num (row index), education (text) and case (target). education is not merged by default as it is not a numerical column.
Example:
import numpy as np
schema = DataSchema.read_schema(path, collapse='all', sep=',',
numeric_dtype=np.float32, #convert all numeric columns to R4
names={0:'row_num', 5:'case'})
print(repr(schema))
#DataSchema([DataColumn(name='row_num', type='R4', pos=0),
# DataColumn(name='education', type='TX', pos=1),
# DataColumn(name='age', type='R4', pos=(2, 3, 4, 6, 7, 8)),
# DataColumn(name='case', type='R4', pos=5)], header=True, sep=',')
print(schema)
#col=row_num:R4:0 col=education:TX:1 col=age:R4:2-4,6-8 col=case:R4:5 header=+ sep=,
Changing a column name
Some datasets have many columns and it is convenient to modify the first ones and let the function handle the rest. Below is an example of how to modify column names.
Example:
schema = DataSchema.read_schema('data.txt', collapse=True, sep=',',
names={0: 'newname', 1: 'newname2'})
print(repr(schema))
#DataSchema([DataColumn(name='newname', type='R8', pos=0),
# DataColumn(name='newname2', type='R8', pos=1),
# DataColumn(name='integer', type='I8', pos=2),
# DataColumn(name='text', type='TX', pos=3)], header=True, sep=',')
print(schema)
#col=newname:R8:0 col=newname2:R8:1 col=integer:I8:2 col=text:TX:3 header=+
Next example renames from column 0 to column 1 into real_0, real_1, …
Example:
schema = DataSchema.read_schema('data.txt', collapse=False, sep=',',
names={(0,1): 'real'})
print(repr(schema))
#DataSchema([DataColumn(name='real_0', type='R8', pos=0),
# DataColumn(name='real_1', type='R8', pos=1),
# DataColumn(name='integer', type='I8', pos=2),
# DataColumn(name='text', type='TX', pos=3)], header=True, sep=',')
print(schema)
#col=real_0:R8:0 col=real_1:R8:1 col=integer:I8:2 col=text:TX:3 header=+
Changing a column type
The read_schema() method uses the dtype argument to change all types or only a few.
We can also use numeric_dtype=np.float32 to change all numeric columns to R4 type.
Example:
schema = DataSchema.read_schema('data.txt', collapse=True, sep=',',
dtype={'real1': np.float32})
print(repr(schema))
#DataSchema([DataColumn(name='real1', type='R4', pos=0),
# DataColumn(name='real2', type='R8', pos=1),
# DataColumn(name='integer', type='I8', pos=2),
# DataColumn(name='text', type='TX', pos=3)], header=True, sep=',')
print(schema)
#col=real1:R4:0 col=real2:R8:1 col=integer:I8:2 col=text:TX:3 header=+
Other Arguments
The sep argument can be used to specify another separator besides ',', which is the default
delimiter. The user can also manually play with the schema himself.
Example:
for col in schema:
print(type(col), col)
#<class 'nimbusml.internal.utils.data_schema.DataColumn'> col=real1:R4:0
#<class 'nimbusml.internal.utils.data_schema.DataColumn'> col=real2:R8:1
#<class 'nimbusml.internal.utils.data_schema.DataColumn'> col=integer:I8:2
#<class 'nimbusml.internal.utils.data_schema.DataColumn'> col=text:TX:3
One Complex Example
In this section, we only show the string representation of the schema for simplicity.
Ranking models require three kind of columns. Two of the columns are the typical Features and
Label columns (of numeric type R4 == numpy.float32) and a third GroupId column which ties
all observations to a specific ranking group. Note that all examples with the same GroupId must
appear sequentially and its type must be TX == str. When reading the file without any additional
information, the raw schema is the following:
col=c0:I8:0 col=c1:I8:1 col=c2:I8:2 col=c3:I8:3 col=c4:I8:4 ... header=- sep=,
But we need to have this:
col=Label:R4:0 col=GroupId:TX:1 col=Features:R4:2-2109 header=- sep=,
Let’s see step by step how to get that and it starts with the raw schema generated using read_schema():
Example:
from nimbusml import DataSchema
from nimbusml.datasets import get_dataset
path = get_dataset('gen_tickettrain').as_filepath()
schema = DataSchema.read_schema(path, sep=',')
print(str(schema))
#col=rank:I8:0 col=group:I8:1 col=carrier:TX:2 col=price:I8:3 col=Class:I8:4
#col=dep_day:I8:5 col=nbr_stops:I8:6 col=duration:R8:7 header=+ sep=,
Let’s rename label and group id:
Example:
schema = DataSchema.read_schema(path, sep=',', header=True,
names={0:'Label', 1:'GroupId'}) # added
print(str(schema))
#col=Label:I8:0 col=GroupId:I8:1 col=carrier:TX:2...
Let’s change the column types. However, this requires to change the type of more than 2000 columns. As types can be changed given a column name and not its position, we use a regular expression to do so.
Example:
schema = DataSchema.read_schema(path, sep=',',
names={0:'Label', 1:'GroupId'},
dtype={'GroupId': str, 'Label': np.float32}) # added
print(str(schema))
#col=Label:R4:0 col=GroupId:TX:1 col=carrier:TX:2 col=price:I8:3
Let’s then merge every columns used later as features into a single name.
Example:
schema = DataSchema.read_schema(path, sep=',',
names={0:'Label', 1:'GroupId'},
dtype={'GroupId': str, 'Label': np.float32},
collapse = 'all') # added
print(str(schema))
#col=Label:R4:0 col=GroupId:TX:1 col=carrier:TX:2 col=price:I8:3-6 col=duration:R8:7 header=+ sep=,
And finally, let’s rename c2 into Features:
Example:
schema.rename('price', 'Features') # added
print(schema)
#col=Label:R4:0 col=GroupId:TX:1 col=carrier:TX:2 col=Features:I8:3-6 col=duration:R8:7 header=+ sep=, #Voila!
Most of datasets are stored in text files. It is usually more convenient to load them in memory with
pandas. But when the datasets is too big, nimbusml has to directly load the data from its
location. It is more efficient to tell the parser which names and types it should use than changing
them by adding transforms in the pipeline. Given the nimbusml.DataSchema generated
above, a nimbusml.FileDataStream can be created to train the model:
Example:
from nimbusml.datasets import get_dataset
from nimbusml import Pipeline, FileDataStream, DataSchema
from nimbusml.ensemble import LightGbmClassifier
path = get_dataset('infert').as_filepath()
schema = DataSchema.read_schema(path, sep=',')
ds = FileDataStream(path, schema = schema)
#Equivalent to
#ds = FileDataStream.read_csv(path, sep=',')
pipeline = Pipeline([
LightGbmClassifier(feature=['age', 'parity', 'induced'], label='case')
])
pipeline.fit(ds)
pipeline.predict(ds)