2.3.1 Column Data Storage

A column data storage file for each column in the source data table MUST be present. A separate file is used to store the column data for each column. An example of a generated file name for a column data storage file for a table that has the identifier "Table1" and a column that has the identifier "Cat" is 4.Table1.Cat.0.idf. For an explanation of the interpretation of the substrings within the file name, see section 2.2.

The system represents each unique data value in a column by an assigned data identifier for that unique value. In the tabular data model file format, the data identifier is always stored, but the data value is not stored. To decode the data identifiers into their actual values, the data dictionary (section 2.3.2) or value hash index (section 2.3.3) is used. The system MAY<9> use any method for assigning data identifiers to values.

The column data file contains an array of the data identifier values that represent the values contained in each row of the column in the source data. In this file, one data identifier is represented per row in the source data column. The order in which the data identifiers appear can vary from the order in which they appear in the rows of the source data table. Partial sorting of values MAY<10> be performed to optimize compression.

The column data storage file is compressed by one of several methods, although it is always compressed by using a XMHybridRLE compression method (except for the special case of the RowNumber column, which uses XM123 compression as part of a hybrid). For a discussion of the types of compression that are available to be used, see section 2.7.

It is necessary to reference the XML metadata file to understand and decode the contents of the column data storage file. An example of a file name for the file that contains the metadata for the data storage file for a table that has the identifier "Table1" is Table1.1.tbl.xml. The metadata file contains the metadata for all the columns of the table. For example, the metadata for the column "Cat" in table "Table1" is found in the Columns collection of the XMSimpleTable object in the metadata file. In the Columns collection, a Column item exists for every column in the table. The Column item in the Column collection for which the value of the name attribute is "Cat" contains the metadata for the "Cat" column. For an explanation of how to interpret the metadata file, see section 2.5.