# Basic R commands and RevoScaleR functions: 25 common examples

Important

This content is being retired and may not be updated in the future. The support for Machine Learning Server will end on July 1, 2022. For more information, see What's happening to Machine Learning Server?

**Applies to: Microsoft R Client, Machine Learning Server**

If you are new to both R and Machine Learning Server, this tutorial introduces you to 25 (or so) commonly used R functions. In this tutorial, you learn how to load small data sets into R and perform simple computations. A key point to take away from this tutorial is that you can combine basic R commands and RevoScaleR functions in the same R script.

This tutorial starts with R commands before transitioning to RevoScaleR functions. If you already know R, skip ahead to Explore RevoScaleR Functions.

Note

R Client and Machine Learning Server are interchangeable in terms of RevoScaleR as long as data fits into memory and processing is single-threaded. If data size exceeds memory, we recommend pushing the compute context to Machine Learning Server.

## Prerequisites

This is our simplest tutorial in terms of data and tools, but it's also expansive in its coverage of basic R and RevoScaleR functions. To complete the tasks, use the command-line tool **RGui.exe** on Windows or start the **Revo64** program on Linux.

- On Windows, go to \Program Files\Microsoft\R Client\R_SERVER\bin\x64 and double-click
**Rgui.exe**. - On Linux, at the command prompt, type
**Revo64**.

The tutorial uses pre-installed sample data so once you have the software, there is nothing more to download or install.

The R command prompt is `>`

. You can hand-type commands line by line, or copy-paste a multi-line command sequence.

R is case-sensitive. If you hand-type commands in this example, be sure to use the correct case. Windows users: the file paths in R take a forward slash delimiter (/), required even when the path is on the Windows file system.

## Start with R

Because RevoScaleR is built on R, this tutorial begins with an exploration of common R commands.

### Load data

R is an environment for analyzing data, so the natural starting point is to load some data. For small data sets, such as the following 20 measurements of the speed of light taken from the famous Michelson-Morley experiment, the simplest approach uses R’s `c`

function to combine the data into a vector.

Type or copy the following script and paste it at the `>`

prompt at the beginning of the command line:

```
c(850, 740, 900, 1070, 930, 850, 950, 980, 980, 880, 1000, 980, 930, 650, 760, 810, 1000, 1000, 960, 960)
```

When you type the closing parenthesis and press *Enter*, R responds as follows:

```
[1] 850 740 900 1070 930 850 950 980 980 880
[11] 1000 980 930 650 760 810 1000 1000 960 960
```

This indicates that R has interpreted what you typed, created a vector with 20 elements, and returned that vector. But we have a problem. R hasn’t saved what we typed. If we want to use this vector again (and that’s the usual reason for creating a vector in the first place), we need to *assign* it. The R assignment operator has the suggestive form *<-* to indicate a value is being assigned to a name. You can use most combinations of letters, numbers, and periods to form names (but note that names can’t begin with a number). Here we’ll use `michelson`

:

```
michelson <- c(850, 740, 900, 1070, 930, 850, 950, 980, 980, 880,
1000, 980, 930, 650, 760, 810, 1000, 1000, 960, 960)
```

R responds with a `>`

prompt. Notice that the named vector is not automatically printed when it is assigned. However, you can view the vector by typing its name at the prompt:

```
> michelson
```

Output:

```
[1] 850 740 900 1070 930 850 950 980 980 880
[11] 1000 980 930 650 760 810 1000 1000 960 960
```

The `c`

function is useful for hand typing in small vectors such as you might find in textbook examples, and it is also useful for combining existing vectors. For example, if we discovered another five observations that extended the Michelson-Morley data, we could extend the vector using `c`

as follows:

```
michelsonNew <- c(michelson, 850, 930, 940, 970, 870)
michelsonNew
```

Output:

```
[1] 850 740 900 1070 930 850 950 980 980 880
[11] 1000 980 930 650 760 810 1000 1000 960 960
[21] 850 930 940 970 870
```

#### Generate random data

Often for testing purposes you want to use randomly generated data. R has a number of built-in distributions from which you can generate random numbers; two of the most commonly used are the normal and the uniform distributions. To obtain a set of numbers from a normal distribution, you use the `rnorm`

function. This example generates 25 random numbers in a normal distribution.

```
normalDat <- rnorm(25)
normalDat
```

Output:

```
[1] -0.66184983 1.71895416 2.12166699
[4] 1.49715368 -0.03614058 1.23194518
[7] -0.06488077 1.06899373 -0.37696531
[10] 1.04318309 -0.38282188 0.29942160
[13] 0.67423976 -0.29281632 0.48805336
[16] 0.88280182 1.86274898 1.61172529
[19] 0.13547954 1.08808601 -1.26681476
[22] -0.19858329 0.13886578 -0.27933600
[25] 0.70891942
```

By default, the data are generated from a standard normal with mean 0 and standard deviation 1. You can use the `mean`

and `sd`

arguments to `rnorm`

to specify a different normal distribution:

```
normalSat <- rnorm(25, mean=450, sd=100)
normalSat
```

Output:

```
[1] 373.3390 594.3363 534.4879 410.0630 307.2232
[6] 307.8008 417.1772 478.4570 521.9336 493.2416
[11] 414.8075 479.7721 423.8568 580.8690 451.5870
[16] 406.8826 488.2447 454.1125 444.0776 320.3576
[21] 236.3024 360.6385 511.2733 508.2971 449.4118
```

Similarly, you can use the `runif`

function to generate random data from a uniform distribution:

```
uniformDat <- runif(25)
uniformDat
```

Output:

```
[1] 0.03105927 0.18295065 0.96637386 0.71535963
[5] 0.16081450 0.15216891 0.07346868 0.15047337
[9] 0.49408599 0.35582231 0.70424152 0.63671421
[13] 0.20865305 0.20167994 0.37511929 0.54082887
[17] 0.86681824 0.23792988 0.44364083 0.88482396
[21] 0.41863803 0.42392873 0.24800036 0.22084038
[25] 0.48285406
```

#### Generate a numeric sequence

The default uniform distribution is over the interval 0 to 1. You can specify alternatives by setting the `min`

and `max`

arguments:

```
uniformPerc <- runif(25, min=0, max=100)
uniformPerc
```

Output:

```
[1] 66.221400 12.270863 33.417174 21.985229
[5] 92.767213 17.911602 1.935963 53.551991
[9] 75.110760 22.436347 63.172258 95.977501
[13] 79.317351 56.767608 89.416080 79.546495
[17] 8.961152 49.315612 43.432128 68.871867
[21] 73.598221 63.888835 35.261694 54.481692
[25] 37.575176
```

Another commonly used vector is the `sequence`

, a uniformly spaced run of numbers. For the common case of a run of integers, you can use the infix operator, :, as follows:

```
1:10
```

Output:

```
[1] 1 2 3 4 5 6 7 8 9 10
```

For more general sequences, use the `seq`

function:

```
seq(length = 11, from = 10, to = 30)
```

Output:

```
[1] 10 12 14 16 18 20 22 24 26 28 30
seq(from = 10,length = 20, by = 4)
```

Output:

```
[1] 10 14 18 22 26 30 34 38 42 46 50 54 58 62 66 70 74
[18] 78 82 86
```

Tip

If you are working with big data, you’ll still use vectors to manipulate parameters and information about your data, but you'll probably store the data in the RevoScaleR high-performance .xdf file format.

### Exploratory Data Analysis

After you have some data, you will want to explore it graphically. For most small data sets, the place to begin is with the `plot`

function, which provides a default graphical view of the data:

```
plot(michelson)
plot(normalDat)
plot(uniformPerc)
plot(1:10)
```

For numeric vectors such as ours, the default view is a scatter plot of the observations against their index, resulting in the following plots:

For an exploration of the shape of the data, the usual tools are `stem`

(to create a stemplot) and `hist`

(to create a histogram):

```
stem(michelson)
```

Output:

```
The decimal point is 2 digit(s) to the right of the |
6 | 5
7 | 46
8 | 1558
9 | 033566888
10 | 0007
hist(michelson)
```

The resulting histogram is shown as the left plot following. We can make the histogram look more like the stemplot by specifying the `nclass`

argument to `hist`

:

```
hist(michelson, nclass=5)
```

The resulting histogram is shown as the right plot in the figure following.

From the histogram and stemplot, it appears that the Michelson-Morley observations are not obviously normal. A normal Q-Q plot gives a graphical test of whether a data set is normal:

```
qqnorm(michelson)
```

The decided bend in the resulting plot confirms the suspicion that the data are not normal.

Another useful exploratory plot, especially for comparing two distributions, is the boxplot:

```
boxplot(normalDat, uniformDat)
```

Tip

These plots are great if you have a small data set in memory. However, when working with big data, some plot types may not be informative when working directly with the data (for example, scatter plots can produce a large blob of ink) and others may be computational intensive (if sorting is required). A better alternative is the *rxHistogram* function in RevoScaleR that efficiently computes and renders histograms for large data sets. Additionally, RevoScaleR functions such as *rxCube* can provide summary information that is easily amenable to the impressive plotting capabilities provided by R packages.

### Summary Statistics

While an informative graphic often gives the fullest description of a data set, numerical summaries provide a useful shorthand for describing certain features of the data. For example, estimators such as the mean and median help to locate the data set, and the standard deviation and variance measure the scale or spread of the data. R has a full set of summary statistics available:

```
> mean(michelson)
[1] 909
> median(michelson)
[1] 940
> sd(michelson)
[1] 104.9260
> var(michelson)
[1] 11009.47
```

The generic summary function provides a meaningful summary of a data set; for a numeric vector it provides the five-number summary plus the mean:

```
summary(michelson)
```

Output:

```
Min. 1st Qu. Median Mean 3rd Qu. Max.
650 850 940 909 980 1070
```

Tip

The *rxSummary* function in RevoScaleR will efficiently compute summary statistics for a data frame in memory or a large data file stored on disk.

### Multivariate Data Sets

In most disciplines, meaningful data sets have multiple variables, typically observations of various quantities and qualities of individual subjects. Such data sets are typically represented as tables in which the columns correspond to variables and the rows correspond to subjects, or cases. In R, such tables can be created as *data frame* objects. For example, at the 2008 All-Star Break, the Seattle Mariners had five players who met the minimum qualifications to be considered for a batting title (that is, at least 3.1 at bats per game played by the team). Their statistics are shown in the following table:

```
Player Games AB R H 2B 3B HR TB RBI BA OBP SLG OPS
"I. Suzuki" 95 391 63 119 11 3 3 145 21 .304 .366 .371 .737
"J. Lopez" 92 379 46 113 26 1 5 156 48 .298 .318 .412 .729
"R. Ibanez" 95 370 41 101 26 1 11 162 55 .273 .338 .438 .776
"Y. Betancourt" 90 326 34 87 22 2 3 122 29 .267 .278 .374 .656
"A. Beltre" 92 352 46 91 16 0 16 155 46 .259 .329 .440 .769
```

Copy and paste the table into a text editor (such as Notepad on Windows, or emacs or vi on Unix type machines) and save the file as **msStats.txt** in the working directory returned by the *getwd* function, for example:

```
getwd()
```

Output:

```
[1] "/Users/joe"
```

Tip

On Windows, the working directory is probably a bin folder in program files, and by default you don't have permission to save the file at that location. Use *setwd("/Users/TEMP")* to change the working directory to /Users/TEMP and save the file.

You can then read the data into R using the `read.table`

function. The argument *header=TRUE* specifies that the first line is a header of variable names:

```
msStats <- read.table("msStats.txt", header=TRUE)
msStats
```

Output:

```
Player Games AB R H X2B X3B HR TB RBI
1 I. Suzuki 95 391 63 119 11 3 3 145 21
2 J. Lopez 92 379 46 113 26 1 5 156 48
3 R. Ibanez 95 370 41 101 26 1 11 162 55
4 Y. Betancourt 90 326 34 87 22 2 3 122 29
5 A. Beltre 92 352 46 91 16 0 16 155 46
BA OBP SLG OPS
1 0.304 0.366 0.371 0.737
2 0.298 0.318 0.412 0.729
3 0.273 0.338 0.438 0.776
4 0.267 0.278 0.374 0.656
5 0.259 0.329 0.440 0.769
```

Notice how `read.table`

changed the names of our original “2B" and “3B" columns to be valid R names; R names cannot begin with a numeral.

#### Use built-in datasets package

Most small R data sets in daily use are data frames. The built-in package, **datasets**, is a rich source of data frames for further experimentation.

To list the data sets in this package, use this command:

```
ls("package:datasets")
```

The output is an alphabetized list of data sets that are readily available for use in R functions. In the next section, we will use the built-in data set *attitude*, available through the **datasets** package.

### Linear Models

The *attitude* data set is a data frame with 30 observations on 7 variables, measuring the percent proportion of favorable responses to seven survey questions in each of 30 departments. The survey was conducted in a large financial organization; there were approximately 35 respondents in each department.

We mentioned that the `plot`

function could be used with virtually any data set to get an initial visualization; let’s see what it gives for the *attitude* data:

```
plot(attitude)
```

The resulting plot is a pairwise scatter plot of the numeric variables in the data set.

The first two variables (*rating* and *complaints*) show a strong linear relationship. To model that relationship, we use the `lm`

function:

```
attitudeLM1 <- lm(rating ~ complaints, data=attitude)
```

To view a summary of the model, we can use the `summary`

function:

```
summary(attitudeLM1)
```

Output:

```
Call:
lm(formula = rating ~ complaints, data = attitude)
Residuals:
Min 1Q Median 3Q Max
-12.8799 -5.9905 0.1783 6.2978 9.6294
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 14.37632 6.61999 2.172 0.0385 *
complaints 0.75461 0.09753 7.737 1.99e-08 ***
---
Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1
Residual standard error: 6.993 on 28 degrees of freedom
Multiple R-squared: 0.6813, Adjusted R-squared: 0.6699
F-statistic: 59.86 on 1 and 28 DF, p-value: 1.988e-08
```

We can also try to visualize the model using `plot`

:

```
plot(attitudeLM1)
```

The default plot for a fitted linear model is a set of four plots; by default they are shown one at a time, and you are prompted before each new plot is displayed. To view them all at once, use the `par`

function with the *mfrow* parameter to specify a 2 x 2 layout:

```
par(mfrow=c(2,2))
plot(attitudeLM1)
```

Tip

The rxLinMod function is a full-featured alternative to lm that can efficiently handle large data sets. Also look at rxLogit and rxGlm as alternatives to `glm`

, rxKmeans as an alternative to `kmeans`

, and rxDTree as an alternative to `rpart`

.

### Matrices and *apply* function

A matrix is a two-dimensional data array. Unlike data frames, which can have different data types in their columns, matrices may contain data of only one type. Most commonly, matrices are used to hold numeric data. You create matrices with the `matrix`

function:

```
A <- matrix(c(3, 5, 7, 9, 13, 15, 8, 4, 2), ncol=3)
A
```

Output:

```
[,1] [,2] [,3]
[1,] 3 9 8
[2,] 5 13 4
[3,] 7 15 2
```

Another matrix example:

```
B <- matrix(c(4, 7, 9, 5, 8, 6), ncol=3)
B
```

Output:

```
[,1] [,2] [,3]
[1,] 4 9 8
[2,] 7 5 6
```

Ordinary arithmetic acts *element-by-element* on matrices:

```
A + A
```

Output:

```
[,1] [,2] [,3]
[1,] 6 18 16
[2,] 10 26 8
[3,] 14 30 4
```

Matrix multiplication uses the expected operators:

```
A * A
```

Output:

```
[,1] [,2] [,3]
[1,] 9 81 64
[2,] 25 169 16
[3,] 49 225 4
```

Matrix multiplication in the linear algebra sense requires a special operator, `%*%`

:

```
A %*% A
[,1] [,2] [,3]
[1,] 110 264 76
[2,] 108 274 100
[3,] 110 288 120
```

Matrix multiplication requires two matrices to be *conformable*, which means that the number of *columns* of the first matrix is equal to the number of *rows* of the second:

```
B %*% A
[,1] [,2] [,3]
[1,] 113 273 84
[2,] 88 218 88
A %*% B
Error in A %*% B : non-conformable arguments
```

When you need to manipulate the rows or columns of a matrix, an incredibly useful tool is the `apply`

function. With `apply`

, you can apply a function to all the rows or columns of matrix at once. For example, to find the column products of *A*, you could use `apply`

as follows:

```
apply(A, 2, prod)
[1] 105 1755 64
```

The row products are as simple:

```
apply(A, 1, prod)
[1] 216 260 210
```

To sort the columns of *A*, just replace *prod* with *sort*:

```
apply(A, 2, sort)
[,1] [,2] [,3]
[1,] 3 9 2
[2,] 5 13 4
[3,] 7 15 8
```

### Lists and *lapply* function

A `list`

in R is a flexible data object that can be used to combine data of different types and different lengths for almost any purpose. Arbitrary lists can be created with either the list function or the c function; many other functions, especially the statistical modeling functions, return their output as list objects.

For example, we can combine a character vector, a numeric vector, and a numeric matrix in a single list as follows:

```
list1 <- list(x = 1:10, y = c("Tami", "Victor", "Emily"),
z = matrix(c(3, 5, 4, 7), nrow=2))
list1
$x
[1] 1 2 3 4 5 6 7 8 9 10
$y
[1] "Tami" "Victor" "Emily"
$z
[,1] [,2]
[1,] 3 4
[2,] 5 7
```

The function `lapply`

can be used to apply the same function to each component of a list in turn:

```
lapply(list1, length)
$x
[1] 10
$y
[1] 3
$z
[1] 4
```

Tip

You will regularly use lists and functions that manipulate them when handling input and output for your big data analyses.

## Explore RevoScaleR Functions

The **RevoScaleR** package, included in Machine Learning Server and R Client, provides a framework for quickly writing start-to-finish, scalable R code for data analysis. When you start the R console application on a computer that has Machine Learning Server or R Client, the RevoScaleR function library is loaded automatically.

### Load Data with *rxImport*

The *rxImport* function allows you to import data from fixed or delimited text files, SAS files, SPSS files, or a SQL Server, Teradata, or ODBC connection. There’s no need to have SAS or SPSS installed on your system to import those file types, but you need a locally installed ODBC driver for your database to access data on a local or remote computer.

Let’s start simply by using a delimited text file available in the built-in sample data directory of the **RevoScaleR** package. We store the location of the file in a character string (*inDataFile*), then import the data into an in-memory data set (data frame) called *mortData*:

```
inDataFile <- file.path(rxGetOption("sampleDataDir"), "mortDefaultSmall2000.csv")
mortData <- rxImport(inData = inDataFile)
```

If we anticipate repeating the same analysis on a larger data set later, we could prepare for that by putting placeholders in our code for output files. An output file is an XDF file, native to R Client and Machine Learning Server, persisted on disk and structured to hold modular data. If we included an output file with rxImport, the output object returned from *rxImport* would be a small object representing the .xdf file on disk (an RxXdfData object), rather than an in-memory data frame containing all of the data.

For now, let's continue to work with the data in memory. We can do this by omitting the outFile argument, or by setting the *outFile* parameter to NULL. The following code is equivalent to the importing task of that above:

```
mortOutput <- NULL
mortData <- rxImport(inData = inDataFile, outFile = mortOutput)
```

### Retrieve metadata

There are a number of basic methods we can use to learn about the data set and its variables that work on the return object of *rxImport*, regardless of whether it is a data frame or *RxXdfData* object.

To get the number of rows, cols, and names of the imported data:

```
nrow(mortData)
ncol(mortData)
names(mortData)
```

Output for these commands is as follows:

```
> nrow(mortData)
[1] 10000
> ncol(mortData)
[1] 6
> names(mortData)
[1] "creditScore" "houseAge" "yearsEmploy" "ccDebt" "year"
[6] "default"
```

To print out the first few rows of the data set, you can use *head()*:

```
head(mortData, n = 3)
```

Output:

```
creditScore houseAge yearsEmploy ccDebt year default
1 691 16 9 6725 2000 0
2 691 4 4 5077 2000 0
3 743 18 3 3080 2000 0
```

The rxGetInfo function allows you to quickly get information about your data set and its variables all at one time, including more information about variable types and ranges. Let’s try it on *mortData*, having the first three rows of the data set printed out:

```
rxGetInfo(mortData, getVarInfo = TRUE, numRows=3)
```

Output:

```
Data frame: mortData
Data frame: mortData
Number of observations: 10000
Number of variables: 6
Variable information:
Var 1: creditScore, Type: integer, Low/High: (486, 895)
Var 2: houseAge, Type: integer, Low/High: (0, 40)
Var 3: yearsEmploy, Type: integer, Low/High: (0, 14)
Var 4: ccDebt, Type: integer, Low/High: (0, 12275)
Var 5: year, Type: integer, Low/High: (2000, 2000)
Var 6: default, Type: integer, Low/High: (0, 1)
Data (3 rows starting with row 1):
creditScore houseAge yearsEmploy ccDebt year default
1 691 16 9 6725 2000 0
2 691 4 4 5077 2000 0
3 743 18 3 3080 2000 0
```

### Select and transform with *rxDataStep*

The rxDataStep function provides a framework for the majority of your data manipulation tasks. It allows for row selection (the *rowSelection* argument), variable selection (the *varsToKeep* or *varsToDrop* arguments), and the creation of new variables from existing ones (the *transforms* argument). Here’s an example that does all three with one function call:

```
outFile2 <- NULL
mortDataNew <- rxDataStep(
# Specify the input data set
inData = mortData,
# Put in a placeholder for an output file
outFile = outFile2,
# Specify any variables to keep or drop
varsToDrop = c("year"),
# Specify rows to select
rowSelection = creditScore < 850,
# Specify a list of new variables to create
transforms = list(
catDebt = cut(ccDebt, breaks = c(0, 6500, 13000),
labels = c("Low Debt", "High Debt")),
lowScore = creditScore < 625))
```

Our new data set, *mortDataNew*, will not have the variable year, but adds two new variables: a categorical variable named *catDebt* that uses R’s `cut`

function to break the *ccDebt* variable into two categories, and a logical variable, *lowScore*, that is TRUE for individuals with low credit scores. These *transforms* expressions follow the rule that they must be able to operate on a chunk of data at a time; that is, the computation for a single row of data cannot depend on values in other rows of data.

With the *rowSelection* argument, we have also removed any observations with high credit scores, above or equal to 850. We can use the rxGetVarInfo function to confirm:

```
rxGetVarInfo(mortDataNew)
```

Output:

```
Var 1: creditScore, Type: integer, Low/High: (486, 847)
Var 2: houseAge, Type: integer, Low/High: (0, 40)
Var 3: yearsEmploy, Type: integer, Low/High: (0, 14)
Var 4: ccDebt, Type: integer, Low/High: (0, 12275)
Var 5: default, Type: integer, Low/High: (0, 1)
Var 6: catDebt
2 factor levels: Low Debt High Debt
Var 7: lowScore, Type: logical, Low/High: (0, 1)
```

### Visualize with *rxHistogram*, *rxCube*, and *rxLinePlot*

The rxHistogram function shows us the distribution of any of the variables in our data set. For example, let’s look at credit score:

```
rxHistogram(~creditScore, data = mortDataNew )
```

The rxCube function computes category counts, and can operate on the interaction of categorical variables. Using the *F()* notation to convert a variable into an on-the-fly categorical factor variable (with a level for each integer value), we can compute the counts for each credit score for the two groups who have low and high credit card debt:

```
mortCube <- rxCube(~F(creditScore):catDebt, data = mortDataNew)
```

The *rxLinePlot* function is a convenient way to plot output from *rxCube*. We use the *rxResultsDF* helper function to convert cube output into a data frame convenient for plotting:

```
rxLinePlot(Counts~creditScore|catDebt, data=rxResultsDF(mortCube))
```

### Analyze with *rxLogit*

**RevoScaleR** provides the foundation for a variety of high performance, scalable data analyses. Here we do a logistic regression, but you probably also want to take look at computing summary statistics (*rxSummary*), computing cross-tabs (*rxCrossTabs*), estimating linear models (*rxLinMod*) or generalized linear models (*rxGlm*), and estimating variance-covariance or correlation matrices (*rxCovCor*) that can be used as inputs to other R functions such as principal components analysis and factor analysis. Now, let’s estimate a logistic regression on whether or not an individual defaulted on their loan, using credit card debt and years of employment as independent variables:

```
myLogit <- rxLogit(default~ccDebt+yearsEmploy , data=mortDataNew)
summary(myLogit)
```

We get the following output:

```
Call:
rxLogit(formula = default ~ ccDebt + yearsEmploy, data = mortDataNew)
Logistic Regression Results for: default ~ ccDebt + yearsEmploy
Data: mortDataNew
Dependent variable(s): default
Total independent variables: 3
Number of valid observations: 9982
Number of missing observations: 0
-2*LogLikelihood: 100.6036 (Residual deviance on 9979 degrees of freedom)
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.614e+01 2.074e+00 -7.781 2.22e-16 ***
ccDebt 1.414e-03 2.139e-04 6.610 3.83e-11 ***
yearsEmploy -3.317e-01 1.608e-01 -2.063 0.0391 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Condition number of final variance-covariance matrix: 1.4455
Number of iterations: 9
```

### Scale your analysis

Until now, our exploration has been limited to small data sets in memory. Let’s scale up to a data set with a million rows rather than just 10000. These larger text data files are available online. Windows users should download the zip version, mortDefault.zip, and Linux users mortDefault.tar.gz.

After downloading and unpacking the data, set your path to the correct location in the code below. It is more efficient to store the imported data on disk, so we also specify the locations for our imported and transformed data sets:

```
# bigDataDir <- "C:/MicrosoftR/Data" # Specify the location
inDataFile <- file.path(bigDataDir, "mortDefault",
"mortDefault2000.csv")
outFile <- "myMortData.xdf"
outFile2 <- "myMortData2.xdf"
```

That’s it! Now you can reuse all of the importing, data step, plotting, and analysis code preceding on the larger data set.

```
# Import data
mortData <- rxImport(inData = inDataFile, outFile = outFile)
```

Output:

```
Rows Read: 500000, Total Rows Processed: 500000, Total Chunk Time: 1.043 seconds
Rows Read: 500000, Total Rows Processed: 1000000, Total Chunk Time: 1.001 seconds
```

Because we have specified an output file when importing the data, the returned *mortData* object is a small object in memory representing the .xdf data file, rather than a full data frame containing all of the data in memory. It can be used in **RevoScaleR** analysis functions in the same way as data frames.

```
# Some quick information about my data
rxGetInfo(mortData, getVarInfo = TRUE, numRows=5)
```

Output:

```
File name: C:\\MicrosoftR\\Data\\myMortData.xdf
Number of observations: 1e+06
Number of variables: 6
Number of blocks: 2
Compression type: zlib
Variable information:
Var 1: creditScore, Type: integer, Low/High: (459, 942)
Var 2: houseAge, Type: integer, Low/High: (0, 40)
Var 3: yearsEmploy, Type: integer, Low/High: (0, 15)
Var 4: ccDebt, Type: integer, Low/High: (0, 14639)
Var 5: year, Type: integer, Low/High: (2000, 2000)
Var 6: default, Type: integer, Low/High: (0, 1)
Data (5 rows starting with row 1):
creditScore houseAge yearsEmploy ccDebt year default
1 615 10 5 2818 2000 0
2 780 34 5 3575 2000 0
3 735 12 1 3184 2000 0
4 713 15 5 6236 2000 0
5 689 10 5 6817 2000 0
```

The data step:

```
mortDataNew <- rxDataStep(
# Specify the input data set
inData = mortData,
# Put in a placeholder for an output file
outFile = outFile2,
# Specify any variables to keep or drop
varsToDrop = c("year"),
# Specify rows to select
rowSelection = creditScore < 850,
# Specify a list of new variables to create
transforms = list(
catDebt = cut(ccDebt, breaks = c(0, 6500, 13000),
labels = c("Low Debt", "High Debt")),
lowScore = creditScore < 625))
```

Output:

```
Rows Read: 500000, Total Rows Processed: 500000, Total Chunk Time: 0.673 seconds
Rows Read: 500000, Total Rows Processed: 1000000, Total Chunk Time: 0.448 seconds
>
```

Looking at the data:

```
# Looking at the data
rxHistogram(~creditScore, data = mortDataNew )
Rows Read: 499294, Total Rows Processed: 499294, Total Chunk Time: 0.329 seconds
Rows Read: 499293, Total Rows Processed: 998587, Total Chunk Time: 0.335 seconds
Computation time: 0.678 seconds.
```

```
myCube = rxCube(~F(creditScore):catDebt, data = mortDataNew)
rxLinePlot(Counts~creditScore|catDebt, data=rxResultsDF(myCube))
```

```
# Compute a logistic regression
myLogit <- rxLogit(default~ccDebt+yearsEmploy , data=mortDataNew)
summary(myLogit)
```

Output:

```
Call:
rxLogit(formula = default ~ ccDebt + yearsEmploy, data = mortDataNew)
Logistic Regression Results for: default ~ ccDebt + yearsEmploy
File name:
C:\Users\RUser\myMortData2.xdf
Dependent variable(s): default
Total independent variables: 3
Number of valid observations: 998587
Number of missing observations: 0
-2*LogLikelihood: 8837.7644 (Residual deviance on 998584 degrees of freedom)
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.725e+01 2.330e-01 -74.04 2.22e-16 ***
ccDebt 1.509e-03 2.327e-05 64.85 2.22e-16 ***
yearsEmploy -3.054e-01 1.713e-02 -17.83 2.22e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Condition number of final variance-covariance matrix: 1.3005
Number of iterations: 9
```

## Next Steps

Continue on to these tutorials to work with larger data set using the RevoScaleR functions: