Edit

Share via

rxSparkListData, rxSparkRemoveData {RevoScaleR}: Manage Cached Data in Spark

  • Article

Description

Use these functions to manage the objects cached in the Spark memory system. These functions are only applicable when using RxSpark compute context.

Usage

  rxSparkListData(showDescription=TRUE,
      computeContext = rxGetOption("computeContext"))

  rxSparkRemoveData(list,
      computeContext = rxGetOption("computeContext"))

Arguments

list

list of cached objects need to be deleted.

showDescription

logical indicating whether or not to print out the detail to console.

computeContext

RxSpark compute context object.

Value

List of all objects cached in Spark memory system for rxSparkListData.

No return values for rxSparkRemoveData.

Examples


 ## Not run:

cc <- rxSparkConnect()

colInfo = list( DayOfWeek = list(type = "factor"))
df <- RxParquetData(file = "/tmp/AirlineDemoSmall.parquet", colInfo = colInfo)

### example for rxSparkListData

## No object in list, because no algorithm has been run 
rxSparkListData()

rxLogit((ArrDelay>0) ~ CRSDepTime + DayOfWeek, data = df)

## After the first run, a Spark data object is added into the list
rxSparkListData()

### example for rxSparkRemoveData

## remove an object
rxSparkRemoveData(df)

## remove all cached objs
rxSparkRemoveData(rxSparkListData())

rxSparkDisconnect(cc)
## End(Not run)