rxHadoopCommand: Execute Hadoop Commands

Article
07/12/2022

Description

Execute arbitrary Hadoop commands and perform standard file operations in Hadoop.

Usage

  rxHadoopCommand(cmd, computeContext, sshUsername=NULL, 
                  sshHostname=NULL, 
                  sshSwitches=NULL,
                  sshProfileScript=NULL, intern=FALSE)

  rxHadoopCopyFromLocal(source, dest, ...) 

  rxHadoopCopyFromClient(source, nativeTarget="/tmp", hdfsDest, 
                         computeContext, sshUsername=NULL, 
                         sshHostname=NULL, sshSwitches=NULL, sshProfileScript=NULL)

  rxHadoopCopyToLocal(source, dest, ...) 

  rxHadoopFileExists(path)

  rxHadoopListFiles(path="", recursive=FALSE, print, computeContext = rxGetComputeContext(), ...)

  rxHadoopMakeDir(path, ...)

  rxHadoopMove(source, dest,  ...)

  rxHadoopCopy(source, dest,  ...)

  rxHadoopRemove(path,  skipTrash=FALSE, ...)

  rxHadoopRemoveDir(path,  skipTrash=FALSE, ...)

  rxHadoopVersion()

Arguments

`cmd`

A character string containing a valid Hadoop command, that is, the cmd portion of hadoop cmd. Embedded quotes are not permitted.

`computeContext`

Run against this compute context. Default to the current compute context as returned by rxGetComputeContext .

`sshUsername`

character string specifying the username for making an ssh connection to the Hadoop cluster.

`sshHostname`

character string specifying the hostname or IP address of the Hadoop cluster node or edge node that the client will log into for launching Hadoop commands.

`sshSwitches`

character string specifying any switches needed for making an ssh connection to the Hadoop cluster.

`sshProfileScript`

Optional character string specifying the absolute path to a profile script that will exist on the sshHostname host. This is used when the target ssh host does not automatically read in a .bash_profile, .profile or other shell environment configuration file for the definition of requisite variables.

`intern`

logical (not NA) specifying whether to capture the output of a Hadoop command as an R character vector in a local compute context. (When using the RxHadoopMR compute context, any output is always returned as an R character vector.)

`source`

character vector specifying file(s) to be copied or moved.

`dest`

character string specifying the destination of a copy or move. If source includes more than one file, dest must be a directory.

`nativeTarget`

character string specifying a directory in the Hadoop cluster's native file system, to be used as an intermediate location for file(s) copied from a client machine.

`hdfsDest`

character string specifying a directory in the Hadoop Distributed File System.

`path`

character vector specifying location of one or more files or directories.

`print`

Deprecation Warning: the print argument in rxHadoopListFiles is now deprecated and is going to be removed in the next release. If FALSE, rxHadoopListFiles will return a character vector of paths; by default it prints paths to the console.

`recursive`

logical flag. If TRUE, directory listings are recursive.

`skipTrash`

logical flag. If TRUE, removal via rxHadoopRemove and rxHadoopRemoveDir bypasses the trash folder, if one has been set up.

`...`

additional arguments to be passed directly to the rxHadoopCommand function.

Details

rxHadoopCommand allows you to run basic Hadoop commands. rxCopyFromClient allows a file to be copied from a remote client to the Hadoop Distributed File System on the Hadoop cluster. rxHadoopVersion calls the Hadoop version command and extracts and returns the version number only. The remaining functions are wrappers for various Hadoop file system commands:

* rxHadoopCopyFromLocal wraps the Hadoop fs -copyFromLocal command.

* rxHadoopCopyToLocal wraps the Hadoop fs -copyToLocal command.

* rxHadoopListFiles wraps the Hadoop fs -ls or fs -lsr command.

* rxHadoopRemove wraps the Hadoop fs -rm command.

* rxHadoopCopy wraps the Hadoop fs -cp command.

* rxHadoopMove wraps the Hadoop fs -mv command.

* rxHadoopMakeDir wraps the Hadoop fs -mkdir command.

* rxHadoopRemoveDir wraps the Hadoop fs -rm -r command.

Value

These functions are executed for their side effects and typically return NULL invisibly.

Author(s)

Microsoft Corporation Microsoft Technical Support

Examples


 ## Not run:

rxHadoopCommand("version") # should return version information
rxHadoopMakeDir("/user/RevoShare/newUser")
rxHadoopCopyFromLocal("/tmp/foo.txt", "/user/RevoShare/newUser")
rxHadoopRemoveDir("/user/RevoShare/newUser")

## End(Not run)