New-AzureHDInsightStreamingMapReduceJobDefinition

[This topic is pre-release documentation and is subject to change in future releases. Blank topics are included as placeholders.]

New-AzureHDInsightStreamingMapReduceJobDefinition

Defines a new streaming Map Reduce job.

Syntax

Parameter Set: Default
New-AzureHDInsightStreamingMapReduceJobDefinition [-Arguments <String[]> ] [-CmdEnv <String[]> ] [-Combiner <String> ] [-Defines <Hashtable> ] [-Files <String[]> ] [-InputPath <String> ] [-JobName <String> ] [-Mapper <String> ] [-OutputPath <String> ] [-Reducer <String> ] [-StatusFolder <String> ] [ <CommonParameters>]

Detailed Description

Defines new job definition object that represents parameters of the Hadoop streaming job.

This topic describes the cmdlet in the 0.8.1 version of the Microsoft Azure PowerShell module. To find out the version of the module you're using, from the Azure PowerShell console, type (get-module azure).version.

Parameters

-Arguments<String[]>

Arguments of the Hadoop job. The arguments will be passed as command line arguments to each task.

Aliases	Args
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-CmdEnv<String[]>

Command line environmental variables that should be set during job execution on data nodes.

Aliases	none
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-Combiner<String>

Combiner executable name.

Aliases	none
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-Defines<Hashtable>

Hadoop configuration values to be set during the job execution.

Aliases	Params
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-Files<String[]>

The collection of files associated with the MapReduce job.

Aliases	none
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-InputPath<String>

WASB path representing location of input files.

Aliases	Input
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-JobName<String>

The name of the MapReduce job. Specifying the name is optional.

Aliases	Name
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-Mapper<String>

Mapper executable name.

Aliases	none
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-OutputPath<String>

WASB path representing location for the job output.

Aliases	Output
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-Reducer<String>

Reducer executable name.

Aliases	none
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

-StatusFolder<String>

Contains the standard and error outputs for the job, including its exit code and task logs.

Aliases	none
Required?	false
Position?	named
Default Value	none
Accept Pipeline Input?	false
Accept Wildcard Characters?	false

<CommonParameters>

This cmdlet supports the common parameters: -Verbose, -Debug, -ErrorAction, -ErrorVariable, -OutBuffer, and -OutVariable. For more information, see  about_CommonParameters (https://go.microsoft.com/fwlink/p/?LinkID=113216).

Inputs

The input type is the type of the objects that you can pipe to the cmdlet.

Outputs

The output type is the type of the objects that the cmdlet emits.

Examples

PS C:\> 

Define a new streaming Map Reduce job.

Create a MapReduce job definition for the streaming job.

PS C:\> $subscriptionName = "<SubscriptionName>"   # Azure subscription name
$clusterName = "<ClusterName>"             # HDInsight cluster name

# Create a MapReduce job definition for the streaming job.
$streamingWC = New-AzureHDInsightStreamingMapReduceJobDefinition -Files "/example/apps/wc.exe", "/example/apps/cat.exe" -InputPath "/example/data/gutenberg/davinci.txt" -OutputPath "/example/data/StreamingOutput/wc.txt" -Mapper "cat.exe" -Reducer "wc.exe"