Creates an HDInsight cluster running Apache Spark 1.4.1.
Creates HDInsight Linux cluster and run custom script action to install Apache Spark 1.4.1
This template creates an HDInsight Linux based cluster and then updates the cluster headnodes with the Apache Spark 1.4.1 binaries(including YARN support).
Additionally, it sets specific environment variables ($SPARK_HOME, updates $PATH) to allow for easy access to the Spark client binaries.
Please be sure to utlize appropriate Spark core, memory, and executor settings based on your chosen deployment size.
To launch Spark interactivly, please SSH into the cluster (clustername-ssh.azurehdinsight.net) and execute the following commands:
Sudo -i
$SPARK_HOME/bin/spark-shell
You should see output similar to the following:
15/10/01 15:21:34 INFO util.Utils: Successfully started service 'HTTP class server' on port 47985.
Welcome to version 1.4.1
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_79)
Type in expressions to have them evaluated.
Type :help for more information.
15/10/01 15:21:40 INFO spark.SparkContext: Running Spark version 1.4.1
...Output Snipped...
scala>
To deploy alternative versions of Apache Spark as published by the HDInsight team, please review follow this link: Click Me
Tags: Microsoft.Network/virtualNetworks, Microsoft.Storage/storageAccounts, Microsoft.HDInsight/clusters