getSplits(conf, numSplits) => max(minSize, min(goalSize, blockSize))
   minSize = conf: mapreduce.input.fileinputformat.split.minsize, defaults to 0
   goalSize = totalSize / numSplits

2. How Spark calculates splits

2.1 Unstructured APIs

val text = spark.sparkContext.textFile("/HdiSamples/HdiSamples/SensorSampleData/hvac/HVAC.csv")
text.getNumPartitions

2.2 Structured APIs

spltiSize = Math.min(defaultMaxSplitBytes, Math.max(openCostInBytes, bytesPerCore))
   defaultMaxSplitBytes = conf: spark.sql.files.maxPartitionBytes, defaults to 128MB
   openCostInBytes = conf: spark.sql.files.openCostInBytes, default to 4MB
   bytesPerCore = totalBytes / defaultParallelism
   defaultParallelism = conf: spark.default.parallelism, defaults to (conf: spark.executor.cores) * (conf: spark.executor.instances)

val hvac = spark.read.format("csv").load("/HdiSamples/HdiSamples/SensorSampleData/hvac/HVAC.csv")
hvac.rdd.getNumPartitions()

3. How to manipulate number of splits for input files in HDInsight

1. What is Spark Streaming?

Spark Streaming is the real time data analytics framework provided by Spark. It is a micro-batch system. In a nutshell, you can think it as an additional layer on top of Spark, which reads data from external data source, periodically pushes batch of data to Spark for real time transformation.

2. How does Spark Streaming work?

The key concept in Spark Streaming is a receiver. A receiver is the interface to a specific external system. Before we move on, you need to understand a few basic concepts of Spark:

Application: A user application submitted to Spark. It consists of a driver and many executors.

Driver: A program that manages all executors for a Spark application, declares data transformation and actions on RDDs.

Executor: A process running on the worker node for a Spark application, which stays up for the duration of the application.

Job: A parallel computing consisting of multiple tasks that are spawned in response to a spark action (e.g. save, collect, etc.).

Stage: Smaller set of tasks in a Job that depend on one another.

Task: A unit of work that is sent to one executor. Note that one executor can execute multiple tasks at the same time.

When you submit a spark streaming application, a driver is started, which launches many executors on the worker nodes through sparkmaster. The first job submitted to the application is the receiver job (usually with Job ID 1), which lasts the duration of the streaming application. Each task in the receiver job is one instance of the receiver.

This is how data enters Spark system through a receiver:

Data Source -> Receiver -> Block Generator -> Block Manager

Receiver reads data from the Data Source, and hands off the received data to a component called Block Generator. The Block Generator keeps the data in memory. Every block interval (spark.streaming.blockInterval, defaults to 200ms), it sends the block to Spark Block Manager as a new partition of a RDD. Block Manager is a component running on every Spark node that manages data blocks both locally and remotely. Every batch interval (specified when creating StreamingContext), the RDD is closed and a Spark job (data processing job) is submitted to process that newly generated RDD.

You can see the receiver jobs and all the subsequent data processing jobs in Spark UI. As can be seen, the Job 1 is the receiver job, which has only 1 task (1 instance of receiver). Job 2,3,4,5,6 are data processing jobs, about 10 seconds apart. This is because the batch interval is set to 10 seconds.

3. How to use Spark Streaming with Azure Event Hubs in HDInsight?

A streaming solution consists of not only the Spark Streaming framework, but also the external data source and the persistent store for processed data. For HDInsight, the recommended data source is Azure Event Hubs. Event Hubs is a pub/sub message broker service in the cloud, equivalent to Apache Kafka. For persistent data store, you can use HDFS (in HDInsight this is WASB, basically Azure Storage), HBase, or SQL database. You can find plenty of online examples/tutorials on this topic. Therefore our focus here is Event Hubs.

There is a fantastic tutorial from HDInsight documentation that shows you how to write an end to end Spark Streaming application using Event Hubs:

https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-csharp-apache-zeppelin-eventhub-streaming/

This is how you create a DStream from a Event Hubs:

This tutorial uses the EventHubsReceiver to receive data from Event Hubs. Event Hubs supports AMQP protocol, which is how the EventHubsReceiver talks to the Event Hubs server. The EventHubsReceiver uses EventHubsClient, which in turn uses the Apache Qpid project as the AMQP library. Note that the Storm EventHubSpout also uses the same EventHubsClient library.

There is a common mistake in running this tutorial. If you don't assign enough cores to the spark streaming application (or Zeppelin if you use Zeppelin notebook), the application will appear stuck and not produce any results. This is because by design each EventHubsReceiver instance only handles one Event Hubs partition. Each receiver instance requires one CPU core to run, and you need to leave some CPU cores to process the received data. So if you set partition count to N in Event Hubs parameters, you need to make sure you assign 2xN CPU cores to the streaming application.

What this means for HDInsight Spark cluster is that for a default configuration, a 4 node cluster can handle at most 8 Event Hubs partitions (each node has 4 cores, and 16 cores can only handle at most 8 partitions).

The "eventhubs.checkpoint.dir" specifies the checkpoint folder to store the message offset. This parameter needs to be an absolute path. You'll see exceptions when you provides a relative path like "mycheckpoint". You need to use "/mycheckpoint" instead. 

The EventHubsReceiver project is submitted to spark-packages repo here:

https://spark-packages.org/package/hdinsight/spark-eventhubs

Spark-packages is a community maintained repository for all third party Spark integration code. The actual source code is shared on github:

https://github.com/hdinsight/spark-eventhubs

The EventHubsClient source code is shared on github:

https://github.com/hdinsight/eventhubs-client

4. Achieving at-least-once semantics

In general there are two types of receivers, the regular receiver and the reliable receiver. Reliable receiver aims at providing at-least-once semantics, meaning each message is guaranteed to be processed at least once. Regular (or unreliable) receiver do not guarantee at-least once semantics.  But with regular receiver you can achieve higher throughput. So if you don't need at-least-once semantics then use the regular receiver.

To the best of my knowledge, the only way to achieve exactly-once semantics in Spark Streaming is to use Kafka as the data source, and use direct stream (DirectKafkaInputDStream.scala) without a receiver.

In HDInsight, we provide both EventHubsReceiver and ReliableEventHubsReceiver so you can choose what you need. The difference in implementation resides in when to do checkpoint. We checkpoint the message offsets in HDFS (in HDInsight this is WASB). Since Block Generator keeps received data in memory for a block interval, it has the risk of losing data when the receiver crashes. ReliableEventHubsReceiver only checkpoints after a block is committed to Block Manager, therefore avoiding data loss. ReliableEventHubsReceiver also needs to work with a Spark Streaming feature called WAL (Write Ahead Log). With WAL enabled, the Block Generator writes a block of data to WAL as well as the Block Manager, avoiding data loss in the later stages of the pipeline. If the driver crashed and restarted, it can reconstruct the RDD partitions from the WAL.

So how do you choose to use the ReliableEventHubsRecriver? You don't need to modify any of the application code (e.g. call EventHubsUtils.createUnionStream() to create a Dstream). You can simply enable WAL (set configuration spark.streaming.receiver.writeAheadLog.enable to true) to enable ReliableEventHubsReceiver.

5. High Availability (HA)

The tutorial we mentioned earlier uses Zeppelin notebook to write a Spark Streaming application. This is quick and convenient, however, it is not production ready. Zeppelin runs the driver on the head node, which will fail in the case of head node failover. Production streaming application needs to run 24x7, regardless of arbitrary node crashes in the cluster.

To achieve High Availability in HDInsight Spark cluster, you need to 1) enable driver checkpointing and 2) RDP to the cluster and submit the Spark Streaming application in cluster mode with supervise:

spark-submit.cmd --deploy-mode cluster --supervise …

This is how HA works in different scenarios:

1) Head node failover

First, we need sparkmaster HA. This means the active sparkmaster saves metadata to Zookeeper, and when head node failover happens, the standby sparkmaster reads metadata from Zookeeper and continue to operate. We have enabled sparkmaster HA by default in HDInsight, so there is no action needed by the user.

Second, we need to submit the application in cluster mode. By default, Spark Streaming driver runs on the head node. During head node failover, the driver is lost and the application is killed. Spark provides a cluster mode in which the driver is shipped to a worker node to run.

2) Worker node failure

If a worker node dies, the tasks assigned to the executors on that worker will be re-assigned somewhere else. There is no problem here. However, if the worker node that runs the driver dies, we need to relaunch the driver. Spark provides a supervise feature in cluster mode that sparkmaster periodically pings the driver and restarts the driver on a different node if it crashed.

Furthermore, the restarted driver should be able to pick up from previous state of the old driver. For this to work we need to enable driver checkpointing. With checkpointing the driver's metadata is stored in HDFS periodically. During startup, the driver tries to read data from the checkpoint folder, and creates StreamingContext from the checkpointed data if found.

For an example of the Spark Streaming application with HA, take a look at this github project:

https://github.com/hdinsight/hdinsight-spark-examples

6. Storm vs. Spark Streaming

Since the inception of Spark Streaming, people are trying to compare it with Apache Storm - an existing successful distributed streaming platform. If you search online, you'll find many articles highlighting the difference as Storm being one-message-at-a-time vs. Spark Streaming's micro-batching implementation. This is only partially true. Regular Storm topology processes one message at a time, which results in low latency but also low throughput. Storm Trident, a software layer on top of Storm, can also do micro-batching and a bunch of high level data transformations.

These are my observations after working on both technologies for a while:

1) Storm is more flexible. With Storm, you can specify the exact processing topology and parallelism levels. With Trident, you can easily achieve exactly-once semantics, which is very hard to achieve in Spark Streaming (as of 1.3).

2) Storm is more reliable. We have been running Storm applications for months and they keep running as expected. As of Spark 1.3, we've observed occasional crashes for Spark Streaming.

3) Spark Streaming is easier to use. Spark Streaming provides higher level abstractions and APIs which make it easier to write business logic. In particular, Spark Streaming provides windowing aggregates out of box, which is not available in Storm.

4) Spark Streaming has an ecosystem. The biggest advantage of Spark Streaming is that it is part of Spark ecosystem. You can use the rich Spark core data transformation functions once the data is pushed into Spark. Another benefit is that if you are interested in implementing an Lambda architecture, your real time and batch data processing can share the same code base.

I also want to stress that Spark Streaming is evolving fast. it is still undergoing heavy development. New features are added constantly. E.g. write ahead log is introduced in 1.2 and direct Kafka stream is introduced in 1.3. So it's reasonable to believe that Spark Streaming will become more feature rich and reliable over time.

So when to use what? My recommendation is this: if you are into Spark and you plan to move your data platform to Spark someday in the future, choose Spark Streaming; otherwise stay with Storm. If you need exactly-once semantics in your application, I would recommend you to use Storm Trident.

Note that EventHubSpout won't be available until Storm 0.11.0. Before that we keep the latest and greatest jar here:
https://github.com/hdinsight/hdinsight-storm-examples/tree/master/lib/eventhubs

Performance Basics

One frequently asked question is about the performance capability of the HDInsight Storm + EventHubs solution. Note that the precise performance depends on the specific Storm topology and the business logic of the application. But in most cases, you'll find EventHubs is the bottleneck. Note that you can always increase the cluster size if Storm is the bottleneck.

The throughput of EventHubs is governed by the concept of Throughput Unit (TU), according to (https://azure.microsoft.com/en-us/pricing/details/event-hubs/). 1 TU means 1MB/s or 1000 message/s ingress, 2MB/s or 2000 message/s egress (The message/s limit for egress is not enforced yet as of 5/15/2015). When Storm pulls data from EventHubs, the receive throughput is governed by the EventHubs egress limit. By default your EventHubs has 1 TU and you can change the throughput unit for your EventHubs namespace in Azure portal to up to 20 TU. Note that there are also protocol overhead in the order of 50 bytes per message. So if your message is 100 byte, with 20 TU, you can achieve 350-400K message/s throughput when pulling data into Storm topology today. When message/s egress limit is enforced at EventHubs server side, you can only achieve 40K message/s. In real time scenario, the throughput also depends on the EventHubs ingress limit because the data has to be sent to EventHubs before it can be consumed by Storm. In that case, the maximum throughput you can get is 20MB/s or 20K message/s.

If you need more throughput, you can call EventHubs support to increase the Throughput Unit to up to 1000. This means that you can achieve up to 1GB/s, or 1M message/s throughput using EventHubs.

EventHubs also support the concept of partitions. EventHubs only guarantee sequence delivery of messages within each partition. Messages from different partitions can be processed in parallel. EventHubs also has 5 TU limit on each partition.

For better performance the number of tasks for EventHubSpout should always match the number of partitions. For best performance, we recommend running 1 EventHubSpout task per node.

In some situations, you may find that the Storm cluster is the performance bottleneck. You can verify this by checking the CPU utilization of the worker node while the topology is running. If it is too high, you need to increase your cluster size. But it is rarely useful to increase the cluster size beyond the number of partitions.

Performance Tuning Tips

In general, we recommend setting the cluster size to be the same of the number of partitions in EventHubs so that each node is dedicated to 1 partition. Having a larger cluster size than the number of partitions means you can run data processing bolts in dedicated nodes. The downside of this situation is that Storm has to shuffle tuples across nodes when sending tuples from spout to bolt, introducing latency and overhead for serialization and de-serialization. If you are not doing much business logic in your topology, you might be safe to reduce the cluster to half or 1/4 of the number of partitions.

By default, Storm processes 1 message at a time (unless you are looking at Trident which does micro-batching), which should result in pretty low latency. If you want to further reduce latency, you need to optimize processing code in bolts and reduce the number of bolts in your topology. You may also consider choosing A8 or A9 node types when creating HDInsight Storm clusters because they are optimized for network speed.

The same tricks to reduce latency also work to increase throughput in the case of high CPU utilization in your cluster. Furthermore, if you are OK with losing tuples occasionally, you may consider turning off ack. This will greatly reduce CPU utilization. To do this, you can set topology.acker.executors to 0 during cluster creation time or in code do something like this: config.setNumAckers(0).

Another trick to improve performance is to use localOrShuffleGrouping() instead of shuffleGrouping() to connect EventHubSpout to the downstream bolts. What this does is to tell Storm to send a tuple to the bolt instance in the same worker process, eliminating the cost of inter-worker communication. For more info on Storm internal messaging, see:
https://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/

Benchmark Results

We wrote a simple benchmark to test the performance of HDInsight Storm with EventHubs. You can find the source code here:
https://github.com/hdinsight/hdinsight-storm-examples/tree/master/EventCountExample

Our topology is a simple 3 layer topology that counts the number of messages received from EventHubs.
EventHubSpout    --(localOrShuffleGrouping)->    PartialCountBolt    --(GlobalGrouping)->    GlobalCountBolt

We use an EventHubs entity of 32 partitions, scaled to 20 TU. Message size is 100 bytes. We first pre-populate data in EventHubs using an event generator Storm topology, then stop that topology and start the EventCount topology. Note that using a Storm topology to generate events in high speed is important: if you pre-load data using a single threaded program, you are not going to achieve high throughput in EventCount topology. This is mainly due to the way how EventHubs arrange and store data as they are sent to EventHubs in different speed. Based on our previous discussion, the theoretical limit of throughput is 400K message/s. And once the egress message/s limit is enforced in EventHubs, this will become 40K message/s.

For the EventCount topology, we set the task number of the spout equal to the number of partitions, the number of workers in the topology equals the number of partitions. These are the results:

1) For an 8 node HDInsight Storm cluster, we can achieve 200K message/s. In this case each node is responsible for processing 4 partitions. We observed that the CPU utilization on worker node of Storm cluster is at 100% therefore the Storm cluster is the bottleneck.

2) For a 32 node cluster, we can achieve about 350K message/s, pretty close to the theoretical limit. The CPU utilization is at about 70%.

We also tested with a privately created 128 partition EventHubs with 100 TU. In this test we achieved 1M message/s with 100 node (default node size) HDInsight Storm cluster.

To submit a Storm topology to an HDInsight cluster, a user can RDP to the headnode of the cluster and run storm command. This is not always convenient. It is actually possible to submit a Storm topology from outside of an HDInsight cluster. The idea is to create an HDInsight Storm cluster with a configured Virtual Networks (VNet), and submit Storm topology from a machine that is connected to the VNet.

There are many types of VNet so with VNet support we can actually submit topology from Azure VM, other Azure services, on-premises infrastructure or developer boxes.

To show case the idea, I’ll show you how to use an Azure VM to submit a Storm topology via VNet.

2. Step-by-step Instructions

1) Create a Cloud-Only VNet in Azure portal. You can use the “QUICK CREATE” button.

2) Create a VM using the created VNet, this will be the VM from which we submit Storm topology. You need to use “FROM GALLERY” button and on page 4 you need to choose the VNet that we just created.

3) Create a Storm cluster using the created VNet. Note you need to use “custom create” and specify the VNet name in Region/Virtual Network section.

4) Find out the FQDN of the active head node of HDInsight Storm cluster using REST API.

This is a Powershell script to help you get the FQDN of the active head node:

function Get-ActiveFQDN(
    [String]
    [Parameter( Position=0, Mandatory=$true )]
    $ClusterDnsName,
    [String]
    [Parameter( Position=1, Mandatory=$true )]
    $Username,
    [String]
    [Parameter( Position=2, Mandatory=$true )]
    $Password)
{
    $DnsSuffix = ".azurehdinsight.net"
    $ClusterFQDN = $ClusterDnsName + $DnsSuffix
    $webclient = new-object System.Net.WebClient
    $webclient.Credentials = new-object System.Net.NetworkCredential($Username, $Password)
    $Url = "https://" + $ClusterFQDN + "/clusteravailability/status"
    $Response = $webclient.DownloadString($Url)
    $JsonObject = $Response | ConvertFrom-Json
    Write-host $JsonObject.LeaderDnsName
}

  
This script will print out something like this:

headnode1.<clusterdnsname>.b1.internal.cloudapp.net

5) RDP to the Azure VM we just created. Copy Storm bits from HDInsight head node (c:\apps\dist\storm-xxx or %STORM_HOME%) to Azure VM (let’s say we copy to c:\storm folder); Install Java 1.7 runtime (either Oracle or OpenJDK is fine).

6) On the Azure VM, make sure the following configurations (environment variable and Storm configurations) are correctly set:

Environment variable:

    JAVA_HOME = "<your java installation path>"

storm.yaml (c:\storm\conf\storm.yaml):

    nimbus.host: headnode1.<clusterdnsname>.b1.internal.cloudapp.net

7) On the Azure VM, submit a Storm topology using storm.cmd command line like this:

C:\storm\bin>storm jar ..\contrib\storm-starter\storm-starter-<version>-jar-with-dependencies.jar storm.starter.WordCountTopology wordcountSampleTopology

Then on the Azure VM you can manage topology status using Storm UI web page (start IE and enter the address like this):

https://headnode1.<clusterdnsname>.b1.internal.cloudapp.net:8772/

This is how easy it is. Enjoy storming!

There are several related memory settings for jobs running in HDInsight cluster which most customers need to pay close attention to. When not correctly set, it will cause obscure failures in Hive/Pig/Mapreduce/Tez jobs.

Note that HDInsight service provides default memory settings, however, the defaults may be subject to change as we tune the service for various workloads or as we move to different VM types and hardware. It is advised that customers explicit set these settings once they know the right settings for their specific jobs. There are two ways to apply these settings: 1) provide it when creating HDInsight cluster using SDK. or 2) set it on a per job basis. E.g. use "-Define" option in Powershell SDK, or "set property=value" in a hive script, or set it in mapreduce code directly. Note that the Yarn level settings can only be changed during cluster creation.

1. Yarn memory settings:

HDInsight 3.x service deploys Hadoop 2.x clusters. Hadoop 2.x (Yarn) introduced the concept of containers. Unlike Hadoop 1.x where each node is assigned a fixed number of "slots", in Hadoop 2.x each Yarn task (mapper, reducer or Tez task) is assigned a container which has an memory limit. This affects 1) how many containers can run in parallel in any given node; and 2) Yarn Node Manager will monitor the memory usage of the task and kill a container when the memory usage exceeds that limit.

Yarn defines how much memory is available for allocation and what is the minimum and maximum container size:

yarn.nodemanager.resource.memory-mb = 5376

yarn.scheduler.minimum-allocation-mb = 768

yarn.scheduler.maximum-allocation-mb = 5376

Note that you can only ask for container size of multiples of minimum-allocation-mb. In the above example settings, if you ask for a container of size 1024MB, you will actually get a container of 1536MB.

2. Mapreduce memory settings:

These are the default Yarn container memory settings in HDInsight for the mapper, reducer and AM(Application Master):

mapreduce.map.memory.mb = 768

mapreduce.reduce.memory.mb = 1536

yarn.app.mapreduce.am.resource.mb = 1536

This is a typical error message in your job attempt log, if these limits are exceeded:

Container[pid=container_1406552545451_0009_01_000002,containerID=container_1406552545451_0009_01_000002] is running beyond physical memory limits. Current usage: 519.1 MB of 512 MB physical memory used; 770.1 MB of 1.0 GB virtual memory used. Killing container. Dump of the process-tree for container_1406552545451_0009_01_000002 : |- PID CPU_TIME(MILLIS) VMEM(BYTES) WORKING_SET(BYTES) |- 4912 15 667648 2502656 |-4712 9546 232022016 97697792 |- 6000 78 1867776 2605056 |- 4780 50234 571166720 438947840 |- 4516 31 1810432 2588672 Container killed on request. Exit code is 137

Aside from the memory monitoring, each Java process has its own heap space settings. These are the default settings for the mapper, reducer and AM:

mapreduce.map.java.opts = "-Xmx512m"

mapreduce.reduce.java.opts = "-Xmx1024m"

yarn.app.mapreduce.am.command-opts = "-Xmx1024m"

This is a typical error message in your job attempt log if these limits are exceeded:

Error: java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded

or

Error: java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space

The Yarn based and Java based settings are related. The Java heap settings should be smaller than the Yarn container memory limit because we need reserve memory for Java code. The best practice is to reserve 20% memory for code. So if settings are correct, theoretically pure Java-based Hadoop tasks should never get killed by Yarn Node Manager unless there is a bug in your Java code somewhere. If the Yarn based error is seen, the cure is to either increase the Yarn container memory or decrease Java heap space. If the Java heap error is seen, you can either increase both memory settings (in which case you'll get fewer tasks running in parallel), or bring down the memory usage if possible.

For streaming jobs, the developer needs to make sure the streaming program don't exceeds Yarn container memory settings. This might require trial run and iterative tuning.

Eventually the memory usage depends on the job you are doing. For example, if you are doing hive query "CREATE TABLE AS SELECT" on huge tables your mapper may demand more memory than 512MB. To increase memory settings for the mapper, you can overwrite these settings in Mapreduce configuration:

mapreduce.map.memory.mb = 1536

mapreduce.map.java.opts = "-Xmx1024m"

There is another memory setting that depends on the container memory size:

mapreduce.task.io.sort.mb = 307

This is the max memory a mapreduce task can use to sort data in buffer during the shuffle stage. This value should be 1/3 to 1/2 of the task heap size.

This is an illustration of various memory settings for Mapreduce to help you visualize the relative size of them:

3. Tez memory settings

These are the default Tez memory settings for Tez AM:

tez.am.resource.memory.mb = 1536

tez.am.java.opts = 1024

For tez container, it uses mapper's memory settings:

mapreduce.map.memory.mb = 768

mapreduce.map.java.opts = "-Xmx512m"

This is an illustration of various memory settings for Tez:

4. Hive memory settings for Tez:

When using Tez for Hive, you can specify Tez task's memory settings, these are default values:

hive.tez.container.size = -1

hive.tez.java.opts = ""

The default settings mean that the actual Tez task will use the mapper's memory setting:

hive.tez.container.size = mapreduce.map.memory.mb

hive.tez.java.opts = mapreduce.map.java.opts

If you want to have Tez use different memory setting from the Mapreduce framework, you can modify these settings for Hive configuration.

There is another important memory setting in hive that depends on Tez container size:

hive.auto.convert.join.noconditionaltask.size = 159519866

This is memory used for caching smaller joining table. During hive join operation, if the smaller one of the two joined tables is smaller than this value, hive will perform map side join, which is significantly faster. This value should fit in the container memory along with the io.sort.memory, as shown in this illustration for Hive on Tez:

5. WebHCat memory settings:

WebHCat(also known as Templeton) uses Mapreduce job (the LaunchMapper) to submit user's Hadoop job. So the Mapreduce memory settings are also applicable to WebHCat's LaunchMapper. However, usually the LaunchMapper will need more memory than the default mapper memory limit of 512MB. For example, for a user's Hive job submitted through WebHCat, LaunchMapper will call hive command line which may start local task that requires close to 1GB memory. In this case you can use this configuration to set LaunchMapper's memory limit to 1GB:

templeton.mapper.memory.mb = 1024

References:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html