• Download Putty if you haven't already
• Open Putty, enter userName@VMpublicIPorDNS and the SSH port for the VM that you identified in Step 1 and 2:

• In the left navigation panel of Putty, go to Connections, SSH, then Tunnels, enter the tunneling port in Putty, click Add, and then click Open to connect to the VM:

>"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --user-data-dir="%USERPROFILE%\chrome-via-socks" --proxy-server="socks5://localhost:1080"

• https://www.whatismyipaddress.com will display that you are using an IP owned by Microsoft
• https://localhost will connect to port 80 of the Azure VM that you are tunneling into
• https://<Azure VNet private IP or host name>:<port> will connect to the specified port of an Azure VM on the same VNet as the VM that you are tunneling into

Back up

1. If you don't already have a data disk dedicated for backups on the VM, attach a data disk, format and mount it.  You can use Azure command line tool or PowerShell to attach a disk to a VM, but for simplicity, we will use Azure portal here.  Navigate to the VM in Azure portal, click on Disks, and Attach new:

You can format and mount the disks however the way you want, below commands are just an example, you need sudo or root to run these commands:

$ fdisk -l (find the newly attached disk, it's likely /dev/sdc if it's the first disk, we assume it's /dev/sdc below)
$ fdisk /dev/sdc (type 'n' at the first prompt, use default for the rest of the prompts, and finally type 'w' to save the changes)
$ mkfs -t ext4 /dev/sdc1
$ mkdir /backupdrive
$ mount /dev/sdc1 /backupdrive
$ blkid (note down the uuid for /dev/sdc1)
$ vi /etc/fstab (add a line like this 'UUID=(uuid from above) /backupdrive ext4 defaults,nofail,noatime, 0 2')

2. If you haven't already backed up the schema for demo_keyspace, back up the schema and save the output file. You only need to run this on one node:

$ cqlsh -e "desc keyspace demo_keyspace" > demo_keyspace_schema.cql

3. Flush data from memory to disk.  On the default deployment of DataStax on Azure, data is stored on the ephemeral disk on each VM in the folder /mnt/data

$ nodetool flush

4. If we have an old test_snapshot, remove it, and list the snapshots folder to make sure it's gone:

$ nodetool clearsnapshot -t test_snapshot
$ ls /mnt/data/demo_keyspace/users-(uuid)/snapshots

5. Take a snapshot and rsync it to the backup disk:

$ nodetool snapshot -t test_snapshot demo_keyspace
$ mkdir /backupdrive/cassandra_snapshots (need sudo or root privilege)
$ rsync -azP --delete /mnt/data/demo_keyspace --include '*/' --include 'snapshots/***' --exclude '*' /backupdrive/cassandra_snapshots

Restore to the existing cluster

1. Stop Cassandra, on our DataStax deployment, run this command:

$ service dse stop

2. Clear commit log and saved cache:

$ rm -f /mnt/commitlog/*
$ rm -f /mnt/saved_caches/*

3. Remove database files for demo_keyspace:

$ rm -f /mnt/data/demo_keyspace/users-(uuid)/*.db

4. Rsync backups to demo_keyspace and move the snapshots two levels up to the directory for the db files:

$ rsync -azP /backupdrive/cassandra_snapshots/ --include '*/' --include 'demo_keyspace/***' --exclude '*' /mnt/data/
$ mv /mnt/data/demo_keyspace/users-(uuid)/snapshots/test_snapshot/* /mnt/data/demo_keyspace/users-(uuid)
$ chown -R cassandra:cassandra /mnt/data/demo_keyspace

5. Start Cassandra:

$ service dse start

Restore to a new cluster

1. Find the tokens of each node on the existing cluster. Go to the Azure portal, and find the public IP of each node:

Note down the tokens for each node by running the following command with that nodes's IP:

$ nodetool ring | grep ip_address_of_node | awk '{print $NF ","}' | xargs

2. Deploy a new cluster with the same number of nodes as the old cluster where the snapshot was taken.  On Azure, you can easily do this by deploying DataStax from Azure Marketplace

3. Copy the backup disks from the old cluster to a storage account in the new cluster. Find the backup disks from the old cluster:

For testing, we can copy to the storage account and container used by the OS disks. We can use Azure command line tool as shown below, or download Azure Storage Explorer, then copy and paste blobs in the UI.

# use "export" on Linux and "set" on Windows to set the environment variables
> export AZURE_STORAGE_ACCOUNT=(destination storage account name) 
> export AZURE_STORAGE_ACCESS_KEY=(destination storage account key)
# create a sas url for the source storage account
> azure storage account sas create --account-name (source storage account name) --account-key (source storage account key) b o r (expiration date, ex: 2016-12-31)
> azure storage blob copy start "https://(source vhd url shown in the above screenshot)?(output from above sas command)" vhds 
# run this to check if copy succeeded
> azure storage blob copy show vhds (vhd blob name)  

4. Attach a copied disk and mount it:

$ fdisk -l (find the newly attached disk, it's likely /dev/sdc if it's the first disk, we assume it's /dev/sdc below)
$ mkdir /backupdrive
$ mount /dev/sdc1 /backupdrive
$ blkid (note down the uuid for /dev/sdc1)
$ vi /etc/fstab (add a line like this 'UUID=(uuid from above) /backupdrive ext4 defaults,nofail,noatime, 0 2')

5. Update cassandra.yaml with the tokens for this node:

$ vi /etc/dse/cassandra/cassandra.yaml 
# uncomment initial_token, and set the value of initial_token to the tokens you noted in step 1

6. Create the schema.  This only needs to run on one node:

$ cqlsh
> source 'demo_keyspace_schema.sql'

7. Restore the snapshots:

# Stop Cassandra:
service dse stop

# Clear systems table
rm -rf /mnt/data/system/*

# Clear commit log and saved caches
rm -rf /mnt/commitlog/*
rm -rf /mnt/saved_caches/*

# Rename the snapshot file from the table name in the old cluster to the table name in the new cluster
ls /mnt/data/demo_keyspace  (copy the new table name)
cd /backupdrive/cassandra_snapshots/demo_keyspace/
mv (old table name) (new table name)

# rsync the snapshot to the data disk and move two levels up to the data folder
rsync -azP /backupdrive/cassandra_snapshots/ --include '*/' --include 'demo_keyspace/***' --exclude '*' /mnt/data/
mv /mnt/data/demo_keyspace/users-(uuid)/snapshots/test_snapshot/* /mnt/data/demo_keyspace/users-(uuid)
chown -R cassandra:cassandra /mnt/data 
# Start Cassandra 
service dse start

1. Go to Cloudera Manager, select HDFS, then Configuration, Search for "core-site", and add the following configuration to Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml, replace with your storage account name and key:

Name: fs.azure.account.key.<your_storage_account>.blob.core.windows.net
Value: <your_storage_access_key>

2. Redeploy stale client configurations, then restart all Cloudera services from Cloudera Manager.

3. To test that Cloudera can access files in Azure storage, put some files in Azure storage.  You can do so by using command line tools like AzCopy, or UI tools such as Visual Studio Server Explorer or Azure Storage Explorer.

4. SSH into any Cloudera node, run the following command.  You may see some warnings, but make sure your can see the files in your Azure storage account.  Note that if you don't specify a destination folder name, you must have the trailing slash in the wasb URL, as shown in the following example:

hdfs dfs -ls wasb://<your_container>@<your_account>.blob.core.windows.net/
 

5. Run distcp on any Cloudera node to copy data from HDFS to Azure Storage.

# Run this command under a user who has access to the source HDFS files, 
# for example, the HDFS superuser hdfs
hadoop distcp /<hdfs_src> wasb://<your_container>@<your_account>.blob.core.windows.net/

Now you should be able to see the source HDFS content showing up in Azure storage:

These steps have been verified on a default deployment of Cloudera CDH cluster on Azure.  At the time of this writing, the deployed CDH is at version 5.7 and Jupyter notebook server 4.1.0 running on Python 2.7 and Anaconda 4.0.0.  The steps should be similar for any CDH cluster deployed with Cloudera Manager. 

1. Go to Cloudera Manager, click the Parcels icon on the top navigation bar, then Configurations, Remote Parcel Repository URLs, add the Anaconda parcel URL "https://repo.continuum.io/pkgs/misc/parcels/" and Save Changes.  

Follow the Cloudera Manager's Parcels wizard to Download, Distribute, and Activate the Anaconda parcel. 

2. SSH into the Spark driver.  If you deployed the CDH cluster on Azure, this is typically the first master node with a host name ending with "-mn0".  For simplicity, we will run the following commands with sudo or as root user.

# install the packages
pip install Jinja2
yum install gcc-c++
yum install python-devel
pip install pyzmq
pip install tornado
pip install jupyter
 
# set environment variables for pyspark
export PYSPARK_DRIVER_PYTHON=/opt/cloudera/parcels/Anaconda/bin/jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --NotebookApp.open_browser=False --NotebookApp.ip='*' --NotebookApp.port=8880"
export PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python
export PATH=/opt/cloudera/parcels/Anaconda/bin:$PATH
 
# create a notebook directory, make sure it's accessible by a hadoop user with sufficient priviledge to hdfs, for example, the hadoop superuser hdfs. 
mkdir /<your_notebook_dir>
chown hdfs:hdfs /<your_notebook_dir>
 
# run pyspark as a hadoop user with sufficient privilege, such as the superuser hdfs.
su hdfs
pyspark

3. SSH into the Spark executors.  These are typically the data nodes in your cluster.  Set the following environment variable on each executor.  Instead of setting it in the console, you want to permanently set it by, for example, adding your custom file under /etc/profile.d. 

export PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python

4. Now go to the Jupyter notebook URL: https://<your_spark_driver>:8880/notebooks, and New a Python2 notebook. 

(If your cluster is deployed with the default configuration in Azure, then port 8880 may be blocked, and you won't be able to access the notebook URL from the Internet.  You can open this port by going to the Azure portal, find the Network Security Groups for the VNet and the master nodes of the CDH cluster, and open this port in their inbound security rules. )

For more information about running Jupyter notebook on Cloudera, please see this documentation. 

1. Enable Azure AD Domain Service. By the end of this step, we will have a classic VNet managed by AADDS.  We will have a Windows VM joined to the domain with Active Directory tools to view and manage the domain services.
2. Connect the Azure classic VNet used with AADDS with an Azure Resource Manager (ARM) VNet in which the VMs will be deployed. By the end of this step, we will have an ARM VNet connected to the classic VNet via VPN and managed by AADDS. AADDS currently only works with Azure classic VNet. Hopefully ARM VNet support will come in the near future, in which case this step will become unnecessary. [Update 8/2017: Azure now supports VNet peering between classic and ARM VNet as long as the two VNets are in the same region. This step can now be greatly simplified with VNet peering if the VNets are in the same region.]
3. Deploy and configure the VMs. By the end of this step, we will have Linux VMs joined to the domain, with Kerberos enabled. We will be able to ssh into the Linux VMs with users defined in AADDS. 
4. Enable Kerberos on Cloudera.

1. Follow this AADDS documentation to enable Domain Service.  Although we will connect the classic VNet used with AADDS with the ARM VNet for the VMs later in Step 2, here are a few things to plan head because they can't be changed after AADDS is enabled -

a. Make sure the classic VNet to be used with AADDS is deployed in a region where your VMs will be deployed to. b. Make sure the address space of the classic VNet to be used with AADDS don't overlap with the address space for the ARM VNet.  For example, use 10.1.0.0/16 for the classic VNet, and 10.2.0.0/16 for the ARM VNet. c. When you enable AADDS, you can choose the dns name for your domain. It doesn't have to be the default "onmicrosoft.com" dns name, you can use, for example, "bigdata.local".

2. Once AADDS is enabled, deploy a Windows VM to the subnet of the classic VNet managed by AADDS.

a. Add AD and DNS tools using "Add roles and features" wizard.

b. Join the VM to the AADDS domain, for example "bigdata.local", using a user that belongs to the "AAD DC Administrators" group created in the previous step.  After rebooting the VM, log in as a domain admin, we can now use the AD and DNS tools to manage the domain.

1. Create an ARM VNet in which we will deploy the VMs, make sure it's in the same region as the classic VNet and that its address space doesn't overlap with that of the classic VNet.
2. Follow these steps to create a gateway in the classic VNet and ARM VNet respectively, then connect the two VNets.  Note that you don't have to use the scripts in the documentation, you can create both gateways and the connection from the Azure portal.  The "shared key" used to make the connection between the two gateways can be accessed from the portal of the classic VNet:

3. The classic VNet should already be configured to use AADDS for DNS in Step 1. Now configure DNS for the ARM VNet to use AADDS as well:

4. Plan for the IP addresses you will assign to the VMs. If you deploy the VMs using this Cloudera ARM template in Step 3 and the address prefix for the VM subnet is, for example, 10.7.0, then the Cloudera master nodes start with 10.7.0.10, and the data nodes start with 10.7.0.20.  Add a Reverse Lookup Zone for the domain controllers and another for the VMs. Make sure to update the PTR records for the domain controllers so that they show up in their reverse lookup zone.

5. Add the DNS records for the VMs

1. For Kerberos to work, we need the FQDN of the domain controllers. The host name generated by AADDS can be found in the DNS tool:

2. In this example, we will use an ARM template originally created for a Cloudera cluster to deploy Cent OS VMs, however, Cloudera specific code is only used in Step 4. You can deploy Cloudera in another way, or you can customize the template to deploy any Linux VMs. In any case, note the following configurations that made the integration between the VMs and AADDS possible:

a. ntp, sssd, samba, and kerberos are installed and configured on each VM (see initialize-node.sh and the configuration templates in the config folder) b. AADDS domain controller names and IPs are passed to the ARM template in the following parameters so they can be automatically set in the configuration files, for example, the following values match the screenshots above. It's also important to set useLdapIDMapping to True when we use sssd rather than NIS (NIS is not supported on AADDS):

"adDomainName": {"value": "bigdata.local"},
"PDC": {"value": "ad-k1kxmrudcuxv"},
"BDC": {"value": "LQ75SYGTK0EFQAA"},
"PDCIP": {"value": "10.1.1.4"},
"BDCIP": {"value": "10.1.1.5"},
"useLdapIDMapping": {"Value": "True"}

Deploy azuredeploy.json. Once done, from the Windows VM on the classic VNet, we should be able to ping the linux VMs using their FQDN, for example contoso-mn0.bigdata.local, and vice versa. We should also be able to ping Linux VMs from each other using their FQDN. To verify reverse DNS works, run "host <linux vm ip>" on a Linux VM, and it should return the FQDN for that IP.

3. Join the Linux VMs to the domain by ssh into each VM and run the following commands with root privilege:

#join the domain with a domain admin credential
#this should be a user in the "AAD DC Administrators" group created in Step 1
> net ads join -Uadadmin@BIGDATA.LOCAL
> authconfig --enablesssd --enablemkhomedir --enablesssdauth --update
> service sssd start
> kinit adadmin@BIGDATA.LOCAL
#if you see an output from the following command, then everything is set up correctly
> getent passwd adadmin@BIGDATA.LOCAL
 

Now we should be able to ssh into each VM using a user defined in AADDS. Up till this point, other than we leveraged the ARM template originally created for Cloudera to deploy the VMs, there's nothing specific we did for Cloudera. In fact, Cloudera bits are not yet installed on the VMs.

1. Install Cloudera on the VMs by deploying azuredeploy_postad.json. The following parameters must match what was specified when deploying azuredeploy.json in the previous step. (This is not necessary if you deployed Cloudera using a different method than our sample ARM template.)

adminUserName adminPassword dnsNamePrefix adDomainName nodeAddressPrefix numberOfDataNodes region tshirtSize

2. Verify Cloudera is installed correctly by RDP into a VM (could be the Windows VM created in the AADDS classic VNet), open a browser, and access https://<dnsNamePrefix>-mn0.<adDomainName>:7180. Use the Cloudera Manager admin credential specified when deploying the template in the previous step.

3. Follow this documentation to enable secure LDAP on AADDS. This is required by Cloudera Manager when issuing LDAPS requests to import KDC account manager credentials.

4. If you haven't already, create an Organization Unit in AADDS. This is required because users in AADDS are created in Azure AD and synced to AADDS, only users under an Organization Unit can be created inside AADDS.

5. Follow this Cloudera documentation to enable Kerberos using the wizard in Cloudera Manager

a. Make sure to install JCE Policy file b. We already installed OpenLdap client library on all VMs using our template.

c. Specify domain info, note that here we specify encryption type rc4-hmac (aes256 doesn't seem to work with AADDS, however, JCE policy file is still needed for hadoop commands to work):

6. Verify Kerberos is set up correctly by running "hdfs dfs -ls /" as the built-in hdfs user on a Cloudera VM, it should render a security error.  Create a hdfs@<domain name> in the Organization Unit created earlier, ssh in as this domain user and run the command again, it should succeed.

These steps have been verified in a default deployment of Cloudera CDH cluster on Azure.  At the time of this writing, the deployed CDH is at version 5.7, HUE 3.9, and Livy 0.2.  The steps should be similar for any CDH cluster deployed with Cloudera Manager.  Note that Livy is not yet supported by Cloudera.   

2. In Cloudera Manager, go to Hue->Configurations, search for "safety", in Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini, add the following configuration, save the changes, and restart Hue:

[desktop]
app_blacklist=

[spark]
server_url=https://<your_hue_server>:8998/

languages='[{"name": "Scala", "type": "scala"},{"name": "Python", "type": "python"},{"name": "Impala SQL", "type": "impala"},{"name": "Hive SQL", "type": "hive"},{"name": "Text", "type": "text"}]'
 

Now if you go to the Hue Web UI, you should be able to see the Spark notebook.  The Spark notebook uses Livy to submit Spark jobs, so without Livy, it's not functioning yet. 

3. In the Hue Web UI, go to Configuration, find hadoop_conf_dir and note it down:

4. SSH to your Hue server, for simplicity, unless specified we'll run the following commands in sudo or as root user:

#download Livy
wget https://archive.cloudera.com/beta/livy/livy-server-0.2.0.zip
unzip livy-server-0.2.0.zip -d /<your_livy_dir>
 
#set environment variables for Livy
export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export HADOOP_CONF_DIR=<your hadoop_conf_dir found in the previous step in Hue configuration>
export HUE_SECRET_KEY=<your Hue superuser password, this is usually the user you use when you log in to Hue Web UI the first time>
 
#run Livy. You must run Livy as a user who has access to hdfs, for example, the superuser hdfs.
su hdfs
/<your_livy_dir>/livy-server-0.2.0/bin/livy-server

5. Now if you go to Hue Web UI again, the warning about Spark in Step 2 should be gone. Go to the root of your Hue Web UI URL, then add "/notebook" in the URL. You should be able to add a notebook to run code:

In the following example, we added a Scala snippet and ran it in the notebook:

If you keep your ssh console with Livy running open, you will see when you start to run your code in a notebook, a Spark job is being submitted. If there's any error, you will also see it in the console.

You can also add Impala snippet, and use the various graphing tools:

For more information about Hue Spark notebook, please see gethue.com/spark.

1. Deploy Active Directory with HA in Azure
2. Deploy Linux VMs for the Cloudera cluster
3. Enable Active Directory DNS on the Linux VMs
4. Sync Linux VMs to Active Directory time service
5. Join the Linux VMs to Active Directory and enable Single-Sign-On
6. Install Cloudera
7. Enable Kerberos on Cloudera
8. Enable Single-Sign-On for Cloudera web consoles

Step 6: Install Cloudera

Step 7: Enable Kerberos on Cloudera

2. We already installed OpenLdap client library on all Linux VMs.

sudo su hdfs
hdfs dfs -ls / 
//should display security error

7. Create the hadoop superuser hdfs in AD in the same NIS domain.  ssh in as hdfs@<domain name>, run the above command again, it should succeed. 8. Create hadoop users in AD in the same NIS domain, create their home directory with hdfs, then log in as a hadoop user, and run mapreduce job.

//log in as hdfs, create home directory for each hadoop user
hdfs dfs -mkdir /user/alice
hdfs dfs -chown alice /user/alice
 
//log in as a hadoop user, for example, then run mapreduce
hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 10000 

Step 8: Enable Single-Sign-On for Cloudera web consoles

1. Follow the Cloudera documentation to enable Single-Sign-On using AD credentials. Once enabled, it will prompt for user credential when we open, for example, Yarn ResourceManager Web UI.  We need to provide the fully qualified user name, for example, someone@bigdata.com.
2. To enable AD authentication for Cloudera Manager console, configure External Authentication:

Note that users must be explicitly added to the AD groups specified here, for example,

Restart Cloudera Manager server:

service cloudera-scm-server restart

1. Deploy Active Directory with HA in Azure
2. Deploy Linux VMs for the Cloudera cluster
3. Enable Active Directory DNS on the Linux VMs
4. Sync Linux VMs to Active Directory time service
5. Join the Linux VMs to Active Directory and enable Single-Sign-On
6. Install Cloudera
7. Enable Kerberos on Cloudera
8. Enable Single-Sign-On for Cloudera web consoles

Step 5: Join the Linux VMs to Active Directory and enable Single-Sign-On

net ads join -U<a domain admin user name>
net ads testjoin -P //verify the VM has joined the domain

authconfig --enablesssd --enablemkhomedir --enablesssdauth --update
service sssd start 

//install the administration tools for Identity Management for UNIX
dism.exe /online /enable-feature /featurename:adminui /all
//install Server for Network Information Server (NIS)
dism.exe /online /enable-feature /featurename:nis /all 
//install Password Synchronization
dism.exe /online /enable-feature /featurename:psync /all 

%hadoopadmin ALL=(ALL) NOPASSWD: ALL

Although we have deployed Linux VMs that have disk configuration and host name conventions configured for Cloudera, the rest of the configuration for joining AD and Single-Sign-On up until this point is not Cloudera specific.  You can join any Linux VMs to AD using the steps above.  The key here is the configuration files we deployed on the Linux VMs.  

• /etc/resolv.conf
• /etc/ntp.conf
• /etc/krb5.conf
• /etc/samba/smb.conf
• /etc/sssd/sssd.conf

1. Deploy Active Directory with HA in Azure
2. Deploy Linux VMs for the Cloudera cluster
3. Enable Active Directory DNS on the Linux VMs
4. Sync Linux VMs with Active Directory time service
5. Join the Linux VMs to Active Directory and enable Single-Sign-On
6. Install Cloudera
7. Enable Kerberos on Cloudera
8. Enable Single-Sign-On for Cloudera web consoles

Step 1: Deploy Active Directory with HA in Azure

Step 2: Deploy the Linux VMs for Cloudera cluster

1. If we want a different subnet for Cloudera than AD, we can go to the Azure portal and create a subnet for Cloudera.  Make sure to use the name and IP address of the VNet created in Step 1 and the subnet created here as the network parameters for the template below.
2. Normally we can deploy a Cloudera cluster in Azure using this template.  However, it uses public DNS and hosts files for domain resolution.  To integrate with AD instead, we need to refactor this template by splitting it into two parts: the deployment of the VMs, and the installation of Cloudera.  The refactored template can be found here.  Deploy azuredeploy.json in this step.  The following parameters must match what we have created in #1.

• • adDomainName, for example, bigdata.com
  • PDC, the host name of the PDC, for example, adpdc
  • BDC, the host name of the BDC, for example, adbdc
  • PDCIP, the static private IP of the PDC, for example, 10.0.0.4
  • BDCIP, the static private IP of the BDC, for example, 10.0.0.5
  • region, the same region as the AD forest
  • virtualNetworkName, the same virtual network as the AD
  • vnetResourceGroup, the resource group where the virtual network is located, not necessarily same as the resource group the Cloudera VMs will live in
  • subnetName, the subnet for Cloudera VM, must be created in #1 if not already exists
  • subnetPrefix, the IP CIDR for the Cloudera subnet
  • nodeAddressPrefix, the first 3 octets of the subnetPrefix

• • cloudera-mn0.bigdata.com
  • cloudera-dn0.bigdata.com
  • cloudera-dn1.bigdata.com
  • cloudera-dn2.bigdata.com

Step 3: Enable Active Directory DNS on Linux VMs

1.  RDP into the PDC, and open the DNS Manager.  Note that if DNS Manager is not installed, install it by "Add Roles and Features" -> Features -> Role Administration Tools -> DNS Server Tools 2.  Add a new reverse lookup zone if there's none that corresponds to the subnet address range where the Linux VMs are located

3.  Add the Linux VMs to the forward lookup zone with their static IPs, check "create/update associated PTR record"

4.  On the Linux VMs, verify VMs are reachable by their DNS names:

hostname -i //should display the static IP of the VM
host <VM static IP> //should perform DNS reverse lookup and display FQDN of the VM
ping <VM name.mydomain.com> //should successfully reach any of the 4 Cloudera VMs within the VNet

Step 4: Sync Linux VMs with Active Directory time service

1.  On the PDC, configure the Windows Time service by running this command, time.windows.com is a valid time server, or you can use another time server of your choice, restart the time service

w32tm /config /manualpeerlist:"time.windows.com" /syncfromflags:manual /reliable:yes /update
net restart w32time

2.  On the BDC, restart the time service by running the following command:

w32tm /config /syncfromflags:domhier /update

3.  On the Linux VMs, verify time syncing with the following command, the "system peer" in the output should point to the PDC:

ntpdc -c sysinfo

Scenario: 

A telecommunications company has a large volume of data for incoming calls.  The company needs to -

1. Obtain insights about customer usage aggregated across geographical regions in real time
2. Detect SIM fraud (multiple calls coming from the same identity around the same time but in geographically different locations) in real time

Ingesting data:

Data ingestion is the same as described in the aforementioned article about Stream Analytics.  Follow these main steps:

Step 1: Create an Event Hubs Input 

1. Go to the Azure portal, click New -> App Services -> Service Bus -> Event Hub -> Quick Create to create a new Event Hub
2. Go to the newly created Event Hub, on the Consumer Groups tab, click Create at the bottom of the page to create a new consumer group
3. On the Configure tab of the Event Hub, under Shared Access Policies, create a new policy with Manage permissions
4. Save the changes, and navigate to the Dashboard, click Connection Information at the bottom of the page, and copy down the connection information

Step 2: Download and configure the sample app that can generate sample call records and push them to Event Hubs

1. Download the event generator sample app from GitHub 
2. Replace the Microsoft.ServiceBus.ConnectionString and EvenHubName values in App.Config with your Event Hub connection string and name
3. Run the solution

telcodatagen [#NumCDRsPerHour] [SIM Card Fraud Probability] [#DurationHours]

The following example will generate 1000 events with a 20% probability of fraud over the course of 2 hours

telcodatagen 1000 .2 2

Step 3: Optionally you can verify that you are able to receive data from Event Hubs using the Java client for Event Hubs.  

There are two reasons you may want to do this:

1. Become familiar with how Java client for Event Hubs works, what packages are needed, and how to build it etc, which can be helpful when we run Spark Streaming in Scala below
2. Make sure the ingestion pipeline works

Processing data with Spark Streaming and Spark SQL on Cloudera:

If you don't already have a Cloudera cluster, you can install it from a simple UI wizard on Azure Marketplace, which is also used in this example.  But the cluster doesn't have to be in any specific environment, as long as you have Spark on YARN installed on the cluster.  Please note that different versions of Spark do require some changes in the code.  Here, we are using Spark 1.3.0 installed with Cloudera 5.4.

You can find the source code for this Spark job on GitHub.   Below are the key steps:

Running the Spark job in Spark-Shell

Before submitting a Spark job in cluster mode, it's a good idea to run Spark interactively with spark-shell for easy troubleshooting.  In the YARN client mode, the Spark driver runs on the local machine, so any additional jars must be available on the local machine.  Any additional jars required by the Spark executors must be available on the Spark executor nodes as well.  For simplicity, we copied all the necessary jars to the /libs folder on both the Spark driver and the executors.

spark-shell --master yarn --deploy-mode client --executor-cores 4 -usejavacp --jars /libs/spark-streaming-eventhubs_2.10-0.1.0.jar,/libs/eventhubs-client-0.9.1.jar,/libs/qpid-amqp-1-0-client-0.32.jar,/libs/qpid-amqp-1-0-client-jms-0.32.jar,/libs/qpid-amqp-1-0-common-0.32.jar,/libs/qpid-client-0.32.jar,/libs/qpid-common-0.32.jar --driver-class-path /libs/sqljdbc4.jar --conf spark.executor.extraClassPath=/libs/sqljdbc4.jar

• You can build spark-streaming-eventhubs jar from the source code of the Event Hubs receiver for Spark Streaming, or you can download it from this Even Hubs Spark Streaming example. 
• You can build eventhubs-client jar from the source code of the Java client for Event Hubs.
• You can download sqljdbc jar.  
• One important thing to note is that the number of executor cores must double the number of Event Hubs partitions so that the executor can receive data from Event Hubs and also process the data at the same time.  If this is not set correctly, you will not see any error, but there will be no output data either.  It will appear that the job is not doing anything. 

Step 1: Receive data in Spark Streaming from Event Hubs

In the interactive command prompt of spark-shell, you can paste the following Scala code to run the Spark Streaming job, replacing the parameters with your own.   

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.streaming.eventhubs.EventHubsUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
 
val streamBatchIntervalInSeconds = 60
val ehParams = Map[String, String](
 "eventhubs.policyname" -> "your event hubs rule name",
 "eventhubs.policykey" -> "your event hubs policy key",
 "eventhubs.namespace" -> "your event hubs namespace",
 "eventhubs.name" -> "your event hubs name",
 "eventhubs.partition.count" -> "2", //executor core count must be twice that of partition count
 "eventhubs.consumergroup" -> "your event hubs consumer group name",
 "eventhubs.checkpoint.dir" -> "your spark checkpoint dir in hdfs", //for simplicity we are not using reliable receiver with checkpoint in this example
 "eventhubs.checkpoint.interval" -> "600")
val ssc = new StreamingContext(sc, Seconds(streamBatchIntervalInSeconds))
val stream = EventHubsUtils.createUnionStream(ssc, ehParams)
val lines = stream.map(msg => new String(msg))
lines.print()
ssc.start()

Start the telcodatagen mentioned earlier to generate some data for input.  If you can see each line received from Event Hubs printed out in the spark-shell, congratulations, you have everything wired up correctly.  

Step 2: Run real time queries in Spark SQL  

Replace "lines.print()" in the above Scala code with the following code.

 //convert international datetime string to unix time
 val isoformat = new java.text.SimpleDateFormat("yyyy-MM-dd'T'hh:mm:ss'Z'")
 def date2long: (String => Long) = (s: String) => try { isoformat.parse(s).getTime() } catch { case _: Throwable => 0 }
 val sqlfunc = udf(date2long)
 
 //define sqlContext
 val sqlContext = new org.apache.spark.sql.SQLContext(sc)
 import sqlContext.implicits._
 import org.apache.spark.sql.functions._
 
 //process the data using Spark SQL
 lines.foreachRDD { rdd => if (rdd.count() > 0) {
 val rawdf = sqlContext.jsonRDD(rdd);
 // convert international time string to unix time for easy calculation
 val df = rawdf.withColumn("callTime", sqlfunc(rawdf("callrecTime")));
 // use the min time of this mini batch as the timestamp for the aggregated entry
 val minTime = df.filter(df("callTime") > 0).select("callTime").first().getLong(0);
 // real time aggregation by region
 val callsByRegion = df.groupBy("SwitchNum").count().withColumnRenamed("count", "callCount").withColumn("callTimeStamp", lit(minTime));
 callsByRegion.show();
 }
 }

Now you should be able to see aggregated results in the spark-shell console output.  Note that the aggregation is applied directly on the Spark SQL DataFrame without having to run a SQL query.  For more complex data analysis, such as detecting fraud in this example, we can run a query like the following:

df.registerTempTable("calls");
val fraudCalls = sqlContext.sql("SELECT CS1.CallingIMSI, CS1.callrecTime as calltime1, CS2.callrecTime as calltime2, CS1.CallingNum as CallingNum1, CS2.CallingNum as CallingNum2, CS1.SwitchNum as Switch1, CS2.SwitchNum as Switch2 FROM calls CS1 inner JOIN calls CS2 ON CS1.CallingIMSI = CS2.CallingIMSI AND (CS1.callTime - CS2.callTime) between 1000 AND 5000 WHERE CS1.SwitchNum != CS2.SwitchNum order by CS1.CallingIMSI, CS1.callrecTime");

Note that this query will fail in Spark 1.3.0, probably related to a bug with self joins in Spark SQL.  We will demonstrate how to run this query in Impala instead later in this article.

Step 3: Store results in Azure SQL Server

Create a table in Azure SQL Server:

CREATE TABLE callsByRegion (
 SwitchNum NVARCHAR (20) NOT NULL,
 callCount INT NOT NULL,
 callTimeStamp BIGINT NOT NULL
);
CREATE CLUSTERED INDEX IX_callsByRegion ON callsByRegion(callTimeStamp ASC);

Add a jdbc connection string variable anywhere before "lines.foreachRDD..." in the above Scala code, replacing parameters with your own:

val jdbccxn = "jdbc:sqlserver://<your azure db server>.database.windows.net:1433;database=<your db name>;user=<your db username>;password=<your password>;encrypt=false;loginTimeout=30;"

Replace "callsByRegion.show()" in the above Scala code with the following code:

//save real time aggregation to sql azure
callsByRegion.insertIntoJDBC(jdbccxn, "callsByRegion", false);

Run the Spark job again and check your Azure SQL Server.  You should see aggregated data in the callsByRegion table in your database.

Running the Spark job with Spark-Submit

Once everything runs fine in spark-shell, for production workload, you would want to submit the job to the Spark cluster.  Using spark-submit, you can run the job either in client mode or cluster mode.  You can build the Scala application into a jar and apply dependencies at run time, or you can build a uber jar that has all the dependencies in it.  Check out this example on GitHub to see how to build the jars in maven.

Running the job in YARN client mode

Note that the order of the parameters in the following commands is important.  --class and app jar must be at the end of each command. 

Using a jar without dependencies:

spark-submit --master yarn --deploy-mode client --executor-cores 4 --jars /libs/spark-streaming-eventhubs_2.10-0.1.0.jar,/libs/eventhubs-client-0.9.1.jar,/libs/qpid-amqp-1-0-client-0.32.jar,/libs/qpid-amqp-1-0-client-jms-0.32.jar,/libs/qpid-amqp-1-0-common-0.32.jar,/libs/qpid-client-0.32.jar,/libs/qpid-common-0.32.jar --driver-class-path /libs/sqljdbc4.jar --conf spark.executor.extraClassPath=/libs/sqljdbc4.jar --class com.sparkeventhub.sample.SparkEventHubSample ./app.jar

Using a uber jar with dependencies:

spark-submit --master yarn --deploy-mode client --executor-cores 4 --driver-class-path /libs/sqljdbc4.jar --conf spark.executor.extraClassPath=/libs/sqljdbc4.jar --class com.sparkeventhub.sample.SparkEventHubSample ./app-jar-with-dependencies.jar

Running the job in YARN cluster mode

Using a jar without dependencies:

spark-submit --master yarn --deploy-mode cluster  --executor-cores 4 --driver-cores 2 --jars /libs/spark-streaming-eventhubs_2.10-0.1.0.jar,/libs/eventhubs-client-0.9.1.jar,/libs/qpid-amqp-1-0-client-0.32.jar,/libs/qpid-amqp-1-0-client-jms-0.32.jar,/libs/qpid-amqp-1-0-common-0.32.jar,/libs/qpid-client-0.32.jar,/libs/qpid-common-0.32.jar --driver-class-path /libs/sqljdbc4.jar --conf spark.executor.extraClassPath=/libs/sqljdbc4.jar --class com.sparkeventhub.sample.SparkEventHubSample ./app.jar

Using a uber jar with dependencies:

spark-submit --master yarn --deploy-mode cluster --executor-cores 4 --driver-cores 2 --driver-class-path /libs/sqljdbc4.jar --conf spark.executor.extraClassPath=/libs/sqljdbc4.jar --class com.sparkeventhub.sample.SparkEventHubSample ./app-jar-with-dependencies.jar

Processing data with Impala

Instead of running queries in Spark SQL, we can alternatively store the data in hdfs, and run queries in real time with Impala.  To store data in HDFS, add a variable to the output file path in HDFS anyhere before "lines.foreachRDD..." in the above Scala code, replacing the path with your own:

import org.apache.spark.sql.SaveMode
val outputDir = "sparkoutput/calls"

Store the data in HDFS by adding the following code at the end of the "lines.foreachRDD..." block:

df.registerTempTable("rawcalls");
val calls=sqlContext.sql("SELECT callrecTime, SwitchNum, CallingIMSI, CallingNum, CalledNum, callTime from rawcalls");
calls.save(outputDir, SaveMode.Append);

Run the job again, and you should be able to see parquet files being stored in the specified HDFS folder.  The folder may be relative to the current HDFS user home directory, for example "/user/hdfs/sparkoutput/calls".  Create an external table in Impala:

create external table calls (callrecTime string, SwitchNum string, CallingIMSI string, CallingNum string, CalledNum string, callTime bigint) stored as parquet location '/user/hdfs/sparkoutput/calls'

Query the data in Impala:

SELECT CS1.CallingIMSI, CS1.callrecTime as calltime1, CS2.callrecTime as calltime2, CS1.CallingNum as CallingNum1, CS2.CallingNum as CallingNum2, CS1.SwitchNum as Switch1, CS2.SwitchNum as Switch2 FROM calls CS1 inner JOIN calls CS2 ON CS1.CallingIMSI = CS2.CallingIMSI AND (CS1.callTime - CS2.callTime) between 1000 AND 5000 WHERE CS1.SwitchNum != CS2.SwitchNum order by CS1.CallingIMSI, CS1.callrecTime

There you have it.  I'm new to Spark Streaming, Spark SQL, and Impala.  Would love to learn if there are better ways to do this.