Big Data Support

Rerunning many slices and activities in Azure Data Factory

JasonHowell — Wed, 31 Aug 2016 20:09:20 GMT

Today someone asked me how to run all the data slices in their data factory on-demand in an ad-hoc fashion, to say run the whole pipeline again from scratch.

For example, if you have a one-time copy data factory that is used to load a data warehouse or a development environment in its entirety, you might want to run it only on-demand, and not have it run on a schedule.

A. Click on each Dataset and click Run on the data slice blade

Then can click on one data set at a time and run it or rerun it. That is still a piecemeal and a bit tedious if you have many datasets and slice times.

1. Select the data set in question. 2. Find the time slice in question. 3. Click the Run button to rerun it.

B. Multiselect from the ADF Monitor and Manage web app.

You can filter the list and rerun the slices from the Monitoring web app. Multiselect (hold shift key and arrow down through the list) will work there in the Activity Windows grid, then you can run many with a single click.

If you have 100 tables being copied for example, you might have a bunch of different data sets and nested slice times as shown.

However, if you have a one-time copy that you did from the Copy Data tool in the browser, you won't see any data sets most likely, so this option is universal for all copy activities I worry.

C. Script it with PowerShell

Another way to rerun many slices is to use a PowerShell script to loop over all datasets in a given data factory or pipeline and reset the status to run them again.

Below is a code sample to do that. This script will rerun all slices on all datasets. If you only want to run some of them (not all slices), you can change the * filter in the where like query.

1. Launch Windows PowerShell ISE from the Windows start menu.

Open the code, or paste it in the script window at the top.
You can press the play button to run the whole script, or highlight lines piecemeal and run selection as needed.

Example of how to highlight the code and run one line at a time:

2. First time Installations

The first time you want to use Azure PowerShell on a given computer, you could either Download the whole Azure PowerShell SDK or launch powershell as administrator and run PowerShell commands to install the needed modules:

Install-Module AzureRM

Install-Module AzureRM.DataFactories

3. Set your active subscription once per session.

The first time you need to get connected to Azure and authenticate, and pick which subscription to work on.
If you use the same PowerShell window on the second time, you can skip these lines, to save time and save typing your password again. Use # pound sign as a comment to comment out the code as needed.

# To login to Azure Resource Manager
Login-AzureRmAccount

# To view all subscriptions for your account
Get-AzureRmSubscription

# To select a default subscription for your current session
Get-AzureRmSubscription -SubscriptionName "your sub" | Select-AzureRmSubscription

4. Example PowerShell code to run.

There are #Comments inline to help you understand the intent.

This sample finds your data factory by name and resource group, then lists all slices, and loops to set the slice status to waiting. That will reset them so that the slices run again.

In case you want to filter just to Output slices, or a certain naming pattern, you can edit the like * clause.

You should adjust the two time stamps highlighted to a reasonable range so that it finds only recent slices, else it may be a bit slow if you have days or years of slices in the history to repeat.

-StartDateTime 2016-01-01T00:00:00.0000000 -EndDateTime 2017-01-01T00:00:00.0000000

#Put your Data Factory name and Resource group here
$RG = "YourResourceGroup"
$DFname = "YourFactory"

#Once per session - comment out to save time
# Login-AzureRmAccount

# Once, select a default subscription for your current session
# Get-AzureRmSubscription -SubscriptionName "your sub" | Select-AzureRmSubscription

# One time per computer - comment out to save time
# Register-AzureRmResourceProvider -ProviderNamespace Microsoft.DataFactory

# Debug option: List all DF in a resource group for debugging if needed
# Get-AzureRmDataFactory -ResourceGroupName $RG

# Find the Data Factory
$df= Get-AzureRmDataFactory -ResourceGroupName $RG -Name $DFname
if ($df -eq $null) { Write-Host "Data Factory " $DFname " cannot be found. Check spelling and resource group name. Error: " $_ -BackgroundColor:Red }

# List all DataSets in the data factory - Add the LIKE filter by name instead of * if needed such as *output*
$DataSets = Get-AzureRmDataFactoryDataset -DataFactory $df | Where {$_.DatasetName -like "*"} | Sort-Object DatasetName

# Loop over matching named DataSets
$i = 1
ForEach($DS in $DataSets)
{
Write-Host $DS.DataFactoryName "--> " $DS.DatasetName -ForegroundColor:Yellow
# List slices
$Slices = Get-AzureRmDataFactorySlice -DataFactory $df -DatasetName $DS.DatasetName -StartDateTime 2016-01-01T00:00:00.0000000 -EndDateTime 2017-01-01T00:00:00.0000000

# Reset all slices to status Waiting for the given dataset, in case there are multiple
ForEach($S in $Slices)
{
$outcome=$false
Write-Host $i ":" $DS.DataFactoryName "--> " $DS.DatasetName "--> Slice Start:["$S.Start"] End:["$S.End"] State:"$S.State $S.SubState -ForegroundColor:Cyan

Try {
$outcome=Set-AzureRmDataFactorySliceStatus -DataFactory $df -DatasetName $DS.DatasetName -Status Waiting -UpdateType UpstreamInPipeline -StartDateTime $S.Start -EndDateTime $S.End
Write-Host " Slice status reset to Waiting so it will run again:" $outcome -ForegroundColor:Green

}
Catch
{
Write-Host " Slice status reset has failed. Error: " $_ -ForegroundColor:Red
}
$i++
}
}

Capture Microsoft Azure Stream Analytics logs

sudhirblog — Wed, 24 Aug 2016 07:50:18 GMT

Microsoft Azure Stream Analytics makes building real time solution very easy. Developers can build Stream Analytics job with few clicks.

While running Stream Analytics jobs, you may encounter an error which may cause Stream Analytics job to go in degrade/stop state. So it requires to figure out the error in job to troubleshoot it. There could be various error you might get into like improper data, issue at source/destination side, query computation fail etc. Before you start fixing the issue, it’s required to figure out the cause of the failing job.

There is a way to figure out the issue from audit logs in azure portal. During our work we figured out another way to capture logs for a Stream Analytics job. In this blog I am showcasing about various technique’s to capture Stream Analytics log including the one we came up using PowerShell.

Microsoft Azure Ibiza Portal (https://portal.azure.com)

Click Audit log and click Filter. Change parameter to show only Stream analytic logs (as shown in picture below). Click Update.

Now we’ll see logs for Azure Stream Analytics. if logs are more, we’ll have Load More option at the end to load more logs information.

Microsoft Azure Classic Portal (https://manage.windowsazure.com)

Either you can go directly to management services (from left side pane) or click Operation Logs from Stream Analytics dashboard.

Change parameters and click run to get all logs.

If there are too many logs then you have to move from one page to another.

Capture Stream Analytics logs through PowerShell

There is another way to download Stream Analytics log which is from PowerShell. The PowerShell script not only capture the logs but also zip it in local system.

Download PowerShell script from here
Open Windows PowerShell ISE and copy script

Run Script
At first step, it will prompt for login information with which Azure subscription is linked. Stream Analytics job should be part of the subscription.

Once authenticated successfully, it will showcase all the Stream Analytics jobs with in the subscription.
Choose one or all jobs for which logs need to be extracted. Press Enter

Provide number of days log need to be extracted. Press Enter

After writing log files it will zip all content.

Open windows explorer and look onto C:\Temp folder. There will be 1) StreamAnalytics folder 2) StreamAnalytics.zip file

Open StreamAnalytics folder. Below are the files available in it.

Description of files.

StreamAnalytics_JobXXXXXXXXXXXXXXXXXXX.txt and StreamAnalytics_JobXXXXXXXXXXXXXXXXXXX.json :- Both contains same information. Later file (with extension JSON) contains information in JSON format. The size of the file depends on number of jobs selected and number of days log required (max 14 days). It contains information like Subscription, Resource Name, Job name, date range, input, output and log.
StreamAnalytics_JobsListing.txt :- This file contains information (mentioned below) about selected job.

I would like to thanks my colleagues (Jason Howell and Dan Shaver) for their contribution. Hope it help in debugging Stream Analytics job.

HDFS gets full in Azure HDInsight with many Hive temporary files

JasonHowell — Mon, 15 Aug 2016 17:24:27 GMT

Sometimes when Hive is using temporary files, and a VM is restarted in an HDInsight cluster in Microsoft Azure, then those files can become orphaned and consume space. In Azure HDInsight, those temp files live in the HDFS file system, which is distributed across the local disks in the worker nodes. This is a different location than where user data is stored in HDInsight, since user data will be kept in Azure Blob Storage or Azure Data Lake Store when using HDInsight. The local disk and HDFS is not used for user data files by default and only the temporary files for Hive are kept there. In the Hive-site.xml, this is the property that controls the location of the temporary files:

hive.exec.scratchdir hdfs://mycluster/tmp/hive

Step 1. Checking Disk space

A> Ambari dashboard shows an HDFS usage percentage in the dashboard.

This is the HDFS file system spread across a number of worker nodes on local disk. From the Azure portal, it should direct you to the URL such as https://yourhdinsightname.azurehdinsight.net/ Ambari should log this kind of text in its metrics collector "/var/log/ambari-metrics-collector/ambari-metrics-collector.log"

2016-08-15 20:01:44,739 [WARNING] [HDFS] [namenode_hdfs_capacity_utilization] (HDFS Capacity Utilization) Capacity Used:[82%, 301787316224], Capacity Remaining:[68437065728]
2016-08-15 20:01:47,317 [WARNING] [HDFS] [datanode_storage] (DataNode Storage) Remaining Capacity:[29831340032], Total Capacity:[86% Used, 210168897536]

B> To see how much local disk space on each worker node and headnode, you could run these kind of commands.

You may ssh into each worker node to run it manually.

> Get free disk space to see if any disk is nearly full.

df -h

Filesystem Size Used Avail Use% Mounted on udev 3.4G 12K 3.4G 1% /dev tmpfs 697M 744K 697M 1% /run /dev/sda1 985G 15G 931G 2% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 3.5G 12K 3.5G 1% /run/shm none 100M 0 100M 0% /run/user none 64K 0 64K 0% /etc/network/interfaces.dynamic.d /dev/sdb1 281G 7.3G 259G 3% /mnt

> List all detailed file sizes starting from the root directory

sudo du / -ch

> See only the total file size from the root directory

sudo du / -ch |grep total 22G total

> Find files that are larger than 1000 MB

sudo find / -size +1000M -print0 | xargs -0 ls -l

These are system files in Linux we can ignore -rw------- 1 root root 7516192768 Aug 15 18:08 /mnt/resource/swap -r-------- 1 root root 140737477881856 Aug 15 19:17 /proc/kcore

> Find files that are larger than 500 MB

sudo find / -size +500M -print0 | xargs -0 ls -l Usually the larger files on the local disks are log files from one of the various Hadoop services can use local significant space. You could delete old log files if you don't need them, or edit the log4j.properties for the service to lower the logging level, such as lowering DEBUG to WARN level or something along those lines. https://logging.apache.org/log4j/1.2/manual.html

Step 2: To see if there are many hive temporary files, you can check HDFS to list the temp folder like this:

> List HDFS files under the hive temp scratch directory

hadoop fs -ls -R hdfs://mycluster/tmp/hive/hive

Notice when the temp files are in use there is a inuse.lck file that Hive maintains to show the folder is in use. Normally once the temp folder is done, Hive will automatically clean it up. In the case Hive crashes, or the server is taken offline abruptly, the files may become dangling / orphaned.

Step 3. Cleaning Up dangling orphaned hive directories in HDFS as needed

A. There is a hive command cleardanglingscratchdir to automate this cleanup safely.

https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-ClearDanglingScratchDirTool Quoting the docs:

The tool cleardanglingscratchdir can be run to clean up any dangling scratch directories that might be left over from improper shutdowns of Hive, such as when a virtual machine restarts and leaves no chance for Hive to run the shutdown hook.

hive --service cleardanglingscratchdir [-r] [-v] [-s scratchdir]

-r dry-run mode, which produces a list on console

-v verbose mode, which prints extra debugging information

-s if you are using non-standard scratch directory

The tool tests if a scratch directory is in use, and if not, will remove it. This relies on HDFS write locks to detect if a scratch directory is in use. An HDFS client opens an HDFS file ($scratchdir/inuse.lck) for writing and only closes it at the time that the session is closed. cleardanglingscratchdir will try to open $scratchdir/inuse.lck for writing to test if the corresponding HiveCli/HiveServer2 is still running. If the lock is in use, the scratch directory will not be cleared. If the lock is available, the scratch directory will be cleared. Note that it might take NameNode up to 10 minutes to reclaim the lease on scratch file locks from a dead HiveCli/HiveServer2, at which point cleardanglingscratchdir will be able to remove it if run again.

1. Dry Run

So you can run this command from the head node to get a dry run list of any files that need to be cleaned up:

hive --service cleardanglingscratchdir -r

-or-

hive --service cleardanglingscratchdir -r -s hdfs://mycluster/tmp/hive/hive

2. Real Run

If that list looks reasonable, and the dates look old and orphaned, then run it for real without the -r dry run.

hive --service cleardanglingscratchdir

-or-

hive --service cleardanglingscratchdir -s hdfs://mycluster/tmp/hive/hive

Note, if you get the error "Cannot find any scratch directory to clear" then it is likely that there is no dangling orphan folder to clean up.

B. If the tool doesn't work, or you are on an older build that doesn't have the cleanup, then you could manually remove the old files

To be very sure none of the files are in use, you should stop Hive services (using Ambari) before running any command, or selectively run the command only on old directories that are not currently in use.

Manual Step 1. Stop Hive – I'm using Ambari to do that

Manual Step 2. Remove the temporary files in HDFS

Option A: All folders at once:

hadoop fs -rm -r -skipTrash hdfs://mycluster/tmp/hive/hive/

Option B: Remove only the folders you are worried about. The date of the hive job is typically listed in the directory name:

hadoop fs -rm -r -skipTrash hdfs://mycluster/tmp/hive/hive/7178b22b-186d-4284-a5a5-6092f165db37/hive_2016-04-14_22-35-31_149_2691354643150374266-1/ hadoop fs -rm -r -skipTrash hdfs://mycluster/tmp/hive/hive/7178b22b-186d-4284-a5a5-6092f165db37/hive_2016-04-14_22-36-56_659_7340036377865464119-4/ hadoop fs -rm -r -skipTrash hdfs://mycluster/tmp/hive/hive/7178b22b-186d-4284-a5a5-6092f165db37/hive_2016-04-14_22-47-26_179_2802662959127515918-4/

Manual Step 3. Now Start Hive again in Ambari using the same menu, then check that it made a fresh scratch file directory.

hadoop fs -ls -R hdfs://mycluster/tmp/hive/hive

That's it for now. Let us know if you run into this problem, how you solve it, and any quirks you faced along the way. Thanks! Jason Howell

How to Find and Kill a running Yarn Application Master in HDInsight with and without SSH access

JasonHowell — Sat, 11 Jun 2016 19:54:28 GMT

Today we faced a challenge in HDInsight not knowing the SSH user password to terminal into the server, and we needed to kill some running Hive jobs that were too far gone and taking too many resources. The user knows the user name and password for Hadoop and Ambari, but does not know the SSH user name and password. HDInsight uses separate accounts and passwords for these two purposes.

Without SSH use cURL to talk REST to kill it

You can use the REST endpoints for Resource manager in HDInsight to kill an application using the Hadoop/Ambari password. 1. Download or build your own cURL https://curl.haxx.se/download.html for Windows, I found it here https://www.paehl.com/open_source/?CURL_7.49.1 Make sure to get one WITH SUPPORT SSL (Since HDInsight only works in HTTPS) 2. To check the application status, you can get a long list of all the applications

curl -u admin:SomePassword -H "Content-Type: application/json" -d "https://servername.azurehdinsight.net/ws/v1/cluster/apps/"

Or focus in on your one application-id

curl -u admin:SomePassword -H "Content-Type: application/json" -d "https://servername.azurehdinsight.net/ws/v1/cluster/apps/application_0000000000_0000/"

3. To Kill a job,on the commandline, run cURL to point to your HDInsight server and the application id, and PUT to change the state to KILLED, effectively killing the Yarn job.

curl -u admin:SomePassword -v -X PUT -H "Content-Type: application/json" -d '{"state": "KILLED"}' "https://servername.azurehdinsight.net/ws/v1/cluster/apps/application_0000000000_0000/state"

Note: for special characters I found I could use an escape character \ before the special character in the password. If you get an error about https, try "" double-quotes around the URL instead of '' single-quotes. More Resource Manager REST options https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

With SSH use Yarn commands to list and kill applications

I am used to killing Yarn applications like this, over a BASH session connected with the ssh account credentials. From SSH paste in the application name and this command should attempt the kill

yarn application -kill application_000000000000000_0001

https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html To find the application ID , you can do yarn application -list or use the resource manager UI from the active head node. From Ambari https://servername.azurehdinsight.net/#/main/dashboard/metrics click Dashbaord, then YARN links tile, click More… > and pick the Active head node, then click the menu for ResourceManagerUI It will jump to the website like this URL https://servername.azurehdinsight.net/yarnui/hn/cluster The first column lists the application ID, which is what Yarn needs to know to kill something.

How to Lock a Resource Group to prevent accidental deletion of resources like HDInsight

JasonHowell — Mon, 16 May 2016 13:43:47 GMT

Did you know it is possible to prevent accidental deletion of resources in Azure? This could apply to any number of resource, HDInsight, Stream Analytics jobs, Data Factories, DocumentDB accounts, etc. We can add a lock to the resource group to prevent resources from being removed inadvertantly. I found out the hard way when someone here tried to delete their HDInsight and it failed. The delete request fails with HTTP status code 409 CONFLICT which indicates the Resource Group is locked.

DELETE/SUBSCRIPTIONS/RESOURCEGROUPS/PROVIDERS/MICROSOFT.HDINSIGHT/CLUSTERS/ Http request failed with ErrorResponseMessageException. HttpStatus: 'Conflict', ErrorCode: 'ScopeLocked' and ErrorMessage: 'The scope '/subscriptions/subscriptionid/resourceGroups/HDInsightRG/providers/Microsoft.HDInsight/clusters/hdinsightclustername' cannot perform delete operation because following scope(s) are locked: '/subscriptions/subscriptionid/resourceGroups/HDInsightRG'. Please remove the lock and try again.'.

References

New-AzureRmResourceLock https://msdn.microsoft.com/en-us/library/mt652515.aspx
Get-AzureRmResourceLock https://msdn.microsoft.com/en-us/library/mt652497.aspx
Remove-AzureRmResourceLock https://msdn.microsoft.com/en-us/library/mt652493.aspx

Here's how to make a lock in PowerShell and prevent resources from being delete.

Perequisite: Download the PowerShell SDK for Azure from here https://azure.microsoft.com/en-us/downloads/ Start > Run > PowerShell ISE I like PowerShell ISE because you can highlight one line at a time, and run it one by one.

Example Code

# 1. Login

# 2. Make a new lock

New-AzureRmResourceLock -LockLevel CanNotDelete -LockName LockHDInsightRG -ResourceGroup HDInsightRG

# 3. List existing locks and their names – it will prompt Yes/No unless you add -Force switch.

Get-AzureRmResourceLock

Name : LockHDInsightRG ResourceId : /subscriptions/subscriptionid/resourceGroups/HDInsightRG/providers/Microsoft .Authorization/locks/LockHDInsightRG ResourceName : LockHDInsightRG ResourceType : Microsoft.Authorization/locks ResourceGroupName : HDInsightRG SubscriptionId : subscriptionid Properties : @{Level=CanNotDelete} LockId : /subscriptions/subscriptionid/resourceGroups/HDInsightRG/providers/Microsoft.Authorization/locks/LockHDInsightRG

# 4. Try to delete an HDInsight cluster – it will Error out

Remove-AzureRmHDInsightCluster -ClusterName MyCluster

Remove-AzureRmHDInsightCluster -ClusterName MyCluster -ResourceGroupName HDInsightRG Remove-AzureRmHDInsightCluster : ScopeLocked: The scope '/subscriptions/subscriptionid/resourceGroups/HDInsightRG/providers/Microsoft.HDInsight/clusters/MyCluster' cannot perform delete operation because following scope(s) are locked: '/subscriptions/subscriptionid/resourceGroups/HDInsightRG'. Please remove the lock and try again. At line:1 char:1 + Remove-AzureRmHDInsightCluster -ClusterName MyCluster -ResourceGroup ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : CloseError: (:) [Remove-AzureRmHDInsightCluster], CloudException + FullyQualifiedErrorId : Microsoft.Azure.Commands.HDInsight.RemoveAzureHDInsightCommand

# 5. Remove the lock by name - it will prompt Yes/No unless you add -Force switch.

Remove-AzureRmResourceLock -LockName LockHDInsightRG -ResourceGroup HDInsightRG

Remove-AzureRmResourceLock -LockName LockHDInsightRG -ResourceGroup HDInsightRG True

# 6. List existing locks and their names – no results

Get-AzureRmResourceLock

HDInsight Name Node can stay in Safe mode after a Scale Down

JasonHowell — Wed, 16 Mar 2016 21:37:23 GMT

This week we worked on an HDInsight cluster where the Name Node has gone into Safe mode and didn't leave that mode on its own. It's not very common, but I wanted to share why it happened, and how to get out of the situation, in case it prevents a headache for someone else.

HDInsight Scaling up & down is so useful

HDInsight offers elasticity by giving administrators the option to scale up and scale down the number of Worker Nodes in the clusters. This allows a cluster to be shrunk during after hours or on weekends, and grown during peak business demands. Let's say there is some batch processing that happens one day a week or once a month, the HDInsight cluster can be scaled up a few minutes prior to that scheduled event so that there is plenty of Memory and CPU compute power. You can automate that with PowerShell cmdlet Set-AzureRmHDInsightClusterSize. Later, after the processing is done, and usage is expected to go down again, the administrator can scale down the HDInsight cluster to fewer worker nodes. From the Azure Portal, find your HDInsight, click the Scale Cluster setting, then on the Scale blade, you just type in the number of worker nodes, and click save, and in a few minutes your HDInsight cluster will have been resized. You can go up or down. In a few occurrences, we have had HDInsight customers shrink their cluster down to the bare minimum 1 worker node as shown above. That works most of the time for them if you start at 1 worker node. But once in a while, scaling from a bunch of nodes all the way down to 1 worker node caused HDFS to get stuck in safe mode during maintenance when they are not expecting it. Maybe when worker nodes are rebooted due to patching, or right after the scale, safe mode can occur during that maintenance timeframe.

The Short Story – If you get in Safe mode, you can manually Leave Safe Mode to ignore the issue (most of the time)

The gist is that Hive uses some scratchdir files, and by default expects 3 replicas of each block, but there is only 1 replica possible if you scale down to the minimum 1 worker node, so the files in the scratchdir are under replicated. HDFS stays in Safe mode because of that at the time when the services are restarted after the scale maintenance. One way to get HDFS out of safe mode is to just leave safe mode (ignoring the cause assuming its benign). For example, if you know that the only reason safe mode is on is that the temporary files are under replicated (discussed in detail below) then you really don't need those files anyway because they are hive temporary scratch files, so it's reasonable to leave Safe Mode "safely" without a worry. HDInsight on Linux: hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode leave ** note that the tick marks/apostrophes are slanted due to blog formatting. Please use plain quote/tick mark. HDInsight on Windows: hdfs dfsadmin -fs hdfs://headnodehost:9000 -safemode leave After leaving safe mode, that you can remove the problematic temp files or wait for Hive to clean them up if you are optimistic.

The Details

Even in HDInsight, the default HDFS configuration is set to require least 3 worker nodes to a copy of each file's blocks to be considered healthy. That is to say HDFS is configured with the dfs.replication setting of 3, so the blocks of those files are under replicated whenever there are less than 3 worker nodes online, because there are not 3 copies of each file block available. We know from working with HortonWorks Support from time to time, that in an Hadoop in a on-premise environment where disks can get full, disks can fail, and networking errors between worker nodes can happen, it is common to see Safe mode from time to time during a catastrophie. But in the Azure cloud, most of those things are not going to happen, so unexpected Safe mode is probably not a hardware problem going on. Other folks have blogged about it Examples of errors we saw when running Hive queries after the safe mode was turned on:

ERROR 1

H070 Unable to open Hive session. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RetriableException): org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive/hive/819c215c-6d87-4311-97c8-4f0b9d2adcf0. Name node is in safe mode. The reported blocks 75 needs additional 12 blocks to reach the threshold 0.9900 of total blocks 87. The number of live datanodes 10 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

ERROR 2

H100 Unable to submit statement show databases': org.apache.thrift.transport.TTransportException: org.apache.http.conn.HttpHostConnectException: Connect to hn0-clustername.servername.internal.cloudapp.net:10001 [hn0-clustername.servername. internal.cloudapp.net/1.1.1.1] failed: Connection refused

ERROR 3

H020 Could not establish connecton to hn0-hdisrv.servername.bx.internal.cloudapp.net:10001: org.apache.thrift.transport.TTransportException: Could not create http connection to https://hn0-hdisrv.servername.bx.internal.cloudapp.net:10001/. org.apache.http.conn.HttpHostConnectException: Connect to hn0-hdisrv.servername.bx.internal.cloudapp.net:10001 [hn0-hdisrv.servername.bx.internal.cloudapp.net/10.0.0.28] failed: Connection refused: org.apache.thrift.transport.TTransportException: Could not create http connection to https://hn0-hdisrv.servername.bx.internal.cloudapp.net:10001/. org.apache.http.conn.HttpHostConnectException: Connect to hn0-hdisrv.servername.bx.internal.cloudapp.net:10001 [hn0-hdisrv.servername.bx.internal.cloudapp.net/10.0.0.28] failed: Connection refused

ERROR 4 - from the Hive logs

WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(442)) - Error starting HiveServer2 on attempt 21, will retry in 60 seconds java.lang.RuntimeException: Error applying authorization policy on hive configuration: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RetriableException): org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive/hive/70a42b8a-9437-466e-acbe-da90b1614374. Name node is in safe mode. The reported blocks 0 needs additional 9 blocks to reach the threshold 0.9900 of total blocks 9. The number of live datanodes 10 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1324)

You can review the Name Node logs from folder /var/log/hadoop/hdfs/ near the time when the cluster was scaled to see when it got stuck in safe mode. Log files like Hadoop-hdfs-namenode-hn0-clustername.* The gist of these errors and this this problems is that even in HDInsight where HDFS is rarely used, Hive does depend on temporary files in HDFS while running queries, and HDFS has gone into "Safe mode". Hive cannot run queries if it cannot write to HDFS. Those temp files in HDFS are kept in the local C: drive mounted to the individual worker node VMs and replicated amongst the worker nodes 3 replicas minumum. Check out how the hive.exec.scratchdir in Hive is configured in HDInsight (on Linux) "/etc/hive/conf/hive-site.xml"

How is this Safe mode issue related to Scale down?

When a scale down attempt happens, HDInsight relies upon the Ambari management interfaces to decommission the extra unwanted worker nodes, which replicates the HDFS blocks to other online worker nodes, and safely scale the cluster down. HDFS goes into a safe mode during the maintenance window, and is supposed to come out once the scaling is finished, normally. However, when you scale to 1 node, there is no longer 3 worker nodes available to host 3 replicas of the HDFS blocks, so the file blocks that were replicated amongst 3 worker nodes or more are now only in 1 worker node, and HDFS considers those blocks "under replicated". Recall that in Azure HDInsight, we tell you not to keep user data in the HDFS file system, and instead put your user data in Azure Blob Storage using wasb:// URI or in Azure Data Lake Store using adl:// URI paths. That azure stored data is accessible to the various Hadoop components much like HDFS without any noticeable difference, maybe even faster than HDFS, and it lives in Azure Storage within the data center, and not on the locally attached disks in the compute VMs. This model of storage in the cloud gives you flexibility to remove HDInsight clusters, and recreate them on demand later, while safely keep your data in tact and stored at a low cost. Its typically only the Hive Scratchdir files that are remaining in HDFS, unless one of the users accidentally saved user data into hdfs:// instead of the normal Azure stores: wasb:// or adl:// You can inventory your HDFS file system by do a list from the console to see if it is the system temporary files, or some user data: hadoop fs -ls -R hdfs://mycluster/

How to see the health and state of your HDFS file system?

You can get a report of status from each name node to show that the name nodes are in Safe Mode or not. SSH into each head node to run this kind of command.

hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode get

Next you can get a report that shows the details of HDFS state when all is healthy and all blocks are replicated to the expected degree.

hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -report

Side bar: The reason we have to give -D switch, is that on HDInsight the default file system is Azure Blob Storage, so you have to specify the local HDFS file system in the commands like this

Let's compare a couple of example output from of a healthy hdfs file system when in and not in safe mode.

hdfs fsck -D 'fs.default.name=hdfs://mycluster/' /tmp/hive/hive

1. Healthy HDFS and no under replicated blocks:

Connecting to namenode via https://hn0-clustername.servername.bx.internal.cloudapp.net:30070 FSCK started by sshuser (auth:SIMPLE) from /10.0.0.14 for path /tmp/hive/hive at Wed Mar 16 23:49:30 UTC 2016 Status: HEALTHY Total size: 0 B Total dirs: 13 Total files: 0 Total symlinks: 0 Total blocks (validated): 0 Minimally replicated blocks: 0 Over-replicated blocks: 0

Under-replicated blocks: 0 Mis-replicated blocks: 0 Default replication factor: 3 Average block replication: 0.0 Corrupt blocks: 0 Missing replicas: 0 Number of data-nodes: 1 Number of racks: 1 FSCK ended at Wed Mar 16 23:49:30 UTC 2016 in 5 milliseconds

2. Healthy HDFS and some under replicated blocks:

There is only one block on this file in this output, but it is under replicated, only exists on 1 worker node, and not 3 as expected by HDFS.

Connecting to namenode via https://hn0-servername. internal.cloudapp.net:30070 FSCK started by sshuser (auth:SIMPLE) from /10.0.0.14 for path /tmp/hive/hive at Thu Dec 31 19:28:03 UTC 2015 . /tmp/hive/hive/_tez_session_dir/62b64823-4d61-48b0-9f36-82bc60ba368f/hive-hcatalog-core.jar: Under replicated BP-2018565019-10.0.0.15-1450760688291:blk_1073741828_1004. Target Replicas is 3 but found 1 replica(s). Status: HEALTHY Total size: 253448 B Total dirs: 31 Total files: 1 Total symlinks: 0 Total blocks (validated): 1 (avg. block size 253448 B)

Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 1 (100.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 2 (66.666664 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Thu Dec 31 19:28:03 UTC 2015 in 8 milliseconds

3. Ambari shows this as well.

Click the HDInsight Dashboard link from the Azure portal, and it will launch the Ambari web site. Ambari will report the block health as well, but is not as detailed as the above commands. https://clustername.azurehdinsight.net/#/main/services/HDFS/summary

How to Prevent HDInsight from getting stuck in Safe mode due to under-replicated blocks

A little foresight can prevent HDInsight being left in Safe mode. A few options are:

Let's go through these possibilities in more detail:

A> Queisce Hive – Stop all jobs

Stop all Hive jobs before scaling down to 1 worker node. If you know your workload is scheduled, then do your Scale Down after Hive work is done. That will help minimize the number of scratch files in the tmp folder (if any).

B> Manually clean up behind Hive by removing scratch files

If Hive has left behind temporary files, before you scale down, then you can manually cleanup the tmp files before scaling down to avoid safe mode. You should stop Hive services to be safe, and at a minimum make sure all queries and jobs are done. You can list the directory hdfs://mycluster/tmp/hive/ to see if it has any files left or not. hadoop fs -ls -R hdfs://mycluster/tmp/hive/hive

sshuser@hn0-servername:~$ hadoop fs -ls -R hdfs://mycluster/tmp/hive/hive drwx------ - hive hdfs 0 2016-03-14 00:36 hdfs://mycluster/tmp/hive/hive/37ebc52b-d87f-4c3d-bb42-94ddf7c349dc drwx------ - hive hdfs 0 2016-03-14 00:36 hdfs://mycluster/tmp/hive/hive/37ebc52b-d87f-4c3d-bb42-94ddf7c349dc/_tmp_space.db drwx------ - hive hdfs 0 2016-03-14 00:05 hdfs://mycluster/tmp/hive/hive/85122ce9-bc22-4ad2-9c34-d497492523bd drwx------ - hive hdfs 0 2016-03-14 00:05 hdfs://mycluster/tmp/hive/hive/85122ce9-bc22-4ad2-9c34-d497492523bd/_tmp_space.db

If you know Hive is done with these files, you can remove them. You can stop Hive services to be extra safe. Note - this can break Hive if some jobs are still running, so don't do this unless all Hive jobs are done. Be sure that Hive doesn't have any queries running by looking in the Yarn Resource Manager page before doing so. Example command line to remove files from HDFS: hadoop fs -rm -r -skipTrash hdfs://mycluster/tmp/hive/

C> Avoid going too low – Three's company, One's a lonely number

If you can't do any of the prior approaches, then you may want to avoid the problem by only scaling down to 3 worker nodes minimum. Don't go lower than that. It will cost some compute hours / money, so its nontrivial I know. With 1 worker node, HDFS can't guarantee 3 replicas of the data will be available. Having 3 workers node is the safest minimum for HDFS with the default replication factor.

D> Fiddle with dfs.replication settings – advanced topic

You could reduce the dfs.replication factor to 1 to prevent the under replicated scenario when using 1 worker node. This is overkill in my opinion, and an advanced option, and has consequences when other regular maintenance is done, so you need to test more thoroughly. If you keep real data (not just the Hive temporary files) in HDFS, you likely do not want to do this!! You could change this in Ambari on the HDFS configuration if you only use HDFS for temp files as recommended in HDInsight.

E> Else plan to leave Safe Mode regularly if it happens from time to time

If you can't prevent it, then if it gets in Safe mode, then tell it to leave safe mode. Once you prove the reason is just Hive temporary files are under replicated, then leave Safe mode without worry. Linux: hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode leave Windows: hdfs dfsadmin -fs hdfs://headnodehost:9000 -safemode leave

That's all folks. Let us know if this post helps you one day. I'm all ears. Regards, Jason Howell (Microsoft Azure Big Data Analytics Support team)

HDInsight Hive Metastore fails when the database name has dashes or hyphens

JasonHowell — Wed, 24 Feb 2016 13:36:00 GMT

Working in Azure HDInsight support today, we see a failure when trying to run a Hive query on a freshly created HDInsight cluster. Its brand new and fails on the first try, so what could be wrong? Our Hive client app fails with this kind of error.

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:445) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:619) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1483) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:63) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2743) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2762) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:426) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1481) ... 13 more Caused by: javax.jdo.JDOUserException: Could not create "increment"/"table" value-generation container meta-store-database.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it. NestedThrowables: org.datanucleus.exceptions.NucleusUserException: Could not create "increment"/"table" value-generation container meta-store-database.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it. at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:549) at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732) at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752) at org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:499) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98) at com.sun.proxy.$Proxy9.createDatabase(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:578) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:598) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:436) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:66) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5509) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:178) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74) ... 18 more Caused by: org.datanucleus.exceptions.NucleusUserException: Could not create "increment"/"table" value-generation container meta-store-database.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it. at org.datanucleus.store.rdbms.valuegenerator.TableGenerator.createRepository(TableGenerator.java:261) at org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obtainGenerationBlock(AbstractRDBMSGenerator.java:162) at org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerationBlock(AbstractGenerator.java:197) at org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractGenerator.java:105) at org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGenerator(RDBMSStoreManager.java:2005) at org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1386) at org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3827) at org.datanucleus.state.JDOStateManager.setIdentity(JDOStateManager.java:2571) at org.datanucleus.state.JDOStateManager.initialiseForPersistentNew(JDOStateManager.java:513) at org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:232) at org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentNew(ExecutionContextImpl.java:1414) at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2218) at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065) at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913) at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217) at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727) ... 34 more

We found out the problem is very simple. The Hive metastore database in HDInsight usually lives in an Azure SQL Database, and the name of that database can be customized. In this case it was a name "hive-meta-store" database name. The error gives us a hint, but the error isn't clear about what caused the failure. Caused by: org.datanucleus.exceptions.NucleusUserException: Could not create "increment"/"table" value-generation container meta-store-database.dbo.SEQUENCE_TABLE since autoCreate flags do not allow it. Apparently in HiveServer2 there is a bug where if the metastore database name is not always are escaped by tick marks 'hive-meta-store' or brackets [hive-meta-store] so instead, Hive gets an incorrect syntax error. On such clusters, the Hiveserver2 wouldn’t start. HiveServer2 wouldn’t start because of table “meta-store-database.dbo.SEQUENCE_TABLE” table

2016-01-16 02:14:10,228 DEBUG [main]: Datastore.Native (Log4JLogger.java:debug(58)) - SELECT NEXT_VAL FROM meta-store-database.SEQUENCE_TABLE WHERE SEQUENCE_NAME=<'org.apache.hadoop.hive.metastore.model.MDatabase'> 2016-01-16 02:14:10,233 INFO [main]: DataNucleus.ValueGeneration (Log4JLogger.java:info(77)) - Error encountered allocating block of IDs : Couldnt obtain a new sequence (unique id) : Incorrect syntax near '-'.

The known issue

The issue is being tracked here https://issues.apache.org/jira/browse/HIVE-6113 and in Hortonworks' internal Jira item HIVE-433. Hortonworks is aware of the issue, and it already fixed in a later build as per reading HIVE-6113 has already upgraded the datanucleus version to 4.x. This upgrade will be available as part of Hive 2.0, which is not out yet in Azure HDInsight. Until that is done, avoid dashes or hyphens in your custom metastore, else HiveServer2 will not start.

How to call a Azure Machine Learning Web Service from NodeJS

carrollwp — Thu, 18 Feb 2016 08:32:00 GMT

Azure machine learning allows data scientists and developers to embed predictive analytics into applications. To learn more about Azure machine learning visit Azure machine learning documentation . A simplified process flow for Azure machine learning is:

Create an Azure machine learning workspace that has an associated Azure storage account.
Login to your Azure machine learning workspace and then login into ML Studio. ML Studio is where you can create your machine learning experiments and publish them as a web services. You need to create an experiment which produces the machine learning model, and then publish it as a web service.
Once you publish your web service, the portal has a page with information on how to call the web service and example code in C#, Python, and R.
Calling the web service allows you to embed your predictive analytics into applications. You will pass new input feature data to the web service. The new feature data will be run through the machine learning model and the web service will send back the prediction.

Recently I had a request on how to call an Azure machine learning web service from NodeJS. The portal did not have an example. In this post I will show the basic code in NodeJS to call an Azure machine learning web service. Below is a simple experiment I created that tries to predict the percent gain ten days from now for the SP 500 Index. At the bottom is a button to set up your web service once you are comfortable with the model. Below is the web service predictive experiment page after you have published it. Below is the information for the published web service. You will need information from this page to call the web service from NodeJS. One piece of information that you will need is the API key. You can also test out your web service here. Under the REQUEST/RESPONSE link is more information that you will need like the POST Request URI. Below you will see the post request URI. You will need this. Also on this page are code examples in C#, Python and R. Now that we have created an experiment published it as a web service and have the Post Request URI and API Key we can start to write the code in NodeJS to call the web service and get a prediction. I am assuming you are somewhat familiar with NodeJS and have downloaded it and installed it. Here is the download link Download NodeJS and web site. You can create the maml-server.js file in notepad or your favorite IDE. From the POST Request URI you can get the information for the host and path variables. The headers will need the API key. The options variable has this information and is created in the getPred function. In the buildFeatureInput function, I just hard coded the new feature data, but you could easily get this information from a web form or a RDMS call. I wanted to keep the example simple. //maml-server.js var http = require("http"); var https = require("https"); var querystring = require("querystring"); var fs = require('fs'); function getPred(data) { console.log('===getPred()==='); var dataString = JSON.stringify(data) var host = 'ussouthcentral.services.azureml.net' var path = '/workspaces/fda91d2e52b74ee2ae68b1aac4dba8b9/services/1b2f5e6f99574756a8fde751def19a0a/execute?api-version=2.0&details=true' var method = 'POST' var api_key = 'vKKR78dSdQeSc9qdMaDmu2Z5bcFqb4TfkZdNgSxzcIjGV9p5OP2uy4k1HfJes1T4Ws3St+EBgQTX/N8vqCs4zg==' var headers = {'Content-Type':'application/json', 'Authorization':'Bearer ' + api_key}; var options = { host: host, port: 443, path: path, method: 'POST', headers: headers }; console.log('data: ' + data); console.log('method: ' + method); console.log('api_key: ' + api_key); console.log('headers: ' + headers); console.log('options: ' + options); var reqPost = https.request(options, function (res) { console.log('===reqPost()==='); console.log('StatusCode: ', res.statusCode); console.log('headers: ', res.headers); res.on('data', function(d) { process.stdout.write(d); }); }); // Would need more parsing out of prediction from the result reqPost.write(dataString); reqPost.end(); reqPost.on('error', function(e){ console.error(e); }); } //Could build feature inputs from web form or RDMS. This is the new data that needs to be passed to the web service. function buildFeatureInput(){ console.log('===performRequest()==='); var data = { "Inputs": { "input1": { "ColumnNames": ["gl10", "roc20", "uo", "ppo", "ppos", "macd", "macds", "sstok", "sstod", "pmo", "pmos", "wmpr"], "Values": [ [ "0", "-1.3351", "50.2268", "-0.2693", "-0.2831", "-5.5310", "-5.8120", "61.9220", "45.3998", "-0.0653", "-0.0659", "-30.3005" ], ] }, }, "GlobalParameters": {} } getPred(data); } function send404Reponse(response) { response.writeHead(404, {"Context-Type": "text/plain"}); response.write("Error 404: Page not Found!"); response.end(); } function onRequest(request, response) { if(request.method == 'GET' && request.url == '/' ){ response.writeHead(200, {"Context-Type": "text/plain"}); fs.createReadStream("./index.html").pipe(response); }else { send404Reponse(response); } } http.createServer(onRequest).listen(8050); console.log("Server is now running on port 8050"); buildFeatureInput(); Once you have saved the code to disk you can run it in NodeJS by issuing, "Node ". I am also displaying the status code from the POST and the results from calling the web service. You would need to parse out the JSON to get the "Scored Labels" column that has the prediction. The model is predicting that the SP 500 will be down 4.71% ten days from now. The SP 500 closed yesterday (2016-02-17) at 1926.82. We can check back on 2016-03-04 and see how well the model did! Machine learning is a big topic but Azure machine learning makes it easy to embed analytics in your applications. I hope this helps some one. I will have changed my API key, so if you try to run the example you might get an authorization error. Bill

Encoding 101 - Exporting from SQL Server into flat files, to create a Hive external table

JasonHowell — Fri, 05 Feb 2016 16:33:00 GMT

Today in Microsoft Big Data Support we faced the issue of how to correctly move Unicode data from SQL Server into Hive via flat text files. The main issue faced was encoding special Unicode characters from the source database, such as the degree sign (Unicode 00B0) and other complex Unicode characters outside of A-Z 0-9. The goal was to get Hive to read those same strings SQL Server had saved out to text files and represent them equally to the Hive consumer. We could have used Sqoop if there was a connection between Hadoop and the SQL Server, but that was not possible, as this was across company boundaries and shipping files was the easier approach. It was tricky to do, but we found a couple of solutions. When the Unicode string value is exported from Microsoft SQL Server via SSIS or the Import Export wizard, they look fine to the naked eye, but SELECT * FROM HiveTable; and the data looks different. Microsoft SQL Server: zyx°°° Looks good Hive: zyx�� Uh-oh! We've got trouble. That Unicode string (NVARCHAR) value in SQL appears as zyx°°° We Export that data to a flat file using SSIS or the Import Export Wizard, then copy the files into Azure Blob Storage. Next using Azure HDInsight, when a Hive table is created atop those files, then the same characters look garbled black question marked - zyx�� as if the characters are unknown to Hive's interpretation. Linux and Hive default to text files encoded to UTF-8 format. That differs from the SSIS Flat File Destination's Unicode output. I found two ways we found to make them compatible.

Change the export options in SQL Server SSIS Flat File Destination to uncheck the "Unicode" checkmark and select code page 65001 (UTF-8) instead.
Keep the Unicode encoding as is in SSIS, but tell Hive to interpret the data differently using serdeproperties ('serialization.encoding'='ISO-8859-1');

Time for a little trial and error

1. Setup an example Database

These are the steps I used to see the issue. You can use SQL Server (any version from 2005-2016). Using SQL Management Studio, I create a database TestHive and table T1 in SQL Server. Insert some data into the NVarchar column, including some special characters. NChar and NVarChar and NVarChar(max) are the double-byte Unicode data type in SQL Server columns, that are used for global language support.

CREATE DATABASE TestHive GO USE TestHive GO CREATE TABLE T1(Col1 Int, Col2 NVarchar(255)); GO INSERT INTO T1 VALUES (1,N'abcdef'), (2,N'zyx°°°'), (3,N'123456' ) GO SELECT * FROM T1

2. Run the SQL Server Import Export Wizard from the Start menu to copy the rows into a text file.

On the Source – point to your SQL Server instance, and select the database and table you want to pull from. On the Destination – choose Flat File Destination, point to a file path on the local disk, and Select the Unicode checkbox on the locale. You could use the same Flat File Destination from within an Integration Services (SSIS) package design and run that if you prefer more control on how the data is transformed in the Data Flow Task, but the Import Export Wizard does the simple copy that we need here. This is the output – so far looks good.

3. Connect to my Azure HDInsight cluster in the cloud to upload the file.

I used a Linux based HDInsight Hadoop cluster, so I will use SSH to connect to the head node. Then I create a directory, and upload the first file into that location in blob store. This is much like saving the file from the local disk into HDFS for the purposes of Hadoop outside of Azure. hadoop fs -mkdir -p /tutorials/usehive/import1/ hadoop fs -copyFromLocal tableexport_ssisunicode.txt wasb:///tutorials/usehive/import1/tableexport_ssisunicode.txt You could also upload the files directly to Azure Blob Storage from tools such as Visual Studio Azure VS 2015 / VS 2013 or Microsoft Azure Storage Explorer / Azure Storage Explorer or Cloud Explorer for Visual Studio 2013

4. Run Hive to read that first folder

Now that the text file is ready in Blob storage, I can run Hive and create a table, and query from that file saved into the import1 folder. From my SSH session I simply run hive. cd /bin hive {

DROP TABLE Import1;

CREATE EXTERNAL TABLE Import1(col1 string, col2 string) ROW FORMAT DELIMITED

FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/tutorials/usehive/import1/' tblproperties ("skip.header.line.count"="1");

SELECT * FROM Import1;

} Notice the strange characters �� where I expected my degree signs. Just to be sure it's not something special with my SSH client (MobaXTerm here) I am trying from the Ambari web dashboard for my Azure HDInsight Cluster, and using the Hive View from the menu icon in the upper right.

5. OK let's try exporting again – this time changing the flat file encoding setting.

In the SSIS Import Export Wizard (SSIS Flat File Destination) choose code page UTF-8. - Uncheck the "Unicode" checkmark. - Choose code page 65001 (UTF-8)

6. Upload that second file to Linux (SFTP) and then copy into HDFS or Azure Blob Storage.

I made a new folder, so I could compare my trials side-by-side. hadoop fs -mkdir -p /tutorials/usehive/import2/ hadoop fs -copyFromLocal tableexport_ssisunicode2.txt wasb:///tutorials/usehive/import2/tableexport_ssisunicode2.txt

7. Now test the Hive table again with the UTF-8 encoded file in the second folder

Run Hive cd /bin hive { DROP TABLE Import2; CREATE EXTERNAL TABLE Import2(col1 string, col2 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/tutorials/usehive/import2/' tblproperties ("skip.header.line.count"="1"); SELECT * FROM Import2; QUIT; } My zyx°°° looks normal now! Success! The Hive View in Ambari likes this data too.

8. An alternative- Tell hive to encode / decode the external files differently

Perhaps you don't want to change the file format to UTF-8 (most universal in Linux and Hadoop), or maybe you cannot change the format at all, because the files come from an outsider. Starting in Hive 0.14 version and above, Hive has a simple way to change the encoding of serialization (for interpreting the bytes encoding in text files for example). The change was explained here https://issues.apache.org/jira/browse/HIVE-7142 1. You can Create from scratch with this code page serializer and override the serialization.encoding property to a code page that best matches your source data encoding. CREATE TABLE person(id INT, name STRING, desc STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES("serialization.encoding"='ISO-8859-1'); 2. Or if you have an existing table, this can be adjusted after the fact. You need to carefully match your encoding to whatever kind of files will be presented in the storage underneath this Hive table. Choose one, or make your own… ALTER TABLE Import1 SET serdeproperties ('serialization.encoding'='US-ASCII'); ALTER TABLE Import1 SET serdeproperties ('serialization.encoding'='ISO-8859-1'); ALTER TABLE Import1 SET serdeproperties ('serialization.encoding'='UTF-8'); To find out which tokens to list next to the equal sign, refer to the Charset code ID is listed in Java documentation https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html I am guessing the right one is this one to match SQL's "Unicode" but it needs to be tested further to be totally sure that ALL characters are interpreted as expected.

ISO-8859-1

ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1

Other tips we didn't have time to try yet:

Someone made a tool to help convert files if it is not possible for them to change the format in SSIS or BCP exports to text files. https://code.msdn.microsoft.com/windowsdesktop/UTF8WithoutBOM-Converter-7a8218af

Hope this helps someone out there. Let us know if it does, or if you still get stuck, post a comment below, or try the Azure forums for help.

Happy Hadooping! Jason

Encoding the Hive query file in Azure HDInsight

JasonHowell — Fri, 05 Feb 2016 14:33:00 GMT

Today at Microsoft we were using Azure Data Factory to run Hive Activities in Azure HDInsight on a schedule. Things were working fine for a while, but then we got an error that was hard to understand. I've simplified the scenario to illustrate the key points. The key is that Hive did not like the Byte Order Mark (first 3 bytes) in the hive .hql file, and failed with an error. Be careful which text editor and text encoding you choose when saving your Hive Query Language (HQL) command into a text file. We are using Azure Data Factory (Feb 2016) with a linked service to Microsoft Azure HDInsight Hadoop on Linux (version 3.2.1000.0) which is the distribution from Hortonworks Data Platform version 2.2.7.1-36 and includes Hive 0.14.0. Error message:

Hive script failed with exit code '64'. See 'wasb://adfjobs@mycontainer.blob.core.windows.net/HiveQueryJobs/92994500-5dc2-4ba3-adb8-4fac51c4d959/05_02_2016_05_49_34_516/Status/stderr' for more details.

To drill into that stderror logging, In the Data Factory activity blade, we see the error text, and see the log files below. We click on the stderr file and read it in the log blade. Stderr says:

WARNING: Use "yarn jar" to launch YARN applications.

Logging initialized using configuration in jar:file:/mnt/resource/hadoop/yarn/local/filecache/11/hive.tar.gz/hive/lib/hive-common-1.2.1.2.3.3.1-1.jar!/hive-log4j.properties

FAILED: ParseException line 1:0 character '' not supported here

That's strange, because the Hive query file looks fine to the naked eye. If you use a binary editor though you can see these two text files are not the same. I have Visual Studio handy, and it can show the hex representation of these files. File > Open > File… Next to the open button there is a drop down – choose "Open with…" Choose Binary Editor. Notice that file2.hql shown below has 3 extra bytes in the beginning. This is called a byte-order mark (BOM).

The Standards Controversy

There is some controversy about whether UTF-8 text files should have a byte order mark or not. The standards folks say don't use it in most cases, but also don't remove it if you already have a BOM. We software developers sometimes forget that and are inconsistent -Microsoft Windows vs. Linux vs. Apache Hive Citing the crowdsourced Wikipedia information here for neutrality: https://en.wikipedia.org/wiki/Byte_order_mark

UTF-8 byte order mark is EF BB BF in hex.

The Unicode Standard permits the BOM in UTF-8,^[2] but does not require or recommend its use.

The standard also does not recommend removing a BOM when it is there, so that round-tripping between encodings does not lose information, and so that code that relies on it continues to work.

In Linux distributions since around 2004 the convention is to generate UTF-8 files without a BOM.

Microsoft compilers^[10] and interpreters, and many pieces of software on Microsoft Windows such as Notepad treat the BOM as a required magic number rather than use heuristics. These tools add a BOM when saving text as UTF-8, and interpret as UTF-8 on reading only when the BOM is present.

The Litmus Test

So let's try the two .hql text files in Hive console directly outside of Azure Data Factory to see if they work in Hive alone. I am using SSH to connect to my Linux head node in HDInsight. I have uploaded the two files using SFTP (easy to do in MobaXTerm if you like that 3^rd party tool – I can't endorse any specific tool, but seems nice). The source command here is running the query file from within the hive app. 1. First I launch Hive on my SSH session.

Cd /bin

Hive

2. Then I run the script from the first file. It works well and returns a list of my Hive tables from my metastore.

source /home/sshuser/file1.hql;

OK

3. Then I run the script from the second file. It fails with a parsing error

source /home/sshuser/file2.hql;

FAILED: ParseException line 1:0 character '' not supported here

The lesson learned

So the lesson learned is that Hive doesn't like the .hql text files when they are encoded in UTF-8 with a Byte-order mark up front in the hex/binary. UTF-8 is the encoding of choice, but the byte-order mark is not desired.

Action Required!

Check your favorite text editor to see if it is marking the BOM or not.

1. Windows Notepad seems to encode UTF-8 files to include a Byte Order Mark, so avoid that one.

2. Visual Studio 2013 / 2015 gives you the option to save "With Encoding" on the drop down beside the Save button, then you can pick which Encoding you want.

First trial - I picked Unicode (UTF-8 with signature) – Codepage 65001. That "signature" means it adds a BOM, which is undesirable for Hive to interpret. Second trial – I picked Unicode (UTF-8 without signature) - Codepage 65001. Looks better now. That's the one without the BOM in the first three bytes, so Hive should be OK to read this one.

3. Linux's common Vi text editor

The hex representation does not have a BOM, so we are good to use this with Hive. Hive can run this one OK

4. Nano – another easy text editor on Linux

Looks good as well, no BOM Hive can run this one OK too. To be continued... There are many more text editors, so please let me know if you have trouble with encoding using another text editor when saving Hive queries to text.

Hope this helps someone out there. Let us know if it does, or if you still get stuck, post a comment below, or try the Azure forums for help.