The way to do this is to pass “enablelogs” as a query string parameter set to true.  Here’s the what the request looks like:

Upon job completion, the logs will be moved into the status directory, under a logs folder with the following structure:

$log_root/list.xml (summary of jobs)
$log_root/stderr (frontend stderr)
$log_root/stdout (frontend stdout)
$log_root/$job_id (directory home for a job)
$log_root/$job_id/job.xml.html
$log_root/$job_id/$attempt_id (directory home for a attempt)
$log_root/$job_id/$attempt_id/stderr
$log_root/$job_id/$attempt_id/stdout
$log_root/$job_id/$attempt_id/syslog

Here’s a screen shot from Storage Studio that shows the folder structure:

If you look in the syslog file here, you’ll see a bunch of goodness about your job execution.  For more complex jobs that spin off multiple map/reduce jobs (eg, Pig, hive, Cascading), you will see the set of jobs recorded there. The root directory will also contain a list.txt with the details of all the jobs, and each jbo will contain a jobs.xml.html which contains all of the details, environment variables, and configuration of the job.  All of this information is useful in debugging and tuning your jobs.

We will be updating the SDK to support this parameter in the next release, but for now, you can submit jobs directly to the cluster and add this parameter to get job logs.

Here’s the text from my execution of a simple Hive query:

2013-07-11 02:48:56,632 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2013-07-11 02:48:56,726 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: c:/hdfs/mapred/local/taskTracker/distcache/1562199005048822745_1328179698_1268869633/namenodehost/hive/scratch/hive_2013-07-11_02-48-38_757_5240085401495207012/-mr-10003/34a2bf31-e18b-440d-8d91-be8d0e445d2e <- c:\hdfs\mapred\local\taskTracker\admin\jobcache\job_201307110233_0003\attempt_201307110233_0003_m_000000_0\work\HIVE_PLAN34a2bf31-e18b-440d-8d91-be8d0e445d2e
2013-07-11 02:48:56,992 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-07-11 02:48:57,289 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.WindowsResourceCalculatorPlugin@a4f5b6d
2013-07-11 02:48:57,726 WARN org.apache.hadoop.hive.conf.HiveConf: hive-site.xml not found on CLASSPATH
2013-07-11 02:48:57,961 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-07-11 02:48:58,101 INFO org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file asv://mwinkletemp37@mwinkle.blob.core.windows.net/hive/warehouse/hivesampletable/HiveSampleData.txt
2013-07-11 02:48:58,101 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
2013-07-11 02:48:58,117 INFO ExecMapper: maximum memory = 954466304
2013-07-11 02:48:58,117 INFO ExecMapper: conf classpath = [file:/C:/hdfs/mapred/local/taskTracker/admin/jobcache/job_201307110233_0003/attempt_201307110233_0003_m_000000_0/classpath-5670276484193870096.jar]
2013-07-11 02:48:58,117 INFO ExecMapper: thread classpath = [file:/C:/hdfs/mapred/local/taskTracker/admin/jobcache/job_201307110233_0003/attempt_201307110233_0003_m_000000_0/classpath-5670276484193870096.jar]
2013-07-11 02:48:58,164 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding alias hivesampletable to work list for file asv://mwinkletemp37@mwinkle.blob.core.windows.net/hive/warehouse/hivesampletable
2013-07-11 02:48:58,164 INFO org.apache.hadoop.hive.ql.exec.MapOperator: dump TS struct<clientid:string,querytime:string,market:string,deviceplatform:string,devicemake:string,devicemodel:string,state:string,country:string,querydwelltime:double,sessionid:bigint,sessionpagevieworder:bigint>
2013-07-11 02:48:58,164 INFO ExecMapper:
< MAP>Id =4
  <Children>
    <TS>Id =3
       <Children>
        <FIL>Id =2
          <Children>
             <SEL>Id =1
              <Children>
                <FS>Id =0
                  <Parent>Id = 1 null<\Parent>
                < \FS>
              <\Children>
              <Parent>Id = 2 null<\Parent>
            <\SEL>
          <\Children>
          <Parent>Id = 3 null<\Parent>
        <\FIL>
      <\Children>
      <Parent>Id = 4 null<\Parent>
    <\TS>
  <\Children>
< \MAP>
2013-07-11 02:48:58,164 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Initializing Self 4 MAP
2013-07-11 02:48:58,164 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing Self 3 TS
2013-07-11 02:48:58,164 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Operator 3 TS initialized
2013-07-11 02:48:58,164 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing children of 3 TS
2013-07-11 02:48:58,164 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing child 2 FIL
2013-07-11 02:48:58,164 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing Self 2 FIL
2013-07-11 02:48:58,179 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Operator 2 FIL initialized
2013-07-11 02:48:58,179 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing children of 2 FIL
2013-07-11 02:48:58,179 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing child 1 SEL
2013-07-11 02:48:58,179 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing Self 1 SEL
2013-07-11 02:48:58,179 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<clientid:string,querytime:string,market:string,deviceplatform:string,devicemake:string,devicemodel:string,state:string,country:string,querydwelltime:double,sessionid:bigint,sessionpagevieworder:bigint>
2013-07-11 02:48:58,179 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Operator 1 SEL initialized
2013-07-11 02:48:58,179 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing children of 1 SEL
2013-07-11 02:48:58,179 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initializing child 0 FS
2013-07-11 02:48:58,179 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initializing Self 0 FS
2013-07-11 02:48:58,195 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Operator 0 FS initialized
2013-07-11 02:48:58,195 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initialization Done 0 FS
2013-07-11 02:48:58,195 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initialization Done 1 SEL
2013-07-11 02:48:58,195 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: Initialization Done 2 FIL
2013-07-11 02:48:58,195 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Initialization Done 3 TS
2013-07-11 02:48:58,195 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Initialization Done 4 MAP
2013-07-11 02:48:58,601 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Processing alias hivesampletable for file asv://mwinkletemp37@mwinkle.blob.core.windows.net/hive/warehouse/hivesampletable
2013-07-11 02:48:58,601 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 4 forwarding 1 rows
2013-07-11 02:48:58,601 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1 rows
2013-07-11 02:48:58,617 INFO ExecMapper: ExecMapper: processing 1 rows: used memory = 87647896
2013-07-11 02:48:58,617 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 4 forwarding 10 rows
2013-07-11 02:48:58,617 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 10 rows
2013-07-11 02:48:58,617 INFO ExecMapper: ExecMapper: processing 10 rows: used memory = 87647896
2013-07-11 02:48:58,617 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 4 forwarding 100 rows
2013-07-11 02:48:58,617 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 100 rows
2013-07-11 02:48:58,617 INFO ExecMapper: ExecMapper: processing 100 rows: used memory = 87647896
2013-07-11 02:48:58,632 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 4 forwarding 1000 rows
2013-07-11 02:48:58,632 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1000 rows
2013-07-11 02:48:58,632 INFO ExecMapper: ExecMapper: processing 1000 rows: used memory = 87647896
2013-07-11 02:48:58,804 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 4 forwarding 10000 rows
2013-07-11 02:48:58,804 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 10000 rows
2013-07-11 02:48:58,804 INFO ExecMapper: ExecMapper: processing 10000 rows: used memory = 87647896
2013-07-11 02:48:59,211 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 4 finished. closing...
2013-07-11 02:48:59,211 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 4 forwarded 59793 rows
2013-07-11 02:48:59,211 INFO org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0
2013-07-11 02:48:59,211 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 finished. closing...
2013-07-11 02:48:59,211 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarded 59793 rows
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: 2 finished. closing...
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: 2 forwarded 0 rows
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: PASSED:0
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: FILTERED:59793
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing...
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 forwarded 0 rows
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 0 finished. closing...
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 0 forwarded 0 rows
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS hdfs://namenodehost:9000/hive/scratch/hive_2013-07-11_02-48-38_757_5240085401495207012/_tmp.-ext-10001/000000_0
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS hdfs://namenodehost:9000/hive/scratch/hive_2013-07-11_02-48-38_757_5240085401495207012/_task_tmp.-ext-10001/_tmp.000000_0
2013-07-11 02:48:59,226 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS hdfs://namenodehost:9000/hive/scratch/hive_2013-07-11_02-48-38_757_5240085401495207012/_tmp.-ext-10001/000000_0
2013-07-11 02:48:59,336 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 Close done
2013-07-11 02:48:59,336 INFO org.apache.hadoop.hive.ql.exec.FilterOperator: 2 Close done
2013-07-11 02:48:59,336 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 Close done
2013-07-11 02:48:59,336 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 4 Close done
2013-07-11 02:48:59,336 INFO ExecMapper: ExecMapper: processed 59793 rows: used memory = 96526440
2013-07-11 02:48:59,336 INFO org.apache.hadoop.mapred.Task: Task:attempt_201307110233_0003_m_000000_0 is done. And is in the process of commiting
2013-07-11 02:48:59,382 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201307110233_0003_m_000000_0' done.
2013-07-11 02:48:59,414 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-07-11 02:48:59,476 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.

We currently use the Web Platform Installer to take care of dependency management of the installation. 

It’s important to point out that there are actually two key pieces that get installed

• The installation of the Hortonworks Data Platform for Windows
• The installation for Microsoft HDInsight

The second has a dependency on the first.  The WebPI feed also contains a number of pre-requisites required to set up IIS and a few other things for the HDInsight dashboard.  Here’s what it looks like on a completely fresh Windows Server 2012 machine.  Most developer machines likely have some or most of the IIS pre-reqs installed.  We’re also working to clean up some of this to minimize installation & setup.

Let’s talk a little bit about what’s in each one.

Hortonworks Data Platform installer

This msi includes the core Hadoop bits (Map/Reduce, HDFS), as well as a number of other Apache projects in the Hadoop ecosystem.  The full list included in the current installer are:

• Map Reduce
• HDFS
• Hive
• Pig
• HCatalog

Each of these projects is packaged into a zip file that contains a PowerShell script that automates the installation and setup of the component.  There are more components in Hortonworks Data Platform, and the teams are working to get these packaged and included.

Microsoft HDInisght Installer

This msi contains bits that are Microsoft specific, and may also contain additional Hadoop projects.  The current install (as of today) contains:

• HDInsight dashboard
• Sqoop
• Isotope.js
• Getting Started content

These are packaged the same way as the Hadoop projects above.  Additionally, there is an installation PowerShell script here which will do some initialization of the single node installer, such as starting the services for the Hadoop components.

Alternate Approaches

The team discussed a number of potential factorings, and we very much welcome feedback here.  A few ideas that we’ve thought about:

• Stable and Experimental packages.  This would allow us to set expectations around quality and stability of the bits. 
• Decomposing every project into an individual msi
• Integrating and building a Chocolatey package for these
  

What Does This Mean For Me?

What this means is that when you install HDInsight out of WebPI, you are installing two different msi’s.  We are revving the Microsoft msi every two weeks to pull in bug fixes (and very shortly include some experimental features).  The Hortonworks msi will be revved on a different schedule, as the team there decides to release an update.  We are partnering closely with the team there and so we will coordinate releases so that the combined installation will always work.

More directly, this implies that if you want to uninstall completely, you will need to uninstall both packages from Add/Remove Programs:

This also means that when we issue an update for the Microsoft HDInsight package, you don’t have to “lose” your cluster by uninstalling both products.  You should be able to simply uninstall & update the Microsoft HDInsight package. 

The team would love to get more feedback on this approach, so, let us know what you think!

What Are We Doing?

Every two weeks, we’re going to take a snapshot of the work in progress and update the public installer with this (provided it passes a basic level of validation).  This gives us the opportunity to rapidly get new bits in front of customers, experiment with new features, and address bugs in a timely fashion. 

We’ve also snapped versions such that releases of the SDK as well as the One-Box installer will all share the same version.  As such, we’ve versioned the msi as 0.2.0.0.

How do I Get This?

Simply by installing HDInisght out of WebPI onto a new machine after uninstalling (in place upgrade is not currently supported).  To completely uninstall previous versions, you will want first uninstall “Microsoft HDInsight Community Technology Preview” and then the “Hortonworks Data Platform”.  We will not preserve data in HDFS using uninstall/reinstall, so archive that prior to uninstalling.

What’s New?

There are no new features in this release, we’ve worked to improve some of the install and setup experience to address some bugs that have been reported on this alias as well as the forums.

• IIS / Dashboard setup issues experienced on certain OS SKU’s
• Addition of HDInsight Dashboard link into Start Menu
• Fix to Hive console for multiple line input errors
• Clean up some uninstall issues
• Minor changes to getting started content

Hortonworks has also shipped an update to the HDP installation to address the bug encountered on Friday (documented here).  By removing both installations above and re-installing, you will get updated bits here.

What if Something Doesn’t Work?

Please report issues on the forums.  We’re also tracking a set of known issues via the release notes.

When Will These Bits Be Updated In Azure?

We’re using this as an opportunity to ship these bits early and often, and they will find their way into future service updates to Azure HDInsight, typically in the next monthly update.

What’s Next?

We’ll continue to address common issues as they are reported.  From a feature perspective, we hope the next update will include updates to the dashboard to include a new and improved Hive console.

This assumes your machine is set up with:

• HDInsight (grab from WebPI here)
• Java build tools (Ant and Ivy on your command path)

Next, let’s build Piggybank by grabbing the Pig source code and checking out the 0.9 branch

• git clone https://github.com/apache/pig.git
• git checkout -b branch-0.9 remotes/origin/branch-0.9

At this point, you should now have a pig directory, move to that and type ant in order to build.

Next, navigate to .\contrib\piggybank\java, and again, type ant in order to build.  This will produce piggybank.jar.

Next, open your HDInsight console window and type pig.  This brings up Grunt, the interactive pig shell.

At this point, you can now use the following in your script:

REGISTER C:\Your\Path\To\piggybank.jar ;

foo = FOREACH entry GENERATE org.apache.pig.piggybank.evaluation.string.UPPER(item_name);

At this point, you can now take advantage of all of the functions in piggybank, and if you’re interested in contributing your own, details are here.

At the same time I saw this, another friend and co-worker was having a fairly crummy day with lots of crazy meetings and requests from co-workers, but that day was made a lot better by sitting down and solving a fun little deployment script problem.  The deployment script is neither here nor there, it was more that so many of the folks in this industry love solving a problem, whether that is getting a query right, chasing down a pesky bug, or getting that a-ha moment when you are trying to design something.  That’s “painting” for a lot of us. 

Every day is a good day when you paint.

[and working auto-tuned Bob Ross into a blog post was just too good to pass up]

This uses the Hadoop streaming capability, which essentially allows for mappers and reducers to be written as arbitrary executables that operate on standard input and output.

The .ps1 scripts are pretty simple, these operate over a set of airline data that looks like this:

0,EV,18,0,MIA,2011-12-18T00:00:00,12,TPA,1227859,2011    
0,EV,17,0,MIA,2011-12-17T00:00:00,12,TPA,1227860,2011    
6,EV,17,4,CLE,2011-12-17T00:00:00,12,ATL,1227861,2011

The schema here is a comma separated set of US flights with delays.

The goal of my job is to pull out the airlines, the number of flights, and then some very basic (min and max) statistics on the arrival and departure delays. 

My mapper:

   1:  function Map-Airline 

   2:  {

   3:      [Console]::Error.WriteLine( "reporter:counter:powershell,invocations,1")

   4:      $line = [Console]::ReadLine()

   5:      while ($line -ne $null) 

   6:      {

   7:          [Console]::WriteLine($line.Split(",")[1] + "`t" + $line)

   8:          [Console]::Error.WriteLine("reporter:counter:powershell,record,1")

   9:          $line = [Console]::ReadLine()

  10:      }  

  11:  }

  12:   

  13:   

  14:  Map-Airline 

My reducer:

   1:  function Reduce-Airlines

   2:  {

   3:      $line = ""

   4:      $oldLine = "<initial invalid row value>"

   5:      $count = 0

   6:      $minArrDelay = 10000

   7:      $maxArrDelay = 0

   8:      $minDepDelay = 10000 

   9:      $maxDepDelay = 0 

  10:   

  11:      $line = [Console]::ReadLine()

  12:   

  13:      while  ($line -ne $null)

  14:      {

  15:          if (($oldLine -eq $line.Split("`t")[0]) -or ($oldLine -eq "<initial invalid row value>"))

  16:          {

  17:              $flightRecord = $line.Split("`t")[1].Split(',')

  18:              if ([Int32]::Parse($flightRecord[0]) -ne 0) 

  19:              {

  20:                  $minArrDelay = [Math]::Min($minArrDelay, $flightRecord[0])

  21:              }

  22:              if ([Int32]::Parse($flightRecord[3]) -ne 0) 

  23:              {

  24:                  $minDepDelay = [Math]::Min($minDepDelay, [Int32]::Parse($flightRecord[3]))

  25:              }

  26:              $maxArrDelay = [Math]::Max($maxArrDelay, $flightRecord[0])

  27:              $maxDepDelay = [Math]::Max($maxDepDelay, $flightRecord[3])

  28:              $count = $count+ 1

  29:              [Console]::Error.WriteLine( "reporter:counter:powershell,"+$oldLine + ",1")

  30:          }

  31:          else

  32:          {

  33:              [Console]::WriteLine($oldLine + "`t" + $count + "," + $minArrDelay + "," +$maxArrDelay + "," + $minDepDelay +","+ $maxDepDelay)

  34:              $count = 1

  35:              $minArrDelay = 10000

  36:              $maxArrDelay = 0

  37:              $minDepDelay = 10000 

  38:              $maxDepDelay = 0 

  39:              [Console]::Error.WriteLine("reporter:counter:powershell,"+$oldLine + ",1")           

  40:          }

  41:          $oldLine = $line.Split("`t")[0]

  42:          $line = [Console]::ReadLine()

  43:      }

  44:      [Console]::WriteLine($oldLine + "`t" + $count + "," + $minArrDelay + "," +$maxArrDelay + "," + $minDepDelay +","+ $maxDepDelay)

  45:  }

  46:   

  47:  Reduce-Airlines

One thing to note on the reducer is that we use the $oldLine variable in order to keep tabs on when our group of results is moving to the next one.  When using Java, your reduce function will be invoked once per group and so you can reset the state at the beginning of each of those.  With streaming, you will never have groups that split reducers, but your executable will only be spun up once per reducer (which, in the sample here, is one).  You can also see that I’m writing out to STDERR in order to get a few counters recorded as well.

The next trick is to get these to execute.  The process spawned by the Streaming job does not know about .ps1 files, it’s basically just cmd.exe.  To get around that we will also create a small driver .cmd file and upload the file with the –file directive from the command line.

AirlineMapperPs.cmd

@call c:\windows\system32\WindowsPowerShell\v1.0\powershell -file ..\..\jars\AirlineMapper.ps1

AirlineReducerPs.cmd

@call c:\windows\system32\WindowsPowerShell\v1.0\powershell -file ..\..\jars\AirlineReducer.ps1

The ..\..\jars directory is where the –file directive will place the files when they execute

And now we execute:

hadoop jar %HADOOP_HOME%\lib\hadoop-streaming.jar 
    -mapper d:\dev\_full_path_\AirlineMapperPs.cmd 
    -reducer d:\dev\_full_path_\AirlineReducerPs.cmd 
    -input fixed_flights 
    -output psMapReduce3 
    -file d:\dev\_full_path_\AirlineMapper.ps1 
    -file d:\dev\_full_path_\AirlineReducer.ps1

And we get our results.

There is still some work to be done here, I’d like to make it a little easier to get these running (so possibly wrapping submission in a script which takes care of the wrapping for me).  Also, on Azure, we either need to sign the scripts, or log into each of the machines and allow script execution.  As I get that wrapped up, I’ll drop it along side our other samples.  We’ll also work to make this easier to get up and running on Azure if that’s interesting for folks.

Here are the sessions we're presenting:

• TechEd (NA and Europe, links are to NA sessions)
• Learn Big Data Application Development on Windows Azure -- Wenming Ye
• Web 2.0 companies have been fully taking advantage of Hadoop based open source tools to tackle Big Data needs. Microsoft now offers the best of both worlds with its own Hadoop solution on Windows Azure with full compatibility and additional rich toolsets. This session is a "getting-started" tutorial on developing Big Data applications on Windows Azure. We cover application scenarios, Hadoop on Azure, tools, and applied data analytics. More importantly, we show you how to put everything together with a couple of sample applications
• Big Data, Big Deal? -- Gert Drapers
• Are you ready for the exploding world of big data? Do you know the difference between Hive and Pig? Do you know why MapReduce is being taught in many universities rather than SQL? If not, pay attention because this talk will help get you started in understanding this new world. While sometimes the Hadoop toolkit (which includes HDFS, MapReduce, Hive, Pig, and Sqoop) is used as an alternative to relational database systems such as SQL Server, more frequently customers are using it as a complementary tool. Sometimes it may be used as an ETL tool or to perform an initial analysis of a freshly acquired data set to determine whether or not it is worth loading into the data warehouse, and sometimes to process massive data sets that are too big to even contemplate loading into all but the very largest data warehouses. In addition to covering the basics of the various parts of the Hadoop stack, this talk discusses the strengths and weakness of the Hadoop approach compared to that provided by relational database systems and explores how the two technologies can be used productively in conjunction with one another.
• Harnessing Big Data With Hadoop
• Attend this session to learn about the Hadoop Big Data solution from Microsoft that unlocks insights on all your data, including structured and unstructured data of any size. Accelerate your analytics with a Hadoop service that offers integration with Microsoft BI and the ability to enrich your models with publicly available data. Finally, learn about our roadmap for Hadoop on Windows Server and Windows Azure and for broadening access to Hadoop through simplified deployment, management and programming, including JavaScript integration
• Hadoop Summit
• Unleash Insights On All Data With Microsoft Big Data -- Tim Mallalieu
• Do you plan to extract insights from mountains of data, including unstructured data that is growing faster than ever? Attend this session to learn about Microsoft’s Big Data solution that unlocks insights on all your data, including structured and unstructured data of any size. Accelerate your analytics with a Hadoop service that offers deep integration with Microsoft BI and the ability to enrich your models with publicly available data from outside your firewall. Come and see how Microsoft is broadening access to Hadoop through dramatically simplified deployment, management and programming, including full support for JavaScript.
• How Klout is changing the landscape of social media with Hadoop and BI -- Denny Lee, David Mariani (VP of Engineering, Klout)
• In this age of Big Data, data volumes grow exceedingly larger while the technical problems and business scenarios become more complex. Compounding these complexities, data consumers are demanding faster analysis to common business questions asked of their Big Data. This session provides concrete examples of how to address this challenge. We will highlight the use of Big Data technologies—including Hadoop and Hive —with classic BI systems such as SQL Server Analysis Services.
  
  Session takeaways:
  • Understand the architectural components surrounding Hadoop, Hive, Classic BI, and the Tier-1 BI ecosystem
  • Get strategies for addressing the technical issues when working with extremely large cubes
  • See how to address the technical issues when working with Big Data systems from the DBA perspective

I think that there is a pretty nice mix of Hadoop, Microsoft plans, and applications across these sessions.  Hope you get a chance to see them (or watch them after the events!)

Not a lot to say right now, but make sure to check out hadoopOnAzure.com !

Come to think of it I also really mis a parallel execution in a flowchart. But other than that flowcharts rock! #wf4 #dmabout 13    hours ago via web

I thought about replying on twitter, but it’s a bit of a longer topic.   When we were building the flowchart for VS2010, we considered a number of possible capabilities, but ultimately the capabilities customers were looking for fell into two categories:

• Arbitrary rework patterns
• Coordination patterns

Arbitrary Rework

We can describe the model of execution in the current flowchart to be one that supports arbitrary rework.  You could also refer to this as GOTO.   One benefit (and downside) of breaking out of the tightly scoped, hierarchical execution model that we see with the Sequential style of workflows (and in most procedural languages) is the fact there exists more freedom in terms of defining the “next” step to be taken.  The simplest way to think about this is the following flowchart:

Now, this isn’t particularly interesting, and most folks who look at this will simply ask “why not a loop?”  which in this case is a valid question.  As a process gets more sophisticated (or if humans tend to be involved), the number of possible loops required gets rather tricky to manage in a sequential flow (consider the following rework scenario which includes “next steps” across conditional boundaries, and from some branches but not others.

Now, mathematically, we can describe the machinery in both of these machines as a set of valid transitions, and it is likely possible that there exists a complete mapping from any flowchart into procedural.

Coordination

The second pattern we consistently saw was the use of a free form style of execution in order to build complex coordination patterns.  The example I consistently return to (pointing back to a blog post I made back in the WF3 days)

Here I want to be able to coordinate a few different things, namely that 3 executes when 1 completes, 5 when 2, 4 when 1 AND 2, 6 when 3 AND 4 AND 5 complete.  Here we’ve created a dependency graph that can’t be handled with the procedural constructs that we have.  How could this happen? Imagine we’re trying to model the ranking process for a series of horse races.  Task 3 can happen when Race 1 completes, as Task 5 can happen when Race 2 completes.  Task 4 represents some work that requires the data from both Races.  When those 3 tasks (3, 4, and 5) complete, I can move forward and take some action (like bet on the next race). 

This type of pattern can be very powerful, and is often described by petri-nets.  There exists a multitude of join semantics from simple AND joins to conditional joins (ranging from a simple “count” to a condition based on data). 

How’d we get to where we are today?

Today, the flowchart in the .NET framework only supports the former pattern, that of arbitrary rework.  How’d we get there.  While we found a lot of value in both patterns what we found when we tried to compose them, we often got into places that became very hard to have predictable execution semantics.   The basic crux of the design issue gets to the “re-arming” of a join.  If I decide to jump back to task 3 at some point, what does it mean for 3 to complete?  Do I execute 6 again right away, do I wait for 4 and 5 to complete a second time?  What happens if I jump to 3 a second time?  Now, there certainly exist formal models for this, and things like Petri-net define a set of semantics for these things.  What we found though was that any expression of the model that took these things into account required a pretty deep understanding of the process model, and we lost one of the key benefits we see from a flowchart, which is that it is a simple, approachable model.  This is not to say that we don’t think that the Coordination pattern is useful, or that petri-nets don’t ultimately hold the unified approach,  we just did not have the time to implement a coordination pattern in VS 2010, and creating an approachable petri-net backed flowchart modeling style was something we’d need more time to get correct. 

Where does this leave us?

If you’re looking at this and saying, “but I want coordination (or a full blown petri-net)”, there are a couple of options:

• Roll up your sleeves and write that activity.  In the blog post above, I outline how you could go about building the coordination pattern, and with the new activity model, this would be simpler.  The harder part is still writing a designer, and there is now some good guidance on free form designers in the source of the state machine on codeplex. 
• Provide feedback on connect, when you do this it opens a bug in our bug database that our team is looking at daily.  Describe your scenario, what you’re trying to do, and what you’d like to see.  This type of customer feedback is vital to helping us plan the right features for the next release.  

If you recall from this post, ModelItem exists to serve as a INotifyPropertyChanged (INPC) aware proxy to the underlying instance being edited within the designer. In exchange for this fairly flexible proxy layer, you get in return a pretty verbose way in which one has to interact to actually get to the data.  Often times there are fairly long incantations of myModelItem.Properties[“foo”].Value.Properties[“bar”].Value.GetCurrentValue() in order to actually return the data (or be able to set the value).  From the beginning, we had decided we wanted to optimize for the WPF XAML consumer of the model item, and so we chose to implement ICustomTypeDescriptor so that WPF binding would be pretty simple (the above would result in a binding syntax of ModelItem.foo.bar). 

One quick note, looking at the definition of ModelItem itself will show that only the INPC interface is implemented.  However, ModelItem is an abstract type, and the implementation subclass ModelItemImpl contains the definition and implementation of the following interfaces ModelItem, IModelTreeItem, ICustomTypeDescriptor, IDynamicMetaObjectProvider

As we worked with some of our customers, it became clear that they were frequently using ModelItem in code as well, and so the programming model was becoming more of a problem (we had somewhat hoped that the majority of customers would be using XAML).  We initially went with the idea that we would create a series of Design time peers to all of the common types in WF (making a SequenceDesign, ParallelDesign, etc) which provided an identical programming surface to its peer, but under the covers in the property getters and setters would redirect to an underlying ModelItem.  This has the advantage that we could create the appearance of a strongly typed API surface, but has the downside that it introduces an extra type for every activity we ship, and it requires customers to create two types as well, and not just for activities, you would want one for customer too.  The other approach we kicked around was using the Dynamic capabilities introduced in .NET 4.

It’s important to note that these are ultimately complimentary approaches, that is, it’s possible in a future framework release (or in a codeplex project if anyone is looking to for a little fun on the side) you could release the strongly typed design time peers. From a timing perspective, it was not likely that we could introduce 30+ new types in the RC/RTM release, and we had questions about the long term sustainability of requiring an additional type for activities.  So, we went ahead with the plan for dynamic support.   In order to do this, we had to add an implementation of IDynamicMetaObjectProvider.  Bill Wagner has a good article here on the basics of doing this.  This requires us to implement one method, GetMetaObject.  In our case, we have an internal, nested ModelItemMetaObject type which inherits from System.Dynamic.DynamicMetaObject. 

In our case, there are basically two important methods that we care about, BindGetMember and BindSetMember.  These wire up to two methods on ModelItemImpl to get and set based on the property name.  The net result of this is that the following code

root.Properties["Residence"].
        Value.
        Properties["StreetAddress"].
        Value.GetCurrentValue()

Can now be written as

dynamic mi = root;
Console.WriteLine(root.Residence.StreetAddress)

Now, with anything there are some tradeoffs, the biggest one is that this is dynamic, I don’t get compile time checking, and I could have a type that will result in an exception being thrown (that said, that problem exists in the verbose form as well).  This does let me write much more succinct code when programming the model item tree though for both gets and sets.  As an added bonus, we found there is actually a decent perf improvement from the WPF XAML data binding that now leverages the IDynamicMetaObjectProvider (mentioned in ScottGu’s post here) in the way that the runtime will only compute the way to resolve the property once (as opposed to every time when it goes through the ICustomTypeDescriptor code path).

Enjoy!