Dave Crook's Dev Blog

Microsoft R Open Training Series

Dave Crook — Sun, 05 Jun 2016 18:02:06 GMT

Hello World! This is another pull from my primary develpment blog, located here. Ever wonder the difference between R and Microsoft R? Considering learning R as a programming language? You should probably watch this video. It is the first in a 4 part series to give you the jump start you need to becoming a professional data scientist with R. Part 1: Introduction to Microsoft R Open. Part 2: Introduction to R Data Structures Part 3: Data Manipulation with R Part 4: Beautiful Visualizations with R

Sigmoid Classifiers Decoded

Dave Crook — Sun, 05 Jun 2016 17:54:08 GMT

Hello World, Another pull from my primary dev blog. Sigmoid really isn't that complicated (once your understand it of course). Some back knowledge in case you are coming at this totally fresh is that the Sigmoid function is used in machine learning primarily as a hypothesis function for classifiers. What is interesting is that this same function is used for binary classifiers, multi-class classifiers and is the backbone of modern neural networks. Here is the sigmoid function: $latex \frac{ 1 }{ 1 + e^{-z}} $

Bounded Function

So lets think about this for a minute. It is used for a variety of classifiers. Sigmoid is basically a bounded function. The output of sigmoid is a value that is between 0 and 1. In the case of binary classifiers, this is straight forward.

Sigmoid's Output

If sigmoid outputs .75, you translate this as there is a 75% probability that this thing is 1 and a 25% probability that this thing is 0. In this scenario we would tell the user "this thing is a 1 with 75% probability". In the case of multi-class classifiers, you use the one-vs-all methodology and pick the highest. So lets say you have classes A, B, C and D. The first call would be "is it A or not", the second would be "is it B or not" etc. You would get back an answer that looks like this: [ .02, .01, .94, .03]. Where .94 is representative of C. We would then return to the user "this thing is a C with 94% probability".

So how does it work?

Well sigmoid is a "trained" function just like our linear regression, so first, be aware of that. If we take a look at the sigmoid function $latex \frac{ 1 }{ 1 + e^{-z}} $ you will notice this z thing. This is the magic parameter. This is z is actually the vector of prediction parameters times the vector of theta values times - 1 or $latex X * \theta * -1 $ What does this mean? Well same as everything else, you are calculating a contribution by defining a polynomial function, some weights and fitting in the values of the sample. So to create a basic linear decision boundary, with 2 parameters, you could do: $latex y = \theta_{x1} * x_1 + \theta_{x2} * x_2 + \theta_b $ you could also define more complex polynomial functions to create more complex decision boundaries or account for potential acceleration between combinations of parameters. As you can see here, the blue line divides two classes of data.

What about the 1 or 0 thing?

Thats what the sigmoid function does. So now that we have our "linear regression". We compress that term such that it can only output values between 1 and 0. Take note $latex \frac{ 1 }{ 1 + e^{-z}} $ since the entire function is 1 / something, the largest it can ever get is 1. the larger z gets, the closer to zero the term goes, the smaller z gets the closer to 1 the result goes.

So where do the magic parameters come from?

I'll do an article later on generating complex polynomials on the fly, but for now, just think "combinations of parameters that can possibly be circular and have other funny curves". so $latex y = \theta_{x1} * x_1 + \theta_{x2} * x_2 + \theta_{x12} * x_1 * x_2 + \theta_{x1-2} * x_1^2 + \theta_{x2-2} * x_2^2 $ is a great example, but looking at your data is crucial, and also remember overfitting can become problematic, so use regularization. Finally the last part is, gradient descent is used for training those magic theta values.

Getting Linear with it

Dave Crook — Sun, 05 Jun 2016 17:52:54 GMT

Hello World! This is a pull from my primary dev blog: www.dacrook.com. This article is Linear Regression Cost Functions with Linear Algebra :). So I wrote an article earlier "Linear Regression From Scratch". Many folks have pointed out that this is in fact not the optimal approach. Now being the perfectionist I decided to re-implement. Not to mention it works great in my own libraries. The following article discussing converting the original code into code that uses linear algebra. Beyond this, it still works in PCL for xamarin, Hoo-Rah Xamarin! Before we get started, note that you will need to install the nuget packages MathNumerics and MathNumerics.FSharp. If you are not compiling to PCL for Xamarin, you should probably just install the entire fslab package, as it has a ton of great stuff for data science and machine learning. So lets take another look at the original Sum of Squares function...

let SumSquaredError (predictionModel:IPredictionModel) (trainingData:UnivariateDataPoint seq) =
trainingData
|> Seq.map(fun point -> point.X
|> predictionModel.PredictValue
|> subtract point.Y
|> square
)
|> Seq.sumBy(fun point -> point)

Here are the issues with this solution: 1. We need this prediction model thing. (more code) 2. Our training data is this data point type. (more code) 3. Seq.map is essentially a for loop going through every object in the sequence and performing the manipulation (slow) 4. We still do this sum thing. (slow) 5. I forgot to divide by the length of the sequence. 6. We can't take advantage of a GPU or massive parallelization Lets see what I have transformed this into.

let SumSquares (theta:Vector) (y:Vector) (trainingData:Matrix) =
let m = trainingData.RowCount |> float
(trainingData * theta)
|> subtract y
|> square
|> divideBy m

Here are the immediate advantages 1. Cleaner and easier to read 2. Maps more closely to mathematical notation 3. Uses the MathNumerics optimizations for Linear Algebra. 4. Deals with polynomials fine without extra code overhead. Lets take a look at the test data

 let tData = matrix [[ 1.0; 2.0]
[1.0; 3.0]
[1.0; 3.0]
[1.0; 4.0]]
let yVals = vector [ 5.0; 6.0; 7.0; 11.0]
let theta = vector [1.0; 0.2]
tData |> SumSquares theta yVals

Ok, so this is different from the other examples usage for sure. We have to extract our features and transform them into these matrix and array things that most folks probably aren't familiar with. So, how in the world did we essentially remove 2 for loops by doing this? Lets dive line by line starting with our example data. The Set Up Data tData is what is considered a 4x2 matrix. It has 4 rows and 2 columns. This is indicative of a polynomial of the form y = mx + b (or the function of a line). This is a standard univariate polynomial. Or in lamest terms, you have one predictor variable such as the square footage of a house predicts the price. Now wait a second, what do you mean one variable? There are two there! Yes, Price = m*squarefootage + 1*yaxismodifier. Where m is how much the square footage affects the price and the yaxismodifier is a boost to get to minimum home size. In all linear regressions, you will at a minimum have this extra 1. So when you extract your data points from the data frame, you will need to append a column of ones. Ok, thats cool, now what about yVals? This is a Vector, which can be thought of as a 1xn matrix. There is 1 row and n columns. yVals are simply the known values of our observed data set. In this instance if you oriented (or transposed) yVals to be a column with 5,6,7,11 and appended it as a column to the end of tData, you would have the actual for each observation. Finally, theta is another vector or 1 x n matrix. This is our list of values, or our weights for how much to affect something. These are the m and b in the equation y = mx + b The Equation

let SumSquares (theta:Vector) (y:Vector) (trainingData:Matrix) =
let m = trainingData.RowCount |> float
(trainingData * theta)
|> subtract y
|> square
|> divideBy m

Lets step through this. First, what is the goal. Well previously we multiplied every sample within training data by the correct theta values, subtracted y, squared them and divided by m. Here we are taking advantage of how Matrix multiplication and subtraction works. In matrix multiplication, you need to multiply 2 matrices of sizes A x B and B x C. trainingData is 4 x 2 and theta is 1 x 2. Or theta, as it is defined as a Vector is a special case of a "ColumnMatrix", which is in fact a 2 x 1. This will produce a 4 x 1 matrix. Ok Ok, thats too much flipping math for my simple brain, what in the world are we really doing here? I tried to explain it, but that didn't work, here is a picture. As you can see above, the way you can think of applying theta to each row of tData, which produces the equation to the right and executes it. This does all of that in one blast shot. This is the type of thing GPUs are amazing at by the way. So as you can see we easily replaced a "for i in 0 .. 3" loop with that single mathematical operation. The resulting data type is a vector. The library we are using takes care of converting vectors into column matrices and then back to vectors again for us when multiplying a matrix by a vector. MathNumerics is built quite well for this construction and deconstruction of matrices. Next up, subtract y. Now remember y is a vector and the out put of tData * theta is also a vector. It simply subtracts from each cell the corresponding y value. Sweet, that part of the for loop replaced. What about square? Well we are giving it a 1 x 4 and a 1 x 4. Again another special case of Matrix multiplication, Vector multiplication, our library takes care of that for us. It does the same thing as before, giving us cell1 * cell1 + cell2 * cell2 + cell3 * cell3 + cell4 * cell4. Hrm, looks like we are squaring, then summing. What do you know? That's pretty nifty. That completes the whole summation portion of the equation without doing a single for loop. Finally we just divide by the total number of data points to get an average error, producing our sum of squares as a total cost function. Next post I will cover gradient descent and we will be using this linear algebra approach from here on out. At some other time I'll address the big data problem, it should be understood that this approach will only work for data up to a certain size. In most cases though you won't have to worry about it.

Intro to Gradient Descent

Dave Crook — Sun, 05 Jun 2016 17:51:18 GMT

Hello World! This is a pull from my privately owned blog: www.dacrook.com -> to find latest and greatest, thats a great place to head. In this article we are going to cover a simple version of Gradient Descent. It is important to note that this version of gradient descent is using Sum of Squares as its cost function to reduce. This implementation utilizes vectorized algorithms. Lets start off with...

What is Gradient Descent

There are a ton of confusing explanations, including wikipedia. Lets just keep it simple. Gradient descent is simply a method to minimize the output of a particular function by taking baby steps in the downward sloping direction. These steps reduce the slope until it reaches a local minimum or the slope becomes zero. This is more easily visualized with the below graphic. You can see here at step 1 we have a very steep slope. We take a big step, overshoot a bit, but at least we are closer, the slope is reduced, take another step, overshoot again, but again we are closer and finally in step 4, we have reached the local minimum of that function, where our slope is zero.

What does slope have to do with the minimum of a function?

Above is a parabolic equation $latex y = x^2 $ the minimum of this function is when x = 0. You can think of step 1 being x = -100, step 2 x = +50, and step 3, x = -25. Each of these steps the slope reduces as x becomes closer to the minimum, which is x = 0. Basically the slope tells us how big of a step we need to take in order to get closer to our minimum as well as in what direction. In step 1, we need to increase x, in step 2 we need to decrease x, and in step 3 we need to again increase it. At step 4, we are just right.

Awesome, so how do we determine slope?

Well now thats a tricky question isn't it? Basically it boils down to remembering your calculus 2 classes from college (or high school). I will cover derivatives and partial derivatives in depth in a seperate article, but the jist of it is that a derivative is the rate of change. In this example, the rate of change for step 1 is very high, while the rate of change for step 2 is lower, step 3 it is lower and step 4 is is zero. Derivatives is a way to understand the rate of change in a localized context for immediately near neighbors. Here is a great article on the topic for now. So lets cut to the chase, derivatives basically have a series of rules and you can read all about them on wikipedia or by taking a calculus class. A great rule though is that if your equation is y = (ANYTHING)^2 (like sum squares), your derivative is y = 2 * (ANYTHING).

Ok, but I have a bunch of variables and weights

Yup, so you need to use "Partial Derivatives", basically partial derivatives allow you to see to what extent and individual component is contributing to the entirety of the slope. See the below graphic for a representation. The above figure is an excerpt from a "contour plot". Basically it allows us to display 3 dimensional information on a 2 dimensional plot, in the instance we have an equation such as $latex y = \theta_1 \cdot x + \theta_2 \cdot z + \theta_3 $ think of $latex \theta_1 $ as green and $latex \theta_2 $ as blue. So to attain the full derivative, which is pictured in red, we can simply take the sum of the 2 partials displayed in green and blue. This gives us the contribution that each component has in the whole. By doing this, we can seperate out each weight and move it into the correct direction an appropriate amount. Partial derivatives give us the power to do this with any polynomial. Ok, thats great, what does it mean for sum of squares? Basically we take the partial derivative with respect to each theta value. The rule here is that you take the anything theta is multiplied with and multiply the entire equation by that as well. So you really end up with this equation for each theta of J. $latex \frac{ \sum_i^m \left(H_{\theta}\left(x\right) - y\right) \cdot x_j } { m } $ where the original sum of squares equation is $latex \frac{ \sum_i^m \left(H_{\theta}\left(x\right) - y\right)^2 } { m } $ You can look up the rules here. It gets hairy quick.

So what does the code look like?

Great question! First I want to mention all code samples in this article are in F# (I will have a python version at a later date for folks). Feel free to read this article on why F# as well as go here to check out the language. Here is the original sum of squares. Notice we are using linear algebra to vectorize our equation. This is an optimization step we are taking. Read more about vectorizing here.

let SumSquares (theta:Vector) (y:Vector) (trainingData:Matrix) =
let m = trainingData.RowCount |> float
(trainingData |> LinearRegressionPredict theta)
|> subtract y
|> square
|> divideBy m

Here is the new optimization equation

let StepThetas (y:Vector) (initTheta:Vector) (alpha:float) (predict:Prediction) (trainingData:Matrix) =
let m = trainingData.RowCount |> float
let err = trainingData
|> predict initTheta
|> subtract y
let nT = (trainingData * err)
|> divideVecBy m
|> multiply alpha
initTheta |> subtract nT

Woah, thats way more stuff...

Sure does. That's because this is the "theta optimization" equation and not simply the partial derivative. That is the code implementation of this mathematical equation: $latex \theta_j = \theta_j - \alpha \cdot \frac{ \sum_i^m \left(H_{\theta}\left(x\right) - y\right) \cdot x_j } { m } $ Notice the partial derivative of sum of squares is right in there. But we have this $latex \alpha $ thing in there and this $latex \theta_j $ thing too.

This is the core of gradient descent

The optimization portion is the meat and potatoes of it. Lets review what we have here. We have a partial derivative, the equation that says "Here is my contribution to the total error for this particular theta". We then multiply by an arbitrary $latex \alpha $ value. This is basically a scaling term that lets us make baby steps towards our goal. Ill speak more deeply about $latex \alpha $ (alpha) later. We then subtract our original theta by this. Lets think about this, subtraction is basically addition of the inverse. So in essence we are adding to our weight the inverse of its contribution to error, thereby reducing it. We multiply by alpha to help ensure the step is of appropriate magnitude.

To Gradient Descent

Here is the final wrapping code.

let GradientDescent (y:Vector) (initTheta:Vector) (alpha:float) (maxIters:int) (H:HypoEquation) (trainingData:Matrix) =
let m = trainingData.RowCount |> float
let mutable nThetas = initTheta
for i = 0 to maxIters do
let r = trainingData
|> StepThetas y nThetas alpha H
nThetas <- r
nThetas

So you can see have a rather naive' implementation of gradient descent. It simply iterates some number, optimizes the weights and returns the trained weights. I do want to call attention to one neat detail in this implementation. Notice the parameter H, HypoEquation. This is the hypothesis equation, or the prediction equation. For linear regression it is:

let LinearRegressionPredict : HypoEquation =
fun(theta:Vector) (trainingData:Matrix) ->
(trainingData * theta)

Simple vectorized equation. Whats interesting here is that HypoEquation is defined as a typed function that takes a Vector and a Matrix and produces a Vector. As long as something adheres to this (which most prediction equations do), our gradient descent will deal with it perfectly fine. This means our implementation of gradient descent for Linear Regression also will work for logistic regression and a variety of others. Sweet, so now we can pump our gradient descent with all sorts of brains such as automatic alpha setting, data normalization etc etc and it will just simply work.

So How do we Test it?

Well lets start with some test data...

let tData = matrix [[ 1.0; 2.0; 5.0]
[1.0; 3.0; 6.0]
[1.0; 3.0; 7.0]]
let yVals = vector [ 5.0; 6.0; 7.0]

Basically, we have a bi-nomial of the form: $latex y = \theta_1 \cdot x + \theta_2 \cdot z + \theta_3 $ We appended our Matrix with a column-vector of 1's to be our third theta. The yVals is our actual values. We also need a set of initial theta values...

let theta = vector [0.0; 0.0; 0.0]

We could probably have picked better initials, but meh, Gradient Descent will optimize anything. Ok, so to the test, lets see it do some work...

let weights =
tData
|> GradientDescent yVals theta 0.001 2000 LinearRegressionPredict
tData
|> SumSquares weights yVals

We create a variable "weights" pass the training data to Gradient Descent, the yvals, the initial theta values. We want an alpha of .001, we want 2000 training steps and pass it the LinearRegressionPrediction function. Finally we want to see how much error our weights have against the fake training data by passing tData to SumSquares, the trained weights (instead of theta) and the actual yVals. Feel free to try various numbers of training steps and watch your errors drop as gradient descent optimizes your values.

Summary

Well this ended up a larger article than I had anticipated, but I think we covered a fair number of key topics.

Functional Languages Rock for ML. I personally like F#.
Derivatives & Partial Derivatives
Cost function vs Optimization Function
Gradient Descent
Implementation

I hope this was useful and please feel free to leave comments, questions, requests.

Pi For Brains Robot - Part 1 Setup

Dave Crook — Thu, 21 May 2015 12:01:39 GMT

This is a pull from my blog: www.indiedevspot.com The other parts will be published there prior to making it here.

Pi for Brains Robot – Part 1 – Setup

Hello World!

Welcome to my blog’s version of the #IDevThis project that will be coming out. This full series will be made available on instructables and channel9 once it has been completed as a full unit.

Pi for Brains Robot (aka Danger-Bot)

These days it seems like I’m always dealing with a robot, they are on the phone, in my car, at the grocery store, just about everywhere. It also seems like every single one of them had Pie for brains, because it just couldn’t do what I told it to. This project is going to build a robot that ACTUALLY has PI for brains! Get out your breadboards, your Raspberry Pis and follow along. We will use a high performance cross platform robotics framework, Cylon, Raspberry Pi, and an Arduino Robotics Kit to build this internet connected robot.

About Part 1

We start from a freshly unboxed Raspberry Pi v1 B+, set up the OS, connect Wifi, enable SSH, setup permissions, install the latest node + npm, enable node to have pin access, and of course it wouldn’t be complete if we didn’t blink an led.

All of these steps should be done exactly in order. Doing these steps out of order may have different results. If you are encountering an issue not addressed, please leave a comment.

What you need for Part 1

Raspberry Pi B+ or v2 Ultimate Starter Kit (cheapest option)
USB Mouse, USB Keyboard, Monitor (HDMI) + Associated Cables
Female Jumper Wires
1. You can hack something together with regular wires, wire strippers, and insulated connectors.
Wifi Network for your Raspberry Pi
Patience

Setup the Pi and Update

Connecting it all up

Insert your Pi into the protective case
Connect the USB of your mouse and keyboard into any USB slots on the Pi.
Connect the USB Wifi (came with the Ultimate Starter Kit) into any USB slot
Connect your Monitor to the HDMI slot on the Pi
Insert the Mini SD Card (came with the Ultimate Starter Kit) into the mini SD Card Slot
Plug the mini USB power supply into the mini USB slot
1. Your Pi should power on at this point and take you to an OS selection page.

Installing Raspbian

Using your mouse, choose Raspbian from the list
1. Ensure no other options are selected
Ensure your keyboard layout is US (or your layout)
Click Install
Click yes on the dangerous looking message
Wait patiently…
When it is done, click OK
1. a console will appear and text will fly, just wait some more…

Basic Configuration

You can always get back to this menu later by typing: “sudo raspi-config” at a command terminal. From this menu, you can only use arrow input from your keyboard. Up and Down flow there the vertical Menu, while LEFT and RIGHT move between cancel and finish. Enter selects the current highlighted item. Any time you type a command, you must execute the command by pressing the enter key.

Select Change User Password
1. a prompt will appear indicating you are about to enter a password, press enter to dismiss it
2. at the bottom you will be prompted in a cmd window for the password and to confirm it.
Enter your new password (I use david1234 for non commercial applications)
1. Press Enter
Enter your password again for confirmation.
1. Press Enter
2. A prompt will appear indicating success, press enter.
Select Advanced Options
Select HostName (this is the name of your raspberry pi)
1. a prompt will appear indicating you have to use normal letters and numbers for a name. Press enter to dismiss
2. a view will appear, use the delete key to delete the old name and enter your own. a-z and 0-9 only.
  1. My Pi’s name is dangerbot
3. use the arrow keys to go to OK and press enter on OK.
  1. You are brought to the first menu again.
Select Advanced Options
Select SSH
1. Select Enable SSH
2. Select OK
Use the right arrow key to navigate to Finish
Select Yes, you would like to reboot now.
Wait some more…
A cmd promp will appear asking you to log in.
1. dangerbot Login:
  1. type “pi” here – just pi, no quotes.
  2. pi is your user name
2. Password:
  1. type the password you configured, I typed: david1234
  2. note that it will not show the letters or even *s
SUCCESS, you are logged in at the Raspberry Pi’s cmd prompt!
type the command: startx
1. this will launch the Raspberry Pi’s Desktop Mode, which is very similar to old Windows.
2. We will do some configuration in desktop mode

Configure Wifi

We will be doing a lot with wifi in this project, for now we need it to download various software packages, but later we will be connecting it to Twitter, Azure, who knows? Nothing is better than a tiny computer backed by a warehouse of servers.

On the desktop, there is an Icon, “Wifi Config”, Click on that.
1. You should see an adapter wlan show up
2. Network should be empty
On the bottom right, click Scan
A new mostly empty window will show up.
On the bottom right here, click Scan as well.
You should see a few networks pop up
Double Click on your network, Mine is Version-29…
1. This should be the same network your laptop is on if you plan to connect to the pi from your laptop.
A new window will appear
1. If you know what these settings are, verify and ensure they are right
2. If you do not, just keep the defaults for the first try. If that doesn’t work, try various combinations.
3. The default settings for me were correct.
In the PSK field, type your network password.
1. I’m not telling mine, as this is my real wifi I travel with.
Click Add
Wait for a while…
1. It may say complete, but it is not done until an IP address appears
2. If it seems to take a while, try clicking connect again.
Verify connection…
1. On the desktop, on the left is an Icon: LXTerminal
2. Double click this, you will be brought to a cmd prompt
3. type: ifconfig
4. look for your ip address
Close the wifi configuration panel.

Updating the Pi

Update your firmware

If you still have the cmd prompt open from checking your ip address, you may use that, otherwise, open a new one.
type the command: sudo rpi-update
1. If this doesn’t do anything, your Raspberry Pi’s firmware is already up to date.
Go get coffee, this might take a while…
When it is done, type the command: sudo reboot

Update your OS

if you had to reboot, login again.
1. Login: pi
2. Password:
3. You don’t need to go to the desktop.
from the cmd prompt, type: sudo apt-get update
1. Go get some more coffee, this might take a while.
from the cmd prompt, type: sudo apt-get upgrade
1. Accept the prompts by typing: y
  1. follow y with the enter key
2. Go get some more coffee, this will take a while.
from the cmd prompt, type: sudo reboot

SUCCESS! Your Raspberry pi is running the latest firmware with the latest version of Raspbian! If you just wanted to know how to set up a Pi, you can end here. If you are interested in building robots using JavaScript and Raspberry Pis, I highly suggest you read on, as this is where things get complicated.

Install and Configure Node

Node is a JavaScript framework that CylonJS (a robotics framework) is installed from. NPM is a Node Package Manager for getting the packages we need for dealing with our robot.

download/install node

Login to the Pi
Get to a command prompt if you are not already on one.
type the command: sudo mkdir /opt/node
Find the latest version of node from: https://nodejs.org/dist/ with the ending -Linux-arm-pi.tar.gz
1. There is currently no easy way to do this. You literally have to look at the dates each folder was updated and attempt to find the latest version for pi.
2. the pi versions are kept behind the regular branches.
3. Cylon and Node both recommend that for the Pi you use the pre-compiled versions.
4. The latest version today is v0.10.28
  1. v0.10.28 is in fact newer than v0.10.4 (it should be v0.10.04)
  2. if you find a newer version, replace references to 0.10.28 with your version.
  3. double check cylon’s site for a version of node if you need to.
type the command: sudo mkdir /opt/node
type the command: wget https://nodejs.org/dist/v0.10.28/node-v0.10.28-linux-arm-pi.tar.gz
type the command: tar xvzf node-v0.10.28-linux-arm-pi.tar.gz
type the command: sudo cp -r node-v0.10.28-linux-arm-pi/* /opt/node

add node and npm to your path

Adding node to your path will make executing node programs much easier. But also once we ditch some of the fake security, we can do it easily with sudo as well.

type the command: cd ~
type the command: sudo nano .bash_profile ..bash_profile
Add the code:

1

2

PATH=$PATH:/opt/node/bin

export PATH
Hold ctrl, then push x.
type y
click enter
1. if you are still in the file…
  1. hold ctrl, then push x
  2. type n if it asks you to save
type the command: source ~/.bash_profile
Test the pathing
1. Type the command: node -v
  1. you should see a version number
2. Type the command: npm -v
  1. you should see a version number

Ditch Security

The only danger posed to us here is the npm install process and 3rd party libraries we use, but we are going to need to give most of these modules low level access anyways. Lets just go ahead and ditch security and make some robots. This will make executing commands much easier. If you are worried about the packages you are installing and the 3rd party libraries you are using doing something malicious, you should probably stop using that library and pick a different one, or write your own.

type the command: sudo chown pi ~/ –recursive
type the command: sudo nano /etc/sudoers
1. At the end of the Defaults section, add the line: Defaults !secure_path
  1. This allows sudo to use the default path variables (we just set up node and npm)
hold ctrl, then push x
type: y
hold ctrl, then push x
type: n
You should be back at the cmd prompt
Test
1. Type: sudo node -v
  1. you should see a version number
2. Type: sudo npm -v
  1. you should see a version number.
type the command: sudo usermod -G gpio pi

type the command: sudo nano /etc/udev/rules.d/91-gpio.rules

type the following code in

SUBSYSTEM=="gpio", KERNEL=="gpiochip*", ACTION=="add", PROGRAM="/bin/sh -c 'chown root:gpio /sys/class/gpio/export /sys/class/gpio/unexport ; chmod 220 /sys/class/gpio/export /sys/class/gpio/unexport'"
SUBSYSTEM=="gpio", KERNEL=="gpio*", ACTION=="add", PROGRAM="/bin/sh -c 'chown root:gpio /sys%p/active_low /sys%p/direction /sys%p/edge /sys%p/value ; chmod 660 /sys%p/active_low /sys%p/direction /sys%p/edge /sys%p/value'"

It might be easier to ssh and copy/paste this in (do this if you are an advanced user)
This must be EXACT

hold ctrl, then push x. Type y , then enter. This saves the file
type the command: sudo nano/etc/modules
add the following lines
1. i2c-bcm2708
2. i2c-dev
hold ctrl, then push x. Type y , then enter. This saves the file
type the command: sudo nano /etc/modprobe.d/raspi-blacklist.conf
comment out all lines by add # to the beginning
type the command: sudo nano /boot/config.txt
add the lines:
1. dtparam=i2c1=on
2. dtparam=i2c_arm=on
hold ctrl, then push x. Type y , then enter. This saves the file
type the command: sudo addgroup i2c
type the command: sudo usermod -G i2c pi
type the command: sudo reboot

There really isn’t much testing we can do here yet. We just have to have faith that it all worked and keep going. But you have effectively removed all the extra security garbage that would ordinarily prevent pi and cylon from doing what it needs to such that we can build robots!

Write Some Code Already!

Create Folders and Install Packages for Coding

Alright, lets get set up for coding…we are going to start by building out the folder structure, installing necessary packages and testing out the cylon framework.

Login using pi and your password
Get to your cmd window if you are not there already.
type the command: sudo mkdir Programs
type the command: cd Programs
type the command: sudo mkdir HelloPi
type the command: cd HelloPi
type the command: sudo chown pi ~/ –recursive
type the command: sudo chown pi ~/Programs –recursive
1. this is just for good measure
sudo reboot
1. this too is just for good measure
login again
type the command: npm install cylon cylon-raspi
1. note that it does not say sudo. If you use sudo, it will not work.
2. This takes a while, wait and go get some coffee…
type the command: npm install cylon-gpio cylon-i2c

Simple Console Output

type nano app.js

Type the code:

var Cylon = require("cylon");
 
Cylon.robot({
    connections: {
        raspi: { adaptor: 'raspi' }
    },
    work: function (my) {
        every((1).second(), function () {
            console.log("led toggle");
        });
    }
}).start();

hold ctrl, then push x. Push y then enter, this saves the file.
type the command: sudo node app.js
you should see the following output:
hold ctrl, then push c to stop execution.

SUCCESS!!! You have not only node working, but also cylon executing. Now we can wire up our LED and instead of writing led toggle to the output, lets actually toggle an led

Wiring the Pi

Ensure your pi does not have power attached to it before you begin wiring.

The pi is a bit confusing to wire. I think of the pins as physical vs logical. Physical pins on the top are even, while physical pins on the bottom are odd. The counting is based left to right with the pi oriented in the fashion I have shown in the diagram above. In this instance, the wire going to the + side of the LED is physical pin 7. This maps to a different logical pin, but because we use cylon, we don’t care :). The ground is connected to physical pin 39. The best site so far that explains the pins can be found here: https://pi.gadgetoid.com/pinout the numbers on this site correlate to physical pins. You always can just count left to right acknowledging even/odd.

Blinking an LED with Code.

login again
type the command: cd Programs/HelloPi
type the command: nano app.js
replace the current code with the following code:

//make sure we have cylon as a dependency
var Cylon = require("cylon");
 
Cylon.robot({ name: 'HelloPi' })
    //setup connection
    .connection('pi', { adaptor: 'raspi' })
    .device('led', { driver: 'led', pin: 7 })
    .on('ready', function (dangerbot) {
    every((1).second(), function () {
        //toggle LED on or off
        dangerbot.led.toggle();
        console.log("led toggle");
    });
}).start();

hold ctrl, push x, type y followed by enter to save the file.
type the command: sudo node app.js
1. Watch your pi light up!

AND THATS IT! You have a Raspberry Pi, with Cylon and GPIO access.

Common Linux Commands

sudo: preceding a command with sudo gives root access
mkdir: makes a directory
man : tells you how to use a specific command
chown: changes the owner for a specified directory
cd: changes a directory
ls: lists the contents of the current directory
wget: downloads a files to the current location
tar: unzips a file
nano: is a very crude text editor
usermod: modifies properties of a user.
node: (not linux specific)executes a node runtime, you can write code directly from here, or execute javascript files.
npm: (not linux specific) installs or uninstalls node packages.

Why use Raspberry Pi and CylonJS

Raspberry Pi can be acquired very cheaply and is a great general purpose computer as well as tinkering platform. It really has a lot of nice components to it for a beginner. I would recommend starting with a Raspberry Pi and then upgrading to an Intel Edison. Why CylonJS? Cylon is JavaScript, has a really nice coding interface for it and is multiplatform. With simple changes, I can move my code written for Raspberry Pi over to an Edison. Beyond just the robotics, Node and JavaScript are skills that will serve you in web as well as database programming. This makes CylonJS an ideal language for learning to program. Raspberry Pi between being affordable and the fact that we were able to do all of this without using a traditional computer at all makes it an ideal educational platform. Not to mention the Raspberry Pi v2 can run Windows 10, which will make programming for it easier.

Summary

So there you have it, we have covered a TON of material in this article. We have covered, Getting started with a brand new Raspberry Pi, how to update the firmware, how to update the Operating System, how to configure wifi. We have covered how to install and configure NodeJS as well as NPM allowing them to access the low level hardware of the device by removing many of the security components that prevent this. We have covered basics of wiring to the pins of a Raspberry Pi and executing node scripts that leverage CylonJS as their robotics platform. We also covered why you would even want to use Cylon and Raspberry Pi specifically. I hope you enjoyed this article. Please leave comments, feedback and questions in the comments section below.

How to Datamine Zillow

Dave Crook — Tue, 12 May 2015 06:02:43 GMT

This is a pull from my blog: www.indiedevspot.com There is a slight delta between this and indiedevspot, where indiedevspot has more content. I also post all of my presentation slide decks at: https://indiedevspot.com/category/events/ NOTE: the formatting may be nicer on www.indiedevspot.com

So on with the article!

Hello World,

As many of you may know at this point, I am relocating to South Florida. Final location to be determined, but will probably be renting around Pompano Beach or Fort Lauderdale while working out of Venture Hive and the Microsoft Fort Lauderdale Offices. So what does this have to do with Zillow? Well, It has EVERYTHING to do with Zillow. What I’ve found while searching for homes is that between Realtors, Zillow and Trulia, they really just don’t have a predictive analytics solution that works for me. So I decided to give a shot at AzureML to mash together a few datasets to send me notifications more to my liking than is currently being sent. So step 1 in this plan is to data mine Zillow. Luckily, Zillow has an api for that. Or if you are feeling particularly frisky, Zillow gets their data from ArcGIS (example for Raleigh). So lets get cracking…

Tools for the Job

Azure: There are literally hundreds of millions of properties for sale. There is no way I’m going to get this done in a timely fashion without millions of computers at my finger tips. Add in the fact that I can right click publish cloud services with 100 nodes…Hell yeah, that’s what I’m talking about… https://azure.microsoft.com/en-us/
F#: F# just is a wonderful tool. The type providers are excellent and seems well suited for this job. I think it might have been overkill for this sample, but the XML Type provider ended up being really useful. F# is my go-to web scraping tool from here on out. https://fsharp.org/
C#: Until F# gets some better tooling dealing with things like SQL Azure (EF support in C# is way better) and ease of publishing Cloud Services, I just am stuck with it. C# just has better libraries and support. Good note is that C# is supposedly getting some new F# features like pattern matching :).
Visual Studio 2013 Community: Free version of Visual Studio: https://www.visualstudio.com/en-us/news/vs2013-community-vs.aspx All of my future development however will be on 2015, it is now acting pretty stable for a beta build and has a host of really awesome new features like performance diagnostics while debugging displayed right next to each stepped over line of code.

Project Structure Breakdown

Note: The project will NOT compile fresh out of the repository. See AnalyzeZillow.Brains breakdown.

I’m used to writing production grade software, so I always break my project apart in a way to make my code highly maintainable and extensible. There are 5 projects in this solution, built in dependency order, and that may seem like a lot, but once you get used to breaking problems apart in this fashion, it seems pretty straight forward. Before looking too deeply here, you might want to download the code from here: https://github.com/drcrook1/ZillowAnalysis

AnalyzeZillow.Core: This is the core basic project with repositories and types. This project has no dependency requirements within the solution. To Create this project from scratch, from VS 2013 Community, you simply Click File -> New Project -> C# Library.
AnalyzeZillow.Brains: This project requires just the types from the Core solution. Working with F#, C# and web has taught me that though you get a performance gain working with F# types, to provide ease of system integration, those types should be C# types. The Brains project simply provides a function that parses the Zillow api response into C# types. To create this project from scratch, from VS 2013 Community File -> New Project -> Other Languages -> F# Library. You will need to install the nugget packages for F# Core and FSharp.Data. Installing the F# Powertools extensions makes life easier as well.
AnalyzeZillow.Host.Role: This is the project that orchestrates using the Brains and the Core Repositories to store data retrieved from the brains and stored into SQL Azure. Role is also responsible for figuring out which instance of the service it is and providing the orchestration of ensuring duplicate data is not retrieved. This gets generated when creating AnalyzeZillow.Host.
AnalyzeZillow.Host: This project simply defines the Azure Cloud Service. It points to the Host.Role and has configuration for number and size of instances. You can see from the config here that we only have 2 instances. To create this project from scratch, from VS 2013 Ensure you have the Azure SDK installed for VS 2013. File -> New Project -> Cloud Service. Name the cloud service what you want, add a worker role (Make sure you edit the name here to Host.Role)
AnalyzeZillow.DB: This project is the SQL Project defining the database structure we will be using for the project. To create this from Scratch File -> New Project -> Other Languages -> SQL Server Database Project.

Lets look at some code already!

So now that we have the project structure and dependencies laid out, we can start looking at some code. I am going to skip the DB project, as that is a visual editor, and I think you can figure it out. If not, I’m working on a separate article (release pending) where we deep dive into those.

AnalyzeZillow.Core

In this project, there is a folder “SQL”. The files under here are auto-generated by code-first from Database. It generates the Entity Framework Context and mappings automatically for you based on the deployed database. You can generate these by right clicking SQL, Add new item, go to the Data section. Notice the “Context” class. This means in the app.config, you will need to replace the connection string with your own. You are probably thinking the app.config for this project, but NO! You need to replace the connection string in the app.config in AnalyzeZillow.Host.Role, as that is the end project being deployed. This is simply a library project. You more or less forget this app.config even exists.

namespace AnalyzeZillow.Core.SQL
{
    using System;
    using System.Data.Entity;
    using System.ComponentModel.DataAnnotations.Schema;
    using System.Linq;
    /// 
    /// Entity Framework Generated Code.  You will need to search in the app.config 
    /// for ZillowDataContext to find the connection string and replace with your own.
    /// 
    public partial class ZillowDataContext : DbContext
    {
        public ZillowDataContext()
            : base("name=ZillowDataContext")
        {
        }
 
        public virtual DbSet Homes { get; set; }
 
        protected override void OnModelCreating(DbModelBuilder modelBuilder)
        {
            modelBuilder.Entity()
                .Property(e => e.State)
                .IsFixedLength();
 
            modelBuilder.Entity()
                .Property(e => e.Latitude)
                .HasPrecision(18, 0);
 
            modelBuilder.Entity()
                .Property(e => e.Longitude)
                .HasPrecision(18, 0);
 
            modelBuilder.Entity()
                .Property(e => e.TaxAssessment)
                .HasPrecision(18, 0);
 
            modelBuilder.Entity()
                .Property(e => e.NumBathrooms)
                .HasPrecision(18, 0);
        }
    }
}

So how about some non generated code? Well there is IZillowDataRepository and ZillowDataRepository. Using an interface allows me the ability to swap out implementations without worries and I should be using a dependency injection framework, but oh well, this is quick. The only code really being leveraged here is “SaveSingle”.

using AnalyzeZillow.Core.SQL;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
 
namespace AnalyzeZillow.Core
{
    public class ZillowDataRepository : IZillowDataRepository
    {
        public ZillowDataContext dbContext { get; set; }
 
        public ZillowDataRepository()
        {
            dbContext = new ZillowDataContext();
        }
        /// 
        /// Notice that this is NOT IZillowDataRepository.SaveSingle,
        /// Doing so would force a private member here.
        /// 
        /// 
        /// 
        public async Task<bool> SaveSingle(Home home)
        {
            try
            {
                dbContext.Homes.Add(home);
                await dbContext.SaveChangesAsync();
                return true;
            }
            catch(Exception e)
            {
                //if something goes wrong, we can't save the changes,
                //but chances are it is still in the context.
                //next time we attempt a save, it will still be there
                //causing another exception.
                //we must remove the bad home (probably a key duplicate)
                //otherwise our program will constantly be failing with a growing 
                //in memory list of homes to add
                dbContext.Homes.Remove(home);
                return false;
            }
        }
 
        public async Task<bool> SaveBatch(ICollection homes)
        {
            foreach(Home home in homes)
            {
                //Add a bunch of homes here
                dbContext.Homes.Add(home);
            }
            try
            {
                //save all of them as a batch
                //this is a HUGE performance gain.
                //I've seen upwards of multiples of minutes 
                //performance increase for larger inserts
                await dbContext.SaveChangesAsync();
                return true;
            }
            catch(Exception e)
            {
                return false;
            }
        }
 
        ICollection IZillowDataRepository.GetHomes()
        {
            throw new NotImplementedException();
        }
 
        SQL.Home IZillowDataRepository.GetSingleHome(int zId)
        {
            throw new NotImplementedException();
        }
    }
}

This repository allows the ability to get and save data. In our instance as of now, we are JUST saving data and therefor we do not need implementations for any of the getters. Entity Framework is really awesome and extremely performant. I highly recommend learning this if you don’t know it yet. It makes dealing with SQL super easy. This repository is what is used by our Host.Role to save data retrieved by our F# Brains.

That’s really it for the Core, on to the Brains!

AnalyzeZillow.Brains

There are really only two files here that matter; Brains.fs and Script.fsx. You will probably notice that when you bring the project down it fails to compile. It fails to compile because I am using the FSharp.Data XML Type provider. It can’t compile, because the xml coming back has a single element “Message”, which says “Where the heck is your api key dummy!?”. Find the following lines of code at the top of the Brains.fs file and replce the {YOURKEY} with your very own Zillow api key. You will see it works.

module Brains =
    []
    let zillowBasicSample = "https://www.zillow.com/webservice/GetUpdatedPropertyDetails.htm?zws-id={YOURKEY}&zpid=48749425"
    let zillowBasicUrl = "https://www.zillow.com/webservice/GetUpdatedPropertyDetails.htm?zws-id={YOURKEY}&zpid="
    []
    let zillowDeepSample = "https://www.zillow.com/webservice/GetDeepSearchResults.htm?zws-id={YOURKEY}&address=2114+Bigelow+Ave&citystatezip=Seattle%2C+WA"
    let zillowDeepsUrl = "https://www.zillow.com/webservice/GetDeepSearchResults.htm?zws-id={YOURKEY}"
    type ZillowBasic = XmlProvider
    type ZillowDeep = XmlProvider

The type providers are really awesome. There are type providers for all sorts of different things. What it does is builds a dynamic type “ZillowBasic” and “ZillowDeep” from the xml returned by the samples. The Url’s are the base url’s for better readability for formulating a request. You will notice that what I do here is a terrible habit (Rats, you caught me). I should pull the XML ahead of time and set a string in another file for it to read the sample from and pass that sample into the provider. This would actually prevent compilation problems as well. Oh well, in this instance, just get your very own api key, put it here, and the code should compile fine.

Ok, so now the project compiles, lets talk about each of the files, Brains.fs and Script.fsx. Script.fsx is not actually compiled, it is a script file that you can use for data exploration and testing. Super awesome to have something like this in the .net run time! Since F# is compatible with C#, you can test out C# stuff like this too (but you have to do it from F#). Highlight the code you want and push ALT+Enter and it will execute in the F# Interactive window. This is how I figured out which calls from Zillow had the data returned that I wanted and could compose the second more complicated request easily and test it real time. Brains.fs is simply the resulting compiled form of what the experimentation led me to.

namespace AnalyzeZillow.Brains
 
open FSharp.Data
open AnalyzeZillow.Core.SQL
open System.Xml.Linq
 
module Brains =
    []
    let zillowBasicSample = "https://www.zillow.com/webservice/GetUpdatedPropertyDetails.htm?zws-id={YOURKEY}&zpid=48749425"
    let zillowBasicUrl = "https://www.zillow.com/webservice/GetUpdatedPropertyDetails.htm?zws-id={YOURKEY}&zpid="
    []
    let zillowDeepSample = "https://www.zillow.com/webservice/GetDeepSearchResults.htm?zws-id={YOURKEY}&address=2114+Bigelow+Ave&citystatezip=Seattle%2C+WA"
    let zillowDeepsUrl = "https://www.zillow.com/webservice/GetDeepSearchResults.htm?zws-id={YOURKEY}"
    type ZillowBasic = XmlProvider
    type ZillowDeep = XmlProvider
 
    let GetHome (id:int, zillowApiKey:string) = 
        let idString = id.ToString()
        let zBasic = ZillowBasic.Load(zillowBasicUrl + idString)
        let address = zBasic.Response.Address.Street.Replace(" ", "+")
        let citystatezip = zBasic.Response.Address.City + @"%2C+" + zBasic.Response.Address.State
        let deepUrl = zillowDeepsUrl + "&address=" + address + "citystatezip=" + citystatezip
        let zFull = ZillowDeep.Load(zillowDeepsUrl + "&address=" + address + "&citystatezip=" + citystatezip)
        let home = new Home();
        home.City <- zFull.Response.Result.Address.City
        home.FIPSCounty <- zFull.Response.Result.FipScounty
        home.HomeSize <- zFull.Response.Result.FinishedSqFt
        home.HomeType <- zFull.Response.Result.UseCode
        home.Latitude <- zFull.Response.Result.Address.Latitude
        home.Longitude <- zFull.Response.Result.Address.Longitude
        home.NumBedrooms <- zFull.Response.Result.Bedrooms
        home.State <- zFull.Response.Result.Address.State
        home.Street <- zFull.Response.Result.Address.Street
        home.TaxAssesmentYear <- zFull.Response.Result.TaxAssessmentYear
        home.TaxAssessment <- zFull.Response.Result.TaxAssessment
        home.YearBuild <- zFull.Response.Result.YearBuilt
        home.zId <- id
        try
            home.NumBathrooms <- zFull.Response.Result.Bathrooms
        with 
            | :? System.Exception -> home.NumBathrooms <- 0.1m
        try
            home.LotSize <- zFull.Response.Result.LotSizeSqFt
        with 
            | :? System.Exception -> home.LotSize <- 0
        try
            home.ZillowEstimate <- float(zFull.Response.Result.Zestimate.Amount.Value)
            home.ZillowHighEstimate <- float(zFull.Response.Result.Zestimate.ValuationRange.High.Value)
            home.ZillowLowEstimate <- float(zFull.Response.Result.Zestimate.ValuationRange.Low.Value)
        with 
            | :? System.Exception -> 
                home.ZillowEstimate <- float(0)
                home.ZillowHighEstimate <- float(0)
                home.ZillowLowEstimate <- float(0)
        home
 
 
    let GetData (something:int) =
        let data = 1
        0

Now I admit, this is not really “functional” coding, but I really just wanted access to the REPL and Type providers F# provides for this. There was no real need for me to put together a series of compose-able functions, that would have been just an academic exercise. I will end up using more F# features during my exploration and analysis post collection. You will notice a fair number of try/catch blocks. One of the biggest things you will find is that very rarely does data come back the way it is promised. Even Zillow has issues.

Some properties have the same id’s, some properties are missing estimates, some properties they just didn’t bother to put any data in at all. So the try/catch go independently around various properties that I noticed had a substantial number of issues so I can hopefully get a substantial amount of data back. Even doing this, it appears there are issues with approximately 80% of the data I parsed through. So some extra effort needs to go into checking that out. Some of the results just simply said “I have no data for this, sorry”. That was actually a fairly large portion of the 80% that had issues (no statistics on what was bad with the 80%).

AnalyzeZillow.Host.Role

So in this instance of our cloud services, I planned to use only 2 cloud services, so I only needed to distribute the workload across even requests and odd requests. I simply determine which node is running the code (0 or 1) and set them infinitely querying Zillow starting at property 1 or 2 and incrementing the request by 2. Notice there is a fair amount of generated code, I just left it there, our code really doesn’t need any of that mess, but I just hate deleting generated code some times, because its tough to tell what it is really doing.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.Storage;
using AnalyzeZillow.Core.SQL;
using AnalyzeZillow.Core;
 
namespace AnalyzeZillow.Host.Role
{
    public class WorkerRole : RoleEntryPoint
    {
        //Generated Code
        private readonly CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();
        private readonly ManualResetEvent runCompleteEvent = new ManualResetEvent(false);
         
        public override void Run()
        {
            Trace.TraceInformation("AnalyzeZillow.Host.Role is running");
            Trace.TraceInformation("Working");
            //DO Work
            var instanceIdSplit = RoleEnvironment.CurrentRoleInstance.Id.Split('_');
        //last token in the id is the integer id of the role _00 - _99
            var roleNumberString = instanceIdSplit[instanceIdSplit.Length - 1];
            int roleNumber;
            int.TryParse(roleNumberString, out roleNumber);
            //82300179 <- Home in Pompano Beach
            if(roleNumber == 0)
            {
                WorkerRole.DoWork(82300178);
            }
            else
            {
                WorkerRole.DoWork(82300179);
            }
 
        }
 
        public static async void DoWork(int i)
        {
             
            bool done = false;
            ZillowDataRepository zRepo = new ZillowDataRepository();
            while (!done)
            {
                try
                {
                    Home h = Brains.Brains.GetHome(i);
                    await zRepo.SaveSingle(h);
                    i += 2;
                }
                catch (Exception e)
                {
                    //done = true;
                    i += 2;
                }
 
            }
        }
        //Generated Code
        public override bool OnStart()
        {
            // Set the maximum number of concurrent connections
            ServicePointManager.DefaultConnectionLimit = 12;
 
            // For information on handling configuration changes
            // see the MSDN topic at https://go.microsoft.com/fwlink/?LinkId=166357.
 
            bool result = base.OnStart();
 
            Trace.TraceInformation("AnalyzeZillow.Host.Role has been started");
 
            return result;
        }
        //Generated Code
        public override void OnStop()
        {
            Trace.TraceInformation("AnalyzeZillow.Host.Role is stopping");
 
            this.cancellationTokenSource.Cancel();
            this.runCompleteEvent.WaitOne();
 
            base.OnStop();
 
            Trace.TraceInformation("AnalyzeZillow.Host.Role has stopped");
        }
    }
}

Also, remember, this is the project, where we need to add the database connection string that case sensitively matches our context in our AnalyzeZillow.Core project. The connection string needs to be set in app.config.

Debugging the project

As the project has 2 cloud service instances, you will need to use the full emulator. To specify the full emulator to use. Right click AnalyzeZillow.Host and select properties. From the properties, on the left tab, click “Web”. This will change the primary view, where you can select the option “Use Full Emulator”. This will simulate multiple instances of cloud services.

Close Visual Studio, and start it again in admin mode. Right click AnalyzeZillow.Host -> Debug -> Start new Instance. Verify everything works, you are putting data in the database etc etc.

Publishing the solution

This is super easy. Right click AnalyzeZillow.Host -> Publish -> New Cloud Service. Enter in your credentials, hit OK and you are off!

Summary

We covered Datamining Zillow through their api using Azure Cloud Services, SQL Azure, C#, F# and Visual Studio! I hope this was useful to you. I use these same techniques for pretty much all data mining activities. The complexity of this is a bit lower than most, which makes it a great starter project.

Intro to Angular, Web API and Azure

Dave Crook — Mon, 27 Apr 2015 18:27:17 GMT

Intro to Angular, Web API and Azure

Hello World,

So I thought it would be beneficial to discuss Angular, Web API and Azure in some form of depth as well as provide an entire set of functioning code. I will start by addressing a few things, What is Angular, What is Web API, and what is Azure? Followed by the code and explanation of the code. The code itself provides a simple website, which has restful routing, requests for processing and lists out data from a database acquired from said processing.

This is a pull from my website: www.indiedevspot.com Check out my other articles!

What is Angular?

Angular is a client side Model, View, Controller or MVC framework. A model being the data in which you wish to show, the view being the format in which you wish to show the data, and the controller facilitates the dynamic aspects of the page. Angular is one of the most popular frameworks out there. It has a bit of magic to it, but is much better to deal with than raw HTML and JavaScript. It provides a framework for dependency management, html injection and dynamic pages. This framework seems to be at the heart of the Single Page App movement. It also works within the context of Cordova, making it a great client framework to learn and understand. That said, in my experience it appears to perform very slowly on semi complex renders, but the speed of modern computers appears to more or less keep it within the bounds of an acceptable performance threshold.

What is Web API?

Web API is an ASP.Net (or .net) framework for providing restful services to clients. These services are easily data mined and therefor very conducive to search engine optimization. Not only that, but they are easily repackaged and sold. Beyond THAT they are also cross platform and follow a standard that most client developers understand. Finally it provides all of the security and goodliness that classic MVC provides. So in short, Web API is just pure awesomeness for a server side developer who wishes to ship product to the world in a timely fashion.

What is Azure?

Azure is the Microsoft Cloud Platform. I have tons of slide decks on my site regarding intro to Azure, Azure Architecture, How to – Azure etc. Go to the Events section and just download nearly any events slide deck and you will find something regarding Azure. In the instance of what we are discussing today, Azure provides the hosting platform for the website as well as the compute space for our web job which scrapes git hub, synchronizing the scale of both together. Azure supports a plethora of technologies and languages on its “Web Apps” platform, which is what we primarily intend to take advantage of. Beyond the Web Apps platform, this solution takes advantage of SQL Azure and Azure Storage. You will notice that all three technologies are within the PaaS world. This is purposeful as PaaS enables me as a developer to push out quality product faster. If there is enough interest, I will post an article about PaaS vs SaaS vs IaaS.

ONTO THE CODE!

The code is located here: https://github.com/drcrook1/Nerds4Hire As of the time of this writing, it is 8:33pm EST on 4/27/2015. Look through the check-in history for that specifically if you see significant changes, as this is a living demo/project.

You should notice in the Solution, three different projects.

NerdsForHire.Services: This is the C# Web API, AngularJS, HTML etc. This comprises the primary components of the Client and Client-Server interaction of the App. This essentially IS the app from a user’s perspective.
NerdsForHire.SQL: This is the SQL code that you need to deploy to your SQL Azure database for use in the Services Project.
NerdsForHire-GitHubScraper: This is the Web Job we deploy along side our app which scrapes GitHub for users data. It uses one of my favorite languages, F# :).

To Start, We should Deploy this solution to your own Azure Account.

Always deploy your database first. This is as simple as going to www.azure.com, signing into your account (or creating a new one) and finding the Databases tab on the left and choosing to create a new one. After which, you can right click the NerdsForHire.SQL project and publish. There will be a series of boxes that you must fill out for this. I’m going to assume that you can figure this out. If you struggle here, you should tweet me @DavidCrook1988 for assistance and I will not only assist you, but also update this blog with more specific steps.

You will also need to create a storage account with a queue. The name of the queue will need to be all lowercase. You will need to change the Web API code and the Scraper code to reflect the name change of this queue.

After the Database, let Deploy the App itself. This is SUPER simple! Right click the .Services project, click publish, and select “Azure Websites”. It will prompt for you to login, do so, and then elect to create a new website with whatever name you feel like. Note, you will need to modify the web.config within the project to point to your database specifically. Change the connection string to point to YOUR SQL Database (mine might not still be around).

Finally, Deploy the Web Job. Again, super simple, right click publish, select the previously created web site. You are golden! Note, on this, you may need to install several F# extensions or plugins. Tweet me @DavidCrook1988 if you run into trouble, or see the following article: https://indiedevspot.azurewebsites.net/2015/02/18/deploying-f-web-job-to-azure/

Explaining the Code

We will only focus on NerdsForHire.Services as the other two are pure Web Job and SQL Script, which is not the core focus of this article.

AngularJS

Within the NerdsForHire.Services Project, you will see a folder called “Scripts”. This is where all of our Angular magic happens. Vendor contains all of the logic provided by third parties. These are dependencies that our client takes on to provide the experience that it does. App.module.js contains all of the core setup and initialization logic, including our routing. You will notice as you click on links you get mywebsite.com/#/routeOne. This is set up by the routing. Controllers contain the logic which is injected into the views and Views contains the actual views for our code.

There is only one view, which is not a .html page, and this is a result of the current ASP.Net templates defaulting to requiring a HomeController and needing an index to route to. This page is basically an html page, which provides the initial landing of our app and you can begin to experience how databinding occurs in Angular.

@*

"app">


 

    FSharp Web Kit

*@
"app">
    "HomePageController as HomePage">
 
        {{HomePage.Title}}
        
            "routeOne">Route One

            "routeTwo">Route Two

            "routeThree">Route Three

        
        @*
                "/#/login">Login

                "/#/register">Register

            *@
 
        
        <code class="js string">"btn_GitHubName"</code><code class="js plain">> 
        

            {{HomePage.GitHubID}}
        
        
 
        
    


@*
*@

Above follows John Papa’s Angular Style guide. This particular HTML is considered the View in the MVC model. To see the C or, controller, take a look at the code below from the controllers folder.

(function () {
    'use strict';
 
    angular.module('app')
        .controller('HomePageController', ['$scope', '$http', function ($scope, $http) {
            var vm = this;
            vm.Title = "FSharp Web Kit";
            vm.GitHubID = "GitHubID";
 
            $scope.scrapeGitHub = function () {
                alert("Attempting to Scrape!");
                var gitId = document.getElementById("btn_GitHubName").value;
                var data = {
                    "Id": 0,
                    "FirstName": "David",
                    "LastName": "Crook",
                    "Specialty": 0,
                    "TagList": null,
                    "githubId": gitId,
                    "Jobs": [],
                    "Specialty1": null,
                    "NerdSpecialtyRefs": []
                }
                $http.post("/api/Nerd/Scrape/", data);
            };
        }]);
})(); 

This page shows hooking up a function to a button press. In this instance, on the push of the button with tag ng-click we fire off the scrapeGitHub function, which posts out this data to our Web API, which then in turn pushes an item on Azure Queue storage, which is picked up by our webjob, processed and pushed into SQL Azure. Bad Ass!

Lets take a quick look at the code that does this…

[ResponseType(typeof(bool))]
[HttpPost]
public async Task Scrape(Nerd nerd)
{
    try
    {
        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(@"DefaultEndpointsProtocol=https;AccountName=nerdsforhirestorage;AccountKey=iqFlEAh3ZkXbU6AMbbtgH11o0PHyNslMFWbNqRQ4jD7OqK7lVClZ7RLJgTaA1TL9v/t/w2W0FoeEIpUBUmHZVw==");
        CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
        // Retrieve a reference to a queue
        CloudQueue queue = queueClient.GetQueueReference("githubscrapeinput");
        // Create the queue if it doesn't already exist
        queue.CreateIfNotExists();
        // Create a message and add it to the queue.
        CloudQueueMessage message = new CloudQueueMessage(nerd.githubId);
        await queue.AddMessageAsync(message);
        //new
    }
    catch(Exception e)
    {
        return Ok(false);
    }
    return Ok(true);
}

Pretty straightforward, we receive a moderate sized packet, and we really only need the githubId portion of that and push it onto the queue. The web job actually then does something with that. Lets go ahead and peek at that then. Note that this code lives in the project: NerdsForHire-GitHubScraper. I am only pasting the code regarding receiving the message and not the actual HTML parsing. You can look at the rest of the program.fs file for that information if you wish, which is located here: https://github.com/drcrook1/Nerds4Hire/blob/master/NerdsForHire/NerdsForHire-GitHubScraper/Program.fs

let ScrapeGitHub(["githubScrapeInput")>]message : string) =
    let repositories = scrapePage(message)
    let dataContext = sqlContext.GetDataContext()
    repositories |> Seq.iter(fun r ->
                                dataContext.``[dbo].[GitRepository]``.Create(r.Description, r.Link, r.Stars, r.Title) |> ignore
                            )
    dataContext.SubmitUpdates()
 
[]
let main argv =
    let _dashboardConn = @"DefaultEndpointsProtocol=https;AccountName=nerdsforhirestorage;AccountKey=iqFlEAh3ZkXbU6AMbbtgH11o0PHyNslMFWbNqRQ4jD7OqK7lVClZ7RLJgTaA1TL9v/t/w2W0FoeEIpUBUmHZVw=="
    let _storageConn = @"DefaultEndpointsProtocol=https;AccountName=nerdsforhirestorage;AccountKey=iqFlEAh3ZkXbU6AMbbtgH11o0PHyNslMFWbNqRQ4jD7OqK7lVClZ7RLJgTaA1TL9v/t/w2W0FoeEIpUBUmHZVw=="
    let config = new JobHostConfiguration()
    config.DashboardConnectionString <- _dashboardConn
    config.StorageConnectionString <- _storageConn
    printf "%s" config.DashboardConnectionString
    let host = new JobHost(config) 
    //let host = new JobHost()
    host.RunAndBlock()
    0 // return an integer exit code

So as you can see here, we make a little bit further use of our Azure Storage account for monitoring and storage of queued messages. This is a requirement of queue storage. Also notice the function with the annotation: []. This indicates we are listening to incoming messages on the queue “githubscrapeinput”. It is camel cased here, but F# takes care of reducing this to lowercase for us. Typically this should be all lower case.

Wow, we are making some serious progress here. Lets talk a little further about displaying this data. routeTwo is where we go to make the call for all github repositories we have stored in our database. So once the webjob finishes, it puts the data into the SQL Database for us, and routeTwo makes a web api call out to go get the data from the database. It has two way databinding to display that data on the routeTwo injected HTML.

Lets look at the HTML first.

<div ng-controller="RouteTwoController as RouteTwo">
    RouteTWO! YAYAYAYAYAY!
    <ul>
        <li ng-repeat="repo in RouteTwo.Repositories">
            <h2>Repositoryh2>
            <span>Title: {{repo.Title}}span>
            <br />
            <span>Stars: {{repo.Stars}}span>
            <br />
            <span>Location: {{repo.Location}}span>
            <br />
            <p>Description: {{repo.Description}}p>
            <h2>-----------------h2>
        li>
    ul>
div>

You can see we define our controller as RouteTwoController. This controller must be defined within the context of our app prior to this html being loaded. Also we rename the controller to RouteTwo, so we can refer to objects within it. We then have a “foreach” in which each repository is named repo and we list out the title, how many stars and the description of each repository.

Lets now look at the Controller powering this.

(function () {
    'use strict';
 
    angular.module('app')
        .controller('RouteTwoController', ['$scope', '$http', function ($scope, $http) {
            var vm = this;
            vm.Repositories = [{}, {}];
            $http.get("api/GitRepositories").success(function (data) {
                vm.Repositories = data;
            });
        }]);
})();

Wow, this is a nice short one. We have a self executing function (per John Papa Styling), in which we define an empty repositories variable, and make a get call out to get ALL repositories from our API. We then assign the resulting data from that api call to our repositories. Due to the Angular two way data binding, we do not have to raise any events. Its all taken care of for us. On loading the page, give it about 10 seconds and it will load (due to being on free tier websites and free tier sql).

So what does the Web API for this look like?

public class GitRepositoriesController : ApiController
{
    private NForHire db = new NForHire();
 
    // GET: api/GitRepositories
    public IQueryable GetGitRepositories()
    {
        return db.GitRepositories;
    }

My goodness do I like Entity Framework right about now. Because that kicks so much ass! That’s it folks, that gets all the repositories from our database. Of course we can use OData options to restrict this back to a page by page result set, but that is for you to look up :).

Summary

We covered AngularJS basics, Web API basics and some Azure. I know this is a ton of info to take in as we covered the whole stack plus web jobs. Tweet me: @DavidCrook1988 for any assistance and I will gladly help you get this rolled out.

So you want to be an Analytics Developer?

Dave Crook — Thu, 23 Apr 2015 09:41:38 GMT

Hello World!

I get a series of questions all the time.

How do I switch careers to be a developer?
How do I become a data scientist?
How do I add intelligence to my code?
How do I get a job in distributed computing?
How do I code more analytically?

The answer to these questions are pretty much all the same. Step 1, learn about it and build one piece of software focused on that goal. Step 2, go for it, just do it. So that said, Microsoft has a fantastic resource, Microsoft Virtual Academy, which provides free training around various topics from entry level to advanced. This article focuses on a learning plan with MVA to attain the goal of becoming an Analytics Developer.

This article is a pull from my website: www.indiedevspot.com I pull some articles from there and repost on my msdn blog for better visibility.

What is an Analytics Developer?

An analytics developer is a developer who focuses on answering deeper questions with software. In traditional development shops, I often see basic CRUD operations, or the extent of the analytics is “Who are the top 5 paying customers”. That’s fine, but that is not adding real analytics to a software product.

An analytics developer aims to take a product from its basic components into “We know what drives our top paying customers, lets deliver the experience to all customers based on criteria focused on them”. Or perhaps it is predicting what the new hot commodity is for your website so you can market it first. Perhaps it is realizing that based on statistics a particular customer has a 90% chance of having poor eyesight and we default an experience with larger font.

An analytics developer is a traditional full stack developer who takes their development skills to the next level to produce meaningful value specific to each and every user, be that user an admin, a customer, or even another developer. Beyond the mindset change for an Analytics Developer, you will need a variety of skills.

So what is the learning plan?

The Baseline

As an analytics developer, you will spend 99% of your time working on backend systems. Therefor there are 2 key programming languages you need.

You need C#. The two below courses should be a good beginner as well as intermediary exposure to the core language you will use when interacting with most api’s. C# will be used primarily to define data types, interact with architectural components and most data access.

https://www.microsoftvirtualacademy.com/training-courses/c-fundamentals-for-absolute-beginners
https://www.microsoftvirtualacademy.com/training-courses/developer-training-with-programming-in-c

You will need F#. F# will be used for data crunching, analysis and distributed computing. It will be your language for writing the core libraries used for the actual intelligence.

https://www.tryfsharp.org/Learn

You will definitely need to understand SQL and Entity Framework for data storage and Access. These are the core technologies I use. SQL can support upwards of low TB, and Entity Framework is a great way to get access to that data quickly.

https://www.microsoftvirtualacademy.com/training-courses/implementing-entity-framework-with-mvc
https://www.microsoftvirtualacademy.com/training-courses/database-fundamentals
https://www.microsoftvirtualacademy.com/training-courses/querying-with-transact-sql

You will need to understand a few important Azure Concepts. Azure is my hosting platform of choice, it also has various PaaS capabilities that are crucial to Analytics Developers.

https://www.microsoftvirtualacademy.com/training-courses/getting-started-with-microsoft-azure-machine-learning
https://www.microsoftvirtualacademy.com/training-courses/microsoft-azure-fundamentals-storage-and-data
https://www.microsoftvirtualacademy.com/training-courses/microsoft-azure-fundamentals-websites

You will need data exploration tools as well. These tools are invaluable in that they help you prove your concepts prior to coding them into production systems.

https://www.microsoftvirtualacademy.com/training-courses/from-data-to-insight-and-impact-the-bi-revolution
https://www.microsoftvirtualacademy.com/training-courses/excel-2013-power-bi-fundamentals

Taking it to the next level

These are what I consider the advanced or specialist topics. They are in this section as they will be needed for specific examples, or can be used to really differentiate yourself as an Analytics Developer. These primarily focus around scaling your solutions or increasing performance.

When your data is too big to handle.

https://www.microsoftvirtualacademy.com/training-courses/implementing-big-data-analysis
https://www.microsoftvirtualacademy.com/training-courses/big-data-with-the-microsoft-analytics-platform-system

When you need to worry about on premise resources. These may be more IT Pro, but an understanding of where your data works and lives is crucial to high performance analytics.

https://www.microsoftvirtualacademy.com/training-courses/microsoft-azure-site-to-site-vpn
https://www.microsoftvirtualacademy.com/training-courses/hybrid-cloud-workloads-sql

Everybody needs to know a little about performance

https://www.microsoftvirtualacademy.com/training-courses/sql-server-2014-security-optimizer-and-columnstore-index-enhancements
https://www.microsoftvirtualacademy.com/training-courses/mission-critical-performance-with-sql-server-2014-jump-start
https://www.microsoftvirtualacademy.com/training-courses/querying-with-transact-sql

Summary

That is a TON of information. Likely that you already know a fair amount of it, but if you are looking to jump into software engineering, my favorite place is definitely as an Analytics Developer. There is something about answering questions that provide value that I just really like. If you are comfortable with the material in at minimum the baseline training section, you should be successful in your endeavors. Real Time analysis and Big data are my next favorite place in taking it to the next step. Good luck!

Intro to C# and Analyzing Government Data

Dave Crook — Mon, 20 Apr 2015 16:32:00 GMT

Hello World!

I have recently been informed that many of my articles may be a bit advanced for folks, so I am going to kick off a series of C# articles dedicated to the Beginner to programming. I have no idea how long this series is going to be, I’ll just keep adding to it as requests come in for various topics. This series is meant to take the absolute beginner to a level in which they can possibly derive value from my other articles.

This is a pull from my personal blog at: www.indiedevspot.com

Tools

Lets start off with some tools. For the purposes of this article, there is really only one tool that you need. Visual Studio. As of today Visual Studio 2015 is currently in preview, so in the heart of all that is beta, we will start with that immediately. I imagine Visual Studio 2015 Community to be coming to general availability soon, but no idea when.

The download for Visual Studio is here: https://www.visualstudio.com/en-us/downloads/download-visual-studio-vs.aspx

References

These are the places that I go when I need help figuring something out, but also places you can go to learn.

https://www.microsoftvirtualacademy.com/training-courses/c-fundamentals-for-absolute-beginners
https://stackoverflow.com/
https://msdn.microsoft.com/library
https://github.com/

Terminology

These are some common terms you will hear in the development world. I want to address some of these right away.

Full Stack Developer: A developer who can code from database to web services to client on a single tech stack
Tech Stack: Usually this is .Net, Node, Open Source. Its the series of technologies that you use to accomplish data storage, web services and UI
.Net: Series of development languages and technologies developed by Microsoft. C# is a language part of the .Net family, as is F# and VB. They run in the “CLR”, which is the common language run time or code execution environment these languages run in, which is common across them.
Data Storage: This is typically in reference to long term data storage or databases, such as SQL.
SQL: A relational database developed by Microsoft for storing tabular data with relationships between the tables.
Web Services: Endpoints on the web that can be accessed that complete tasks and return results. Typically Restful.
Endpoint: Typically a globally accessible or resolvable address on the internet, such as https://www.mywebsite.com/api/Cars
Restful: Type of internet calls for endpoints. These are typically Get, Post, Delete. A series of known verbs control this. It is typically manifested in a url as: https://www.mywebsite.com/api/Cars/Black/1 (would get a black car with an id of 1 if following normal conventions).
Object Oriented: A paradigm of coding in which objects are the primary feature of the language.
Object: A section of code that contains data as well as methods to manipulate data. Manifested as pieces of memory that can be instantiated in your code. Objects are common in structure, but differ in mutable types.
Mutable type: Data that can be changed that lives in memory, typically in the context of an object.
Logic: The intelligence to your code that manipulates data to produce a desired result.

Getting Started

Start by opening Visual Studio 2015.

On the top left, you should see a series of buttons, click File -> New -> Project

Expand Templates, click on Visual C# and select ConsoleApplication. Name the application as seen below.

Click OK.

You should now have a view like below…

Lets take a minute to talk about this.

I will explain concepts in need to order.

namespace Tutorial: This defines that code within these curly brackets ({ }) will be referenced by the name Tutorial.
using System; : System is a namespace, we are using the namespace System, such that we can have access to objects within it.
class Program: This is the default class built for us, other code will reference this via Tutorial.Program
static void Main(string[] args): this is the “entry point” of the application. This is where everything starts.

Now lets start with the super simple “Hello World” before we jump into something way more interesting.

namespace Tutorial
{
    public class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
            Console.ReadKey();
        }
    }
}

Change your code to reflect above (keep your using statements). This simply will write to the console “Hello World!” and then wait for you to push any key before moving on.

To run the program, on the very top bar in Visual Studio is a “Debug Button”. Click this to execute your code. It is identifiable by being a green Arrow |> pointing to the right with Start next to it.

You will see a console pop up and state “Hello World”, you can push a button and it will go away.

Doing something Real now.

So now lets do something interesting, like figure out the best and worst place to live in the United States based on the open data on data.gov :)

Download the data from here: https://onedrive.live.com/redir?resid=478A64BC5718DA47!300&authkey=!AP7WdDyZ_Inbfas&ithint=file%2cxls

This is real data pulled from data.gov.

First, we need to bring in a library that helps us parse Excel files.

Open Solution Explorer

On the right hand side of the screen is Solution Explorer, if you don’t have it there, it can be found at the top button bar View -> Solution Explorer.

Add the Excel Nuget Package. Right Click the .csproj file or Tutorial, highlighted in blue here

After right clicking Tutorial, a submenu will pop up, click “Manage Nuget Packages”. A new dialogue will fill the screen. In the search text box, type “Excel”.

Click on ExcelDataReader and click Install.

Close that tab.

Now we are ready to do something useful!

Lets start by adding new using statements that we need to the top of the file:

using Excel;
using System;
using System.Collections.Generic;
using System.Data;
using System.IO;

Lets build a few types…

Within the Tutorial namespace, add the following code:

//Enum Type
//These are actually integer values, but can be named.
//Useful for switch statements.
public enum Stats
{
    Employed,
    Unemployed,
    MedianIncome,
    LaborForce,
}
/// 
/// Data Class that we can use for holding data.
/// 
public class AreaStatistic
{
    /// 
    /// Data property of type string, holds the state this area is in.
    /// 
    public string State { get; set; }
    /// 
    /// Name of the area within the state.
    /// 
    public string AreaName { get; set; }
    /// 
    /// List of statistics for each year of this particular area.
    /// It is a List of a Tuple, which is a pair of objects, in which the first
    /// object is an integer and the second object is a tuple.
    /// The second Tuple is a pair of objects in which the first is a Stat enum and the second is
    /// a float?  a float? is a nullable floating point number
    /// 
    public Listint, Tuplefloat?>>> YearlyStats { get; set; }
 
}

This code defines the data types we will be using for our analysis. The AreaStatistic defines a class, which, when instantiated is an object.

Lets now write out our code for getting the statistics for a year.

/// 
/// Gets stats for a particular year.
/// 
/// Data Row you want to get stats for for that year
/// Year you want to have data for
/// index starting point for that year.
/// 
public static Listint, Tuplefloat?>>> GetStatsForYear(DataRow r, int year, int i)
{
    //Create the empty object which holds the stats.
    Listint, Tuplefloat?>>> stats = new Listint, Tuplefloat?>>>();
    //Begin big region for making sure we deal with empty or null data.
    string s = r.ItemArray[i].ToString();
    string s1 = r.ItemArray[i + 1].ToString();
    string s2 = r.ItemArray[i + 2].ToString();
    float? f;
    float? f1;
    float? f2;
    if (string.IsNullOrEmpty(s) || string.IsNullOrWhiteSpace(s))
    {
        f = null;
    }
    else
    {
        f = float.Parse(s);
    }
    if (string.IsNullOrEmpty(s1) || string.IsNullOrWhiteSpace(s1))
    {
        f1 = null;
    }
    else
    {
        f1 = float.Parse(s1);
    }
    if (string.IsNullOrEmpty(s2) || string.IsNullOrWhiteSpace(s2))
    {
        f2 = null;
    }
    else
    {
        f2 = float.Parse(s2);
    }
    //End big area of checking for bad data.
 
    //get data for LaborForce this year
    stats.Add(
        new Tuple<int, Tuplefloat?>>
        (year, new Tuplefloat?>
        (Stats.LaborForce, f)));
    //get data for # employed this year.
    stats.Add(
        new Tuple<int, Tuplefloat?>>
        (year, new Tuplefloat?>
        (Stats.Employed, f1)));
    //get data for # unemployed this year.
    stats.Add(
        new Tuple<int, Tuplefloat?>>
        (year, new Tuplefloat?>
        (Stats.Unemployed, f2)));
    //return the yearly stats.
    return stats;
}

Wow! That is a TON of code! Most of it is error checking, man I am missing F# pattern matching right now…The rest simply gets sections out of the row and adds it to a stats type. So what calls this code?

/// 
/// Converts a single row into the AreaStatistic Type.
/// Note, this is based on the format of the file being known.
/// 
/// Data Row
/// AreaStatistic
public static AreaStatistic ConvertRowToStat(DataRow r)
{
    //create an empty stat object we can populate
    AreaStatistic stat = new AreaStatistic();
    //Get the state code from the data row.
    stat.State = r.ItemArray[1].ToString();
    //if the state code is not of length 2, it is not a data row.
    if(stat.State.Length != 2)
    {
        //Throw an exception so the code stops here and doesn't continue.
        //An exception will be printed to the console due to the line of code that
        //wraps this.
        throw new Exception("Not data row");
    }
    //get the area name.
    stat.AreaName = r.ItemArray[2].ToString();
    //Build an empty object for the yearly stats.
    stat.YearlyStats = new Listint, Tuplefloat?>>>();
    //Stat years are 2000-2013.  The columns start at column 9,
    //There are three rows of data and 1 skipped row.
    //Therefor we do 9 (for year 2000 and add 4 * the current iteration to that)
    for (int i = 0; i < 14; i++)
    {
        //Add to the list of stats the stats for this year, total of 13 stats.
        stat.YearlyStats.AddRange(Program.GetStatsForYear(r, 2000 + i, 9 + (i * 4)));
    }
    //row 65 has income data.
    string s = r.ItemArray[65].ToString();
    float? f;
    //sometimes data is blank, we need to deal with that.
    if (string.IsNullOrEmpty(s) || string.IsNullOrWhiteSpace(s))
    {
        //representation of no data.
        f = null;
    }
    else
    {
        //we should be good to parse this.
        f = float.Parse(s);
    }
    //add the data.
    stat.YearlyStats.Add(
        new Tuple<int, Tuplefloat?>>
        (2013, new Tuplefloat?>
        (Stats.MedianIncome, f)));
    //Return the stats for this area.
    return stat;
}

Wow, a bunch more code! Again, notice that 99% of this code is dealing with poorly formed data and also comments. Notice the amount of code re-use we get out of the GetStatsForYear method. This is great, always try to extract methods out like this that you can use. So Finally, we get into the reading of the data in. This also has been extracted out, so readers can simply download the .xls file, put it where they want, and point the code to where it is on your laptop.

/// 
/// Method we define that allows our Main function to look cleaner.  This reads in the data and returns
/// a list of statistics
/// 
/// location on our drive where the file is.
/// List of statistics for each area.
public static List ReadInData(string url)
{
    //Initialize the stats object as a new list that holds objects of type AreaStatistics
    List stats = new List();
    //Using ensures that the reader is disposed of from memory when we are done with it.
    using (StreamReader reader = new StreamReader(url))
    {
        //This is from the ExcelData library we brought in, that parses the excel data for us.
        IExcelDataReader excelReader = ExcelReaderFactory.CreateBinaryReader(reader.BaseStream);
        //Converts the stream into a dataset type.
        DataSet d = excelReader.AsDataSet();
        //There is only 1 table, but we iterate as if there were multiples anyways.
        foreach (DataTable dt in d.Tables)
        {
            //Iterate each row in the table.
            foreach (DataRow r in dt.Rows)
            {
                try
                {
                    //add to our stats list the statistic after we convert it.
                    stats.Add(Program.ConvertRowToStat(r));
                }
                catch(Exception e)
                {
                    //if something goes wrong, write it to the console.
                    //tell me what happened.
                    Console.WriteLine(e.Message);
                }
            }
        }
    }
    //return the list of stats
    return stats;
}

This doesn’t seem so bad. You can see the catching of the exception, this is where if any transient missed exceptions get thrown will be caught such that our program won’t just up and crash on us. The downside is that if an exception is thrown due to bad data, that row will not be added to our stats, and instead the console will write out the message of what happened.

Finally, we take a look at our code entry point:

/// 
/// Entry point to the program.
/// 
/// Command line arguments passed in.
static void Main(string[] args)
{
    //We populate the type "stats" with data we read from the .xls file.
    List stats = Program.ReadInData("C:\\Unemployment.xls");
    //just lets us see some output.
    Console.ReadKey();
}

From here, you can see our high level task, we wanted to read in the statistics into this “stats” object. We now have a list of “stats” that we can do some real analysis with :).

The next article will focus on doing interesting things with this data for more insights into it. Maybe we can see if we can verify the claims about unemployment rates. Maybe there are interesting insights from this data we can see from a collection standpoint, or growth rates in total labor vs employed vs unemployed vs income.

Please use the comments section for any questions you might have. I know this might be a bit more advanced than I intended. The full code is pasted below…

Summary

We have covered some C# basics, as well as our first dive into analyzing government data.

using Excel;
using System;
using System.Collections.Generic;
using System.Data;
using System.IO;
 
namespace Tutorial
{
    //Enum Type
    //These are actually integer values, but can be named.
    //Useful for switch statements.
    public enum Stats
    {
        Employed,
        Unemployed,
        MedianIncome,
        LaborForce,
    }
    /// 
    /// Data Class that we can use for holding data.
    /// 
    public class AreaStatistic
    {
        /// 
        /// Data property of type string, holds the state this area is in.
        /// 
        public string State { get; set; }
        /// 
        /// Name of the area within the state.
        /// 
        public string AreaName { get; set; }
        /// 
        /// List of statistics for each year of this particular area.
        /// It is a List of a Tuple, which is a pair of objects, in which the first
        /// object is an integer and the second object is a tuple.
        /// The second Tuple is a pair of objects in which the first is a Stat enum and the second is
        /// a float?  a float? is a nullable floating point number
        /// 
        public Listint, Tuplefloat?>>> YearlyStats { get; set; }
 
    }
    /// 
    /// Primary program that lets us do stuff.
    /// 
    public class Program
    {
        /// 
        /// Entry point to the program.
        /// 
        /// Command line arguments passed in.
        static void Main(string[] args)
        {
            //We populate the type "stats" with data we read from the .xls file.
            List stats = Program.ReadInData("C:\\Unemployment.xls");
            //just lets us see some output.
            Console.ReadKey();
        }
        /// 
        /// Method we define that allows our Main function to look cleaner.  This reads in the data and returns
        /// a list of statistics
        /// 
        /// location on our drive where the file is.
        /// List of statistics for each area.
        public static List ReadInData(string url)
        {
            //Initialize the stats object as a new list that holds objects of type AreaStatistics
            List stats = new List();
            //Using ensures that the reader is disposed of from memory when we are done with it.
            using (StreamReader reader = new StreamReader(url))
            {
                //This is from the ExcelData library we brought in, that parses the excel data for us.
                IExcelDataReader excelReader = ExcelReaderFactory.CreateBinaryReader(reader.BaseStream);
                //Converts the stream into a dataset type.
                DataSet d = excelReader.AsDataSet();
                //There is only 1 table, but we iterate as if there were multiples anyways.
                foreach (DataTable dt in d.Tables)
                {
                    //Iterate each row in the table.
                    foreach (DataRow r in dt.Rows)
                    {
                        try
                        {
                            //add to our stats list the statistic after we convert it.
                            stats.Add(Program.ConvertRowToStat(r));
                        }
                        catch(Exception e)
                        {
                            //if something goes wrong, write it to the console.
                            //tell me what happened.
                            Console.WriteLine(e.Message);
                        }
                    }
                }
            }
            //return the list of stats
            return stats;
        }
 
        /// 
        /// Converts a single row into the AreaStatistic Type.
        /// Note, this is based on the format of the file being known.
        /// 
        /// Data Row
        /// AreaStatistic
        public static AreaStatistic ConvertRowToStat(DataRow r)
        {
            //create an empty stat object we can populate
            AreaStatistic stat = new AreaStatistic();
            //Get the state code from the data row.
            stat.State = r.ItemArray[1].ToString();
            //if the state code is not of length 2, it is not a data row.
            if(stat.State.Length != 2)
            {
                //Throw an exception so the code stops here and doesn't continue.
                //An exception will be printed to the console due to the line of code that
                //wraps this.
                throw new Exception("Not data row");
            }
            //get the area name.
            stat.AreaName = r.ItemArray[2].ToString();
            //Build an empty object for the yearly stats.
            stat.YearlyStats = new Listint, Tuplefloat?>>>();
            //Stat years are 2000-2013.  The columns start at column 9,
            //There are three rows of data and 1 skipped row.
            //Therefor we do 9 (for year 2000 and add 4 * the current iteration to that)
            for (int i = 0; i < 14; i++)
            {
                //Add to the list of stats the stats for this year, total of 13 stats.
                stat.YearlyStats.AddRange(Program.GetStatsForYear(r, 2000 + i, 9 + (i * 4)));
            }
            //row 65 has income data.
            string s = r.ItemArray[65].ToString();
            float? f;
            //sometimes data is blank, we need to deal with that.
            if (string.IsNullOrEmpty(s) || string.IsNullOrWhiteSpace(s))
            {
                //representation of no data.
                f = null;
            }
            else
            {
                //we should be good to parse this.
                f = float.Parse(s);
            }
            //add the data.
            stat.YearlyStats.Add(
                new Tuple<int, Tuplefloat?>>
                (2013, new Tuplefloat?>
                (Stats.MedianIncome, f)));
            //Return the stats for this area.
            return stat;
        }
        /// 
        /// Gets stats for a particular year.
        /// 
        /// Data Row you want to get stats for for that year
        /// Year you want to have data for
        /// index starting point for that year.
        /// 
        public static Listint, Tuplefloat?>>> GetStatsForYear(DataRow r, int year, int i)
        {
            //Create the empty object which holds the stats.
            Listint, Tuplefloat?>>> stats = new Listint, Tuplefloat?>>>();
            //Begin big region for making sure we deal with empty or null data.
            string s = r.ItemArray[i].ToString();
            string s1 = r.ItemArray[i + 1].ToString();
            string s2 = r.ItemArray[i + 2].ToString();
            float? f;
            float? f1;
            float? f2;
            if (string.IsNullOrEmpty(s) || string.IsNullOrWhiteSpace(s))
            {
                f = null;
            }
            else
            {
                f = float.Parse(s);
            }
            if (string.IsNullOrEmpty(s1) || string.IsNullOrWhiteSpace(s1))
            {
                f1 = null;
            }
            else
            {
                f1 = float.Parse(s1);
            }
            if (string.IsNullOrEmpty(s2) || string.IsNullOrWhiteSpace(s2))
            {
                f2 = null;
            }
            else
            {
                f2 = float.Parse(s2);
            }
            //End big area of checking for bad data.
 
            //get data for LaborForce this year
            stats.Add(
                new Tuple<int, Tuplefloat?>>
                (year, new Tuplefloat?>
                (Stats.LaborForce, f)));
            //get data for # employed this year.
            stats.Add(
                new Tuple<int, Tuplefloat?>>
                (year, new Tuplefloat?>
                (Stats.Employed, f1)));
            //get data for # unemployed this year.
            stats.Add(
                new Tuple<int, Tuplefloat?>>
                (year, new Tuplefloat?>
                (Stats.Unemployed, f2)));
            //return the yearly stats.
            return stats;
        }
 
 
    }
}

Deploying F# Web Job to Azure

Dave Crook — Wed, 18 Feb 2015 13:52:57 GMT

Hi All,

This is another pull from my blog over at: www.indiedevspot.com

This article is one of those that is going to help remind me how to do this deployment, as it can be a bit tricky. If you are working with F# for web jobs, like I have started doing, there are a few steps.

Create a new console application
Add proper nuget packages
Manually add a .dll reference and copy said .dll to output
Zip up and deploy entire output folder.

Create new Console Application

This is easy. See picture.

Add Proper Nuget Packages

Microsoft.Azure.WebJobs.Core
Microsoft.Azure.WebJobs

These packages contain the framework for Azure Web Jobs.

Add Proper References

So I discovered that I needed FSharp.Core in my solution. I got the craziest errors that seemed to have nothing to do with this, until I added a printf statement to see what my connection string was. Anyways, you need to manually add a reference to FSharp.Core v4.3.1 . You also need to copy this to the output directory of your solution, as setting the properties to copy currently does not actually copy this library as expected.

Zip Up Entire Folder and Deploy

This should be straight forward. Using any other article out there in the world, you can deploy, but I will walk through here as well.

Go to your Azure Portal https://azure.microsoft.com/en-us/
Navigate to the website you wish to deploy this webjob with
Click on the WebJobs section
Click the Add button
Name your web job
Select your zipped up output folder
Set your job per type you want (I use continuously as I link to queues mostly)
Click the check mark
If it stays on “Running” everything went well.

Summary

That’s it! If you want, you can navigate to this repository, which has a functional web job you can try out: https://github.com/drcrook1/Nerds4Hire Note that you will need to go through all of these steps with this same project.