Architecture overview of the AUBP

Default report generated by AUBP

We highly recommend to have general knowledge on Azure Resource Usage and RateCard APIs and Developer’s guide to auth with Azure Resource Manager API to develop similar projects or customize this one.

    In this post we will mention:

• Develop Python application in a Microsoft Windows environment with Visual Studio
• Publish it to a RaspberryPi (RPi) that is running Raspbian OS
• Run the app on RaspberryPi and remotely debug it with Visual Studio on Windows

In this link you can find some of the answers to the question "Why to develop Python app in Visual Studio?" We will use fresh Raspbian installation and show step by step preparation of the Raspberry Pi device.

Requirements

• Windows machine with Visual Studio Community Edition (Free for developers)
• Python Tools for Visual Studio (Open Source)
• Raspberry Pi 2 Model B with Raspbian OS (+Power Supply, Micro SD Card, Ethernet Cable or WIFI dongle)
• Wired or Wireless Internet connection
• Putty SSH Client

Preparation

• Install Visual Studio then Python tools for Visual Studio (If not already installed), connect the machine to internet
• Install Putty on your desktop (actually just copy the single EXE file on to your Desktop)
• Install Raspbian on RaspberryPi (setup guide)
• Connect RaspberryPi to Internet (wired or wireless). (Howto wireless connection)

Be sure that both the development machine (with Visual Studio) and the RaspberryPi connected to the same network. You should be able to ping RaspberryPi from the windows machine.

Get IP Address

We need to know the IP address of the RaspberryPI device so we can deploy our Python script and debug it remotely. To get the IP address of the RPi device follow the steps mentioned here. In this post our RPi device's IP address is: 192.168.0.107

Get Putty & Open a Terminal Window

We don't want to switch between one monitor or keyboard to another. So we will use the development machine to access the RPi device remotely. To do so open a Putty window and enter your RPi's IP address as shown in the below screenshot. Keep the rest of the settings as they are.

    Click the "Open" button to open a terminal window to the RPi device.

Prepare RPi for Python Project

• Login to the RPi device through the terminal window opened in the previous step.

  Default username is: pi

  Default password is: raspberry

   

  

   

• Create folder named "pythonapp" and change its access mode to "0777" so anyone can access/read/write in this folder. Note its path by typing "pwd" command. In our case the path is: "/home/pi/pythonapp"

   

  

  Commands used in order:

      mkdir pythonapp

      chmod 0777 pythonapp

      cd pythonapp

      pwd

   

• Update all packages to its recent version by typing the following command in the terminal window:

  sudo apt-get update

   

• Install opensource ptvsd remote debugging libraries on RPi device by typing the following command in the terminal window:

  sudo pip install ptvsd

   

Enable Windows file share on RPi device

• Install Samba software package by typing the following command in the terminal window:

  sudo apt-get install samba samba-common-bin

   

• Configure Samba installation by typing the following command in the terminal window:

  sudo nano /etc/samba/smb.conf

   

  above command will open an editor with samba configuration file. Scroll down and add the following lines to the end of the configuration file.

   

  

   

  lines added to the end of the configuration file:

  [pythonapp]

  path=/home/pi/pythonapp

  browsable=yes

  writable=yes

  only guest=no

  create mask=0777

  directory mask=0777

  public=yes

   

  Switch back to windows, open Windows File Explorer in the development machine. Type "\\192.168.0.107\pythonapp" in the address bar. You should be able to explore the empty folder in RPi device from a windows machine.

   

  

   

Develop & Publish Python App

• Open Visual Studio, create a new project.

  

   

• Select the "Python Application" from the templates, and create a blank project.

  

• Change project's Python environment by right click on the "Python Environment" node under the project name in the solution explorer window. Click on the "Add/Remove Python Environments…" item in the pop-up menu.

  

   

• Select just the "Python 2.7" in the pop-up window and click OK button.

  

   

• Right click on the "Python 2.7" node under the "Python Environments" in the solution explorer window. Click on the "Install Python Package…" item in the pop-up menu.

  

   

• Type "ptvsd" in the package to install edit box and click OK.

  

   

• Enter the following lines of Python code in to the "pythonapp.py" file through the editor.

  import ptvsd

  ptvsd.enable_attach(secret='my_secret')

   

  first_num = int(raw_input('Enter first number: '))

  second_num = int(raw_input('Enter second number: '))

  sum = first_num + second_num

  print('{0} + {1} = {2}: '.format(first_num, second_num, sum))

   

   

  

   

• Click the "Start" button on Visual Studio toolbar to run the script locally.

  

   

• Console window will open and waits you to enter two integer value, then shows the sum of these two numbers.

  

• Now lets make required settings to be able publish the app to RaspberryPi device. Open the project properties window by right clicking on the project name under solution explorer window and then click on the "properties" menu item.

  

   

• In the properties window, switch to the "Publish" tab, enter publish location as "\\192.168.0.107\pythonapp" (or whatever in your case) then press "Publish Now" button.

   

  

   

  This will copy the latest version of your Python script into the Raspberry Pi device.

   

• Any time you update the Python script, just right click on the Project name in the solution explorer window and click on the "Publish" button afterwards. This will also copy the latest version of your Python script into the Raspberry Pi device.

  

Remotely Debug the Python script that is running on Raspberry Pi Device

• Switch back to the Putty terminal window. If session timed-out, re-enter your credentials and go to the "pythonapp" folder. Type the following command in the SSH terminal window:

  python pythonapp.py

   

• Previous command will run the python script. Before entering first number switch back to "Visual Studio" window.

  

• In Visual Studio, put a breakpoint on the line that starts with "sum = first…"

  

   

• In the "Debug" menu, click on "Attach to Process…" menu item.

  

   

• In the "Attach to Process" window, select "Python remote (ptvsd)" as Transport. Enter "tcp://my_secret@192.168.0.107" as Qualifier and press ENTER. Available Python processes on the RPi device will be listed. Select the one to Debug and press "Attach" button in this window.

   

  

   

  Here the keyword "my_secret" in the Qualifier must be the same keyword that we entered in the second line of the Python source code.

   

• Now go back to the SSH terminal to the RPi device and enter the two integer value to calculate their sum. When you enter the second number and press ENTER, Visual studio will flash and the execution cursor will stop on the breakpoint. You can browse local variables, threads, their values etc. on the Locals, Watch, Threads, Call Stack etc. windows. You can execute the script line by line remotely.

   

  

• How to develop simplest Azure Machine Learning (AML) experiment?
• How to publish AML Experiment as a scalable Web Service?
• How to consume the Web Service from an R/Python script?

How to develop simplest Azure Machine Learning (AML) experiment?

In our initial post on ML series, using the R studio, we have developed a sample ML solution with linear R model. In this post we will develop the same solution completely in AML with minimum effort.

Referring to our previous post, create a new "blank experiment" and add (drag/drop) the following modules on the empty canvas:

• "Enter Data Manually": used to enter training data
• "Linear Regression": Linear Regression Model
• "Train Model": Module to train the Model with provided data

Below is the screen shot from the canvas with the above modules connected to each other. To connect the modules with each other, just press the mouse left button on the modules output port, and drag/drop it on to the input port of the other module. You will see a directed arrow will be drawn. Referring to the below screenshot, if you can't find the above modules, just type their names inside the "Search experiment items" search box, it will be filtered out for you.

On the right side of the window you will see a child window (like the toolbox) with the caption "Properties". One by one, clicking on (making it selected) each of the three module, you can see corresponding properties of that module in the properties window. Starting from the first module "Enter Data Manually", we will update their properties.

First we will generate a training dataset. Using the same method mentioned in previous post, with an Excel sheet generate data series with random noise. Plotting this data should look like a noisy line with slope = 1 as shown below. This data consists of two columns: X and Y. For each value in X column you will see the corresponding Y value. You should observe that almost all Y values are (+/- 50) same to the X value.

Aim is to develop an ML solution to predict corresponding Y value of any X value which does not exists in our training dataset.

 How to generate training data video:

[View:http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-56-12/8838.ml04Blog.mp4] 

After settings the properties of the modules, now it is time to train our model by pressing the run button. On the bottom of the page.

It will take around a minute to run the solution and after a successful run you will see green checkmarks on the right side of each module.

How to publish AML Experiment as a scalable Web Service?

How will we use this ML solution? Will it take ~1 minute each time to train the model for such small amount of data for training? No, we don't need to train our model every time. Once it is trained, if there is no major change in your training dataset, then you can keep the trained model as it is. To be able to use this trained model, you need to publish it as a web service. To do so, first click on the "Set up web Service" button near the "RUN" button. In the menu windows, select "Create Predictive Experiment". With this command it will create a new experiment by adding "Web Service Input" and "Web Service Output" modules.

I also made small modification by manually adding two "Project Columns" modules. You can omit them, there will not be a problem in your solution. Reason for putting these two modules is to filter out the Y column in the training dataset. As we already trained our model, we don't need Y column anymore, and we don't want it to be visible in our new web service as an input parameter. Because we want to predict this value, so there is no need to provide it as an input parameter. Same for the output. We filter out the output data by just allowing "Scored values" – predicted values to be visible in the web service output.

Now you have to "Run" this experiment once again to be able to publish it as a web service. After a successful run, you will see "Deploy Web Service" button near the "RUN" button, press it. After few seconds your web service will be ready.

Press the "TEST" button which will pop out an input parameters dialog box. Type any "X" value that you want to predict its corresponding "Y" value. i.e. 3578, which is not in our training set.

After few seconds you will see the output of this web service on the bottom of the page as:

How to consume the Web Service from an R script?

Just click on the "Response/Request" link near the "TEST" button. It will open a new page with sample codes. Copy/Paste the R code into a new page in RStudio.

In this sample code you have to update 2 lines. One is the key (password) to access the web service that we created. Copy the key in the web service management page:

And type this key into the following line by deleting the "abc123"

api_key = "abc123" # Replace this with the API key for the web service

Also you should update the web service input parameters. Following line

"Values" = list( list( "0" ), list( "0" ) )

with

"Values" = list( list( "3578" ), list( "6000" ) )

In this sample two different input values provided, 3578 and 6000. You can provide as much as you want by putting comma between them.

If you have just installed the RStudio (we mentioned how to install it in the first ML post) on your machine, then you will get two error message when you run the sample code:

1. Error in library("rjson") : there is no package called 'rjson'
2. Error in library("RCurl") : there is no package called 'RCurl'

Our sample code uses these packages and they are not installed to RStudio by default. You can install them from Tools->Install Packages menu.

Just type the missing package name in the text box and press install. Do it both for "RCurl" and "RJson"

After all, you can press CTRL+ALT+R button combination to run the R code. In the console window you will see the output of the web service with predicted values of the input Xs.

…

$Results$output1$value$Values

[1] "3606.16504920019" "6030.31943595806"

…

Following experiments will answer the above questions and we will develop our first AML experiment with a custom R script.

In our previous blog post, we mentioned about R and R Studio and also developed a simple R script that runs an ML model.

(quote from prev. post) You can get the free open source development environment and libraries from https://www.r-project.org. Moreover, a free IDE with GUI can be found at https://www.rstudio.com. There also exist hundreds of open source algorithms, software libraries under https://cran.r-project.org.

1. Open R Studio and type the following simple code on the console window and press ENTER

    > version

    output will be something like:

    Here we run the R command named "version" which returns the R version related details and print them on the command window. version.string or minor/major values in the output window are the version of R environment installed on our machine. In our case local R version is: 3.1.0

1. Now try to do the same in AML. Referring to our previous blog post, open Microsoft Azure ML Studio, create a blank Experiment.

    

   

    

2. From the toolbar/modules bar on the left, drag and drop an "Execute R Script" module on the empty canvas. If you can't find the "Execute R Script" module within tens of other modules, just type one of the words in "Execute R Script" into the search box on the left top corner of the window on modules bar.

    

   

    

3. Referring the above screenshot, click on the module (make the "Execute R Script" module selected). Then you will see properties window on the right side of the main window. Under properties window, you can write your custom R script within the "R Script" text box.

    

   Select all text inside "R Script" textbox and delete them. Now copy/paste the following custom R code inside the textbox.

           

1. Now press on the RUN button on the bottom of the window to run this experiment. Once the experiment run successfully, you can see the green checkmark on right top corner of the canvas and each module in the experiment.

   

2. So what is the output of this experiment? In AML, almost every module has input and output ports. Each module is a part of execution pipeline. To see the output of this R module, click on the port number 1 on "Execute R Script Module". Then select "Visualize" command from the popup menu items.

   

3. Clicking on the Visualize command will open another window with the results that we expected. Evaluating the result window, output is same as our local result. We are running exactly the same R environment as AML.

   

4. What about the installed R packages on AML? Typing command "installed.packages()" in R Studio will print out the local installed package list. To do the same in AML, just copy paste the following script as we do for "version" and view the result from port number 1

TODO: In our upcoming post, we will mention about installing and running custom R packages. Developing own R learning model and using it in a native AML experiment.

AML service have the capability to run both custom R and Python scripts along with several preinstalled Python, R software libraries. With Free and Standard service tiers, it is possible to create an AML workspace and start using the service immediately. Please note that the free tier doesn't cover all available features and designed for developers to have them general overview about the service. To start using any tier of this service you need a Microsoft account and then following the https://studio.azureml.net/Home address you can activate your free subscription. For testing the whole AML features, you need a Microsoft Azure account.

Machine Learning Studio

Machine Learning Studio (MLS) is a browser-based tool used to develop predictive models by dragging, dropping, and connecting predefined or custom modules / scripts. Once you created a free AML service account you can start using MLS directly from https://studio.azureml.net. Or If you have an Azure subscription, want to test the full features of AML, then you first create an Azure Storage where you will store your AML data, custom scripts, saved models etc. and a machine learning workspace that will be a base to the MLS for developing your custom solutions.

Follow the below pictorial guide to create an AML workspace. As highlighted on the image, you can select any existing azure storage account from the listbox. Your custom data uploaded to be used in experiments, the experiment results, python notebooks etc. all will be stored in this storage area.

Once you have created an ML workspace, you can access the Machine Learning Studio web page, browser based tool for creating predictive analytics models.

Below image is a screenshot of the landing page of AML studio. Description of the numerated regions are as:

1. Tabs that you can access your experiments, published web services, Python notebooks, your and third-party dataset, ready to use trained models and settings like user authorization to your AML Studio.
2. List all the saved experiments of your own.
3. Sample ML solutions developed by Microsoft for your reference.
4. List of filtered solutions.
5. Third party ML solutions developed and published by AML users.
6. Copy selected solution from one of your workspace to another.
7. Preview of the selected solution diagram.
8. List of workspaces that you are authorized to access. Can switch to any other workspace from this menu.

For creating a new experiment, click on the Experiments tab and then click on the button named NEW on the bottom left corner of the window. If you haven't done yet, you should definitely click on the "Experiment Tutorial" tile to see step by step animated tutorial about creating a new experiment.

If you already experienced the "experiment tutorial" then click on the "Blank Experiment" to create a new experiment that we will be using to show some of the AML features on it.

In the future ML posts we will be creating new AML experiments based on this guide.

• Messages in the queue seems not to be processed in the same order they are placed in the queue. Check Step20 in the prev. post. Message 3 seems to be processed after message 4 and 5.
• When a message arrived to the queue, it is not triggering WebJob method immediately. It is processed after few seconds (check step 14 in the prev. post)
• When you have several messages waiting in the queue, some of them processed concurrently.
• After 5^th unsuccessful message processing attempt, it is moved into another queue called Poison.

Is there a way to control this behavior? Yes. Just add the following lines into your WebJob's Main method and update the values as desired.

static void Main()

{

JobHostConfiguration config = new JobHostConfiguration();

config.Queues.BatchSize = 8;

config.Queues.MaxDequeueCount = 4;

config.Queues.MaxPollingInterval = TimeSpan.FromSeconds(15);

var host = new JobHost(config);

// The following code ensures that the WebJob will be running continuously

host.RunAndBlock();

}

In the above code, lines 5-7 are the default code generated by VisualStudio. We added lines 1-4 and here is their description.

BatchSize: This parameter defines the number of messages to be processed concurrently. If you set it to 1 in the previous blog post scenario, you will see that the messages in the queue will be processed one by one in the queue order. If you have powerful WebApp instance in Azure, you can increase this number to process multiple messages at the same time without so much performance drop in your WebApp.

MaxDequeueCount: In the prev. blog post step 20, we saw that after 5^th unsuccessful message processing attempt, message is moved into Poison queue. This parameter sets how many trials needed before putting a message into Poison queue. Important: Unless there exist other messages waiting in the queue that are not in concurrent processing stage, there is no time interval between trials! So if there exists enough resources, retry occurs immediately. In your case, if you need other processes to be completed before the next trial, just use thread.Sleep inside the trigger method.

MaxPollingInterval: In the prev. blog post step 14, we observed that the message arrived to the queue are not processed immediately. Because WebJob checks the queue every MaxPollingInterval time span to see if there is a new message arrived. You can adjust this frequency by MaxPollingInterval parameter.

Briefly if WebJob method triggered by a queue message, the message is processed by that method successfully and the message automatically removed from the queue. IF there will be an exception occurred during the processing of the message, then WebJob will retry several times, and after some unsuccessful retrial, the message is moved to Poison queue. Below we will showcase this scenario.

1. Create a new azure WebJob project

2. Create a new storage account to create a test scenario.

3. Give a name (i.e. "testscenario") and set parameters of the new storage account to be created.

4. Copy the storage connection string. (PS: I have deleted this account/password before the blog post)

5. Open App.Config file from project explorer and paste the connection string into the corresponding fields.

6. In the server explorer window, right click on the "testscenario->Queues" item and select "Create queue". Give a name "queue" to the new queue service. (It must be same as the name given in function.cs file)

7. In a new window it will ask you storage account key. Paste just the accountKey value that you can find either in the properties window of the storage account or inside the connectionstring that you paste into App.Config file.

8. Now when you double click on your newly created queue service name, management window will open. In this window you can see existing messages waiting in the queue, add new message to the queue etc.

9. Now without making any changes (except the ones that we made in App.Config file), start the WebJob application in Debug mode locally.

10. Console window will open and waits a new message to arrive to the queue. Once a new queue message arrives, it will process it and finish its job (check Function.cs file)

11. Screenshot of the console application waiting for a new queue message.

12. Now open the queue management window and add a new test message to the queue manually.

13. When you added a new message, you can see the message on the queue management window.

14. Few seconds (1, 5, 10… will mention more on the next post) after adding the new queue message, WebJob will receive it and process it. In the console window you can see the information message telling that a message caught.

15. Press the Refresh button on the Queue management window. You will see that the processed message become invisible, deleted from the queue automatically after got processed.

16. Now for testing our scenario, add the following lines of code by updating function.cs file. This code will print out the received message content in the console window (or in the WebJob output window on Azure management portal) and If the content of the message contains "Hello" than it will throw an exception.

public static void ProcessQueueMessage([QueueTrigger("queue")] string message, TextWriter log)

{

log.WriteLine(message);

Console.WriteLine("Received message: {0}", message);

     if (message.Contains("Hello"))

throw new Exception();

}

17. Before starting the WebJob application (or if it is running, then close it) lets add some messages to the queue. In the below example the third of four message that is added to the queue has the content "Hello".

18. Now start the WebJob in Release mode (if in Debug mode, every time the exception occurs the execution breaks. You need to press continue button in the debug window each time. So we continue our scenario in Release mode) and wait messages to be processed.

19. When you press Start button in Release mode, there will be a warning message, press "Continue Debugging". By this way our application will run without any break in an exception case.

20. In the console window, we will see that all messages are processed. Messages with the content "Hello" are re-processed 5 times and after unsuccessful 5^th processing attempt, they are moved into Poison Queue.

21. If you open the "Server Explorer" and refresh the Storage Queue node by right click and then Refresh command, you will see the newly created "Queue-Poison" named queue. Inside this queue you can find the messages that are not successfully processed after 5^th processing attempt.

For more details on retrial counts, concurrent queue message processing etc. please refer http://blogs.msdn.com/b/mustafakasap/archive/2015/12/26/azure-webjobs-amp-jobhostconfiguration.aspx

Imagine you want to make weather forecast and based on previous data (data that you have) you want to predict tomorrows weather status. If you have just few data samples from the past and all these samples are collected during the winter time, then you may not predict tomorrow's data precisely. You need data covering not only winter time period but all days, seasons of the year. On the other hand, if you have data with whole coverage of the seasons but only previous year's daily data, it may also not be enough for precise weather forecast because you need more historical data. You need not only last year data but may need years of past data. Also for better prediction not only the comprehensive past weather data may be enough but also you may need additional data, parameters such as air pollution data per region, moisture, soil humidity etc.

to summarize, you may need:

• Massive amount of past data (not only previous few days or years).
• Comprehensive data. (data homogeneously distributed over cyclic time period, not only from specific session but from all sessions).
• Additional parameters/data that affects the data value to be predicted (not only temperature but humidity data etc.).

In almost every programming language you can develop programs with machine learning algorithms but there is a dedicated programming language and environment for this purpose called R. R is a language for statistical computing and graphics. You can get the free open source development environment and libraries from https://www.r-project.org. Moreover, a free IDE with GUI can be found at https://www.rstudio.com. There also exist hundreds of open source algorithms, software libraries under https://cran.r-project.org. In the next series of this post, we will mention about Microsoft Azure Machine Learning service which is running custom R, Python scripts together with pre-installed CRAN libraries.

Let's start working on a sample project where we will make data prediction based on existing dataset, set of data samples gathered from observations, either by manual processing or from sensor like devices. Assuming you have Microsoft Excel or similar tool; create two columns X and Y, and 30 rows with header and data values from 1 to 29. Here there is no specific reason of selecting 1 to 29 range, it might be 1 to 100 or -50 to -15 whatever you prefer. Below image is the plot of these X, Y data pairs on an Excel chart.

It is very intuitive that for any X value, the corresponding Y value is same as the X. Here the Y values are labels that corresponds to X values; which means you know the label of a specific X value. Now create a new column named YNoise where you will create 29 random numbers each range between -1 to 1. You can use "=IF(RANDBETWEEN(0,1), -1 * RAND(), RAND())" or a better function to generate these random values. Finally create two new more columns named X and YwNoise, copy original X values to the new X column and then take the sum of original Y and YNoise columns and put the result in new YwNoise column. Plot the resulting values on a new chart.

Assume the original dataset (without noise) is the ideal set that we are trying to model, predict. In real world scenario, you don't have any clue about this ideal dataset and with ML techniques you attempt to find the best model that mimics the ideal dataset. For the sake of this example we provide a portion of this ideal dataset together with its formulation which is X = Y. Based on this formula if we take X = 100, which is not in our existing dataset (we have X values that range from 1 to 29, not 100), we can directly say that the corresponding Y value is 100. Because we know the formula, we can predict the corresponding value of any X.

In real life, most of the time you don't have ideal data. Your labels (in above sample Y values) are affected from various, mostly unpredicted, dynamic parameters / noise. To simulate this dynamic parameters, that's is the reason we add random noise to the ideal dataset. With this noisy dataset (X and YwNoise), it is not that easy to find the formulation or relation between these two columns (especially in the case of a more complex and in big data). When we plot this noisy dataset on to chart, it is intuitive that there is a linear relation with X and Y values. Using Linear Regression method, if we fit (best fit) a line on the noisy dataset, we can get the formula of this line (something like y = mx+b, for this specific example m = 1 and b = 0) and using this formula, we can predict corresponding Y value of any other X value which is not in our existing dataset.

Dataset that we used is very simple and in two dimension. In real life scenario you may have more complex and multidimensional data. i.e. you may have X1, X2, X3, …XN and Y columns, where Y is the label of X1 to XN value set. Here X1 maybe price value of a car, X2 maybe engine power, X3 max speed, X4 time to reach 100 mph/sec and Y is the brand of the car. Having this car dataset, in the future you may have any X1 to X4 values and predict the possible brands of the car (Y value).

Summary: Having a dataset where you don't have any relation information within the data, you may build a model to predict missing values of a new data sample that is not in your initial dataset.

As mentioned earlier, size of the dataset is also important. Considering the below sample, if we have only 4 samples in our dataset then following the above procedures to find the best fit line (linear regression) and its formula over these three point is more difficult. As seen on the chart below there may be many alternative lines (L1, L2, L3) passing through these three points. And each line will take us to different formulation, so completely different result set. i.e. for x = 27, taking L1 formulation the y = 40, L2 formulation the y = 26, L3 formulation the y = 21.

Below is another sample but with larger size of dataset. There are 30 points but if we look at this data cluster, which is cumulated in a specific region, from a larger scale, again we may draw more than one line fitting on the dataset. In closer scale, it seems that there is no big difference but from a larger scale, again the result will be very different among the lines that we fit on the dataset.

Linear or Non-Linear

In the next sections we will go in detail about using the right model, algorithm in our ML solution. For now, we will try to keep the simplicity and mention about the basics. For the sample dataset case that is mentioned earlier, we used linear regression method. Without having knowledge about the ideal dataset, intuitively we decided about linear method. What if we have a dataset in larger scale that spread on different clusters and not linear? Take a look at the below chart:

For the above dataset, linear regression is not the best model. Still we can find best fitting line that passes from these two cluster but it will not give precise results about finding the missing values. Using non-linear models will give more precise results, i.e. fitting 3^rd degree cubic polynomial may be the right decision. Below image represents the 3^rd degree polynomial fitted on our dataset.

Again we can't guarantee that the cubic polynomial is the best option. What if there were other data values in another cluster that is not on the path of the curve? Than we have to find another method. This shows us the importance of data size, quality and also the model that we are using. If we have some insights about the data that it is somehow linear, then using linear model is the best option because of its simplicity and computation resource requirements. Using more complex models for simple cases, datasets is also another option but it is waste of computation resource and time consuming approach. So it is very important to use the correct model in the right place.

All we mentioned are about two dimensional data and not always it is possible to build a model with polynomials. Because the higher degree the polynomial more complex the computation it requires. What if we have multidimensional data? Imagine the below surface with parameters x and y, then the labels as z.

Depending on the complexity, computation time constraint, Accuracy, computation memory constraint etc. there exist many algorithms like neural networks, decision trees and we have to find the best algorithm to build our solution.

RStudio

RStudio is one of the most commonly used Open Source IDE for developing programs in R language. In this section we will use this tool to get a brief overview about the R language, its syntax and capabilities. You can copy/paste the below R code into a new RScript editor in RStudio and run it with CTRL+ALT+R key combination. This self-explanatory code generates a dataset, plot it on a chart and write it in a CSV file for further use.

# Generate noisy data for supervised learning sample

# Generate dataset that ranges from -20 to 20 with step value 1

x <- seq(-70, 70, 3)

# Generate a point cloud around the line

# with n slope of 1 and a y-intercept of 0

noise_magnitude = 5

m <- 1

b <- 0

y <- m * (x + noise_magnitude * rnorm(x)) + b;

# plot point cloud on a chart

plot(x, y)

# combine two columns to create data grid

linoise <- cbind(x, y)

# write out to a CSV file

write.csv(linoise, file = "linoise.csv", row.names = FALSE)

Using the dataset generated with the above R code, we will train a linear regression model by using an R function "LinearModel" with "lm" command. Next we define three new random x values which does not exist in our dataset. Here I selected 95, 51 and 33. You can pick any other number of random values that you wish. In this example try to pick numbers between -100 to 100 otherwise don't forget to update the axis limit parameters of the "plot" command in the following code. With the below code, aim is to plot our dataset on a chart, fit a line on this dataset. Predict the corresponding y values of the random x values that you picked and plot them on the chart with red color marks.

#read data into variable

csvdata <- read.csv("linoise.csv")

#create a linear model using the existing data

fit <- lm(y ~ x, csvdata)

# define one or more random X value that you

# want to find their corresponding Y labels

rnd <- data.frame(x = c(95, 51, 33))

# Using the linear model fit, predict Y values

py = predict(fit, rnd)

# Plot existing data on the chart

plot(csvdata$x, csvdata$y, xlim=c(-100, 100), ylim=c(-100, 100), xlab = NA, ylab = NA)

# find and plot best fit line on the point data

abline(fit)

# plot the 3 randon points and their calculated labels as RED dots on the same chart

par(new = T) # draw on existing chart

plot(rnd$x, py, pch=19, col="red", xlim=c(-100, 100), ylim=c(-100, 100), xlab = NA, ylab = NA)

Executing the above code will result in the following chart. It is clear that the linear model that we constructed is working as expected because plot of the randomly picked 3 x values and their corresponding y values lies exactly on the line.

To conclude, along with the practical ML definition, we have demonstrated a simple ML sample scenario with a simple code written in R language.

I would love to hear what you think about ML. Please leave your thoughts and questions in the comments box below.

1. Introduction to Machine Learning, Ethem Alpaydin

Sensörler (duyargalar bkz. www.tdk.gov.tr) teknolojideki gelişmeler ile çok daha düşük maliyete üretilebilmekte. Örneğin Texas Instruments’ın üzerinde 6 duyarga (sıcaklık, nem, basınç, ivme, jiroskop ve manyetometre), kontrol ünitesi ve bluetooth low energy veya bluetooth 4.0, kısaca BLE vericisi olan CR2032 saat pili ile uzun yıllar çalışabilen SensorTag ürününü 25$’a satın alınabilmekte. Bu duyargalardan bir veya birkaçı bilgisayar oyunlarından, ev otomasyonuna kadar günlük hayatın birçok alanında kullanılmaktadır. Bulut bilişim gibi hizmetlerin sunduğu yüksek depolama (Microsoft Azure Depolama) ve hesaplama (Microsoft Azure Hesaplama) gücü gibi özellikler bu duyargalardan sürekli olarak çekilecek anlık verinin arşivlenip analiz edilmesi için oldukça ucuz ve hızlı olanaklar sunmakta, yeni ufuklar açmaktadır.

SensorTag Developer Kit unboxing. (SD Kart boyut karşılaştırması için eklenmiştir)

Bu yazımda sizlere Microsoft Windows Phone 8.1 veya BLE destekli Windows tablet kullanarak SensorTag duyargalarından gelen veriyi toplayıp Microsoft Azure depo alanına atmaktan bahsedeceğim. Eğer sizlerde ileride yayınlayacağım IoT konularına temel oluşturacak bu uygulamayı denemek, geliştirmek isterseniz https://tisensortag.codeplex.com adresinden kaynak kodlarını yayınladığım projeyi inceleyebilirsiniz. Bu kaynak kodların derlenip Windows Phone uygulama mağazasında yayınlanmış haline ise bu adresten ualaşabilir, bir SensorTag cihazı ile birlikte test edebilirsiniz.

SensorTag üzerindeki IR Thermometre ile çay suyu sıcaklığı ekranda.

Uygulama belirlediğiniz sıklıkta (örneğin her saniye) 6 duyargadan aldığı verileri Microsoft Azure Storage hizmeti altında Microsoft Excel tablosuna benzer bir formatta ölçüm zamanı bilgisi ile saklamaktadır. Bir yıl boyunca her saniyede yapacağımız ölçüm bilgisinin tamamının Microsoft Azure’da saklanmasının maliyeti ise ~1TL dir (Farklı saklama yöntemi, verinin arşivden okunması, farklı hizmetler kullanarak üzerinde analizler yapılmasının maliyeti ayrıdır, ilerideki yazılarımızda bu konulardan da bahsedeceğiz).

Peki, bir veya onlarca yıl boyunca her saniye yapılan sıcaklık, nem, basınç, ivme ve jiroskop ölçüm bilgisinin bizim için anlamı nedir? Bu kadar çok bilgiyi nasıl anlamlandırabiliriz? Nasıl analiz edebiliriz? Bundan çok ama çok daha fazla miktardaki veriyi Microsoft Azure HDInsight hizmetini kullanarak çok kısa sürede analiz edebilir, görsel raporlar halinde analiz sonuçlarını sunabiliriz.

Örneğin yatağa yerleştirilecek SensorTag ile uyku kalitesini, oda iklim koşullarını zamana göre analiz etmek, cepte taşınacak SensorTag ile zamana göre ne kadar hareket edildiğini gözlemlemek vs. senaryolardan sadece bazılarıdır. Bu verilerin 1-2 kişi değil de ülke genelinde binlerce kişiden geldiğini düşünürsek toplumda hangi profilde (Microsoft Azure Mobil Servisler ile sosyal medya hesabından alınacak profil, örneğin cinsiyet, bilgisi.)  hangi kesimi daha fazla hareket ediyor, deniz seviyesinden farklılığa göre uyku kalite haritası vs. çıkarılabilir.

Tabii ki bu duyargalardan gelecek veriler sadece büyük veri analizi için kullanılmamaktadır. Örneğin buzdolabına yerleştirdiğimiz duyarga elektrik kesintisinde veya buzdolabı arızasında Microsoft Azure Hizmet Veri Yolu (ServiceBus) hizmeti yardımı ile ilgili olabilecek herkesin marka model den bağımsız olarak ister cep telefonuna, ister masaüstü bilgisayarına bir uyarıda gönderebilir.

IoT konusunda devam edecek olan bu yazı dizisinde Microsoft Azure Servislerinin farklı kullanım senaryolarını anlatıyor olacağız.

 Proje kaynak kodlar: https://tisensortag.codeplex.com 

Windows Phone Uygulaması: http://www.windowsphone.com/en-us/store/app/ti-sensortag/b88323c5-6e1b-4e6a-a9ae-dd2d3ba14790 

HDInsight işlemci kümesi yaratmak

Hadoop kurulumu için ihtiyacımız olacak temel kaynak “Storage” hizmetidir. Verilerin saklanacağı bu alan için Azure yönetim sayfasından, Storage hizmet sayfasına (Resim 2.1) geçip, “New” (Resim 2.2) seçiminde bulunarak yeni bir veri saklama hizmeti oluşturulmalı.

Resim 2: Windows Azure ana sayfa.

Yeni “Storage” hizmeti oluşturmak için açılan sihirbaz yardımı ile “Storage” hizmetine bir isim vermeniz gerekir (Resim 3.1). İsim tanımı yapıldıktan sonra “Create Storage Account” diyerek Hadoop için gerekli veri saklama alanını oluşturabilirsiniz. Bu hizmet sistem yoğunluğuna göre saniyeler içerisinde oluşturulacak ve sonrasında Hadoop kurulumu için temel gereksinim tamamlanmış olacak.

Resim 3: Azure Storage oluşturma sihirbazı.

HDInsight servis kullanım talebinizin kabulü ve servis kullanım portalinin aktivasyonu sonrası Windows Azure portalinden “HDInsight” (Resim 4.1) sekmesine tıklayınız. Açılacak sayfada “Create an HDInsight Cluster” (Resim 4.2) düğmesine tıklayınız. Bu sayfadan aynı zamanda varsa mevcut HDInsight kümelerinizin detaylarını da görüntüleyebilirsiniz.

Resim 4: HDInsight hizmet görüntüleme sayfası.

HDInsight kümesi yaratma seçimi yapıldıktan sonra çıkacak sihirbaz yardımı ile kurulacak işlemci kümesinin özelliklerini ayarlayabilirsiniz. Sihirbaz aracılığı ile HDInsight kümesine bir isim (Resim 5.1), bu küme içerisinde yer alacak düğüm sayısı (Resim 5.2) ve yaratılacak kümenin yönetim portaline erişim için şifre (Resim 5.3) ile hangi “Storage” hizmeti üzerine kurulacağını (Resim 5.4) belirtmeniz gerekir. Bu hizmet için Resim 3.1’de tanımladığınız “Storage” ismini listeden seçiniz.

Düğüm sayısını arttırarak daha fazla işlemci gücü elde edebilirsiniz fakat Azure hizmetini ücretli olarak kullanıyorsanız bu sayının arttırılması maliyeti de arttıracaktır. Ücretsiz deneme hizmeti kullanıyorsanız, bu sayının arttırılması sizin için ayrılmış aylık X dolarlık yada X saatlik hizmet kredisinin daha hızlı tükenmesine neden olacaktır. Test amaçlı kullanımlarda düğüm sayısını en az oranda tutulması kredinin daha geç tükenmesini sağlayacaktır.

Resim 5: HDInsight hizmet kümesinin kurulumu.

Oluşturulacak HDInsight servisinin yönetim paneline erişim için kullanılacak yönetici ismi “admin”, şifresi ise Resim 5.3’teki alanda tanımlayacağınız şifre olacaktır. Yönetim paneline erişim için gerekli URL adresini kurulum tamamlandıktan sonra Resim 4’te belirtilen sayfadan görebileceksiniz.

Yukarıda belirtilen özelliklerin tanımından sonra “Create HDInsight Cluster” bağlantısına tıklayarak kurulum işlemini başlatabilirsiniz. Kurulum tamamlandıktan sonra Resim 6’da belirtilen “Status” kolonunda “Running” ifadesi yer almalı. Resim 6.1 ile gösterilen bağlantıyı takip ederek Hadoop işlemci kümesindeki ana bilgisayara uzak bilgisayar bağlantısı ile bağlanabilir, işlem güncesini inceleyebilir, direkt komut satırından işlemlerinizi başlatabilirsiniz. Bu tarz işlemleri direkt ana bilgisayar üzerinden yapmak yerine, Internet’e bağlı herhangi bir bilgisayardaki Web gezgini üzerinden sihirbaz aracılığı ile yapmak isterseniz Resim 6.2 ile gösterilen bağlantıyı takip etmeniz gerekmektedir. Bu bağlantı ile açacağınız web sitesinde Hadoop işlemci kümesini kontrol edebilir, küme içerisinde yer alan makinelerin performans vs. analizlerini grafiksel olarak takip edebilirsiniz.

Resim 6: Windows Azure HDInsight ana sayfası.

Hadoop işlemci kümesine web arayüzü ile erişim için Resim 6.2 deki bağlantıyı takip ederek yönetim paneline giriş yapabilirsiniz. Bunun için Resim 7’de görüntülenen kullanıcı giriş ekranındaki ilgili alanlara Resim 5.3’de belirttiğiniz kullanıcı adı ve şifreyi giriniz. Resim 5’deki gibi sihirbazı kullanarak Hadoop kümesi yarattıysanız kullanıcı adınız “admin” olarak belirlenir.

Resim 7: Yönetim Paneli

Yönetim paneli (Resim 8) üzerinden sistemdeki düğümler ile “remote desktop” bağlantısı kurabilir, düğümlerdeki iş durumlarını görüntüleyebilir Örnek kod – projeleri mevcut küme üzerinde çalıştırabilirsiniz.

Resim 8: Yönetim Paneli

Bir sonraki yazı dizimizde Hadoop üzerinde büyük ölçekli bir metin bilgisi içerisindeki kelimelerin kullanım sıklık oranlarının nasıl çıkarılacağı üzerinde bir örnek yapacağız.