Some theory

Important bits

Technical side of things

Summary

First hurdle

Second hurdle

Third Hurdle

Testing

Conclusion

Resources

Power Bi Dashboards with Azure Functions

1. Power BI Embedded

2. Using Publish To Web

3. Using Power BI Rest Api's to push data into Power BI

 4. Using Power Bi Visuals javascript  library

function getData() {     $.ajax({         url: '/api/MyDashingBoard?code=MYCODE',         type: 'GET',         contentType: 'application/json',         dataType: 'json',         success: function(data,text) {            updateVisual(data.l1,data.l2);         },         error: function (request,status,error) {             $("#errormsg").html(request.responseText);             $("#failureinfo").removeClass("hidden");             console.log(request.responseText);         }     }); }

And this code results to something like this on my dashboard:

Pretty neat , huh ?

Get the bits:

Adding User Authentication

Benefits of declarative Auth

Who's there

Summary

Serverless Azure Architecture, Simple Forms

Scripting, not software development project

Serverless doesn’t mean free

Some implementation notes

Resources

Whats next ?

Setup

Sharp 2Y0A02 comes from a large family of sensors for distance measuring. There are several versions of the same thing optimized for different needs and use cases. I happened to have this partiular version lying in my junkbox so without too much thought thats what I'll be using for this project. It is optimized for distances between 10cm to 100Cm and it gives out the measurement results as an analog voltage. (Sharp GP2D02 does the same thing but You can use a digital protocol to talk to it just in case You are using some microcontroller that has no analog input pins like Raspberry).

I also have a spare Arduino Leonardo from another experiment so I'll use that . Besides its very cheap and capable.

Connectivity

As You can see from the picture I have USB cable attached my Arduino. It is primarily used to push compiled application to the oboard chip but it can also be utilized by my own software as a serial port.
So I'll be using that USB to talk to my PC-software which does the actual talking to Azure IOT infrastructure. Arduino just takes care of measuring and communicates result to pc.
There are other options :

• Wifi shields enable Arduino to have its own connection to internet and send data directly to Azure
• http://arduino.cc/en/Main/ArduinoWiFiShield
• One could use wireless connection to PC instead of usb (software stays exactly the same since these boards usually pretend to be serial connections)
• http://playground.arduino.cc/Learning/Tutorial01

So in my case I decided to use USB since it also provides +5v to Arduino. 

Arduino code 

The code reads the voltage from our sensor, linearizes it and sends it over serial line to PC. (there will be a link in resources)
 

I am using a miniprotocol where I prepend the readings with sensorid, in this case "A", then separator ":" , then the actual value and finally a newline as terminator. This because I will attach more sensors later on.
Codewise there is really nothign spectacular going on here. Just plain old plain old.

PC-code

So the PC is listening to the Arduino and talking to Azure whenever it gets new values. I am also visualizing the received values with a progres bar.The most notable thing here is maybe the pattern of using Action to communicate between receiving thread and ui thread.The actual Azure communication bit is probably very familiar by now and we've been covering it in previous IOT postings in this blog.

As You can see from the code we parse the lines received from Arduino and add the data to send Queue which will automagically sends the events to Azure.
Take a look at the actual file for more details on how to setup serial connection and so forth.

Azure side

On Azure I have setup a Azure Stream Analytics job which stores the received lines into TableStorage and in csv-file for later usage by HDInsight (MS version of Hadoop).
I have blogged earlier about how to setup the Azure infrastructure for capturing events, so take a look at them if You are unsure about these details.
When looking at the data with storage Explorer here is what we see.

Conclusions ,resources and next steps

This article is about data capture and storage which are definitely needed steps in every succesfull IOT solution but the business value usually doesn't lie here but more in the realm of data analysis and machine learning.
My next IOT-articles will concentrate more on the usage of the gathered data since we pretty much have proven that data capture and storage are very close to nobrainers when using Azure PaaS infrastructure.
 

Arduino code file

C# code file

Part I of this article describes IOT in general and Microsoft PaaS offering around it in more general terms.
Take a look if You feel the need for some theory: Part I

Sorry it took me so long to get back to this, but time flies when youre having fun ...
Anyways, we are here now.

On the menu today :

• Simulating events with nice gui and generating credible traffic
• Designing your data so that it plays nicely with the storage infrastructure
• Storing data in several places and forms at one go
• Retrieving data from relational database (Azure Sql) to webApp and graphing it
• Retrieving data from Azure Tablestore to webapp

Set up

The basic structure of an IOT solution in "Azure PaaS"-land usually consists of following things:

1. Someone who knows how to send event using either AMQP or REST over https
1. Our simulator is using AMQP and threading for efficiency
2. Provisioned PaaS infrastructure to catch the traffic
1. Event Hub service in our Azure subscription
3. Optional real-time analytics and processing
1. Averaging the events in 30 second windows
4. Permanent storage on one or more places
1. Storing averaged events to Azure Sql and storing all (unsummed) event to Azure Blob Storage and Azure Tablestore
5. Some sort of client to see the data
1. Our web app which does rudimentary visualization of collected data

We are intentionally not doing any higher level analytics or Machine Learning here. We are leaving that for the possible sequel: "Part III"

 Sending events in - the simulator

I programmed the simulator as a WebApi-project in VIsual Studio. I wrote a bunch of REST-Services and a HTML5/Jquery- interface on top of it.
You might prefer some other technologies for client but I do mine like this since I can whip them up pretty fast and they behave quite ok. (and i know HTML and Jquery very well)

Basically the whole program is a no-brainer but I'll go over some of the most interesting bits here.
First of all we need to have some structure that we are sending back and forth, here's mine:

public class MeasurementEvent : TableEntity
    {
        [DataMember]
       public long deviceid { get; set; }

        [DataMember]
       public DateTime arrivaltime { get; set; }

        [DataMember]
       public long temperature { get; set; }

        [DataMember]
       public long pressure { get; set; }

        [DataMember]
       public long vibration { get; set; }

        [DataMember]
       public long value { get; set; }

        [DataMember]
       public string type { get; set; }

        [DataMember]
       public long serial { get; set; }

       public MeasurementEvent()
       {
       this.serial = AppModel.getSerial();   
       }
    }

As You can see I'm using the same structure to carry all the event types in my demo. In real life You might be using few types instead.
The other notable thing is that the class is decendent from TableEntity, it doesn't have to be, but I'm doing it because I'm later on reusing this same class when querying the TableStore for events and LINQ wants it like this.

Here is my sender-function that is being called once a second when the simulator is active :

(sorry for including this as an image but the editing system screws my VS-pasted code blocks pretty good, I'll put the code in resources at the tail of this article)

The SensorData is just a structure to convey the HTML-fields from the screen to the backend .So nothing special there. The randomizer is a simple block generating some randomly drifting values. The dice generates (-1,0,1) and the values are drifting accordingly:


The timing is implemented by the client using good old window.setInterval since it doesn't have to be rock solid. Add some basic field checks and the simulator is pretty much a done deal.

 Catching the events

For instruction on how to setup the infrastructure for Azure Stream Analytics please see Part I of this article. I'll show here only the most interesting bits

Stream analytics dashboard:

Averaging query:

 Basic query for storing into Blob storage:

So at runtime we have three active jobs consuming the same input. Pretty cool, it just works that way.
Maybe one thing worth noting in Tablestore output definition is that the fields "Partition Key" and "Row Key" are referring to column names in Your input, not values.

Here's a piece of data from blob storage:

After adding a terminating "]" this is basically a valid json file that can be used by tools like HDInsight (Hadoop) or Azure Machine Learning (we might look in to that in Part III).

TableStore looks like this:

This is a more traditional view of the same data.
Actually this is newer picture and there are a couple fields worth mentioning. I added a thing called avain (key in Finnish) which is a decending number since there is no "order by" when selecting rows from the TableStore which just gives You rows starting from the first one. So making the key decending turns the default sorting (PartitionKey,RowKey) upside down so that when I take only 40 lines I get the latest ones.
Other thing to note is that the PartitionKey is used as the datatype label. Very much like table name would be in relational world.

Here's looking at You kid

I wanted to view how my pumps and ekg-monitors are doing so I needed to build a little dashboard to show the latest values coming from the relational db.

Remember that it was averaged data in db so we are looking at trends here, not actual line data.

One can go totally crazy and build wild visualizations of this data using BI-tools or whatever. I'm using just HTML and flot here. There's a little REST-function on the backend that spits out lates monitoring data from tga db and jquery feeds it to flot diagrams. Easy as pie.

Another view of the same data comes from TableStore :

Here we are presenting all the latest rows unsummed which is a good starting point for later analysis.

The TableStore is approached by this little code snippet:

 Note the usage of the same MeasurementEvent-class we used sending data in Event Hub . (the full code is in resources)

So there You have it !

This is how You programmatically create events, send them , capture them, analyze them and finally show them in various forms.
What's missing is the business case here and the real value that comes from applying analysis to this data.
We'll probabply look into those tools in the next part of this article.

Get the code resources to help Your typing.

If You are a lazy programmer like me this will blow Your mind  .... I promise.

When developing software I'd like to write only the stuff that absolutely definitely needs to be written, nothing more.
So I have a long standing hate-hate relationship with those verbose xml-configs and convoluted frameworks since all they represent extra work that I need to do. In the worst case such work that has no value to the actual customer (remember customer anyone ?) , but work that just needs to be done to make the wheels go round.

I hate writing boilerplate code/config/definition with a passion.

So Imagine my smile when I found out that I can consume the services of a modern Authorization stack (including multi factor authentication) with a simple click of a button.
Doesn't sound like much does it ? Wait, let me show You ...

Setting the scene

 
I’m writing this from the viewpoint of an LOB – developer. More specifically I want to write responsive HTML5 single page applications so I’m using Bootstrap on the front end side and some jquery (optionally some knockout,angular,…) and some sort of backend to service my front ed app some REST JSON APIs.
Simple enough.

In this scenario my backend needs to spit out some html but mostly serve my API calls. And when doing that the backend needs to know who the user is !

The users can’t be stored on the LOB-applications own database anymore (we all know that) so how about using some modern structure like oAuth and Azure Ad and so on … That’s when everything starts
getting convoluted !

The minute You start dabbling with authorization things slow down like in Matrix bullet-time and You lose all productivity and Your hair starts getting gray very rapidly.

But luckily there’s a really nice rabbit hole to bypass oll this grief when operating with Azure websites:


The red bar is the real superhero of this story. It intercepts all calls to my application, regardless if it is a page request or a REST call to one of my APIs It makes sure that only those people who have successfully logged in with my AD can pass. And when they do pass my application is being given a nice context describing the user …
And all this without writing any code. … well almost no code , since I need to write code to acces and use the userinfo I’ve been so graciously given.

Enough talk, show me the money !

For the sake of simplicity I’ll only present the skeleton portion of an LOB, next time well continue from that.

Visual Studio 2013  / New Project /  Cloud / ASP.NET Web application / Web API / No Authentication :


So now we have a nice skeleton. Lets tweak it just a little.

 And hit run (F5) . Once it runs go to http://Your_app/api/values. Mine is: http://localhost:44752/api/values

You should get some JSON back. Opening mine : ["value1","value2",""] – Exactly as it should be

So now we are done developing !!!

Header:

 using System.Security.Claims;
 using System.Threading;
 using System.Web.Configuration;

Code:      

 public IEnumerable<string> Get()
 {
 String res = "";
 var claimsPrincipal = Thread.CurrentPrincipal as ClaimsPrincipal;
 if (claimsPrincipal != null && claimsPrincipal.Identity.IsAuthenticated)
 {
 foreach (var claim in claimsPrincipal.Claims)
 {
 res += "<div>type:" + claim.Type + ":" + claim.Value + "</div>";
 }
 res += "<p>Logout:" + WebConfigurationManager.AppSettings["WEBSITE_AUTH_LOGOUT_PATH"] + "</p>";
 }
 return new string[] { "value1", "value2", res };
 }
 

Let’s deploy this to azure.

If You didn’t already create a publishing profile while creating the product do it now:

1. Stop the debug, Right click Your project name and choose “publish”
2. As publish target choose “Microsoft Azure Website”
3. If You don’t have existing websites choose new.
4. Either way now You have a website picked, Validate the connection
5. On File publish options choose : “Remove additional files” and hit publish
6. After few minutes Your site should come up. go to the same relative url “/api/values”
7. You should get the exact same response as when running locally

Fine . Now for the magic ….. (drumroll ,please) ….

Goto to : manage.windowsazure.com/ , find Your website and go to configure and find this:

Choose Your directory and “Add Application”.
That’s it. You’re done . 


Next take a new browser or inPrivate browsing or whatever it is called in Your browser. The idea being that You should have a browser where You have no ongoing sessions to azure.
This time You are presented with a login screen and once You clear that you go to the same url http://yourapp/api/values

Notice that the returned JSON has all kinds of interesting claims now. Amongst them are your name , email and so on …

Now You know that for every api or page request the caller is going to be identified against Your Azure AD (You can manage the AD as You wish) and You are being told who the caller is.
Isn't this wonderful.

Adding Multi Facet login capabilities

Let's get a little greedy and add some cool stuff while we're at it.
Take a look at here on how to setup Your AD to support multi facet logins : http://azure.microsoft.com/en-us/documentation/services/multi-factor-authentication/
I have Azure AD premium so I just enabled the feature and made one of my users to require some additional authentication.

Then I logged in with that user and was presented with a request to use my mobile app for extra credentials.

Sorry , this is in Finnish, but You probably can guess what is says.

So the application in my mobile came up and I chose “verify” and Boooom … I got in to my little sample application .
And sure enough, the returned JSON message from "/api/values" had my webusers details in it.

Easy breezy multi-factor-authentication-implemented-with-no-code

This is so cool .
Next time we’ll start to implement the actual application itself since we now got this super cool authentication implemented in no time.
There are still things to wish for. Group-support is one of them but it is coming.

Read more from here: http://azure.microsoft.com/blog/2014/11/13/azure-websites-authentication-authorization/

Until next time …

Well, in the meantime us regular dudes can actually do something without spending huge amounts of time or cash to get our feet wet in IOT by utilizing ... You guessed it ... good old PaaS approach.

So today we are going to build an infinitely scalable input point and send in some random temperature readings.

Here's a picture of the overall architecture of a generic IOT solution running on Azure PaaS offering.

We have our gadgets that produce events, optionally some gateways to aggregate messages or transform protocols, Event hubs to catch all the traffic, then some more or less realtime analysis and storage.
So what's new ? ... nothing really.
Other than the fact that now we can do this in an afternoon without building a costly infrastructure first. We do this by utlizing PaaS components and by doing so we get the benefit of world class scalability and security by design. But looking at this purely from a functional standpoint ... it's all stuff that we have been doing for a long ... long time. 

Enough talk, let's actually do something...

1. Create Service Bus namespace to contain our event hub
2. Create event hub to catch the message from field
3. Build a small Simulator to send in events
4. Create a Stream Processing Job to do some mild analysis
5. Store the data for Part II where we will try to do some reporting and analysis

Namespace

The namespace is a manageable unit that can house several different services like Queues,Topics and Event Hubs, which we will be using today. One interesting thing about the namespace is that it controls the scaling of the Services it contains.

Namespace does this via Throughput Units. One TU can take in roughly 1MB per second. Given a 12 hour day that makes about 50GB per day. Not bad.
TU's are also a billable resource. One TU in standard mode cost about 0.0224€ per hour, which totals about 17€ per month. SO cost isn't the thing keeping Your company from testing IOT.
Pay attention to this ! This is the point that decides how scalable Your solutions intake will be and how costly it will be to run. With this slider You can easily provision up to 20 TU's for running Your ingress.

 Event Hub

An Event Hub is the actual endpoint to which all the gadgets send their events.You can use Partitions and parition keys in Your data to splice the data up, but if not used the partitions will be used evenly for all data. 
Remember to configure Shared Access Policy for Your Event Hub or the Job will not be able find it later. I created a poly called "manage" and ticked  all the boxes for access.

 Stream Analytics Job

 A Job is actually very simple thing connecting the ingreaa queue with long term storage and performing some optional analysis on the fly.

Jobs monitoring storage account should reside in same region as the job itself or You will be paying for the traffic.
A Job consists of three moving parts :
Input: Here You will reference Your Event Hub.
Output: The long-term storage can be various things I chose Sql Database in the same region that the Job is in. Remember to enable Access for other Azure Services in DB's config ot You will spend some time wondering why the Job is not running.

There needs to be a table in the Sql database that matches the structure that Your "query" produces and it needs to be named in the outputs-config, so here goes:
CREATE TABLE [dbo].[AvgReadings](
    [WinStartTime]   DATETIME2 (6) NULL,
    [WinEndTime]     DATETIME2 (6) NULL,
    [DeviceId]      BIGINT NULL,
    [AvgTemperature] FLOAT (53)    NULL,
    [EventCount] BIGINT null
);
CREATE CLUSTERED INDEX [AvgReadings] ON [dbo].[AvgReadings]([DeviceId] ASC);

Query: This is where the realtime analysis and the overall definition of the data structure happens. Basically You create a query and whatever it produces will be the new datatype and it will be store in "output".
A simple query could be like this : "SELECT DeviceId, Temperature FROM input" - this would just store the values received into output-table for later usage.
If You want to do some aggregation or math with the data before it is written to storage You can use some clever sql:http://msdn.microsoft.com/en-us/library/azure/dn834998.aspx

In this case we will just average some values with 5-second window:
"SELECT DateAdd(second,-5,System.TimeStamp) as WinStartTime, system.TimeStamp as WinEndTime, DeviceId, Avg(Temperature) as AvgTemperature, Count(*) as EventCount  FROM input GROUP BY TumblingWindow(second, 5), DeviceId"
Note that the "input" needs to refer to the alias-name that You gave when defining Your Job input !!!

Go to Your Jobs Dashboard and hit Start ... and our system should be ready to accept some input.

Creating some events

The most important things in our little Simulator client are:
NamespaceManager to connect to our namespace ,credentials stored in app.config.
EventHubClient for accessing the api
EventData to house our data structure.
EventHubCLient.SendAsyc for actually sending.

Don't worry there will be a link from where You can get the client code from.
Running this guy feeds some events in our pipeline.

Note that the second parameter needs to be the name of Your Event Hub inside Your namespace.

Querying tha database will show that we indeed get some events coming in and we are able to perform some light operations on it too.
 

 Conclusions and next steps

So , IOT isn't that hard after all.
By using some PaaS Services we were able to make a solution that can ingest millions of  messages per second. By using some good old SQL we were also able to enrich and analyze the incoming data stream.
Not bad.

Next part of this article will elaborate on the client side, namely we will be sending events from scripting language, probabply from Python, We'll use a Raspberry pi as a client if all goes well .
We will also do some reporting and maybe some Machine Learning.
Stay tuned.


In case You have problems finding Your way around Azure management screens there is a more step-by-step oriented tutorial available here, it also has the source code for the client used:
http://azure.microsoft.com/en-us/documentation/articles/stream-analytics-get-started/