• It needed to be portable enough so I can run it on my Raspberry PI
• Since Flukso is open source I want to make my code available publicly too
• In order to make sure other people can benefit from it as easily as possible, I wanted to use Infrastructure as Code

• An export of the LastFM data you are interested in
• Azure Databricks account
• Power BI

• .NET Core
• Visual Studio Code
• Docker
• WSL
• LastFM Account + API Key
• Azure Account

dotnet add package Newtonsoft.Json

dotnet add package Microsoft.Extensions.Configuration
dotnet add package Microsoft.Extensions.Configuration.EnvironmentVariable
dotnet add package Microsoft.Extensions.Configuration.Json

az login
az group create -l westeurope -n RGLastFMReader

az container create -g RGLastFMReader --name lastfmreader --image wesback/lastfmreader --cpu 1 --memory 1 --restart-policy Never --location=westeurope -e lastfmkey=<LASTFMKEY> storageaccount=<STORAGEACCOUNT> storagekey=<STORAGEKEY> lastfmuser=<USER>

• Microsoft expands artificial intelligence (AI) efforts with creation of new Microsoft AI and Research Group with 5000 engineers: http://bit.ly/2gg5mZp
• How Bing and Azure are using an AI supercomputer in the cloud: http://bit.ly/2fU2LR8
• The Microsoft Cognitive Toolkit is a free, open-source toolkit that trains deep learning algorithms: http://bit.ly/2fkaydw      
• Leading genome research center migrates to DNANexus on Azure: http://bit.ly/2fw7Oc8
• Microsoft and OpenAI are joining forces as OpenAI will run most of its large-scale experiments on Azure: http://bit.ly/2eX8jZ6
• How Uber is using Cognitive Services to improve security: http://wapo.st/2cZDRzh

So how easy is it to connect to a Spark cluster running on Azure?  Let's find out!

First go to Get Data and select Databases

You will get the list of data sources where we currently support a live connection.  Here you obviously have to select Spark and select click Connect.

Enter the details of your cluster (clustername.azurehdinsight.net)

That's basically it, you are now connected to the Spark cluster and you can start digging into the data.

Another option is to use the Power BI Desktop if you want to enrich the data with additional calculations or transformations. 
You will have to download the Spark ODBC driver if you want to use Power BI Desktop to connect to your Spark cluster, you can get that here.

Now let's try to see if I had some influence on my daughter (on her music choice at least :-)).  Let's start by connecting to the Spark cluster from our Power BI Desktop.

Let's look at the dashboard I have built to see the influence of my father on me and myself on my daughter.  First thing I noticed is that my listening moments are completely different from my father's.  You can see when my wife comes home by looking at the time I stop listening to music ;-)  I was very happy to see the top songs for me and my daughter and I really hope she will keep on listening to this kind of music.  Looking at the chart for me and my father I'm pretty sure he is thinking the same :-)

Since we are using the Power BI In-Memory model here I can also use Q&A to investigate trends.

I hope these posts make clear that you can get up and running incredibly quickly with Spark using Azure and go from raw data to beautiful visualizations.

Nathan and myself are already thinking about our next blog post about Spark... stay tuned!!!

As Microsoft we have a history of embracing Big Data technologies on our Azure platform.  We offer templates in our IAAS platform for HortonWorks, Cloudera and MapR among other Big Data Services.  We also offer HDInsight, our 100% Apache Hadoop-based service in the cloud.  By using HDInsight you are up and running in minutes with your own Big Data cluster, these clusters can come in different types such as Hadoop (HortonWorks), HBase, Storm and more recently Spark.  If you want to read more in detail about this please refer to https://azure.microsoft.com/en-us/blog/interactive-analytics-on-big-data-with-the-release-of-spark-for-azure-hdinsight/.

If you want to learn how easy it is to setup a Spark cluster on Azure please follow the instructions here: https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-zeppelin-notebook-jupyter-spark-sql/

We will go through a practical example on how you can take an input file and process it using Spark, for this I will call in the help of someone with a lot of experience in Spark (Nathan Bijnens - @nathan_gs).

My previous post (I know it has been way to long since I blogged :-)) was about using Last.FM data to see if my father had influenced me.  As I am a father myself I figured I should now see how I am influencing my daughter, so I created her an account on Last.FM too.  The goal of this post is to show how easy it is to work with Spark on Azure, we are not trying to prove the performance of Spark since there are more than enough benchmarks done.  Because we want to work with a raw file on Spark I have written an Azure WebJob that reads the Last.FM API every night and outputs this to an Azure Storage Account (in JSON).  This will be the starting point for our journey from raw file to Power BI Dashboard. 

In part 1 we will see how you can take this JSON file and prepare it to be used in Power BI (as a Hive table).  Attached some screenshots of the Zeppelin Notebook, if you want some of the source code you can find it on my OneDrive @ https://onedrive.live.com/redir?resid=F8448F26F719FCAC!192419&authkey=!AB4ZLro6lU-ysos&ithint=file%2ctxt)

First step in this exercise was finding out how I could get data from Last.FM to start my analysis.  I turns out Last.FM has quite a nice API which could give me all the information I needed to do my analysis.  There are 2 outputs possible, JSON and XML.  In my first attempt I started using the XML output because it is more readable in the browser, this is important if you don't know the output yet.  However, it also has more overhead and that's why I decided to go for JSON instead.  Data Explorer supports JSON very well so it was easy to move to the JSON output.

There was a specific call I was very interested in, user.getRecentTracks which gives the tracks you listened too with some attributes like the time and - interesting for Power View - a URL to the artist image on Last.FM.  One challenge however is the fact that it also accepts a parameter called limit which defined the number of rows per page (which has a max limit of 200) and a parameter page.  You can probably guess what I needed next, loop over the API with a different page number so I can get my entire history.  Turns out this isn't to difficult in the Data Explorer language, just just need to define a function that takes a parameter.  The easiest way to do this, is to build the query from within the UI and then take that as a starting point for your function. 

My query started with the following statement:

let
    Source = Json.Document(Web.Contents("http://ws.audioscrobbler.com/2.0/?method=user.getrecenttracks&api_key=myapikey&format=json&limit=200&page=1&extended=1&user=dis4ea"))
    ...

I changed it to

(page as number) =>

let
    Source = Json.Document(Web.Contents("http://ws.audioscrobbler.com/2.0/?method=user.getrecenttracks&api_key=myapikey&format=json&limit=200&page=" & Number.ToText(page) & "&extended=1&user=dis4ea"))
    ...

After that you can call the 'function' by starting a new query with the following:

let
    Source = Table.Combine(List.Transform({1..10}, GetLastFMTracksByPage))
in
   Source

*(GetLastFMTracksByPage is the name of the previous query)

You can learn more about this trick on Kasper De Jonge's blog.

Now this was a good start but remember the original question was to compare it with my dad's listening habits.  So I actually needed a second parameter which is the user name and that's when it became interesting, I was trying to get that working but couldn't find a good solution immediately.  So I reached out to my friends in the product team and they came up with a couple of interesting answers (many thanks to Faisal Mohamood and Clemens Szyperski for their help!).  Adding a second parameter wasn't too difficult but you need to have the right syntax to loop over it. 

First I added the second parameter user name to the definition:

(page as number, username as text) =>

let
    Source = Json.Document(Web.Contents("http://ws.audioscrobbler.com/2.0/?method=user.getrecenttracks&api_key=myapikey&format=json&limit=200&page=" & Number.ToText(page) & "&extended=1&user=" & username))
    ...

Now it's a matter of finding the right syntax for the Table.Combine statement:

let
    Source = Table.Combine(List.Transform({1..99}, each GetLastFMTracksByPage(_ , "dis4ea")))
in
    Source

That solved the first problem of having a dynamic parameter, but it would be even better to have the list for my dad and myself merged into a single table.  So how can we do this?  Faisal came up with the great idea to use a table and a custom column to solve this one.  I created a table in Excel with the following layout:

Username, Page
dis4ea, 1
dis4ea, 2
master_cobra, 1
...

Then I created the following query starting from this table:

let
   Source = Excel.CurrentWorkbook(){[Name="ParameterTable"]}[Content],
   InsertedCustom = Table.AddColumn(Source, "Custom", each GetLastFMTracksByPage([Page], [UserName]))
in
   InsertedCustom

This made it possible to have all the data in one table (with an extra column UserName) and it also makes it easy to extend it with other user names.  Now it was just a matter of visualizing the information in Power View and get the answer to my question, did my dad influence my music choice?  If you look at the screenshot below I think I can safely conclude that he did (for the record, my user name is dis4ea).

I have placed the new file here and the original file is still available here.

If you are interested in having this as a Tabular Model (for SQL Server Analysis Services) please let me know.  You can easily do this yourself if you like of course by importing the model in the SQL Server Data Tools.

During the last Techdays here in Belgium, I have had the privilege of being able to present a session on Lync. I had chosen as a topic for that session “Deploying Lync: Notes from the Field, and more”, where I talked about the top ten questions I run into when talking to customers about deploying Lync in their environment. I tried to answer the following questions:

• Lync Online – Lync On Premises: where do we go?
• Lync is a Puzzle, really?
• Lync and NAT: Yes or No?
• Whats’s up with the Picture in Lync?
• Are there any tools available to help me design hardware and bandwidth requirements for Lync?

And then the final topic I touched upon was monitoring, and that’s when I was happy to introduce one of my Technology Advisor colleagues, Wesley Backelant, who knows almost everything about PowerPivot and Power View and SQL and BI to the IT Pro Lync audience And we promised to blog about it, so here it is, the details on the web

Monitoring Server in Lync, why would you use it?

In short, the Monitoring Server role in Lync is optional, and can be added after finishing the deployment of all the other server roles within Lync. The Monitoring Server enables you to capture both call detail records (CDR) for Enterprise Voice and Audio/Video conferences and data collections about your Audio/Video Quality of Experience, also incorporating data on  file transfers, application sharing, and remote assistance. It does NOT archive the conversations itself (if you want to keep the IM conversations in your organization, you should consider deploying the Archiving server role).

Some typical questions on the Monitoring Server role in Lync:

• Can the Monitoring server role be combined with any other Lync Server role?

  Yes, as taken from TechNet:

  “A Monitoring Server can be collocated with an Archiving Server, with a SQL Server store of an Enterprise Edition Front End pool, or with a file store of a Front End pool. The Monitoring Server requires a database, but the database can be collocated on the Monitoring Server, with the database server for the Archiving Server, or on the Back End Server of an Enterprise Edition Front End pool. A Monitoring Server cannot be collocated with a Standard Edition server in a production environment.”

• Can you collocate the monitoring database with a Lync back-end database?

  The answer is yes, but the back-end database needs to reside in a separate SQL instance.

• Do you require SQL Reporting Server?

  No,  but if you do you will be able to take advantage of the built-in Monitor Server Reports, which you can customize as wanted.

• What’s the advantage of using the Monitoring Server role with Microsoft System Center Operations Manager?

  By installing the Microsoft System Center Operations Manager, which uses the Monitoring CDR and QoE data, you can enable the generation of near real-time alerts showing the health of call reliability and media quality, and define actions upon these.

• Do I need a monitoring server per Lync Pool, and even per Site?

  No, as taken from the same Technet Article linked to before:

  “Multiple central sites can also share any of the following that you deploy in one central site:

  • Archiving Server
  • Monitoring Server
  • Stand-alone Mediation Server or pool
  • Edge Server or Edge pool”

• Any documentation available on how to deploy the Monitoring Server?

  Yes: Microsoft Lync Server 2010 Monitoring Deployment Guide, available for download here: http://www.microsoft.com/download/en/details.aspx?id=8207, which described the different steps envolved:
  • Installing Message Queuing for Monitoring
  • Installing SQL Server (with Reporting Services)
  • Adding a Monitoring Server to the Topology using Topology Builder
  • Installing the Monitoring Server
  • Deploying Monitoring Server Reports (if wanted)
  • Configuring Monitoring Server Settings

• Is there a document that described how to use the Monitoring Server Reports in a Lync Server 2010 deployment?

  Sure there is, there is Microsoft Lync Server 2010: Using Monitoring Server Reports whitepaper available for download here: http://www.microsoft.com/download/en/details.aspx?id=890

Built-In Reports examples:

What does the Dashboard give me?

And yes you can export the reports, and you can customize them…but if you know have the feeling as in “this is not exactly what I want to show my manager”, “I would like to have some more fancy reports” “I’m looking for an easier way to use the gathered data”…then please read along, because now it’s time to dive into PowerPivot and Power View!

Let me start by saying that the solution provided here is obviously for demo purposes.  It is not a perfect solution and we are not responsible for the production use of this file.

Technologies Used

First a couple of words on the technology used in this solution.

PowerPivot is a powerful data mashup and data exploration tool based on xVelocity in-memory technologies providing unmatched analytical performance to process billions of rows at the speed of thought.  It comes with a client part, a free add-in for Excel 2010, which allows you to import and combine data in an easy way.  We used Excel in this particular case to get information from the Lync monitoring databases and create some interesting calculations.  In order to get your hands on PowerPivot please visit the official website. 

The second part of the solution uses SharePoint and Power View to share and visualize the information in a very interactive way.  Sharing PowerPivot files in a scalable and controlled way can be achieved by using the Excel Services functionality of SharePoint.  With Excel Services you have the ability to share Excel files to a broad audience without the need to install Excel 2010 (and in this case PowerPivot) on every machine.  If you need more information about Excel Services please read this.  When integrating SQL Server Reporting Services 2012 in SharePoint 2010, you will get a new powerful data visualization tool called Power View.  Power View is really all about the interactive and stunning experience.  Understanding Power View is actually the easiest by just watching this demo.

Now that you know the technology part let’s dive into some of the details on how we created this.  Of course we started by interviewing customers to understand what they would like to get out of the information available.  Some of the topics that came up were duration distribution, average response time, number of sessions by type, uptake of Lync and a lot more interesting stuff!

Most of the information you need is available in the LcsCDR which is fully documented here.  The leading table is SessionDetails so you better figure out how this table is structured.  One additional set of data you may need is a date table, I have included a hidden sheet with dates but you could also use the DateStream feed from our Windows Azure Marketplace.  A couple of other tables we used are SIP, Users, ClientVersions and UriTypes

The other database we used was rtcab1 to get display names.  This is a bit less straightforward to retrieve but here is how you can extract it:

SELECT avname.Value as 'DisplayName', avuri.Value as 'UserUri'
  FROM AbUserEntry u
  INNER JOIN AbAttributeValue avname ON avname.UserId = u.UserId AND avname.AttrId = 3
  INNER JOIN AbAttributeValue avuri ON avuri.UserId = u.UserId AND avuri.AttrId = 8

The entire model looks like this

Now that we have all the information in place it is time to visualize some of it.  A couple examples of the things we built (remember it was built on a test database too so don’t freak out by the unanswered call rate ).

Taking this PowerPivot file to the server to share it with a broad audience is a matter of saving it on an Excel Services enabled SharePoint Server.  This would give you the following view

But we promised a more compelling and interactive way of visualizing this information and that is exactly what you will do with Power View.  Power View can be used on PowerPivot files published on SharePoint or on solutions deployed to SQL Server Analysis Services.  As a little side note, you can import the PowerPivot model or build this model from scratch in SQL Server Data Tools to make it a real server-side solution running on SQL Server Analysis Services.

The most popular view however is the adoption rate of Lync in the organization, this can be achieved by using the Scatter Chart functionality in Power View.

Please find the PowerPivot file here.  In order to get data from your enviroment go to the PowerPivot window, select “Design” and “Existing Connection”.  Edit both connections to point to your SQL Server and when done use “Refresh All”.  In the Excel Windows go to Data" and select “Refresh All” and you are done.  If you want to use Power View just upload the file to a SharePoint where this feature is enabled.

There is a LOT more you can get out of the Lync databases, so we hope this sparks a whole wave of creativity!  Ilse’s favorite for example is the QoEMetric database to see the call quality for different users and devices.  If I can find some time I’ll see if we can publish this PowerPivot file too. 

Ilse & Wesley