Sri Lanka Tech Geeks

Analyze Twitter #tags using Sentiment Analysis and Visualize in Power BI

Sasindu Kavinda — Wed, 27 Jun 2018 11:13:06 GMT

So in this post I'm going to discuss you on using Microsoft Flows and Sentiment Analysis API to analyze twitter posts and visualize them in Microsoft Power BI. I have mentioned you about Microsoft Flows in my previous posts. Text analytics (or sentiment analysis) is the automated processing of texts to determine topics, key phrases and the opinion of the writer (positive, negative, neutral). Text analytics (or sentiment analysis) is the automated processing of texts to determine topics, key phrases and the opinion of the writer (positive, negative, neutral). Microsoft Cognitive Services are a new set of cloud APIs that can be used from practically any technical platform (Android apps, iOS apps, Windows apps, websites, etc) to leverage capabilities such as natural language processing, speech, vision, knowledge exploration, etc. These capabilities, like the whole field of artificial intelligence, are easy for humans but difficult for computers. Now you can leverage the results of Microsoft Research and powerful cloud computing to power your own applications with these capabilities. The Text Analytics API, offered by Microsoft as part of Cognitive Services, has the following functionality:

Determine the language of a text.
Determine the sentiment of a text. The output is a score between 0 (very negative), 0.5 (neutral) and 1 (very positive).
Detect key phrases in a text.
Detect the top topics for a collection of texts.

The Text Analytics API itself is built with Azure Machine Learning. No training data is necessary, since it is already trained by Microsoft. You can easily test it on the Text Analytics API webpage. So for this demo, I'm going to use Sentiment Analysis from the above capabilities of Text Analytics API. As the step 01, you have to create a power bi dashboard and streaming dataset. This isn’t going to be a full tutorial on how to use PowerBI, but hopefully I can take you through just enough steps that you can visualize this data. In order to do that we are going to login to the Power BI service, and In work space, top right, click the + Create and select Streaming data in the menu. As per the above video, you can create a streaming data set. So what you can do is, just push data to that API as post request with that json data. As per the above video, you can visualize the data. You will see the actual data once you push your data to Power BI. Now let's create Text Analytics API Service in Azure Grab and store that API key with you. As the step 02, you have to create a new flow. If you haven’t already, go to flow.microsoft.com and have a look around. You can find thousands of templates there. The flow creating will shown in the following video. So after any tweet triggered based on that keyword we have mentioned, the Power BI will show it to you. This is just simple clicks!! No need coding ❤️

Create and Host your Node.js API in Azure

Sasindu Kavinda — Mon, 28 May 2018 02:31:46 GMT

Node.js is an Open source, Cross-platform, and runtime environment for executing JavaScript code outside the browser. We can build Web apps and APIS using Node.js. This is Super-fast and highly scalable e.g. PayPal, Uber, Netflix Builds twice as fast with fewer people 33% fewer lines of code 40 % fewer files 2x request/sec 35% faster response. So first let's create an API using Node.js. Before we proceed, we must ensure we have the following installed in your computer : Node.js To check if you have Node.js installed, run this command in your terminal: node-v To confirm that you have npm installed you can run this command in your terminal: npm-v After the confirmation, let's create a new folder and go into it by following commands, mkddir nodejsAPI cd nodejsAPI Initiate a new NPM project with this command npm init and provide all the details requested for. So let's add ExpressJS framework to create our API. To use ExpressJS in our application, we need to execute following two commands. npm install express --save npm install body-parser --save Then we have to create a new file named: app.js And let's add the content as following, //-------------------

var express = require("express");

var bodyParser = require("body-parser");

var routes = require("./routes/routes.js");

var app = express();

var port=process.env.PORT || 3000;

app.use(bodyParser.json());

app.use(bodyParser.urlencoded({ extended: true }));

routes(app);

app.listen(port);

console.log('Server Listening at port'+port);

//--------------------------

As per the code this will get expressjs framework and body parser to get the http request body. And this will run on localhost with port 3000. Now let's create routes.js which is our code going to write. So for that create new folder and name it as "routes", then create the routes.js inside the routes folder. Then add the content of the routes.js file as follows. //-------------------------

var appRouter = function (app) {

app.get("/", function (req, res) {

res.status(200).send({ message: 'Welcome to our restful API' });

});

app.get("/getItems/:num", function (req, res) {

var num =req.params.num;

var data = {

'item1': 'http://public-domain-photos.com/free-stock-photos-1/flowers/cactus-76.jpg',

'item2': 'http://public-domain-photos.com/free-stock-photos-1/flowers/cactus-77.jpg',

'item3': 'http://public-domain-photos.com/free-stock-photos-1/flowers/cactus-78.jpg'

}

res.status(200).send(data);

});

app.get("/getText/:txt", function (req, res) {

var text =req.params.txt;

var data = {

'text': text

}

res.status(200).send(data);

});

}

module.exports = appRouter;

//------------------------- So as per the router.js, There are three APIs.

Index
getItems -- with parameter number
getText -- with parameter text

So now you can navigate to the folder by cmd and type "node app.js" So your node API is now up in your local machine. Let's change your settings to publish your application to Azure. First change your package.json as follows

{

"name": "demonodeproject",

"version": "1.0.0",

"description": "writing first node app",

"main": "app.js",

"dependencies": {

"express": "^4.14.0"

"scripts": {

"start": "node app.js"

}

-----------------

Here the scripts section will run after the app published to azure.

Then let's add web.config file.

------------------

<system.webServer>

handlers>

<rules>

rule>

rule>

rules>

rewrite>

<files>

files>

defaultDocument>

system.webServer>

configuration>

-------------------

So now your folder structure needs to be like this. :)

Let's create Web App Service in Azure.

So your app service is up and running.

Now what you have to do is, just zip all the content in your nodejs folder.

Then navigate https://.scm.azurewebsites.net/ZipDeploy

and upload your zip file.

(https://testingnodeapp.scm.azurewebsites.net/ZipDeploy)

And also you can use https://testingnodeapp.scm.azurewebsites.net to manage your web app in Azure.

After successful upload of your zip file, Kudu service will extract and publish your files to the Azure web app.

Now you can use you Node.js API. as https://.azurewebsites.net

(https://testingnodeapp.azurewebsites.net/)

So as per the previous local URLs, you can use this also in the same way.

(https://testingnodeapp.azurewebsites.net/getText/Hello user)

(https://testingnodeapp.azurewebsites.net/getItems/2)

I'm sharing my github repository of the API I've created.

https://github.com/kavindasas/Node-JS-API-with-Azure

Hope you understood :)

References: https://docs.microsoft.com/en-us/javascript/azure/?view=azure-node-latest

Make Your FAQ Bot Application By Simple Clicks

Sasindu Kavinda — Sat, 12 May 2018 19:42:37 GMT

Microsoft Bot Framework is an amazing framework that people are using it more and more to build Bots these days, because it is so simple to put together a Bot very quick. When talking about the bot framework, and chatbots in general, people usually assume that these are all using some clever logic and Natural Language Processing (NLP) to deliver a chunk of business logic via a natural language interface. With the bot framework this is most likely implemented by wiring up the Language Understanding Intelligent Service (LUIS): originally a stand-alone, (optionally) self-training, natural language understanding service, but now part of Microsoft Research’s Cognitive Services – previously “Project Oxford” – a collection of extremely powerful machine learning APIs for processing images, video, text, speech, to extract meaning. As a simple bot application let's create a conversational bot. So this article will be based on creating a bot application which supports QnA or else known as FAQ bot. As the first step you have to login to your azure portal and need to create a new service => Search for "QnA Maker" and then create the service with selecting the resource group and pricing tires. So after the deployment, you can view your created bot service in your azure portal. Then you can navigate to https://www.qnamaker.ai/Create to create the QnA Bot. There you need to login from the account you have created your service. Now you will see what are the available Azure directories and according to the directory, your subscription and your available bot services you created. As the next step you have to give a name for you knowledge base. Next you can populate FAQ list as your knowledge base or you can manually add new question and answer pair to the knowledge base to train. After that you can train your bot service upto the given questions. Then it will show you the extracted questions and answer pairs in your model. And also from here, you can just add new question and answer pair. Like below image, you can just add new questions. Then you can Save and Train your QnAs. And the most impressive thing is, you can see a live testing of your created bot by click the "Test" at your right upper corner. Now you can publish your Bot Service. So, now you can see the request URI of your deployed bot application. You can just request this uri and get back the response as the answer. Like wise you can create this as a sample bot application. In addition you can use Luis service to understand the user's input and match words and then it will return an answer with much confidence. Happy Coding :) References: https://www.luis.ai/ , https://qnamaker.ai/

Azure Custom Vision Service : How to do a simple custom image recognition.

Nissanka Seneviratne — Fri, 27 Apr 2018 12:58:29 GMT

Custom vision service is a part of the cloud based azure cognitive service API.While the traditional computer vision API is effective on generic images it performs poorly when we want the results for a particular use case or a domain. This is where Custom Vision API comes in. We can train the image recognition algorithm from our own labelled data set thus achieving higher accuracy. Through this tutorial I aim to go through on how to make your image recognition simple through the custom vision API. As the first step login to https://www.customvision.ai/projects using your Microsoft account. Choose the domain that you prefer. The Food domain is best optimized to detect food items, Adult domain is best optimized to recognize Adult related content. If you are not sure on which domain to use it choose the general domain. The domains with an exclamation mark next to them can be exported and used somewhere else. I'll be using this service to identify tomato diseases. Then you will have to upload your image set to the cloud service from your local device.When uploading first upload the related images first. For example in my data set I will first add the tomato blight data set. In the next step we have to add the labels for this data. This is the most important step that affects your accuracy. So please make sure to add the proper labels that describe your image here. In my data set i'll label it as tomato blight as i uploaded images with tomato blight disease. After the image finishes uploading click Add images option again and add the other data set. I will be uploading the data set with healthy tomato leaves and giving it the relevant label.Make sure you upload at least 5 images per each tag. Then click on the train button and wait for some time which will vary based on size of your data set. Then using cross validation of the provided data set the precision and recall values are given.These values will differ based on the size of the data set and the labeling of the data. Thereafter click on the quick test option to upload an image which is not in the dataset. Using this option we can test out whether our service works in the way that we intended. This is the result that i got when i uploaded an image with tomato blight.It gave the prediction with an accuracy of 94% which is very reliable. You can publish this as a web service and use it in connection with your app. Microsoft Azure has simplified the process of prediction for you and now all what you need is a data set with proper labels to create the next big image recognition app. Please post in the question board if you have any queries or contact me through astroanju80@gmail.com

Connect Your IoT Devices to Cloud with Azure IoT Hub

Sasindu Kavinda — Sat, 24 Mar 2018 14:40:55 GMT

So IoT is not an extreme new, in fact it's been something companies have been doing since before it was named IoT. The Microsoft Azure IoT Hub is an IoT suite in the Azure cloud, which offers several services for connecting IoT devices with Azure services, processing incoming messages or sending messages to the devices. From a device perspective, the functionalities of the Microsoft Azure IoT Hub enable simple and safe connection of IoT devices with Azure services by facilitating bidirectional communication between the devices and the Azure IoT Hub. This has many features which you need not any much effort to work the IoT with cloud.

Provides multiple device-to-cloud and cloud-to-device communication options. These options include one-way messaging, file transfer, and request-reply methods.
Provides built-in declarative message routing to other Azure services.
Provides a queryable store for device metadata and synchronized state information.
Enables secure communications.
Provides extensive monitoring for device connectivity and device identity management events.
Includes device libraries for the most popular languages and platforms.

For the IoT services, there are two kind of Hubs in Azure. IoT Hub and Event Hub. So they have some differences in the connectivity and the usage with the IoT device it self. So IoT hub supports two way communication where Event hub mostly used only device-to-cloud service. And when we take the file uploading scenario. event hub will not supports. IoT Hub Provides device SDKs for a large variety of platforms and languages, in addition to direct MQTT, AMQP, and HTTPS APIs. And in Event Hub, it supports on .NET, Java, and C, in addition to AMQP and HTTPS send interfaces. You may wonder why we have to use this Azure IoT Hub for the IoT device connectivity? So in this IoT hub, we can create our devices with a unique device name(known as device Id) which allows you to send messages from your raspberry pi or Arduino. So the IoT Hub contains all the information of the device which registered. As a example, the state of the device, configuration and all needy informations saved. Also this has a high secure connectivity between the device and the cloud. So each of the device registered in the cloud has own authentication and secured connectivity. This will allow you to send messages from device easily. No dozens of lines of codes needed. And the best thing is the support of various languages with the support of SDKs. So this supports C#, Java, JavaScript and Python. So as a scenario which we need to get sensor data from 1000 devices all over over the country. You ever knew that this supports 500,000 devices simultaneously. So we can send data from 500,000 devices per second. So this is pretty awesome. So wait till next blog which will show how to send and receive data from Azure IoT Hub with a real example. References: https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-get-started-simulated

Enabling Search Abilities to Your Apps with Azure Search

Sasindu Kavinda — Sun, 25 Feb 2018 10:50:33 GMT

Azure Search is a cloud service that provides indexing and querying capabilities for uploaded data. This removes the infrastructure requirements and search algorithm complexities traditionally associated with implementing search functionality in an application. Definitely, you may face situations to integrate search facilities in your mobile, web or any application you develop. It’s search. Below are a few examples where you would use Search.

In a Blog Site or Mobile App, you would able to integrate a search option by entering some relevant keyword.
Web Site or Mobile App You might want to search some relevant data, such as location data, food prices etc.
E-Commerce sites You would search for different gadgets, books etc. depending on your needs.

Everything depends on "Search". So this is a component of the Microsoft Azure Cloud Platform providing indexing and querying capabilities for data uploaded to Microsoft servers. The Search as a service framework is intended to provide developers with complex search capabilities for mobile and web development while hiding infrastructure requirements and search algorithm complexities. For everything,

We don’t need to manage the infrastructure.
Everything related to the required infrastructure for provisioning the search will be taken care of by Microsoft Azure.
Reduced time and efforts both in terms of development and management of the functionalities.

Azure Search supports over 50 language analyzers to choose from. This can be exposed as a rest API. Let's create a Search Service in Azure Portal. Create A Resource --> Azure Search Click Create and you can fill the data like below. :) Now you can see the dashboard of your created search service. For the testing purposes, you can import a sample data set provided by Azure. As you can see in the video, you can connect any data source or multiple data sources to feed data when we search through the index. (An index is like a database table that holds your data and can accept search queries. You define the index schema to map to reflect the structure of the documents you wish to search, similar to fields in a database.) As you can see in the following image, when you importing your data set, you can define what are the retrievable, searchable columns. And you can change the key if you need as in the time you import your dataset. Likewise, you can do your search service using Azure Search. You can use this in your application as a rest call. Ref: https://docs.microsoft.com/en-us/rest/api/searchservice/ Comment Below if has any issue :)

How to host WordPress site on Microsoft Azure

Yasintha Perera — Tue, 20 Feb 2018 12:44:43 GMT

Greetings! Today I’m going to install WordPress on Azure. Let’s see how it going to work.

After you logged into the Azure portal navigate to create resources and search WordPress.

then press create

Then you’ll get website Configuration and enter the following information :

website name (example.azurewebsites.net)
Subscription
Resource Group
Database provider (Azure Database for MySql)
App service plan
Database (In here you have to configure database server)

Now a new WordPress site is being deployed to Azure websites and will take a few minutes to create the resource group and provision Website, Database and Storage resources within this group. Once you are notified in the notifications section that the deployment went through successfully. You can see it on the dashboard. In my case sampleblog123.azurewebsites.net

After clicking it, You can see the website settings.

Click the URL and setup WordPress web as following figures.

Select your preferred language

Set website title and WordPress admin portal username password and email.

All done. Let’s log in to our WordPress web now. :)

your WordPress admin portal URL will be example.azurewebsites.net/wp-admin

Thanks. Reference https://medium.com/@yasintha/how-to-host-wordpress-site-on-microsoft-azure-dd66e8a2e376

Automate your every step using Microsoft Flows

Sasindu Kavinda — Sun, 18 Feb 2018 18:18:44 GMT

If you're not using Microsoft Flow for process automation, are you really as efficient as you think you are?As we move through our daily routine, it's natural to wonder how you can become more efficient. How do you automate simple tasks? Where can you save time within a process without sacrificing quality or reliability of the results? Microsoft Flow answers the call for the automation of workflows. For automating these processes, you can use your outlook account or O365 account. This is simply free for use. As the usage, you can automate your emails, get a notification when someone submitted a form, get a notification when someone posted on your facebook or twitter etc. Actually, this is super awesome. You don't need to worry about anything. Just wanted to do connect everything in flow wise. You can use "Schedule" trigger also if you need to run some flow to run as an example in 15 minutes, per day likewise. So get started working with flows. Go through with flows.microsoft.com They will ask you to sign-in. After you sign-in to MS Flows, you can see the following screen. There you can see "MyFlows", "Approvals", Templates", "Connectors". MyFlows are the flows which you created previously. Approvals are used to create a flow to the approval processes. Templates are the sample templates which we can use to create flows. Those are common scenarios in day-to-day life. As in Connectors, those are the things we need when creating our flows. Those are the access tokens, keys for the external sources. Such as if we want to use outlook email, You need to authorize the outlook.com account as a connector. You also can add those connectors in the time you create your flow. Let's create a flow to send an email once in 15 minutes. You can see how to create that flow. [video width="2560" height="1600" mp4="https://msdnshared.blob.core.windows.net/media/2018/02/Untitled-9.mp4"][/video] This is the email received, So as it works fine. Let's create a flow which reads the twitter posts with a #tag and send analyze the twitter post body. Then the twitter body will send directly to Text Analytics API to get the "SENTIMENT" and send an email with the sentiment percentage. For this first you need a Text Analysis API key, you can grab a free key from here: https://azure.microsoft.com/en-us/try/cognitive-services/?api=text-analytics Then you need to add a connector to your twitter account and outlook account. After all, you can see your connection are in active status. Finally, you will receive an email saying "Sentiment Detected" with the body including the sentiment score of the tweet. In the send email flow item, I set the email body just as the Returned Sentiment Score. You can change it as Tweet Body to get the Tweet Post included email. For that, you can attach the previous item's results as in the email. And like here, you can just update your flow as follows, You can edit the flow as a team if you are using an O365 account. Likewise, you can connect anything from here and there. Not just Microsoft techs, you can send email to Gmail or connect google drive as well dropbox. This is really simple to start and workout. This is a really awesome tool. Use this and reduce your day-to-day workload.

Azure Functions Usage

Sasindu Kavinda — Sun, 11 Feb 2018 12:26:10 GMT

In this post, I'm going to discuss Azure Functions. Azure Functions is a solution for easily running small pieces of code, or "functions," in the cloud. Azure Functions enables you to write serverless code to handle events at scale, with minimal overhead and cost. Azure functions work with both C#, F# and JavaScript functions. You may use many different triggers and binding types supported by Azure functions including monitoring queues. And it works with blob storage, sending emails, and able to develop in Visual Studio or from the command line with a text editor, if you prefer. As an awesome feature, this provides the ability to automate deployments, debug and monitor our functions. As per the working of Azure Functions, you don't need to write, deploy, run in different places and deploying the application getting it as a big task. Because the Azure Functions gives you every feature to work with different combinations with the support of C# F# codes and JavaScript codes. Will go through with the implementation of Functions. In the Azure portal, you can search "Function Apps" You will need to enter the application name which you can access later through the URL. The resource group also needed with an active subscription. Then you can select either you need to deploy this in Windows or Linux Os. Then with new storage, you can simply create the Azure Function. When you create the app you can see like this. After that, you can search with the created name to find your azure function to open the developing of your function. Then you can open the application. Now you can see your application is now running mode. With the name you created, you can navigate to your Azure Function using the URL. Eg: https://testingfunctionappdemo.azurewebsites.net/ In the function section, you can create a new function with your own preferred language. Let's create "HTTP Trigger" which we can write C# or JavaScript or F# code to run a code as our need. As in the authorization level, there are three options;

Function: You need to provide function specific API key.
Annonymous: No need of any authorizations.
Admin: You need to provide an admin key.

So you can create Function with Annonymous Authorizations. This will show "run.csx" file where the full code that we need to run. This is the C# script. The sample c# script will shows. This script will return "Hello {Name}". Name is the parameter that we need to provide when we calling the URL. Like: https://testingfunctionappdemo.azurewebsites.net/api/HttpTriggerCSharp1?name=sasindu Then this will return "Hello Sasindu" in tags as in XML format. This is the function you created. Then you can create a trigger inside the function to trigger the function as a timer. In the timer function, you can set the schedule which we need to trigger the script code inside the trigger function. Let's edit the time trigger to call the 1st created function. Use below code : ----------------

using System;

public static async Task Run(TimerInfo myTimer, TraceWriter log)

{

var client = new HttpClient();

var response = await client.GetAsync("https://testingfunctionappdemo.azurewebsites.net/?name=Sasindu");

var result = await response.Content.ReadAsStringAsync();

log.Info($"Executed HttpTriggerCSharp1: {DateTime.Now}, Result: {result}");

}

---------------

//Please use your url as: https://testingfunctionappdemo.azurewebsites.net/

Then you can see the output in the log.

Actually, this is a sample function application.

And in the "run.csx" file, you can edit that script as you need. You can use Azure SQL database queries as well. To use SQL DB, When you click the Function Apps Name, you can see "OverView" and "Platform Features". In here, you can find "Application Settings". Then when you go down, you can see the "Connection String" area. Here you can add the connection string as in we previously used in other c# applications. And just you can call that connection string in the run.csx file as below: This is a simple update query. As you can see we can do many more things with the serverless compute service. References: https://docs.microsoft.com/en-us/azure/azure-functions/

Azure SQL Database - Database as a Service

Sasindu Kavinda — Sun, 04 Feb 2018 20:56:36 GMT

Microsoft SQL Server is a relational database management system developed by Microsoft. So as in the usage of Microsoft SQL Server, the first thing comes to our mind is SQL Database which we can create and consume in SQL Server. In the reason of most .Net and other developers intend to use MS SQL Server Databases, the versions of SQL Server also has now come with SQL 2017 as their newest release which was started from MS SQL Server 7.0 and 6.0. In an era which every technology is in the cloud, Microsoft has decided to add new service: SQL Database as a Service. It was just the previous SQL Server Databases which we used as on-premise versions. This was announced in the year 2009. And in the year 2010 – Service went live. So now we can use the Azure SQL DB as a central DB for any usage. Such as if a developer needs to use his application as web and mobile, he can use one single Database for his usage without many configurations at a low cost. Azure SQL Database is offered either as a Standalone database or Elastic database pool and is priced in three tiers: Basic, Standard, and Premium. Each tier offers different performance levels to accommodate a variety of workloads. So the Basic will best suit for a small database, supporting typically one single active operation at a given time. Examples include databases used for development or testing or small-scale infrequently used applications. and the Standard will most cloud applications, supporting multiple concurrent queries. Examples include workgroup or web applications. And the final or Premium is designed for high transactional volume, supporting many concurrent users and requiring the highest level of business continuity capabilities. Examples are databases supporting mission-critical applications. For more on pricing: https://azure.microsoft.com/en-us/pricing/details/sql-database/ Will create a sample database and check how it will go. First, you have to log in to the Azure portal. Then you need to select new Item -> Databases -> SQL Database as follows So you will need an active subscription to create the database. Then you can directly Configure the Server and just create the database. Then it will deploy in your workspace. This is what you see after the deployment of the DB. Here you can see the DB connection string, resource group, active status and the basic details of your DB. As a tip, for the usage of the Database for development purposes, you have to add the IP ranges of the devices which are going to access the database. Otherwise, those connections will automatically refuse by the Azure DB. To configure that, you can see "Set server firewall" button in the middle of the menu. When you click that button you will redirect to a form as below shown. Here you can add "New Rule" mentioning the IP range as Start IP and End IP. So that you can configure the firewall settings. In this post, this is the most eye-catching thing in Azure SQL DB, which is the "Query Editor". Did you ever think that you would able to use SQL Databases without Visual Studio or SQL Server? Yes, you can use it now from this Query Editor. You just need to log in using the User Name and the Password given when you create the server for the Database. Then you will see the Tables, Views, Stored Procedures as you can see in MS SQL Server management studio. The difference is you have to do everything using queries. Just give a try and use Azure SQL DB. You can have free Azure credits with new account from here: https://azure.microsoft.com/en-us/free/ For more details: Azure SQL