Other day I was reading about change feed feature of Azure CosmosDB and thought of possibilities can be driven from it. That triggered me to test a scenario which we deal with in our day to day life which is getting notification of any activity happened on Bank account.

Use case

Typically, in a banking sector, Any bank keep account holders information stored in their database and provides various methods to spend (through debit card), withdraw (through ATM), money transfer (Online or mobile app) and deposit (any channel). The balance will get adjusted based on the transaction type. At the end of successful transaction, as a bank customer we get SMS/email regarding the transaction took place in our account. This will make sure customer is updated of any transaction happening on their account and also if there any fraudulent transaction takes place they will act.

With that thought, I am going to write about how to build the solution using Azure and give information about the services.

Technologies Involved

Azure CosmosDB:- This is heart of this solution. Azure CosmosDB is a multi-model geo replicated database. It provides various APIs like SQL, MongoDB, Cassandra, Graph and make sure core features like security, multi-region replication, consistency, partition, scalability, indexing and log monitoring applies to all. Azure CosmosDB doesn’t enforce the learning of new tools but provides an ability to developer to continue using existing tools for example if a developer is working on GrpahDB, he/she canstill use gremlin console to interact with it.

Change Feed: - This feature once enabled, creates a lease collection. This collection keeps track of changes happening. It stores changes the way they happen on the collection. All the inserts and updates get captured however for deletes, it requires workaround like adding additional property to decide delete. Change feed can be accessed by Azure function (we’ll cover in this blog), Azure Cosmos DB SDK and Azure Cosmos DB Change Feed Processor library. Below is the example of document stored in lease collection.

Azure Functions: - Based on serverless architecture, Azure Functions provide ease of running micro code without worrying about setting up infrastructure, scheduling job, security, multitasking etc. One can quickly build a function and run it on any scale required. This architecture can be used in various domains like IoT, retail, finance etc. For more features and capabilities of Azure functions please refer here. Before production deployment please refer here to make sure the programming language that you are planning to use (.Net, JS, Java etc.) is in GA and not in experimental.

PowerBI: - PowerBI is the representation layer. It gives a a visualization capacity for users to make sense of different datasets and computations done on them. A dashboard shows the story of data it’s presenting. Microsoft PowerBI has evolved a lot since the time I started working with it. It captures all the different verticals and audiences. The ability to develop once and render it on any device makes it more powerful. For more information please refer here.

Architecture

• Active Azure Subscription
• Visual Studio 2017

• Go to www.portal.azure.com
• Click Create a resource -> Databases -> Azure Cosmos DB

• Once page is open, Enter ID
• Select API as SQL
• Select Subscription
• Create/Select Resource Group
• Select Location
• Select Pin to Dashboard
• Click Create

• Once the page is open, click Overview
• Click Add Collection

• Let’s create Database and a collection to store account master information. Provide Database id
• Provide Collection Id
• Select Storage Capacity as Fixed (10 GB)
• Click OK

• Once Collection created successfully, Click New Collection

• Enter Database id

• Enter Collection Id
• Select Storage capacity
• Select Throughput
• Click OK
• Once Database and collection created, screen will look like below

• Click Close (X)

Adding Account Master Details

• Go to https://github.com/rawatsudhir1/AzureCosmosDBChangeFeedUseCase and either clone the repo or download it as zip
• In the repo under SupportingFiles, open CustomerAccountInfo.txt file. This file has some records or json document (contains customer account master information) which we will upload through portal. The other approach is to create a web-app or mobile app (with proper security enabled) to send this data.

*Note :- I am not following any data modeling technique. Generally, when you are storing data in Azure CosmosDB please decide, whether application will perform read or update extensive operation.

• Copy first record from the file (from line 1 to line 8 as per the above image)
• Goto Azure CosmosDB account, created in earlier step.
• Click Data Explorer

• Click SummaryInfo
• Click Documents
• Click New Document
• Paste the record copied from github in earlier step

• Click Save
• Repeat Steps to add more records (copy from github repo)
• After adding all six records, here is how screen look like

• Close (X) Data Explorer

• Click Create a Resource
• Click Compute
• Click Function App

• Once blade is open, Enter App name
• Select Subscription
• Create new or Use existing Resource Group
• Select Windows as OS
• Select Consumption Plan as Hosting Plan
• Select Location (Select the same region where Azure CosmosDB is created)
• Let default values for Storage
• Click Pin to dashboard
• Click Create

• Once Function App is created, Click on New (+)

• Click Custom Function

• Choose a template, enter cosmos in search

• Select Cosmos DB trigger
• On New Function, Select C# as language
• Provide Name of the function
• On Azure Cosmos DB account connection click new and select the cosmodb account created earlier
• Provide Collection name
• Provide Database name

• Let Collection name for leases as it is
• Click Create
• Once function is created, Click on function app and then Application Setting

• Under Application settings, add two variables endpointUrl and authorizationKey. These variables hold values to connect CosmosDB to retrieve and update Bank Account Master table. Copy values for both the variable from CosmosDB account under Keys section.
• Add endpointUrl , authorizationKey with values and Save it. This is how it will look like.

• Click on function (if you choose default name then it will be CosmosTriggerCSharp1)
• Copy code from https://github.com/rawatsudhir1/AzureCosmosDBChangeFeedUseCase/tree/master/AzureFunctionCode and paste it in function. Click Save

• Open Visual Studio 2017, create a console application
• Copy code from here (Program.cs) and paste it.
• Include Microsoft.Azure.DocumentDB.Core and Newtonsoft.Json nuget package.
• Add endpointurl and authorizationKey as shown below.

• Run/F5 to run console application. This action will post a transaction to Azure CosmosDB

• Switch to Azure portal
• Look at Logs window in function

• Please make sure to setup twilio test account to send sms notification.
• Switch to Azure portal and open Azure function
• Click Integrate
• On Outputs click New Output

• Select Twilio SMS and click Select

• Keep message as Message parameter name
• Keep acctsid as Account SID setting. The variable acctsid will define under application setting which hold value of Account SID (got from Twilio dashboard)
• Keep authtoken as Auth Token setting. The variable authtoken will define under application setting which hold value of Authentication token (got from Twilio dashboard)
• Provide To number. Make sure this phone number is verified in Twilio
• Provide From number. This information available in in Twilio dashboard. If first time created Twilio account then it will provide a number to use for some time.

• Let’s add variables in Application settings. Click function and Application settings

• Add acctsid and value (Account SID from Twilio Dashboard)
• Add authtoken and value (Authorization Token from Twilio Dashboard)

• Click Save
• Copy code from github and paste in
• Click Save. Make sure there is no error in log
• Run console program and post a transaction

• Once it run successful, a SMS will be sent to the mobile number. If not received make sure defined number is not in DND (Do Not Disturb) mode.

Building PowerBI Report

• Follow here to build a PowerBI dashboard.

Summary

In this blog, we built a use case to showcase how Azure COSMOSDB can be used in Banking Sector. However, there are another scenario which can be build using this feature like create request for vendors once an order is received (Retail domain), Immediate action on bad feedback (Service domain), Alert when a new quotation request is received (Insurance domain) etc.

Thanks for reading and I hope you liked it.

Thanks to my colleague Gandhali for her suggestions.

Eat Healthy, Stay Fit and Keep Learning.

In my role I work with many customer working on different technologies. With BOT, Cognitive, AI bubbling up customers keep asking me about it. One of the scenarios I keep hearing how to build knowledge base to serve their customer effectively. For example

• How I can setup IT help desk for my customer/employee? Specially on mobile incase something goes down due to hardware/software failure on PC, they can still chat to get some help?
• Getting information about product, licensing etc. on the way on mobile?

This triggers me to think about build a solution and showcase how it can be done. After my initial research I figured out services I am going to use “QandA maker” and “Xamarin” to build it. Here is the architecture I zeroed down to work on.

Architecture:- Let’s look onto the architecture. The developer will build, train and publish knowledge base using QnA maker. Then build Xamarin chat assistance application which will connect to QnA maker and response customer query. Once development of application is done, developer will release it for various platform. Same can be download by end user and they can chat from their devices.

Let’s start building it. First, setup knowledge base. We’ll be using Microsoft QnA maker.

• Go to https://qnamaker.ai/ (it's in preview as of today).
• Click “Create new service”. It may ask to login with your Microsoft account.

• Provide “Service Name”
• Either provide “FAQ URL” or upload file containing question and answer or add question and answer manually.
• On the left side there is a “Test” option incase you want to test it.
• Click “Create”

• Add question and answer in knowledge base if you want and click “Save and retrain”
• Click Publish
• On next page click Publish
• Once completed, copy the “Sample HTTP request” information.

• Download chat user interface sample

Changed the code to call QnA service and make async call for better user experience. You can modify the response based on the score as well. Though it's not implemented here but you get score in response. Download the code from https://github.com/rawatsudhir/Building-Chat-Assistance-Solution. Feel free to contribute to github if you have any idea/suggestion.

Here is how final outcome looks like.

Problem Statement:- After launching product in the market, a company want to capture users sentiment from social media.

There were two challenges here 1) capture tweets from twitter or comments from Facebook (assuming product has page listed)  2) Figure out the sentiment of the tweets.

 Technologies Involved:- Microsoft Azure Logic apps, Azure SQL DB, Azure WebJob, Cognitive services and PowerBI. 

 Prerequisite :- Microsoft SQL Server Management Studio, Microsoft Visual Studio, Azure SDK for .NET (Visual Studio 2015), Microsoft Power BI Dashboard and download source code from GitHub and store it in local drive

As a first step, let’s set up  Azure SQL database to store tweets.

1) Go to Azure Portal https://portal.azure.com

2) Once login to the Azure Portal, Click New, Databases, SQL Database

3) Provide Database name, Subscription, Resource group (recommend to Create new for this demo ) , Select source (as Blank database), Server (Create a new Server as mentioned below), Want to use SQL elastic pool? (Not now), Pricing tier (as Basic) and Collation. Check Pin to dashboard and click Create

*Note:- Please note down Server admin login and Password.

4) Once created, open database (Double click) from Azure dashboard.

5) Go to Properties and then SERVER NAME

6) Copy SERVER NAME. 

7) Click Start (Windows sign on bottom left) and search for Microsoft SQL Server Management Studio. Click to open Microsoft SQL Server Management Studio.

8) Paste SERVER NAME (copied in step 6) in Server name, Login and Password

*Note:- Keep Server Name, Database Name, login and Password in a Notepad file. It will help through out this sample.

9) Click Connect. If you encounter below warning then need to setup firewall rule at Azure SQL level. It will allow SQL Server management studio installed on machine to connect SQL Server on cloud.

10) Click Sign in… login to Azure

11) Once successfully logged in, Select Add my client IP (XXXXXXX) and click OK

12) Once logged in, click New Query

*Note:- Please make sure to select the tweetdb/created earlier database from the drop down box.

13) Copy create table script from the folder (DB Script) downloaded from GitHub or copy it from here.

14) Paste query in query window, click Execute

Once Azure SQL DB is set up, now let’s move to next step which is capture tweets real time and store it in Azure SQL db. 

Here I am using Microsoft Azure Logic App. With Logic apps one can easily design workflows on cloud. I created a workflow in logic apps which will capture tweets (contains #MyDemoProduct hashtag) after every 3 minutes. If interested read more about Microsoft Azure Logic Apps here.

Let’s go step by step and set up workflow on Microsoft Azure using Logic App.  

4) Enter Name, Subscription, Create/Select Resource group (select existing one which created in earlier step), Location

5) Click Pin to Dashboard and Create

6) Once validation done, Azure will setup new Logic App service and open designer to create the workflow

7) Click Blank LogicApp

8) Search for twitter connector in Show Microsoft managed APIs. Select Twitter – When a new tweet is posted

9) Login to Twitter account. If you don’t have twitter account please create it.

10) Enter #MyDemoProduct or any other text which you want to capture in *Search text. Set *Frequency is Minute and *Interval is 3. When we run this app it will capture tweets (based on *Search text) in every 3 minute. Click + New step and Select Add an Action

11) In Show Microsoft managed APIs, Search for SQL and select SQL Server – Insert row

12) Enter *CONNECTION NAME, *SQL SERVER NAME, *SQL DATABASE NAME, *USERNAME and *PASSWORD. If you haven’t copied these information in notepad as suggested earlier then get it from Azure portal. Click Create.

13) Select table name TweetFromTwitter from the *Table name drop down box

14) Next step to map table columns with twitter fields. Move to every column and click Add dynamic content and select twitter field as described below. Leave Score column as blank. Click Save

15) Close Logic App Designer. Click Run Trigger

16) For test purpose tweet with #MyDemoProduct or any other text you provided earlier (when setting up Logic Apps)

16) In status, notice Logic App execution

17) Click Disable as it will fill up Azure SQL database table quickly based on the frequency of tweet post

18) Switch to Microsoft SQL Server Management Studio and Execute query (Select * from TABLE_NAME). Notice Score column is NULL

*Note:- Since I tweet from dummy account, it capture same location. I changed manually Location column value for some records.

Next Step is to setup Cognitive Services to get sentiment of tweets captured in Azure SQL Database. Also let’s setup Azure web job.

1) Go to Azure Portal https://portal.azure.com. Click New, Intelligence + analytics, Cognitive Services APIs 

2) Enter Account name, Subscription, API Type as Text Analytics (preview), Location as West US (only available region at present), Pricing tier as Free and Resource group as Use existing (choose one which is created earlier). Check Pin to dashboard and click Create

3) Once done, it will show below screen indicates Microsoft Cognitive Services is created successfully. 

4) Click Keys. Copy Key 1 and paste in Notepad.

Microsoft Cognitive Service is setup. Next step is to make changes in code and deploy it as Azure webjob. If not already, please download the source code from https://github.com/rawatsudhir/ProductSentimentAnalysis

1)  Open Microsoft Visual Studio 2015. Provide Azure SQL DB credential and Subscription key (of Microsoft Cognitive Service)

2) Click Start. This code took all tweets from Azure SQL DB and pass it to Microsoft Cognitive Services get sentiment of each tweets and store it

3) Switch to Microsoft SQL Server Management Studio. If not open already please go to Start menu and click Microsoft SQL Server Management Studio.

4) Connect to Azure SQL DB and run Select top (100) * from TABLE_NAME

2) Once created below screen will show up

3) Switch to Microsoft Visual Studio. Right click on project and click Publish as Azure WebJob

1) Click windows button, search and click Power BI Desktop

2) Once Power BI dashboard opened, Enter Server and Database name. Click OK

3) Click Database, Provide Username, Password. Click Connect

4) Select TweetFromTwitter and click Load 

This blog post shows how it can be done but when you execute program it will prompt for authentication which will not work when you want to automate the entire process.

We want to automate the process means no manual intervention is required to provide credential for authentication. In below steps I am setting up a application in Azure AD (Active Directory) which is a must if you want to automate the process, providing access to resource group and then .net code setting up HDInsight cluster.

First step is to create Azure Active Directory application and service principal to access resources on Azure. Please follow here to setup.

Once Azure Active Directory application and role assignment  is configured it will look like below.

I blogged about how one can load incremental data from Azure Table Storage to Azure SQL using Azure Data Factory in my teams blog. Please read full post from here.

Hope it will be helpful. Thanks for your time. Cheers!!!!

- Eat Healthy…Stay Fit and Keep learning.

This blog post is going to be a quick one.

Today I was working on a issue and a part of it was to add additional storage account to HDInsight cluster (ARM). For me it’s little tricky to find out so I thought to put it here so incase someone finds it handy.

Hope it will be helpful.

- Eat Healthy…Stay Fit and Keep learning.

Hi again. While I was working on Azure Stream Analytics, I got into a situation where I need to parse array data from incoming JSON event. So I blogged about it in my teams blog here.

Hope it will be helpful. Thanks for your time. Cheers!!!!

- Eat Healthy…Stay Fit and Keep learning.

Getting Started…

I recently did a community event on IoT and thought to share my learning's.

My journey with IoT starts when I started working on Microsoft Azure Stream Analytic Service. It’s interesting how real time reporting brings lot of insights for business decision maker to get real sense of their business and make effective decisions. There are lot of real time services available like storm, MapR, Microsoft Azure Stream Analytic Service etc which help building real time solutions. I worked on Apache storm and when I started working on Azure Stream Analytic Services, one thought came to my mind which is “This is really simple”. I am not doing any comparison here which many people might be interested in. I will probably write another blog on such comparison but for now some information can be found here.  

During my research phase, I read lot about IoT (Internet of Things). So I started getting deeper and deeper and wonder how technology benefit world in real time. How this concept can help us to make a better world. There are lot of work going on in this direction.   

I started looking onto sensors, board, devices etc. so that I can get involved in it. I looked onto various boards available in the market like CubieBoard, Banana Pi, MinnowBoard Max, Intel Galileo, Raspberrypi etc. I bought raspberry pi because it’s cheap   and I thought to use it as my entertainment device at home. I setup my raspberry pi device with sensor and send all information to PowerBI through Microsoft Service bus and Azure Stream Analytics. Here are the steps.

When I received my raspberry pi 2 kit, here what I got in the packet.

1) Raspberry Pi 2 Model B with 1GB RAM, QuadCore CPU

2) 16 GB Micro SD Card with NOOBS (New out of box software)

3) Plastic Case

4) 5V 2Amp Power Adapter with Micro USB cable

5) High Quality HDMI Cable

6) 4Pole 3.5mm to 3 RCA Video/Audio Cable

7) Cat5E Network Patch Cable

The 16 GB Micro SD Card I received have ‘new out of box software’ includes various operating system like Raspbian, Pidora, OpenELEC, OSMC, RISC OS, Arch Linux. We can choose to install any one the OS provided in Micro SD Card. Though I have Raspbian on the card but I thought to format it, download the bits and install it. Here are the steps…

Setting up device…

1) Put SD Card in a card reader.

2) Put SD card reader into the laptop.

3) Format the card.

4) Download Raspbian image on a separate disk.

5) I am using “win32 Disk Imager” to the write image (downloaded in previous step) on the card.

6) Put SD card to raspberry pi

7) Connect network, Power, Display, Keyboard and Mouse to raspberry pi and switch it on.

8) It will start booting up the device and pause incase you want to change anything in configuration. The default keyboard layout is British so you may want to change it. That can be done even after.  Click finish.

9) Once it finish the configuration, it open the console to run queries. Type “startx” and press enter.

10) The above command will open Linux UI (below).

Next I want to test some code to connect to event hub and send events. Let’s explore how that can be done.

Installing Azure SDK for python…

11) Open LX Terminal.

12) Setting up python setuptools. Type following command sudo easy_install pip

13) Install azure sdk by running following command sudo pip install azure

14) Once installation is done. It’ll be good to reboot. Type sudo reboot and press enter.

15) After Reboot, it will prompt for login and password. Type pi as login and raspberry as password. Press enter.

16) Type startx and press enter.

Setting up Service on Microsoft Azure…

Before we exploring python code to send events to Event hub, we need to setup service bus on Microsoft Azure.

17) Open browser on raspberry pi or on any other device.

18) Goto https://manage.windowsazure.com

19) Login with your account which is associated with Microsoft Azure Subscription.

20) Once you login, Click on Service Bus and click CREATE A NEW SERVICEBUS. You can use existing service bus as well.

21) Enter Service Bus Name, Region, Type and Messaging Tier as mentioned below.

22) Once service bus set up, double click on newly created service, click on Configure and copy the Primary Key which is required in next step.

We are not going to create event hub. That will be done through python code. For more information on service bus can be found here.

Sending events from Python code…

23) Open Python IDE

 

24) Click File, New Window

25) Copy & Paste the following code

#import random because using one of the functions in code
import random
from azure.servicebus import ServiceBusService

# SharedAccessKeyName from Azure Portal
key_name ='RootManageSharedAccessKey'

# SharedAccessKey from Azure Portal
key_value=''

# Pointing to Microsoft Azure Service Bus
sbs = ServiceBusService('SERVICE_BUS_NAME', shared_access_key_name=key_name, shared_access_key_value=key_value)

# Event Hub Creation
sbs.create_event_hub('hubcreationfromlinux')

# Some randomg values
devicecounts = 5
eventcounts = 1000

for eventcount in range (1, eventcounts) :
    deviceId = "dev-" + str(random.randrange (1, devicecounts))
    temperature = str(float("{0:.2f}".format(random.uniform (0,50))))
    sbs.send_event('hubcreationfromlinux', '{ "DeviceId":"' + deviceId + '", "Temperature":"' + temperature +'"}')
    print "DeviceId : " + deviceId + "  Temperature : " + temperature

The above code I put up in github here.

26) Click File –> Save

27) Click Run –> Run Module

28) Once code start executing, it will first create a Event hub and then start sending the events.

29) Switch to Microsoft Azure portal and click on newly created event hub. Under the dashboard, it will show events are coming up.

 

That’s all for now. I hope you like it (though it’s little longer ). I will write another blog on same line where I will showcase how to use sensors with raspberry pi and sending events to event hub and then how to present that information on PowerBI. Signing off….Have a great day!

- Eat Healthy…Stay Fit and Keep learning.

In this short article I’ll just explain what above means and show core calculation.

So let’s take an example if you have 740 HDInsight Core on your subscription and 740 HDInsight Core being used by 8 clusters (167 nodes). Each worker node consumes 4 cores and each head node consumes 8 cores.

Here is the calculation:-

So if you try to create cluster at least with 1 node it requires 12 cores which is not available to subscription. That’s why you get error “The cluster can have at most 0 data nodes” and message “99% of 740 HDInsight cores” in the dashboard.

In case it’s require to increase the core, one of the options is to raise ticket from the portal.

The other option is reuse existing cluster in case you are not using it.

HTH.