Technoloy Evangelist – Loves New tech, and a his https://docs.microsoft.com/archive/blogs/brijrajsingh/feed.xml Just another Developer Network site Wed, 17 Apr 2019 03:20:27 GMT en-US hourly 1 Multi Column GroupBy Aggregate using R in Azure ML studio https://docs.microsoft.com/archive/blogs/brijrajsingh/multi-column-groupby-aggregate-using-r-in-azure-ml-studio Mon, 31 Jul 2017 04:50:52 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=155 here and get some basics. Its the most ultimate experience in my knowledge that orchestrates the machine learning process's each level very intuitively. Quite recently I was working on some data which required GroupBy aggregation at different column levels. Consider the data in this format  
RegionCode RegionName StoreCode Category ProductCode ProductName Quantity Size Gender Season PricePointName ProductCategoryName
3 Michigan 499 SHOES 487369 KWANGO 1 6 Men FW 09 FTW < 10000 FOOTWEAR
3 Michigan 499 TOPS 498510 ADIPURE BRA 1 L Women SS 14 FTW 6000-6999 APPAREL
3 Michigan 499 SANDALS/SLIPPERS 499408 ADI SUN 1 8 Men FW 15 FTW 1000-1999 FOOTWEAR
3 Michigan 500 SANDALS/SLIPPERS 499429 ADI SUN 2 8 Men FW 15 APP 2000-2499 FOOTWEAR
3 Michigan 500 SHOES 500228 DURAMO 6 LEA M 1 11 Men SS 15 FTW 6000-6999 FOOTWEAR
3 Michigan 500 SHOES 500284 HOWZAT J V 1 3 Kids-Boys FW 14 FTW 3000-3999 FOOTWEAR
3 Michigan 500 PANTS 541832 ESS 3S KN PANT 3 M Women SS 14 APP 1500-1999 APPAREL
3 Michigan 499 PANTS 544313 NEW FIREBIRD TP 1 34 Women SS 15 APP 1500-1999 APPAREL
3 Michigan 499 PANTS 544314 NEW FIREBIRD TP 2 40 Women SS 15 APP 2500-2999 APPAREL
I wanted to get a group by aggregate SUM on Quantity group by multiple columns like RegionCode, StoreCode, and Category. It would only take a line of code using R, Just drag and drop the Execute R Script component and write this simple statement
# Map 1-based optional input ports to variables dataset1 <- maml.mapInputPort(1) # class: data.frame data.set = aggregate(dataset1$Quantity, by=list(RegionCode=dataset1$RegionCode,StoreCode=dataset1$StoreCode,Category=dataset1$Category), FUN=sum)
# Select data.frame to be sent to the output Dataset port maml.mapOutputPort("data.set");
Run the experiment, right click the "Execute R Script" component and click on "Result Dataset --> visualize"
[caption id="attachment_166" align="aligncenter" width="919"] Visualize Data[/caption]
     ]]> Downloading an Azure blob storage container - Python https://docs.microsoft.com/archive/blogs/brijrajsingh/downloading-a-azure-blob-storage-container-python Sat, 27 May 2017 08:12:55 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=136 stack overflow post . The post however doesn't give a complete resolution but gives a pretty good idea of what needs to be done. What does your humble next door TE does in this situations? Well, he rolls up his sleeves (already rolled up in this sweltering Delhi Heat ) and wrote a small script to help you download a blob container with paths and sub-paths to your local machine while preserving the directory structure.  ]]> Installing Azure CLI 2.0 on Windows https://docs.microsoft.com/archive/blogs/brijrajsingh/installing-azure-cli-2-0-on-windows Thu, 02 Mar 2017 04:53:25 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=125
  • ACS (Azure Container services)
  • Compute
  • Networking
  • Storage
    • The Azure CLI can be used on Linux, MacOS and Windows. While the official documentation here mentions all the steps required for installation in all supported operating systems, for windows people may face this issue when right after installation when you try to run an az command it says "az : The term 'az' is not recognized as the name of a cmdlet, function, script file, or operable program." The issue is because the azure cli 2.0 is installed in location - C:\Users\<username>\AppData\Local\Programs\Python\Python35\Scripts\  and this path isn't added to the PATH variable. Well! until that happens here is the fix First make sure you have python installed in your machine. If you don’t have the original CLI (or python) at all, you need that first. Download and install it from here: https://www.python.org/downloads/release/python-352/
    • Uninstall Azure CLI earlier versions with command - pip uninstall azure-cli
    • Re-install Azure CLI 2.0 - pip install --user azure-cli
    • Add the path C:\Users\<username>\AppData\Local\Programs\Python\Python35\Scripts\ to PATH variable                                                                                                                                                                                                                                    setx PATH "%PATH%;C:\Users\<username>\AppData\Local\Programs\Python\Python35\Scripts\"
    • check if the az command is working - az --help
    Ideally the tool must have removed the older cli and made necessary changes in the PATHs. If it still doesn't work just drop me a comment or DM me @brijrajsingh]]>         Chat Bots — Designing Intents and Entities for your NLP Models https://docs.microsoft.com/archive/blogs/brijrajsingh/chat-bots%E2%80%8A-%E2%80%8Adesigning-intents-and-entities-for-your-nlp-models Sun, 29 Jan 2017 15:52:48 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=115 1. Casual Intents — I also call them ‘Small talk’ Intents. These intents are the opener or closer of a conversation. The Greetings like “hi”, “hello”, “Hola”, “Ciao” or “bye” are the opening or closing statements in a conversation. These intents should direct your bot to respond with a small talk reply like “Hello what can I do for you today” or “Bye thanks for talking to me”. The casual intents also comprise of Affirmative and Negative intents for utterances like “Ok”, “yes please”, “No not this one but the first one”, “Nope”. Having General affirmative and negative intents help you handle all such intents and rather take them in context with the conversation bot just had with the client. For ex — if the Bot just asked a question to end-user — you should expect either an affirmative or a negative intent and if its anything else Bot can ask the same question again. Your affirmative and negative intents should be able to handle most such utterances. 2. Business Intents — These are the intents that directly map to business of the bot. For eg — if it’s a Movie information Bot then an utterance from client like “When was Schindler’s list released?” is a business intent that intends to find out the Release year of Schindler’s list and you should label it accordingly with an understandable name like “GetReleaseYearByTile”. Ideally you should think more about business intents because rest of small talk like saying hellos or affirming choices is taken care by general casual intents. Entities Business intents have metadata about the intent called “Entities”. Let’s take an example for an Intent “GetReleaseYearByTitle” — Sample Utterance “When was Schindler’s list released ?” Here “Schindler’s list” is the title of the movie for which the user “intends” to find out the release year. The process of finding the entities can be understood at Part of sentence (POS) tagging. However as a user of NLP as a service you don’t need to get into the technicalities of knowing how POS tagging work but if you do want to here is a nice paper on it https://nlp.stanford.edu/software/tagger.shtml Whenever user is thinking about designing their intents the entities must also be identified and labelled accordingly. Again, in entities you can have general entities labelled for use throughout the intents like metrics (including quantity, count, volume), Dates and most of NLP as service allows you to tag entities of such general types without any big hassle. Some entities may be labelled as composite entities that is having more than one entities (component entity) inside it. As a science, it doesn’t matter if you don’t have this feature with your NLP service as long as you have simple entity labelling. One must define component entities before labelling composite entities. For ex:- Find me a pair of Size 8 Red Adidas Sport shoes. Intent — SearchProduct Entities — Composite Entity — ProductDetail Component Entity — Size — 8 Brand — Adidas color — Red Category — Sport Shoes Training for Intents and Entities Ideally one should train the NLP as a service with some real corpus, so if you have some chat messages with your clients over Facebook or skype or whatever channel you work with those messages/utterances can help training for intents, otherwise you can think of training for intent with your own “Manufactured” utterances for any intent. For ex — Training for intent of “GetReleaseYearByTitle” can have utterances like “what was the release year of movie Pulp fiction” “in which year Pulp fiction was released” “when did pulp fiction came” — Bad English I know :) “When was Pulp fiction released” The training with Manufactured utterances helps in bootstrapping the system but one must re-train the NLP service when it starts getting some real utterances. The process of re-training the system should keep going on until the error rate reduces. The more variance of utterances you receive from real conversations the better you can train your NLP service for intents. Ideally Minimum 5 or optimally 10 utterances per intent are good enough to bootstrap the system. The NLP services go through a routine of Supervised — Unsupervised — Supervised learning phases. Each supervised learning phase serves as a feedback loop through which the course correction is done for the NLP models. User trains the system with utterances — this is supervised learning, then the NLP service learns on its own on the basis of supervised learning and about 10% of what the NLP Service has learnt during unsupervised learning is asked back from the NLP service to the user if what it has learnt has been correct or not, the user again trains or affirms/negates the unsupervised learning results of NLP service and re-trains the model. This process keeps going on and eventually user will find less and less questions by NLP service with more and more confidence towards the questions its asking the user to affirm. Key Takeaways -
    • Identify Intents in advance — differentiate between general/casual and business intents.
    • Identify Entities — differentiate between metric related and noun related entities.
    • If possible train Intents with original corpus of conversations, otherwise train with manufactured utterances. Minimum 5 utterances, optimum 10 utterances.
    • Train, Converse, re-train — feedback loop must continue in order to train your NLP models.
    Interesting reads -
    • https://www.nlpca.com/DCweb/settingintent.html
    • https://research.google.com/pubs/NaturalLanguageProcessing.html
    • https://nlp.stanford.edu/software/tagger.shtml
    • https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45678.pdf
    Similar article at Medium - https://medium.com/@brijrajsingh/chat-bots-designing-intents-and-entities-for-your-nlp-models-35c385b7730d#.dj39gyhtl]]>         Android Sample - Direct Line API - Microsoft Bot Framework https://docs.microsoft.com/archive/blogs/brijrajsingh/android-sample-direct-line-api-microsoft-bot-framework Tue, 10 Jan 2017 05:40:21 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=105 Bot framework allows you to integrate the bots in your custom applications be it web or Mobile. While for Web I was able to find some straight forward samples, unfortunately I couldn't find any sample for Integration of an android app. Direct Line API allows you to communicate with your bot using the REST API or WSS (Web socket) based calls. The API documentation is available here https://docs.botframework.com/en-us/restapi/directline3/ I wrote a very basic Android sample that shows a Chat like interface where you can send the messages to Bot and receive/render the response messages from Bot. The Android code is available at my Github Repo The code is written with synchronous REST API (ain't that a bada** way to write a sample code :) ), I must change it to asynchronous and write a WSS based version of the same sometime soon. ]]> Open source on Azure - CSI Coimbatore Chapter, August 2016 https://docs.microsoft.com/archive/blogs/brijrajsingh/open-source-on-azure-csi-coimbatore-chapter-august-2016 Wed, 10 Aug 2016 11:50:20 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=85 CSI Coimbatore's August event. The community comprises of people from Academia, professionals from local tech industry, and students, the mix is quite even; you don't get to mingle so easily with so many PhDs in one room :), they even have their own office! [caption id="attachment_95" align="aligncenter" width="300"]CSI Coimbatore Office CSI Coimbatore Office[/caption] I was giving a session about Azure on open source, as it turns out there were quite a good chunk of people who were OSS enthusiasts like me, the session was success and Q&A session was pretty good as well.]]> Introduction to Iron Python at - PyDelhi, August 2016 https://docs.microsoft.com/archive/blogs/brijrajsingh/introduction-to-iron-python-at-pydelhi-august-2016 Wed, 10 Aug 2016 11:33:32 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=76 Pydelhi I gave a crisp talk over what is Iron Python and how one can use an Iron python code in .net and vice versa; in fact one can integrate with any one of the CLI language implementations like C#, VB, J# ... The code demos I used are in Github in the repository there are 3 projects built using Iron Python, showcasing the interoperability using IronPython. 1.netClassLib - A C# class library project, that is referenced by the TestIronPython project 2.TestIronPython - An Iron Python project consuming the class library produced by the netClassLib 3.PytoDotNet - A C# project that references the IronPython libs, loads a given Python program using Python expressions or Python files with classes, and executes the given expression or creates an object for the given class in python and calls methods on it. ]]> ARM template for LAP (Linux + Apache + Php) and Neo4j Deployment https://docs.microsoft.com/archive/blogs/brijrajsingh/arm-template-for-lap-linux-apache-php-and-neo4j-deployment Mon, 30 May 2016 09:31:52 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=65 Neo4j which is accessible only to the first VM on a private port. The Sample website in the Apache node can echo out a message if it could get connected successfully. The repo is yet to be merged with the Azure repo, but for now one can find it here  https://github.com/brijrajsingh/azure-quickstart-templates/tree/master/101-lap-neo4j-ubuntu. Lets discuss the construct of our ARM template, This is the folder structure of our ARM template [caption id="attachment_67" align="alignnone" width="467"]ARM Template folder structure ARM Template folder structure[/caption]   azuredeploy.json, azuredeploy.parameters.json, metadata.json and README.md are mandatory file for any template to be a valid ARM template, you can find the rules in the github repo here https://github.com/Azure/azure-quickstart-templates#readmemd. The installation of Apache, PHP and code to check connectivity as well as installation of Neo4J is done by the custom script extensions. install-lap.sh - is the script to
    • Install Apache
    • Install php5
    • Install Composer - Globally
    • Write a Info.php file - to emit phpinfo()
    • Write a file neo4jtest.php - which echo out 'connected successfully' if it can connect to the Neo4J VM on a static IP - 10.0.0.10
    install-neo4j.sh - is the script to
    • Add Neo4J to the packages source list
    • Update apt-get
    • Install Java 8 (Silently) - Neo4J needs JDK or JRE - as in Neo4J(J is Java)
    • Install Neo4j
    • Enable remote connectivity to Neo4J and restart Neo4j
         ]]>         GIDS 2016 - Azure Data Lakes and Optimizing MEAN Stack https://docs.microsoft.com/archive/blogs/brijrajsingh/gids-2016-azure-data-lakes-and-optimizing-mean-stack Mon, 02 May 2016 04:40:27 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=55 GIDS 2016 (Great Indian Developer Summit) at Bangalore, JN Tata Auditorium. I was one of the speakers, but highlight of event was the presence of none other but Douglas crockford himself. I delivered 2 sessions, the first one was about "Performance optimization of MEAN Stack" , a very dear topic to me as I have spent a lot of time with NodeJS and MongoDB and actually had the issues first hand that I spoke about in my session, Mongo is not a secret sauce that may just help your application perform, its like any other DB when it experiences load and I had some really good tips to share about this. The solution I used is already at the Github. Later I had some pretty hearty disucssion with some people from audience about sharding techniques of MongoDB and how to chose the right sharding keys. The Next session was about "Leveraging Data Lake for your Big Data", Now Azure Data Lake is still in preview and till 2 weeks back I just knew about the product but didn't had access to it, So I shot an email to Saveen Reddy and Alan Tam and finally got access in just a couple of days. Data lake for me is a product that just demoratizes the big data market, it has all the components that make it good enough to work with hadoop, hdfs, sqoop, pig, hive kind of big data products yet it has the capabilities that a person who knows nothing about big data may yet dump in huge amounts of data to the data lake store and yet run easy to understand U-SQL queries. Most from my audience were just thrilled to see how easy could it be to write an agreegate query just like SQL and yet not need to learn what goes in the background and how much more easier is to control the degree of parallelism of the jobs that you are running. The sample I used is also on My Github Page. I am sure the ppt decks will be available in a few days at the GIDS 2016 website, until next year...]]> Hygiener- IOT Hack https://docs.microsoft.com/archive/blogs/brijrajsingh/hygiener-iot-hack Fri, 22 Apr 2016 05:56:28 GMT https://blogs.msdn.microsoft.com/brijrajsingh/?p=15 Hygiener Architecture Hygiener Architecture[/caption] The product consists of an ultrasonic sensor based device and a door toggle device. The ultrasonic sensor based device is placed on the disinfectant bottles and if someone goes quite near the bottle to get the disinfectant the ultrasonic sensor readings will tell us. The door toggle device on the other hand provide the information about if the door was opened or closed, a couple of these readings will tell you that approximately how many people entered the area, it may not be too accurate as more than one people can enter a workplace while one person opens the door, but lets say its a corner case for us for now, later on one can use techs like Kinect based sensor to find out exact number of people entering or exiting. Both sensors are attached to Raspberry Pi's running Raspbian, and registered with an Azure IOT hub. HCS04 The Wire diagram for the connecting is as follows [caption id="attachment_36" align="alignnone" width="797"]HC-sr04 wiring with Raspberry PI HC-sr04 wiring with Raspberry PI[/caption]   while making toggle switch was easy, its wiring was as follows [caption id="attachment_45" align="alignnone" width="716"]Raspberry Pi with toggle switch Raspberry Pi with toggle switch[/caption]   Code for both sensors to send data to Azure IOT hub is available at GitHub repository. Once the data is sent to Azure IOT Hub, for both sensors the data is sent to Azure blobs, as well 2 separate stream analytics for both sensors process this data, with cool down periods in mind (sensor may send multiple reading very quickly when someone is near it, or the door toggle happens too quickly). The processed data is -
    • How many people used the disinfectant - as in how many people got near the bottle enough.
    • How many times the door was operated
    Both processed outputs are connected to PowerBI and a dashboard is prepared to see a nice chart (will share the snapshot soon..) is shown. DevOPS for IOT - A question that kept bugging me all the time was this - "How do you push code to thousands of IOT devices ?" for now I needed something which is simple, push the code to git and pull it automagically at the raspberry pi for which I wrote a simple shell script that can pull the code by checking for the update at the git repo; essentially there is hardly a way by the git commands to find the update at server, I did some research at SO and nothing substantially came out, I did a simple thing thus, Pull the latest from the git repo to a temp folder, check the git branch -v at the temp and at the deployment folder, if there is a mismatch, you move temp to deployment, and clear the temp folder. Keep checking for the same at periodic interval by scheduling your DevOPS script in the cron scheduler. The code, diagrams and scripts will be available at GitHub.]]>