Gaurav Agarwal https://docs.microsoft.com/archive/blogs/gagarwal/feed.xml Empowering you and your business with Big data and Analytics Tue, 16 Apr 2019 05:08:08 GMT en-US hourly 1 Making Impact on Oil and Gas Industries - Microsoft Advance Analytics https://docs.microsoft.com/archive/blogs/gagarwal/making-impact-on-oil-and-gas-industries-microsoft-advance-analytics Wed, 25 Jul 2018 16:47:20 GMT https://blogs.msdn.microsoft.com/gagarwal/?p=1525
  • Drilling & Production Optimizing the drilling process and reducing the failure will let to the profitability to the organization. Analyzing the data on real time bases and generating the real time reports dashboard. Leverage the data from all the sensors and creating the self service analytics to drive the production and minimize the safety risks. Below is how to visualize real time data using some of the Azure services:
  • Predictive Maintenance The continues monitoring for the most critical and a complex equipment's is required so that the proactive maintenance or any fatal breakdown can be detected. The predictive maintenance approach required the historical data pointers for the particular type of machine which will help to build the predictive model. Azure Machine learning based model can be one of the key tool for building this type of system along with other tools and technologies.
  • Enterprise Security In the recent past, oil and gas companies have invested heavily to protect their property from any unwanted threats. Big data and analytics can be used to manage internal network threats. Correlating network events with metrics and identifying patterns to predict cyber-terror is critical. By using artificial intelligence, companies can now anticipate IT security breaches and are strengthening security with data from video monitoring. This can be achieve using Microsoft Cognitive Services and creating AI based system. Microsoft Cognitive Services is a set of intelligent APIs for building equally intelligent applications. It currently offers 21 different API for analyzing images, videos, text, and much more. For more information, visit https://www.microsoft.com/cognitive-services
  • Building End to End Data Lake Solution As data is continues to grow and we need a centralized repository to store any size, any type with enterprise-wide hyper-scale capability to process and analyze this data. Microsoft Azure data Lake Store and was architected from the ground up for cloud scale and performance. With Azure Data Lake Store you can analyze all of its data in a single place. It can be integrated seamlessly with your existing IT infrastructure providing you complete cloud big data and advanced analytics platform that helps you with everything from data preparation to doing interactive analytics on large-scale datasets.
    • Every method has its own pros and cons. It's fascinating that we have so many options to develop a particular solution. However I would strongly recommend to try these methods based on the use case of your systems. Gaurav Agarwal]]>         Azure Data Warehouse:Understanding the DWU, workload management, resource and memory allocation https://docs.microsoft.com/archive/blogs/gagarwal/azure-data-warehouseunderstanding-the-dwu-workload-management-resource-and-memory-allocation Tue, 18 Apr 2017 11:49:03 GMT https://blogs.msdn.microsoft.com/gagarwal/?p=1395
  • DWU
  • Concurrent queries
  • Concurrency slots
  • Resource class
  • Memory allocation
  • Distribution
    • Let's try to understand what are there building blocks and how they are important to maximize the performance. We all know that if we go for a higher level for DWU units we will get the highest performance. But we need to also understand how to differentiate a DWU100 with a DWU2000. Do I really need DWU100 or do I need DWU2000 or DWU6000? If I try to compare the DWU Vs the Max No. of concurrent quires, this will look something like as shown in the graph below: If you have noticed that DWU100 has max 4 concurrent queries, from DWU1000 till DWU6000 we are getting max 32 concurrent queries. Now I will say, "Why I should consider DWU beyond 1000, when the max no. of concurrent queries remains the same as 32?" Here comes the concept of concurrency slot, More no. of DWU more no. for Concurrency slots. A DWU6000 you will get upto 240 concurrent slots. The next confusion "What are these slots and how they going to affect the performance of a concurrent query"? There comes another concept called resource class. A resource class will help you in defining the and controlling the memory allocation and CPU cycle allocated to a query. This can be controlled and assigned using the database roles. There are four resource classes smallrc, mediumrc, largerc, and xlargerc. So as the name suggest user with smallrc will be allocated smaller amount of memory but can have more concurrent slot. The concurrency level may decrease when users are assigned to a higher resource class or when SQL Data Warehouse is configured with low DWU. Some queries, like DMV queries, are always allowed to run. Lastly, we need to understand the concept of Distribution. To achieve the highest throughput and query performance a SQL Data warehouse stores the data into 60 different distribution. Each distribution is referred as a single database unit called as distribution. Distribution setting are defined at table level in a SQL Data Warehouse selecting any of the two-method available: Round robin or Hash. By default, if you do not define a distribution, the table will be distributed using the round robin distribution method. So how to find how much memory is allocated to the entire data warehouse? Let's say I have Data warehouse with DWU1500 and the resource class allocated to it is xlargerc which is 3200 MB memory Total memory allocated = 3200*60 distributions/1024 = 187.5 GB Conclusion: This blog is to help in understanding the Data warehouse workload management and how the memory allocation taken care based on the resource class assigned to a user. In order to understand the SQL Data warehouse overview and architecture I strongly recommend to go through this What is Azure SQL Data Warehouse? Gaurav Agarwal]]>         Real-time Prediction using Azure Machine Learning with Stream Analytics and Power BI https://docs.microsoft.com/archive/blogs/gagarwal/integrating-azure-machine-learning-model-stream-analytics-with-power-bi-real-time-prediction Wed, 21 Dec 2016 02:55:56 GMT https://blogs.msdn.microsoft.com/gagarwal/?p=1345 Azure Stream Analytics (ASA) and Power BI integration provide us to analyze high volume streaming data in a real-time. We know already how we can integrate an Azure Stream Analytics job output to a Power BI dataset. Now lets make this more interesting, how about doing a Real-time prediction for your streaming data and display the result in a Power BI Dashboard. This is a typical scenario when your business model need to apply Artificial Intelligence in your data and perform real time data analytics using machine learning algorithms. The overall data integration show below: Following are the steps to be followed to configure AML web service with Azure Stream analytics: (Note: Assuming that you are already aware of how to configure ASA with Power BI, and you have already deployed your Azure Machine Learning model as a Web service)
    1. Create a New Azure Stream analytics job
    2. Define the Job Topology Input as appropriate. I have used Source as Event hub with Source type as Stream
    3. Define the Job Topology Output as Sink to Power BI so that it will automatically create the dataset in Power BI portal
    4. Here is the most important step where you will define a Function for calling the Azure Machine Learning Web Service. Click on Function on left pane of the Stream Analytic job and Add New -> Function Alias -> Function Type as Azure ML -> Subscription-> URL of your Azure Web Service-> API KPI. Once its make the successful call to the Azure ML Web Service it will automatically populate the function Signature
    5. Finally define the Job Topology Query for the ASA job, you need to call the AML function in the Select clause which you have created in the query as show below:
    6. Save the query and you can now Start the job, this will create a Power BI dataset along with AML function output prediction.
    I hope this blog will help you to know how to configure and perform Real time prediction using Azure Machine Learning and integrating it to a Power BI dashboard. Gaurav Agarwal

    ]]>         Installing R package in Azure Machine Learning https://docs.microsoft.com/archive/blogs/gagarwal/installing-r-package-in-azure-machine-learning Sat, 03 Dec 2016 04:59:47 GMT https://blogs.msdn.microsoft.com/gagarwal/?p=1075 https://studio.azureml.net/) supports for R language which makes our life easier to use our own existing R code with in AML studio. There are over 400 most popular R packages pre-install in the Azure Machine Learning Studio. How to find the list of packages that are currently supported or installed, you can use the below script: Use Execute R script task in AML studio and copy paste the below script: out <- data.frame(installed.packages(,,,fields="Description")) maml.mapOutputPort("out")  If you want, you can also convert the output to a CSV file using Convert to CSV: Now the biggest issue is when we try to execute our existing R code which have R packages not pre-install or supported in AML. You will end up getting Error: could not find function "<packagename>" or package '<packagename>'is not available or there is no package called '<packagename>' You need to upload/install the R package with Azure Machine Learning Studio to execute you code. Here are the Steps How to install/upload outside R package in Azure Machine Learning studio: I will be using dummies R package in my example for the installation:
    1. Download the R package dummies or which you want to install, you can use R studio to do this if the package is not yet install in the R package library. Location will be: C:\Users\<username>\Documents\R\win-library\3.2
    2. Put dummies package into a .zip folder as shown below:
    1. Now the most important step where most of us making mistake is we need to again put the .zip folder to another .zip folder. Which means dummies.zip -> dummies (2).zip
    2. You can rename the final zip folder as per your wish let say in my case I renamed it to pack.zip
    1. Upload the final pack.zip file to the Azure Machine Learning Data Set:
    2. Under saved data set drag and drop the .zip package file
    3. Use Execute R script task in AML studio and copy paste the below script:
      install.packages("src/pack/dummies.zip", lib = ".", repos = NULL, verbose = TRUE) library(dummies, lib.loc=".", verbose=TRUE)  
    4. This will successful install the dummies R package.
    Please note if you are getting the below error while running you R script, please do zip of you zip, as mention in the step 3.

    Error 0063: The following error occurred during evaluation of R script:

    ---------- Start of error message from R ----------

    zip file 'src/dummies.zip' not found

    I hope this blog will help you to know how to check the pre-install, and how to install you own R package within Azure ML Studio and handle ErrorCode":"0063. Gaurav Agarwal  ]]>