SQL BI https://docs.microsoft.com/archive/blogs/karang/feed.xml My Experience - Troubleshooting SQL BI Stack Sun, 18 Nov 2018 10:56:49 GMT en-US hourly 1 Azure Data Factory V2– Pipeline Functional Testing https://docs.microsoft.com/archive/blogs/karang/azure-data-factory-v2-pipeline-functional-testing Sun, 18 Nov 2018 09:59:40 GMT https://blogs.msdn.microsoft.com/karang/?p=1515 Note:- This article applies to Azure Data Factory V2. This article assumes that you have gone through Introduction to Azure Data Factory and the QuickStart Tutorial. Credits - Thanks to ADFv2 Product Group (Anudeep Sharma, Hermine Vermishyan & Abhishek Narain ) for reviewing and all assistance with my ADFv2 and my partner in awesomeness David Kuo for testing the script Overview During development phase of an Azure Data Factory pipeline you have a debugging functionality in UX which you can use to debug a pipeline and find out if anything is failing due to reasons like incorrect connection string, object not present in database, wrong query, etc. Keeping same idea in mind I have extended the functionality of debugging from pipelines' development stage to release stage, where you can use PowerShell script to perform pipelines' functional testing in Test or Integration or UAT environment ensuring all issues are caught at that stage instead of hitting in Production environment and in this article series will share the process of achieving same with PowerShell script and good news is that you can use same script in your environment without making any code change. Concept Function test with ADF can be used to validate if pipeline is working and if anything related to functionality is failing then stop release, fix issue and iterate. It's more aligned with build verification ensuring objects like linked service, tables, storage, etc. which are required for a pipeline are successfully validated before releasing to production environment. Scope Testing functionality of pipelines ensuring configurations like databases, server, storage, etc. are configured correctly and pipeline is not failing due to misconfiguration like database not exist or something on same lines. Errors due to data quality, performance or anything related to data sanctity is outside the scope of functional testing. How it Works Let's go in details of understanding working of functional testing with ADFv2. First of all you need to understand how debugging works in pipeline and good news is that there is very informative video done by Gaurav Malhotra from ADF team (highly recommended if you want to get insight of pipeline debugging): Iterative development and debugging using Data Factory As you have seen in this video that for debugging you need to run a pipeline and for running a pipeline it requires parameters defined for your pipeline and run the pipeline in debug mode with an output showing whether pipeline succeeded or failed, keeping same approach in mind I attempted to write this functionality with following flow:- Functional Testing Flow: - clip_image002 Now let's look into each step in this flow:- 1. Azure Active Directory (AAD) Authentication - AAD is required for silent authentication of PowerShell script which is used during automation of testing and PowerShell script needs to access Azure Data Factory for getting list of pipelines so we need to ensure Application ID should also have access to Azure Data Factory. There are few handy articles which you can follow for creating Application ID mentioned as follow:- a. How to create an AAD application and service principal b. How to give access to Application ID i. Go to Azure Data Factory in Azure Portal ii. Under All settings select Users and click Add as shown in following figure: - Settings: - clip_image004 iii. After hitting Add it will open another windows which will allow you to add Application ID which you had crated in Step1, as there is need of only getting list of pipelines so we are making Application member of Reader Role as shown in following figure where Application name is FunctionalTesting and added member of Reader Role:- clip_image006 iv. In PowerShell we will be leveraging Application ID for silent authentication and its recommended to use Key Vault for pulling secrets in this case we need a Application ID secret (password), in PowerShell I have parameterized secret field so you can either pass it as plain text or make a call to Key Vault during release process and pass along the secret. Code snippet as follow in which we are doing silent authentication using Application ID which is also knowns as Service Principal, you can refer this article for getting TenantID clip_image008 2. Functional Testing Logic - a. Get list of Pipeline from ADF clip_image010 b. Check for Pipeline Exclusion clip_image009 Example of exclusion file:- clip_image012 c. Pipeline Test file - Following is an example of test file, in my case I have a pipeline named as pl-testValidation in data factory and that pipeline expects a parameter known as SubFolder, I need to ensure then I am creating a parameter file with respect to my pipeline which will be used a functional test file by testing logic, also need to ensure that test file name matches with the pipeline name and should have all parameters stated in a pipeline, and if this file is missing then it will be marked as failed with missing test file Example of test file as the pipeline name is pl-testValidation so I have created test file as pl-testValidation.json (PipelineName.json) with a parameter consumed by pipeline and TestTimeout value as I want to run this test for 80 seconds, if I will miss this value then it will default to 30 seconds. clip_image014 d. In following code if you notice we are iterating through pipelines and checking if PipelineName.json exist and also checking what is the TestTimeout value if missing then defaulting it to 30 seconds. clip_image016 e. Running Pipeline test by using following command clip_image018 f. Using value of $runId to get current status (refer GIT - PS script for full code) clip_image020 g. Using value of $runId to get status (refer GIT - PS script for full code) clip_image021 h. Fail test if it exceeds timeout setting clip_image023 i.If you are integrated with Release Manager then it will generate exception in-case of failure resulting in failed release and give an output of failed pipelines which needs to be fixed before deploying to production environment but don’t be dis-heartened you can still code even though you are not using Release Manager by using following command. .\AzureDatafactoryFunctionalTesting.ps1 -dataFactoryName "Your Factory Name" -resourceGroupName "Factory Resource Group Name" -azureSubscription "Subscription ID" -applicationSecret "Application Secret from Point 1" -applicationID "Application ID from Point 1" -tenantID "Tenant ID from Point 1" -excludedPipelinesFilePath ".\" -functionalTestJsonFilePath ".\" Conclusion As you have seen in this article I have gone through step by step of achieving functional testing with ADFv2 and in subsequent article I will be discussing more about how to integrate same with Release Manager so that you can have this testing framework as a part of Pipeline Release process to Production. Code Location Git – feel free to download and contribute  ]]> ADF - Azure Data Factory Learning Path https://docs.microsoft.com/archive/blogs/karang/adf-azure-data-factory-learning-path Wed, 10 May 2017 23:08:38 GMT https://blogs.msdn.microsoft.com/karang/?p=1235 If you are looking to learn Azure Data Factory (ADF) it’s a good path to start, as it covers various components of ADF like all kind of supported flows with code samples.

I would suggest to start with Portal as you don’t require any coding experience on same and if you are coming from DevOps or DBA background then you can follow PowerShell path, further if you are coming from pure development background then you can use APIs provided by ADF through Nugget.

Here is link of learning path:-
https://azure.microsoft.com/en-us/documentation/learning-paths/data-factory/

Following link shows how Microsoft Studios (XBOX) team is using Azure Data Factory:-
https://channel9.msdn.com/Blogs/raw-tech/Halo-Minecraft-and-More-How-Microsoft-Studios-Processes-Gaming-Event-Data

Happy Learning

]]> SQLDW: Could not allocate a new page for database https://docs.microsoft.com/archive/blogs/karang/sqldw-could-not-allocate-a-new-page-for-database Thu, 16 Feb 2017 07:15:56 GMT https://blogs.msdn.microsoft.com/karang/?p=1225 Issue:

Recently I faced an issue where Stored Procedure running on SQLDW failing with error as follow

An error occurred in Stored Procedure Activity execution. Diagnostic details: Database operation failed on server ‘xxxyyyy123.centralus1-a.worker.database.windows.net,11087' with SQL Error Number '1101'. Error message from database execution : Could not allocate a new page for database 'Distribution_11' because of insufficient disk space in filegroup 'PRIMARY'. Create the necessary space by dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.

 

Resolution:

After working with SQLDW Support team found that issue can occur when you hit max size of the database which is by default 10TB(if not stated while creating a database) and it can be resolved by increasing the size of database using following command:-

ALTER DATABASE [SQLDWDB] MODIFY ( MAXSIZE=245760 GB );

In this case I set it to 240 TB but you can size it based on your requirement.

Note:

Before you run these examples, make sure the database you are altering is not the current database. The current database must be a different database than the one you are altering, therefore ALTER must be run while connected to the master database.

 

Credits:

P V Suhas from SQLDW Support Team

]]> HDInsight: How to find Long Running Queries https://docs.microsoft.com/archive/blogs/karang/hdinsight-how-to-find-a-long-running-jobs-2-2-2 Wed, 25 Jan 2017 09:40:00 GMT https://blogs.msdn.microsoft.com/karang/?p=1215 Coming from SQL background where you have dependency on SQL Profiler Trace for understanding what’s going on SQL Server like who is running non optimized long query, where its scanning through all partitions and yes who is using non indexed columns for filter right? So coming to HDInsight world wondering do we have something like SQL Profiler for tracking bad jobs – Yes we do, as a part of Apache Ambari Project – (as per wiki) Whole objective of Ambari is to make Hadoop management easy by providing a mechanism of managing or monitoring Hadoop  clusters with it's easy to use  UI, in this Post I will show how to use Ambari for getting long running or resource intensive jobs.

Objective: How to find long running or resource intensive Hive query?
Steps:-

clip_image001[4]

 

  • Click Yarn > Quick Links > Active Head Node >>Resource Manager UI

clip_image002[5]

Or if you don’t want to go through the whole hassle just type https://<HDInsightClusterName>.azurehdinsight.net/yarnui/hn/cluster, in popup provide security information admin and its password - and it would open Resource Manager - it provides you all information about jobs (running/pending/finished/etc.) on HDI cluster

  • In following screenshot - you can notice multiple options for checking Jobs like in this case I have this specific job which is consuming:-
    • A - Memory - 2.2 TB
    • B - % of Queue - 64%
    • C - % of Cluster - 60%

image

  • In this case filtered job may be problem making query which is causing other jobs to wait in queue as its consuming 60% of resource and 2.2 TB of total 3.75TB allocated for this cluster. I can further look into job by clicking into job id (Job ID > Logs and Pull query from there dag) and check how's query look like and what's its doing under the hood - is it scanning TBs of partitions or filtering on non-partitioned fields or what.

clip_image004

 

  • Like SQL You can Kill this job from Resource Manager it self – Click on Job and on top you will find a button for killing same query, like this job is running for last 7 hours, you can Kill it by clicking Kill Application

image

 

Hope this will come handy.

Feel free to add comments, may be in next post I can take a deep dive into each log and action.

Happy Learning.

(Published on my  blogs https://karanspeaks.com/ & https://blogs.msdn.microsoft.com/)

]]> SSAS Monitoring Tool–SQL 2014 ASTrace https://docs.microsoft.com/archive/blogs/karang/ssas-monitoring-tool-sql-2014-astrace Thu, 01 Dec 2016 04:27:40 GMT https://blogs.msdn.microsoft.com/karang/?p=1005  

Refer SSAS Monitoring Tool– ASTrace to learn more about ASTrace, this post was pending for a long time as I got tons of email and comment around SQL 2014 and 2016 ASTrace, whith this blog like to mention that I have shared a source code on git so anyone can contribute to same.

 

Code for ASTrace 2014 is tested with latest build of SQL Server 2014 but for 2016 I had tested with SQL 2016 Preview and didn’t get chance to test with RTM so in coming few days will test and update the code as well if required, but feel free to update if you like as its Git so anyone can contribute.

 

Here is Git Link - https://github.com/karanspeaks/SQL-Server-Analysis-Services-Samples

]]> Halo, Minecraft, and More! How Microsoft Studios Processes Gaming Event Data https://docs.microsoft.com/archive/blogs/karang/halo-minecraft-and-more-how-microsoft-studios-processes-gaming-event-data Wed, 17 Feb 2016 23:36:36 GMT https://blogs.msdn.microsoft.com/karang/2016/02/17/halo-minecraft-and-more-how-microsoft-studios-processes-gaming-event-data/ My first video on Channel 9, it talks about how we Handle various Events generated from Games………..

https://channel9.msdn.com/Blogs/raw-tech/Halo-Minecraft-and-More-How-Microsoft-Studios-Processes-Gaming-Event-Data

]]> Azure Data Factory: Detecting and Re-Running failed ADF Slices https://docs.microsoft.com/archive/blogs/karang/azure-data-factory-detecting-and-re-running-failed-adf-slices Thu, 12 Nov 2015 23:00:03 GMT https://blogs.msdn.microsoft.com/karang/2015/11/12/azure-data-factory-detecting-and-re-running-failed-adf-slices/  

Recently I came across  a scenario where I need to detect failed slices of all Datasets in Azure Data Factory, in my case I need to detect for last 3 months and the number of slices was around 600+, they failed due to validation error as the source data wasn’t present and after a number of re-try slices were marked as failed.

In such cases its difficult to perform re-run from the  Portal as you need to right click on each slice and run it explicitly.

Solution: I wrote a following PowerShell Script, this script will detect all failed slices in a given Azure Data Factory and re-run same with your consent.

You can use same script not only for failed slices but for any status, you just need to change the Dataset status in filtering of slices, shown in following script.

I am also planning to write a solution which will run as a service in a worker role and automatically detect failed slices in a given time and re-run same.

Question can be asked, that in ADF you already have re-run logic they why you need to go through the hassles of writing and running script.

Yes we do have but after x number of re-runs a slices is marked as failed and only way is to run is through portal or programmatically.

So, here is my contribution to ADF Community.

Pre-requisite – Azure Resource Manager PowerShell (https://azure.microsoft.com/en-us/blog/azps-1-0-pre/)

Copy following code in text file and save it as file.ps1

You can also download the script and save it as PS1 – Click Here

#Begin Script

Login-AzureRmAccount

$slices= @()
$tableName=@()
$failedSlices= @()
$failedSlicesCount= @()
$tableNames=@()

$Subscription="Provide Subscription ID" 
 
  Select-AzureRMSubscription -SubscriptionId  $Subscription   
$DataFactoryName="Provide Data Factory Name"
$resourceGroupName ="Porvide Resource Group Name for Data Factory"
 
$startDateTime ="2015-05-01" #Start Date for Slices
$endDateTime="2015-08-01" # End Date for Slices


#Get Dataset names in Data Factory - you can explicitly give a table name using $tableName variable if you like to run only for an individual tablename
$tableNames = Get-AzureRMDataFactoryDataset -DataFactoryName $DataFactoryName -ResourceGroupName $resourceGroupName | ForEach {$_.DatasetName}

$tableNames #lists tablenames

foreach ($tableName in $tableNames)
{
    $slices += Get-AzureRMDataFactorySlice -DataFactoryName $DataFactoryName -DatasetName $tableName -StartDateTime $startDateTime -EndDateTime $endDateTime -ResourceGroupName $resourceGroupName -ErrorAction Stop
}


$failedSlices = $slices | Where {$_.Status -eq 'Failed'}

$failedSlicesCount = @($failedSlices).Count

if ( $failedSlicesCount -gt 0 )
{

    write-host "Total number of slices Failed:$failedSlicesCount"
    $Prompt = Read-host "Do you want to Rerun these failed slices? (Y | N)"
    if ( $Prompt -eq "Y" -Or $Prompt -eq "y" )
    {

        foreach ($failed in $failedSlices)
        {
               write-host "Rerunning slice of Dataset "$($failed.DatasetName)" with StartDateTime "$($failed.Start)" and EndDateTime "$($failed.End)""
            Set-AzureRMDataFactorySliceStatus -UpdateType UpstreamInPipeline -Status Waiting -DataFactoryName $($failed.DataFactoryName) -DatasetName $($failed.DatasetName) -ResourceGroupName $resourceGroupName -StartDateTime "$($failed.Start)" -EndDateTime "$($failed.End)"
            $failed.DatasetName

        }
    }
           
}
else
{
    write-host "There are no Failed slices in the given time period."
}

#End Script
]]> Microsoft Azure Essentials: Azure Machine Learning https://docs.microsoft.com/archive/blogs/karang/microsoft-azure-essentials-azure-machine-learning Thu, 16 Apr 2015 15:41:19 GMT https://blogs.msdn.microsoft.com/karang/2015/04/16/microsoft-azure-essentials-azure-machine-learning/  

Microsoft Azure Essential: Azure ML Free book is pretty informative if you wanted to learn about ML.

Link as follow:

Download the PDF (5.84 MB)

Additionally I recommend to go through free courser from University of Washington available through coursera.

Step by Step course of going in depth of Machine Learning.

Link as follow:

https://www.coursera.org/course/machlearning

]]> Informative HDInsight Links https://docs.microsoft.com/archive/blogs/karang/informative-hdinsight-links Tue, 20 Jan 2015 16:48:13 GMT https://blogs.msdn.microsoft.com/karang/2015/01/20/informative-hdinsight-links/

Tried to collate informative articles related to HDInsight.

In comments section please add URLs of article / video which  you came across and like me to add it to this list.

Links as follow:-

Here are the Hadoop technologies in HDInsight:

  • Ambari: Cluster provisioning, management, and monitoring
  • Avro (Microsoft .NET Library for Avro): Data serialization for the Microsoft .NET environment
  • HBase: Non-relational database for very large tables
  • HDFS: Hadoop Distributed File System
  • Hive: SQL-like querying
  • Mahout: Machine learning
  • MapReduce and YARN: Distributed processing and resource management
  • Oozie: Workflow management
  • Pig: Simpler scripting for MapReduce transformations
  • Sqoop: Data import and export
  • Storm: Real-time processing of fast, large data streams
  • Zookeeper: Coordinates processes in distributed systems

Other popular Videos

Books

Introducing Microsoft Azure HDInsight
In Introducing Microsoft Azure HDInsight, we cover what big data really means, how you can use it to your advantage in your company or organization, and one of the services you can use to do that quickly—specifically, Microsoft’s HDInsight service. We start with an overview of big data and Hadoop, but we don’t emphasize only concepts in this book—we want you to jump in and get your hands dirty working with HDInsight in a practical way.

Download the PDF (6.37 MB)

]]> Harnessing the Power of both worlds https://docs.microsoft.com/archive/blogs/karang/harnessing-the-power-of-both-worlds Sun, 16 Nov 2014 19:01:50 GMT https://blogs.msdn.microsoft.com/karang/2014/11/16/harnessing-the-power-of-both-worlds/  

Harness the Power of Both Worlds –SQL Engine and Analysis Services

Introduction

The combination of the SQL Engine and Analysis Services is awesome because it lets you harness the power of both worlds: columnstore Indexes in the SQL engine and real-time or relational OLAP (ROLAP) in Analysis Services.

Recently I have done some testing around a much talked about feature of the SQL engine – columnstore indexes. As you can learn in BOL, in SQL Server columnstore indexes store and manage data by using column-based data storage and column-based query processing in-memory. Columnstore indexes work well for data warehousing workloads that primarily perform bulk loads and read-only queries. By using the columnstore index you can achieve up to 10x query performance gains over traditional row-oriented storage, and up to 7x data compression over the uncompressed data size.

A columnstore index is a technology for storing, retrieving and managing data by using a columnar data format, called a columnstore. SQL Server supports both clustered and nonclustered columnstore indexes. Both use the same in-memory columnstore technology, but they do have differences in purpose and in features they support.

Benefits

Columnstore indexes work well for mostly read-only queries that perform analysis on large data sets. Often, these are queries for data warehousing workloads. Columnstore indexes give high performance gains for queries that use full table scans, and are not well-suited for queries that seek into the data, or that search for a particular value.

Columnstore indexes have these benefits:

· Columns often have similar data, which results in high compression rates.

· High compression rates improve query performance by using a smaller in-memory footprint. In turn, query performance can improve because SQL Server can perform more query and data operations in-memory.

· A new query execution mechanism called batch-mode execution has been added to SQL Server that reduces CPU usage by a large amount. Batch-mode execution is closely integrated with, and optimized around, the columnstore storage format. Batch-mode execution is sometimes known as vector-based or vectorized execution.

· Queries often select only a few columns from a table, which reduces total I/O from the physical media.

Solution with ROLAP and Columnstore Index

So how can you leverage this new shining star of SQL Server for business intelligence solutions?

I began by combining clustered columnstore indexes with Analysis Services ROLAP feature, enabling my fact tables for ROLAP, and then tested the performance of aggregations. As expected, I got pretty awesome performance with 500 million records.

So I doubled the number of records to 1 billion! At this scale, basic aggregates such as SUM and COUNT were returned in 2 to 3 seconds. The acid test of course lies in a certain trouble -making measure – you guessed it, Distinct Count.

So I did testing with Distinct Counts on the ROLAP solution and results were pretty positive, with around 1.5 billion records (1555618627), and around 15,638,865 distinct values. On a cold cache I got results within 2 seconds and on a warm cache in 16 milliseconds.

This was pretty awesome so I decided to try the combination of warm-caching with ROLAP in order to meet my SLA for the solution dashboards, which was around 4 seconds.

Recommendations

There are some important settings that you need to consider when implementing a BI solution based on in-memory columnstore indexes and Analysis Services ROLAP:

· EnableRolapDistinctCountOnDataSource – This is a very important property when we are using Analysis Services ROLAP. Set this property to 1 when using ROLAP with distinct count calculations. The property enables calculation of distinct count at the source (the relational database) instead of fetching rows in Analysis Services and calculating distinct count using Analysis Services.

· Statistics – In the absence of SQL Statistics Query Optimizer you will not be able to generate the right execution plan of a given query. Therefore it is very important to do timely updates of the statistics for the columnstore table; otherwise DML operations can degrade performance.

Hardware

For this experiment I used a Virtual Machine with 16 logical processors and 120 GB Ram.

References

Columnstore Indexes: https://msdn.microsoft.com/en-us/library/gg492088.aspx

]]>