SELECT
 SUM(CASE WHEN r.[status] = 'Running' THEN 1 ELSE 0 END) [running_queries],
 SUM(CASE WHEN r.[status] = 'Running' THEN rw.concurrency_slots_used ELSE 0 END) [running_queries_slots],
 SUM(CASE WHEN r.[status] = 'Suspended' THEN 1 ELSE 0 END) [queued_queries],
 SUM(CASE WHEN rw.[state] = 'Queued' THEN rw.concurrency_slots_used ELSE 0 END) [queued_queries_slots]
FROM
 sys.dm_pdw_exec_requests r 
 JOIN sys.dm_pdw_resource_waits rw ON rw.request_id = r.request_id
WHERE
 ( (r.[status] = 'Running' AND r.resource_class IS NOT NULL ) OR r.[status] ='Suspended' )
 AND rw.[type] = 'UserConcurrencyResourceType';

• running_queries - this is the number of queries actively executing in the data warehouse.
• running_queries_slots - this is the number of concurrency slots consumed by the running queries.
• queued_queries - this is the number of queries queued.
• queued_queries_slots - this is the number of concurrency slots requested by the queued queries.

T-SQL for monitoring queries

SQL Data Warehouse Git Hub repo: https://github.com/Microsoft/sql-data-warehouse-samples

Using the sys.dm_pdw_resource_waits view, you can craft a query to see if any query is queued waiting for system resources. Using the sample queued_queries T-SQL script, we can see what is waiting for execution.

            
SELECT
    *
    , [queued_sec] = DATEDIFF(MILLISECOND, request_time, GETDATE()) / 1000.0
FROM
    sys.dm_pdw_resource_waits
WHERE
    [state] = 'Queued'
ORDER BY
    queued_sec DESC;

The results lists all queries that are queued with the number of concurrency slots requested and the duration of the queue.

1. Create a server level login
2. Create a database level user
3. Add the user to the resource class
4. [Optional] Grant permissions

Create a Server Level LOGIN

CREATE LOGIN cloudsaxl WITH PASSWORD = ‘<use a strong password>’;

Create a Database Level USER

CREATE USER cloudsaxl FOR LOGIN cloudsaxl;

Add the User to the XLargeRC resource class

EXEC sp_addrolemember 'xlargerc', 'cloudsaxl';

[Optional] Grant permissions

EXEC sp_addrolemember 'db_owner', 'cloudsaxl';

Prerequisites

• You have successfully configured your Azure Active Directory and connected it to SQL Data Warehouse (see the Use Azure Active Directory Authentication for authentication with SQL Database or SQL Data Warehouse topic for details on configuration).

Creating a User for an individual

CREATE USER [<alais@domain>] FROM EXTERNAL PROVIDER WITH DEFAULT_SCHEMA = [<schema>];

CREATE USER [user@company.com] FROM EXTERNAL PROVIDER WITH DEFAULT_SCHEMA = [sales];

Create a User for a security group

CREATE USER [<Security Group Display Name>] FROM EXTERNAL PROVIDER WITH DEFAULT_SCHEMA = [<schema>];

CREATE USER [Sales Team] FROM EXTERNAL PROVIDER WITH DEFAULT_SCHEMA = [sales];

CREATE USER [salesteam@company.com] FROM EXTERNAL PROVIDER WITH DEFAULT_SCHEMA = [sales];

Msg 33130, Level 16, State 1, Line 4
Principal 'salesteam@company.com' could not be found or this principal type is not supported.

Getting Started (the portal way)

Getting Started (the T-SQL way)

ALTER DATABASE [DemoDW] SET ENCRYPTION ON;

Verify Encrypted Status

SELECT name, is_encrypted FROM sys.databases;

Next Steps

Database Design

1,Cables & Power,
2,Camera & Photo,
3,Cell Phones, That latest mobile devices.
4,Computers,
5,GPS & Navigation,
6,Headphones,
7,Home Audio,
8,Office Electronics,
9,Security & Survelliance,
10,Software,
11,Tablets, All of your tablet needs.
12,Televisions, The latest in LCD/LED and 4K.
13,Wearable Technology,

CREATE TABLE DimProducts
(
 [ProductID] INT NOT NULL,
 [Name] VARCHAR(50) NOT NULL,
 [Description] NVARCHAR(256) NULL
)
WITH
(
 HEAP,
 DISTRIBUTION=ROUND_ROBIN
);

Code Sample

static void Load(string path)
{
    // Load the file
    var reader = new StreamReader(path);

    // Get the Data Table to hold the rows
    var datatable = GetDataTable();

    // Setup our SQL Connection to our SQL Data Warehouse
    var connection = GetConnection();

    // Iterate through the file
    string row;
    while ((row = reader.ReadLine()) != null)
    {
        // Split the row by comma
        var values = row.Split(',');

        // Add the row values to the Data Table
        datatable.Rows.Add(values[0], values[1], values[2]);
    };

    // Open the connection to SQL Data Warehouse
    connection.Open();

    // Create a Bulk Copy class
    var bulkCopy = new SqlBulkCopy(connection);

    // Define the target table
    bulkCopy.DestinationTableName = "dbo.DimProducts";

    // Write the rows to the table
    bulkCopy.WriteToServer(datatable);

    ...
 }

SELECT * FROM DimProducts;

Download

Now why do you care? When using CCI with SQL Data Warehouse, you’ll often be presented with this problem: I loaded n MB/GB/TB of data into the service, ran DBCC PDW_SHOWSPACEUSED, summed the DATA_SPACE column and see that the space used is dramatically smaller than the amount of data you put in.

An Example

• Scenario #1: You have a 750 MB csv file (represents 1 hour of data) that you generated from your business and have loaded the data via PolyBase into SQL Data Warehouse. You check the database via DBCC PDW_SHOWSPACEUSED and see that you have about 1.4 GB of data.
• Scenario #2: You use that same 750 MB csv file with the other 23 from that day (to represent a full day of data about 14GB of data), loaded in SQL Data Warehouse, and the results of DBCC PDW_SHOWSPACEUSED is 850MB.

So what magic is happening to make the data on disk so much smaller? Our friend, CCI!

A Simple Explanation

In scenario #1, there are not enough rows in the data to meet the criteria for compression. SQL Server will optimize and move the data into Clustered Index (CI) format which does not enable compression. The difference in size is due to the table storage, CI storage, and the possibility that all of the data pages are not fully packed.

In scenario #2, there are enough rows in the data to meet the criteria for compression. In this case the data is very compressible and we achieve around 14x compression.

Best Practice

• CCI works best with data sets above 1M rows. This gives optimal row group sizes for compression and query performance.

Monitoring

To monitor CCI data, you can inspect the metadata for the table to determine the open rowgroups and rows (min, max, and average) as an indicator if CCI is the correct indexing strategy. We have a great tutorial on Managing Columnstore Indexes that walks you through how to determine rowgroups, rowgroup rows/counts, and a couple of strategies for tuning your system to get the best possible performance.

Next Steps

Visit the SQL Data Warehouse Overview to learn more about Microsoft’s scale out relational data warehouse.

SQL Data Warehouse is a multi-node scale out relational SQL Server data warehouse. PolyBase has the capability of using all of the nodes in the SQL Data Warehouse cluster to import or export data in parallel. This is great for performance reasons (multiple nodes working in parallel can produce very large ingestion rates) but can cause some confusion if you're not aware of this design.

Let's take a look at an example.

Exporting Data Default

When exporting data, you'll use a T-SQL command to create an external table from a SELECT statement. We call this a CETAS (Create External Table As Select). The processing of moving data from one table to another via CETAS will invoke PolyBase to create an external storage mechanism for the data to land into. In this example, a set of files on Azure Blob Storage.

CREATE EXTERNAL TABLE dbo.Orders
WITH
(
    LOCATION = '/Orders/',
    DATA_SOURCE = AzureStorage,
    FILE_FORMAT = TextFile
)
AS
SELECT
*
FROM
dbo.OrderHistory;

This will produce 60 text files in Azure Blob Storage. This by design as PolyBase as the capability of reading multiple files in a directory or directory tree as the backing data for an External Table (a handy feature we’ll blog about in the future).

Exporting a Single File

But wait, what if I want to export all of my data to a single file? There is a handy trick that you can use – the TOP clause. Using the TOP clause forces the data to the control node in the service and the output will be sent to a single file:

CREATE EXTERNAL TABLE dbo.Orders
WITH
(
    LOCATION = '/Orders/',
    DATA_SOURCE = AzureStorage,
    FILE_FORMAT = TextFile
)
AS
SELECT TOP [N]
*
FROM
dbo.OrderHistory;

Note: You’ll need to replace the [N] value with the row count of the table (if you’re doing a full export).

After executing the CETAS with the TOP clause, you’ll be left with a single file in Azure Blob Storage.

Next Steps

Visit the SQL Data Warehouse Overview to learn more about Microsoft’s scale out relational data warehouse.