SQL Server Customer Advisory Team

SQLCAT at PASS Summit 2018

Dimitri Furman — Mon, 05 Nov 2018 16:23:47 GMT

This week in Seattle, PASS Summit 2018 will be happening for the twentieth time. As for many years before, the SQLCAT (a.k.a. DataCAT) team will be at the conference. This year, we will present three sessions: On Thursday Nov 8 at 4:45 PM in room 6B, Alexei Khalyako, John Hoang, and Mike Weiner will talk about Learnings and Best Practices in Building an Enterprise Data Warehouse Using Azure SQL DW. Our team has had the opportunity to work with a number of customers moving their data warehouse workloads to Azure SQL DW. In this session we’ll share learnings around the most common design patterns with Azure SQL DW and go deeper on data loading and performance tuning and optimization considerations with Azure SQL DW (Gen2). Come to this session to learn from our real-world experiences in working with customers on adopting Azure SQL DW. On Friday Nov 9 at 8:00 AM in room 615, Arvind Shyamsundar will show how to tackle Advanced SQL Server Troubleshooting with SQLCallStackResolver. This Level 400 session is a product of many real-world customer engagements that our team has worked on. Arvind will show you some tools and techniques to tackle the hardest-to-investigate performance issues in SQL Server and he will demo one such scenario involving spinlock contention on high-end servers. Most importantly, the information in the session is very practical and directly usable. Guaranteed to help you be much more efficient and precise when troubleshooting SQL Server performance! On Friday Nov 9 at 9:30 AM in the TCC room Tahoma 5, Kun Cheng, Dimitri Furman, and Mike Weiner will talk about SQL DB Managed Instance - Best Practices and Lessons Learned. For well over a year, since the early private preview in summer of 2017 and through General Availability in fall of 2018, we have been collaborating with Microsoft customers who are the early adopters of Managed Instance, and with the Managed Instance engineering team, to ensure successful implementations and to improve the offering. If you want to learn how Managed Instance works and how to use it optimally, from performance tips to migration best practices, this session is for you. In addition to presenting, we will also be at the Data Clinic, where together with our colleagues from the SQL Server engineering team, the Tiger team, the CSS team, and many others we will help you solve real life problems and answer hard (or easy 😊) questions about SQL Server and other data technologies we work with. See you at the Summit!

Real-time performance monitoring for Azure SQL Database Managed Instance

Denzil Ribeiro — Wed, 26 Sep 2018 16:38:40 GMT

This article was co-authored by Dimitri Furman and Denzil Ribeiro Reviewed by: Danimir Ljepava, Borko Novakovic, Jovan Popovic, Rajesh Setlem, Mike Weiner

Introduction

In our ongoing engagements with the SQL DB Managed Instance preview customers a common requirement has been to monitor database workload performance in real time. This includes not just monitoring of a single (or a few) performance metrics, but being able to look at many relevant metrics at once, within seconds of events occurring. While much of the needed performance data is available in various DMVs, consuming it via T-SQL queries and looking at the results in SSMS query windows is quite cumbersome, and does not provide a simple way to look at real time data, or at recent historical data and trends. More than a year ago, we had similar monitoring requirements for SQL Server on Linux, and had blogged about it here: How the SQLCAT Customer Lab is Monitoring SQL on Linux. For monitoring Managed Instance, we used Telegraf. Telegraf is a general-purpose agent for collecting metrics, and is broadly adopted in the community. It can be used for monitoring SQL Server on both Windows and Linux using the SQL Server plugin. In Telegraf version 1.8.0, this plugin has been updated to support SQL DB Managed Instance as well. Telegraf connects to Managed Instance remotely, and collects performance data from DMVs. InfluxDB is the time series database for the monitoring data collected by Telegraf, while new Grafana dashboards, specific to Managed Instance, were developed to visualize this data. Big thanks to Daniel Nelson and Mark Wilkinson for accepting pull requests with the changes to the Telegraf SQL Server plugin needed to support Managed Instance. This blog will describe a solution for real time performance monitoring of Managed Instance in sufficient detail for customers to use it for the same purpose. The code we developed is open source and is hosted in the sqlmimonitoring GitHub repo. We welcome community contributions to improve and expand this solution. Contributions to the Telegraf collector should be made via pull requests to the SQL Server Input Plugin for Telegraf. For changes to Grafana dashboards, pull requests should be made to the sqlmimonitoring GitHub repo. The detailed discussion below is intended specifically for real time performance monitoring and troubleshooting. For other monitoring needs, such as query-level performance monitoring, historical trend investigations, and resource consumption monitoring, consider using Azure SQL Analytics. In near future, Azure Monitor will provide support for monitoring of Managed Instance metrics and sending notifications.

Dashboard examples

Before describing the solution in detail, here are a few examples of Managed Instance Grafana dashboards, captured while a customer workload is running. Figure 1. Performance dashboard. Additional panels not shown: Memory Manager, Memory Clerks, PLE, Buffer Manager, Access Methods, SQL Statistics. Figure 2. Database IO Stats dashboard. Additional panels not shown: Data File Size, Log File Size, Log File Used. Figure 3. HADR dashboard.

Setup and configuration

Here is the conceptual diagram of the solution. The arrows show the flow of monitoring data from source to destination. Telegraf pulls data from Managed Instances and loads it into InfluxDB. Grafana queries InfluxDB to visualize the data on web dashboards.

Prerequisites

Basic knowledge of Linux.
Linux VM(s) with internet connectivity, which is needed to download Docker images during initial configuration. In a typical configuration shown on the diagram, a single VM runs all three components (Telegraf, InfluxDB, and Grafana).
Network connectivity between the Telegraf VM and monitored Managed Instances. Traffic from the Telegraf VM to the Managed Instances on TCP port 1433 must be allowed. The Telegraf VM can be in the same VNet where monitored instances are, or in a peered VNet, or on premises if the on-prem network is connected to the VNet over ExpressRoute or VPN. For details on supported Managed Instance connectivity scenarios, see this documentation article.
Inbound traffic to the VM(s) on the following ports must be allowed. If deploying in Azure, modify NSGs accordingly.
- TCP port 8086 – this is the default InfluxDB port. This is only required on the InfluxDB VM, and only if lnfluxDB is running on a machine different from the Grafana machine.
- TCP port 3000 – this is the Grafana default web port to access the web-based dashboards. This is only required on the Grafana VM.

In our test environment, we use a single VM in the same Azure VNet as the monitored instances. If monitoring just a couple of instances, the VM can be quite small and inexpensive. A single F1 VM (1 vCore and 2 GB of memory) with one P10 (128 GB) data disk for InfluxDB was sufficient for monitoring two instances. However, what we describe is a central monitoring solution, and as such it can be used to monitor many Managed Instance and SQL Server instances. For large deployments, you will likely need to use larger VMs, and/or use a separate VM for InfluxDB. If dashboard refreshes start taking too long, then more resources for InfluxDB are needed.

Step 1

Install Git and Docker, if they are not already installed, otherwise skip this step. Ubuntu: [code language="text"] sudo apt-get install git -y wget -qO- https://get.docker.com/ | sudo sh [/code] RHEL/CentOS: [code language="text"] sudo yum install git -y sudo yum install docker -y sudo systemctl enable docker sudo systemctl start docker [/code] Note: On RHEL, SELinux is enabled by default. For the InfluxDB and Grafana containers to start successfully, we either have to disable SELinux, or add the right SELinux policy. With the default RHEL configuration, SELINUX configuration is set to enforcing. Changing this configuration to permissive or disabled will work around this issue. Alternatively set the right SELinux policy as described here. See this documentation article for more information. [code language="text"] sudo vi /etc/selinux/config [/code]

Step 2

Clone the GitHub repo. [code language="text"] cd $HOME sudo git clone https://github.com/denzilribeiro/sqlmimonitoring.git/ [/code]

Step 3

Install, configure, and start InfluxDB. a. When creating the Azure VM, we added a separate data disk for InfluxDB. Now, mount this disk as /influxdb, following the steps in this documentation article (replace datadrive with influxdb). b. Optionally, edit the runinfluxdb.sh file and modify the INFLUXDB_HOST_DIRECTORY variable to point to the directory where you want the InfluxDB volume to be mounted, if it is something other than the default of /influxdb. [code language="text"] cd $HOME/sqlmimonitoring/influxdb sudo vi runinfluxdb.sh [/code] If you are new to vi (a text editor), use the cheat sheet. c. Pull the InfluxDB Docker image and start InfluxDB. [code language="text"] cd $HOME/sqlmimonitoring/influxdb sudo ./runinfluxdb.sh [/code]

Step 4

Pull the Grafana Docker image and start Grafana. [code language="text"] cd $HOME/sqlmimonitoring/grafana sudo ./rungrafana.sh [/code]

Step 5

Optionally, if the firewall is enabled on the VM, create an exception for TCP port 3000 to allow web connections to Grafana. Ubuntu: [code language="text"] sudo ufw allow 3000/tcp sudo ufw reload [/code] RHEL/CentOS: [code language="text"] sudo firewall-cmd --zone=public --add-port=3000/tcp -permanent sudo firewall-cmd --reload [/code]

Step 6

Install Telegraf version 1.8.0 or later. Support for MI was first introduced in this version. Ubuntu: [code language="text"] cd $HOME wget https://dl.influxdata.com/telegraf/releases/telegraf_1.8.0-1_amd64.deb sudo dpkg -i telegraf_1.8.0-1_amd64.deb [/code] RHEL/CentOS: [code language="text"] cd $HOME wget https://dl.influxdata.com/telegraf/releases/telegraf-1.8.0-1.x86_64.rpm sudo yum localinstall telegraf-1.8.0-1.x86_64.rpm -y [/code]

Step 7

Create a login for Telegraf on each instance you want to monitor, and grant permissions. Here we create a login named telegraf, which is referenced in the Telegraf config file later. [code language="sql"] USE master; CREATE LOGIN telegraf WITH PASSWORD = N'MyComplexPassword1!', CHECK_POLICY = ON; GRANT VIEW SERVER STATE TO telegraf; GRANT VIEW ANY DEFINITION TO telegraf; [/code]

Step 8

Configure and start Telegraf. a. The Telegraf configuration file (telegraf.conf) is included in sqlmimonitoring/telegraf, and includes the configuration settings used by the solution. Edit the file to specify your monitored instances in the [[inputs.sqlserver]] section, and then copy it to /etc/telegraf/telegraf.conf, overwriting the default configuration file. [code language="text"] sudo mv /etc/telegraf/telegraf.conf /etc/telegraf/telegraf_original.conf sudo cp $HOME/sqlmimonitoring/telegraf/telegraf.conf /etc/telegraf/telegraf.conf sudo chown root:root /etc/telegraf/telegraf.conf sudo chmod 644 /etc/telegraf/telegraf.conf [/code] b. Alternatively, edit the /etc/telegraf/telegraf.conf file in place. This is preferred if you would like to keep all other Telegraf settings present in the file as commented lines. [code language="text"] sudo vi /etc/telegraf/telegraf.conf [/code] Uncomment every line shown below, and add a connection string for every instance/server you would like to monitor in the [[inputs.sqlserver]] section. When editing telegraf.conf in the vi editor, search for [[inputs.sqlserver]] by typing “/” followed by “inputs.sqlserver”. Ensure the URL and database name for InfluxDB in the outputs.influxdb section are correct. [code language="text"] [[inputs.sqlserver]] servers = ["Server=server1.xxx.database.windows.net;User Id=telegraf;Password=MyComplexPassword1!;app name=telegraf;" ,"Server=server2.xxx.database.windows.net;User Id=telegraf;Password=MyComplexPassword1!;app name=telegraf;"] query_version = 2 [[outputs.influxdb]] urls = ["https://127.0.0.1:8086"] database = "telegraf" [/code] c. Note that the default polling interval in Telegraf is 10 seconds. If you want to change this, i.e. for more precise metrics, you will have to change the interval parameter in the [agent] section in the telegraf.conf file. d. Once the changes are made, start the service. [code language="text"] sudo systemctl start telegraf [/code] e. Confirm that Telegraf is running. [code language="text"] sudo systemctl status telegraf [/code]

Step 9

In Grafana, create the data source for InfluxDB, and import Managed Instance dashboards. In this step, [GRAFANA_HOSTNAME_OR_IP_ADDRESS] refers to the public hostname or IP address of the Grafana VM, and [INFLUXDB_HOSTNAME_OR_IP_ADDRESS] refers to the hostname or IP address of the InfluxDB VM (either public or private) that is accessible from the Grafana VM. If using a single VM for all components, these values are the same. a. Browse to your Grafana instance - https://[GRAFANA_IP_ADDRESS_OR_SERVERNAME]:3000. Login with the default user admin with password admin. Grafana will prompt you to change the password on first login. b. Add a data source for InfluxDB. Detailed instructions are at https://docs.grafana.org/features/datasources/influxdb/ Click “Add data source” Name: influxdb-01 Type: InfluxDB URL: https://[INFLUXDB_HOSTNAME_OR_IP_ADDRESS]:8086. The default of https://localhost:8086 works if Grafana and InfluxDB are on the same machine; make sure to explicitly enter this URL in the field. Database: telegraf Click “Save & Test”. You should see the message “Data source is working”. c. Download Grafana dashboard JSON definitions from the repo dashboards folder for all three dashboards, and then import them into Grafana. When importing each dashboard, make sure to use the dropdown labeled InfluxDB-01 to select the data source created in the previous step.

Step 10

Optionally, configure a retention policy for data in InfluxDB. By default, monitoring data is maintained indefinitely. a. Start InfluxDB console. [code language="text"] sudo docker exec -it influxdb influx [/code] b. View current retention policies in InfluxDB. [code language="text"] use telegraf; show retention policies; [/code] c. Create a retention policy and exit InfluxDB console. [code language="text"] create retention policy retain30days on telegraf duration 30d replication 1 default; quit [/code] In this example, we created a 30-day retention policy for the telegraf database, setting it as the default policy.

Step 11

You are done! The dashboards should now display data from all instances specified in the Telegraf config file. You can switch between monitored instances using the dropdown at the top of each dashboard. You can change the time scale by using the pull-down menu in upper right. You can switch between different dashboards using the “Other Managed Instance Dashboards” button.

Troubleshooting examples

Here we will show some examples of actual troubleshooting scenarios on Managed Instance where this monitoring solution helped us identify the root cause.

A six-minute workload stall

In this case, we were testing an OLTP application workload. During one test, we noticed a complete stall in workload throughput. During the stall, we saw no current waits on the instance at all, so this was not the kind of problem where the workload was stalled because of waiting for some resource, such as disk IO or memory allocation. Then, after about six minutes, the workload resumed at the previous rate. Looking at the waits again, we now saw a very large spike in LCK_IX_M waits, dwarfing all other waits. This immediately pointed to a blocking problem, which was indeed the case. As a part of each transaction, the application was inserting a row into a history table. That table was created as a heap. We found that during the stall, a single DELETE statement affecting a large number of rows in the history table was executed under the serializable transaction isolation level. Running a DML statement targeting a heap table under serializable isolation always causes lock escalation. Thus, while the DELETE query ran for six minutes, it held an X lock at the table level. This OLTP workload inserts a row into the history table as a part of each transaction, therefore it was completely blocked. What is interesting about monitoring this kind of typical blocking scenario is that the LCK_IX_M waits did not appear on the dashboard until after the stall has ended. This is because SQL Server reports waits in sys.dm_os_wait_stats DMV only once they have completed, and none of the LCK_IX_M waits completed until the X lock went away and the workload had resumed.

Periodic drops in throughput

The next example is more interesting, and is specific to Managed Instance. During an otherwise steady high volume OLTP workload on a General Purpose instance, we observed sharp drops in workload throughput at five-minute intervals. Looking at the graph of wait types, we saw corresponding spikes in BACKUPIO and WRITELOG waits, at the same five-minute intervals. Since log backups on Managed Instance run every five minutes, this seemed to imply a strong correlation between drops in throughput and log backups. But why do log backups cause drops in throughput? Looking at the Database IO Stats dashboard, we saw corresponding five-minute spikes in Read Bytes/sec specifically for the transaction log file, which is indeed expected while a transaction log backup is running. The magnitude of these spikes in read throughput pointed to the root cause of the problem. The spikes were close to the throughput limit of the transaction log blob. At the high level of workload throughput we were driving (~ 8K batch requests/second), the combined read and write IO against the log blob during log backup was enough to cause Azure Storage throttling. When throttling occurs, it affects both reads and writes. This made transaction log writes slower by orders of magnitude, which naturally caused major drops in workload throughput. This problem is specific to General Purpose Managed Instance with its remote storage, and does not occur on Business Critical Managed Instance. For General Purpose Managed Instance, the engineering team is looking for ways to minimize or eliminate this performance impact in a future update.

Conclusion

In this article, we describe a solution for real time performance monitoring of Azure SQL Database Managed Instance. In our customer engagements, we found it indispensable for monitoring workload performance, and for diagnosing a broad range of performance issues in customer workloads. We hope that customers using Managed Instance will find it useful for database workload monitoring and troubleshooting, as they migrate their on-premises applications to Azure. We hope many of you will start your journey with Managed Instance soon, and are looking forward to hearing about your experiences, as well as feedback and contributions to this monitoring solution.

Storage performance best practices and considerations for Azure SQL DB Managed Instance (General Purpose)

Dimitri Furman — Fri, 20 Jul 2018 09:44:28 GMT

Reviewed by: Kun Cheng, Borko Novakovic, Jovan Popovic, Denzil Ribeiro, Rajesh Setlem, Arvind Shyamsundar, Branislav Uzelac In this article, we describe database storage architecture on Azure SQL Database Managed Instance (MI), for General Purpose (GP) instances specifically. We also provide a set of best practices to help optimize storage performance. At the time of this writing (July 2018), Managed Instance, both General Purpose and Business Critical, is in public preview. Some aspects of MI storage architecture will likely change as MI progresses from the current preview to general availability and beyond. This article reflects the current state of the offering.

Database storage architecture on MI GP

MI GP uses Azure Premium Storage to store database files for all databases, except for the tempdb database. From the perspective of the database engine, this storage type is remote, i.e. it is accessed over the network, using Azure network infrastructure. To use Azure Premium Storage, MI GP takes advantage of SQL Server native capability to use database files directly in Azure Blob Storage. This means that there is not a disk or a network share that hosts database files; instead, file path is an HTTPS URL, and each database file is a page blob in Azure Blob Storage. Since Azure Premium Storage is used, its performance characteristics, limits, and scalability goals fully apply to MI GP. The High-performance Premium Storage and managed disks for VMs documentation article includes a section describing Premium Storage disk limits. While the topic is written in the context of VMs and Azure disks, which is the most common usage scenario for Azure Premium Storage, the documented limits are also applicable to blobs. As shown in the limits table in the documentation, the size of the blob determines the maximum IOPS and throughput that can be achieved against the blob. For MI GP, this means that the size of a database file determines the maximum IOPS and throughput that is achievable against the file. The disk/blob size shown in the limits table is the maximum size for which the corresponding limit applies. For example, a blob that is > 64 GB and <= 128 GB (equivalent to a P10 disk) can achieve up to 500 IOPS and up to 100 MB/second throughput. The current implementation of MI GP does not use blobs smaller than 128 GB (P10). The system will use 128 GB (P10) blobs even for very small database files, to avoid negative performance impact that would be likely with smaller blob sizes (P4 and P6). Additionally, when allocating blobs in Premium Storage, MI always uses the maximum blob size within a storage performance tier. For example, if database file size is 900 GB, MI GP will use a 1 TB (P30) blob for that file. In other words, blob size is snapped up to the maximum size of each storage performance tier. If the file grows above that limit, the system automatically increases blob size to the maximum of the next performance tier. Conversely, if the file is shrunk below the maximum size of a performance tier, blob size is automatically reduced as well. For billing purposes, MI uses the configured instance file size limit (8 TB or less), not the larger blob size allocated in Azure Premium Storage. As mentioned earlier, the tempdb database is not using Azure Premium Storage. It is located on the local SSD storage, which provides very low latency and high IOPS/throughput. This article focuses on databases that use remote storage. Managed Instance Business Critical (BC) instances do not use remote Azure Premium Storage, but instead use local SSD storage. Storage performance considerations discussed in this article do not apply to BC instances.

Azure Storage throttling

When the IOPS or throughput generated by the database workload exceed the limits of a database file/blob, storage throttling occurs. For MI GP instances, a typical symptom of storage throttling is high IO latency. It is possible to see IO latency spike to hundreds of milliseconds when being throttled, while without throttling, average storage latency at the OS level would be in the 2-10 millisecond range. Another symptom of storage throttling is long PAGEIOLATCH waits (for data file IO) and WRITELOG waits (for the transaction log file IO). Needless to say, when storage throttling is occurring, there is a substantial negative effect on database performance. If you observe the symptoms of storage throttling when running workloads on MI GP, we recommend increasing database file size to get more IOPS/throughput from Azure Premium Storage. More specific recommendations, including a script to determine if storage IO generated by the database workload approaches Azure Premium Storage IOPS/throughput limits, are provided further in the article.

Storage limits on MI GP

There are two storage-related limits that should be considered when using MI GP. The first limit is on the total size of all database files on the instance. System databases including tempdb are counted towards this limit, and all data files and transaction log files are considered. Currently, this limit is 8 TB per instance, though a customer can configure their instance to use a lower limit. Note that this limit applies specifically to file size as it appears in the size column in the sys.master_files DMV, not to the space actually used to store data within each file. The second limit is the Azure Premium Storage limit on the maximum space in a storage account, which is currently limited to 35 TB. Each MI GP instance uses a single Premium Storage account. The up to 8 TB file size limit implies that if there are many databases on the instance, or if a database has many files, then it may be necessary to reduce the size of individual database files to stay within this limit. In that case, the IOPS/throughput against these files would be reduced as well. The more files an MI instance has, the more pronounced is the impact of instance file size limit on storage performance. In the worst case, the size of individual database files may have to be reduced to the point where they all end up using 128 GB (P10) blobs, even if the workload could take advantage of better performance available with larger files. Whether this limit is relevant for a particular workload depends on its storage performance needs. Many workloads will perform sufficiently well even with smaller database files. The IOPS/throughput that can be obtained for a particular file size can be determined in advance from the Azure Premium Storage limits table. In the context of migration to MI, customers can measure actual IOPS/throughput on the source system being migrated, to see if it is necessary to increase database file size on MI GP to provide comparable storage performance. For SQL Server on Windows, use the Disk Transfers/sec and Disk Bytes/sec PerfMon counters, for IOPS and throughput respectively. For SQL Server on Linux, use iostat, looking at the sum of r/s and w/s values for IOPS, and the sum of rKB/s and wKB/s for throughput. Alternatively, the sys.dm_io_virtual_file_stats() DMF can be used as well. The script referenced in the best practices section below provides an example of using this DMF to determine IOPS and throughput per database file. The fact that MI GP does not use blobs smaller than 128 GB (P10), coupled with the fact that the blob size is snapped up to the maximum size of its performance tier, can make the 35 TB limit relevant for some MI GP deployments. As described in documentation, when there are many database files with sizes smaller than the sizes of the blobs they use, then the 35 TB blob size limit may be reached sooner than the 8 TB file size limit. To see the total file and blob sizes for an instance, customers can use the following query. It returns a single row with two columns, one showing the total file size, where the 8 TB limit is relevant, and the other showing the total blob size, where the 35 TB limit is relevant: [code language="sql"] WITH DatabaseFile AS ( SELECT database_id AS DatabaseID, CAST(size * 8. / 1024 / 1024 AS decimal(12,4)) AS FileSizeGB FROM sys.master_files ) SELECT SUM(FileSizeGB) AS FileSizeGB, SUM( IIF( DatabaseID <> 2, CASE WHEN FileSizeGB <= 128 THEN 128 WHEN FileSizeGB > 128 AND FileSizeGB <= 256 THEN 256 WHEN FileSizeGB > 256 AND FileSizeGB <= 512 THEN 512 WHEN FileSizeGB > 512 AND FileSizeGB <= 1024 THEN 1024 WHEN FileSizeGB > 1024 AND FileSizeGB <= 2048 THEN 2048 WHEN FileSizeGB > 2048 AND FileSizeGB <= 4096 THEN 4096 ELSE 8192 END, 0 ) ) AS BlobSizeGB FROM DatabaseFile; [/code] For example, this result set shows different file and blob total sizes for a MI GP instance: [code language="text"] FileSizeGB BlobSizeGB ----------- ----------- 2048.4474 2944 [/code]

Comparison with SQL Server in Azure IaaS VM

In a typical deployment of SQL Server on an Azure IaaS VM (SQL VM), database files are placed on Azure disks attached to the VM. Sometimes, a Storage Spaces pool (on Windows) or an LVM volume (on Linux) is created using multiple disks, to aggregate IOPS/throughput provided by multiple disks, and to increase storage size. There are notable differences between this model, and the model used by MI GP. For SQL VMs, available IOPS/throughput are shared among all database files using a disk (or a storage pool), whereas for MI GP, each file/blob gets its own allocation of IOPS/throughput. Each model has its pros and cons. When there are multiple databases on the instance, MI GP makes it possible to provide a fixed allocation of IOPS/throughput per database file, avoiding the “noisy neighbor” effect of other databases on the instance. On the other hand, each database on a SQL VM can take advantage of higher aggregated IOPS/throughput provided by multiple disks, assuming that all databases do not create spikes in storage utilization at the same time. One other important difference between MI GP and SQL VM is that the per-VM IOPS/throughput limits, documented for each VM type as Max uncached disk throughput: IOPS / MBps, do not apply to MI. When we talked about throttling earlier in this article, we referred specifically to Azure Premium Storage throttling, and not VM-level throttling. VM-level throttling can occur on SQL VM when cumulative IO requests against all Azure disks attached to the VM exceed the per-VM limits. However, MI GP does not use Azure disks for database files; therefore, the per-VM disk limits are not applicable, and MI instances do not experience VM-level throttling in the way that SQL VMs do.

Performance impact of database file size on MI GP – an example

To illustrate the performance impact of database file size on MI GP, we ran a series of tests using a generic OLTP workload on an 8 vCore instance with 8 TB storage. This example is not intended as a benchmark, and the numbers shown should not be taken as representative for MI GP in general. Storage performance is highly dependent on workload characteristics, and the results obtained using the same MI instance type and storage layout used in these tests, but a different workload, may be very different. These results are shown only to demonstrate the relative difference in performance when different file size configurations on MI GP are used. In these tests, we used a database with 3 data files and 1 log file. In the initial test, the size of every file was 32 GB (as noted earlier, the system actually uses 128 GB P10 blobs in this case). In following tests, we first increased the size of the transaction log file, and then gradually increased the size of data files. Finally, we increased the size of all files to over 1 TB (to use 2 TB P40 blobs) to get maximum storage performance possible with this file layout. We observed the changes in workload throughput and behavior using metrics such as batch requests/second (BRPS), CPU utilization percentage, PAGEIOLATCH and WRITELOG wait percentage relative to all other non-ignorable waits, and write latency for data files and log file (measured using the sys.dm_io_virtual_file_stats() DMF). At the storage level, the workload had a 60/40 read/write ratio with cold cache. The database was small enough to fit in memory, so during the test, that ratio changed to nearly 100% writes as most of the data pages became cached in the buffer pool. For this reason, we show only write latency in test results. Test results are presented in the following table:

File layout	Average BRPS	CPU utilization (%)	WRITELOG waits (%)	PAGEIOLATCH waits (%)	Average data write latency (ms/write)	Average log write latency (ms/write)
3 x 32 GB data files, 32 GB log file	1457	10	21	70	132	16
3 x 32 GB data files, 512 GB log file	1732	12	13	85	477	4.5
3 x 256 GB data files, 512 GB log file	2422	12	12	80	155	4.7
3 x 512 GB data files, 512 GB log file	2706	12	12	77	165	4.4
3 x 1.1 TB data files, 2 TB log file	7022	46	74	4	49	4.5

We can see that for this workload, incrementally growing data and log files up to 512 GB in the first four tests provided gradual improvements. Using 512 GB files, workload throughput (BRPS) was almost twice as high as with the 32 GB files used initially. But increasing the size of all files from 512 GB to over 1 TB provided a 4.8x increase in workload throughput, compared to the initial test throughput. In this last test, removing the bottleneck in data file IO drastically increased both BRPS and CPU utilization, and shifted the dominant wait type from PAGEIOLATCH to WRITELOG as many more write transactions were processed. To reiterate, the degree of performance improvement from increasing file size on MI GP is workload-dependent. For some workloads, sufficient performance levels can be achieved without using large files. Furthermore, other factors besides storage performance could affect the overall workload performance. For example, increasing file size on a CPU-constrained MI GP instance would not necessarily provide a significant performance increase.

Storage performance best practices for MI GP

In closing, we present a set of storage performance best practices for MI GP instances. These are not absolutes. We encourage the readers to understand the reasons behind these best practices that were discussed earlier in the article, and consider each recommendation in the context of their specific application/workload. 1. Determine if the storage IO generated by the database workload is exceeding Azure Premium Storage limits. A script that examines IOPS/throughput per database file over a period of time, and compares them against the limits, is available on GitHub. If storage IO is near the limits, allocate more IOPS/throughput for the database as described below. 2. When possible, increase space allocation at the MI instance level. This will allow you to increase the size of individual database files, resulting in higher IOPS/throughput limits per file. 3. Increase the size of individual database files as much as possible to utilize available instance space, even if a large portion of each file would remain empty. But consider Azure Premium Storage limits as well. For example, if the size of a file is 150 GB, then increasing it up to 256 GB would not provide any additional IOPS/throughput. MI already uses a 256 GB (P15) blob in this case. This larger file would take some of the instance storage space that may be better used to increase IOPS/throughput for other database files on the same instance. In this example, to get the advantage of the next storage performance tier for this file, it would be necessary and sufficient to increase file size to 257 GB. MI GP would then snap up the blob size for this file to 512 GB (P20). 4. For many write-intensive workloads, increasing the size of the transaction log file provides higher performance improvement than increasing the size of data files. 5. Make all data files equally sized with the same growth increment. While this recommendation is not specific to MI, it is particularly relevant here. On MI GP, a single small file may negatively impact performance of the entire workload, due to storage throttling becoming more likely for that small file. 6. Avoid using many small files for a database. While the aggregate IOPS/throughput from many small files may be comparable to the IOPS/throughput provided by fewer large files, each individual small file is more likely to encounter storage throttling, which affects the entire workload. 7. Consider not only increasing file size, but also adding data files. For example, consider an MI instance with a single database. If this database has a single data file, it can get at most 7500 IOPS for data IO, by making that file larger than 1 TB in size (a P40 or P50 blob would be used depending on the exact file size). But using N files, each larger than 1 TB, would provide up to 7500*N IOPS for data IO, and 7500 IOPS for log IO. For example, you could use six 1.1 TB data files, and one 1.1 TB log file. The total file size would be 7.7 TB, while the total blob size would be 14 TB, remaining within the 8 TB and 35 TB limits respectively. Keep in mind that simply adding empty files to an existing database would not improve performance for most workloads; it is also necessary to reallocate existing data over all files.

Conclusion

In this article, we show how database file size and layout can have a significant performance impact on MI GP instances. We provide recommendations to help customers achieve optimal storage performance, and to avoid common pitfalls. This article applies to General Purpose MI instances. The complexities and limitations we describe do not apply to Business Critical instances, which are recommended for mission critical workloads with high storage performance requirements. If sufficient storage performance cannot be achieved on a General Purpose instance even after following best practices described above, then switching to a Business Critical instance is recommended. Update 2018-12-18: As of this writing, both the General Purpose and Business Critical MI service tiers implement log rate governance at the instance level. As documented in the Managed Instance resource limits article, this limits log throughput to 22 MB/s per instance for General Purpose instances, and to 3 MB/s per vCore for Business Critical instances. When the maximum instance log rate is reached, one of the dominant wait types will be INSTANCE_LOG_RATE_GOVERNOR. For write intensive workloads such as bulk data loads, this limits write throughput regardless of the transaction log blob size. There are two potential workarounds to improve data load performance on MI. One is to use tempdb as a temporary staging area for data loads. Tempdb is not subject to instance log rate governance, and provides minimal logging as well. Another workaround is to use clustered columnstore indexes (CCI), which reduce the volume of transaction log writes due to high data compression.

DataCAT team at Data Platform Summit 2018

Arvind Shyamsundar — Mon, 16 Jul 2018 18:44:59 GMT

Team members from Microsoft AzureCAT (a.k.a. DataCAT / SQLCAT) will be presenting a full-day pre-conference on “Building A Scalable Data Architecture In Azure” at the upcoming DPS 2018 conference in Bengaluru, India. We are very excited to be part of the 4th edition of Asia’s largest Microsoft data-focused conference, with over 1000 attendees expected overall.

The motivation for this full-day pre-conference training is to share with attendees typical scenarios and considerations in architecting and implementing solutions on Microsoft Azure. Whether you are migrating an existing on-premises data pipeline, or implementing a greenfield project in the cloud, the variety of choices in terms of technologies can be bewildering and confusing. We intend to share scenarios and patterns and practices which have worked in those scenarios. With this background, the attendee has a head-start on the technical front and more importantly, confidence that the architectural patterns work in real-world scenarios.

The pre-conference is structured into the following sections:

Overall Azure Data Services landscape
Transactional systems and structured data with Azure SQL DB
Not Only SQL systems (NoSQL) with Cosmos DB
Relational Data Warehousing using Azure SQL DW
Handling streaming data in Azure with Event Hubs, Kafka etc.
Building Data Lakes in Azure
Data Analytics and AI / ML at scale using Apache Spark, Azure Data Lake Analytics etc.
Options for orchestrating data pipelines: Azure Data Factory etc.
Interactive Querying using Power BI, Azure Analysis Services etc.
Top lessons learnt from real-world customer engagements

Each section will be accompanied by demos highlighting practical considerations and recommendations beyond just the basics. The team members who are part of this conference are all very experienced in the industry and are true believers in sharing information and learnings:

Sanjay Mishra leads the DataCAT group within AzureCAT and is based in the Microsoft HQ at Redmond, USA. His current focus is on overall architecture patterns in the cloud.
Rangarajan Srirangam is part of AzureCAT based in Bengaluru, India. He has helped major customers adopt and use Microsoft Azure. He brings to the table practical considerations and architectural recommendations based on his hard-earned experiences.
Arvind Shyamsundar is part of the DataCAT group based in Redmond, USA. His focus areas include relational (SQL) systems and distributed computing using Apache Spark.
Mandar Inamdar is part of the AzureCAT team and is based in Mumbai, India. His focus areas are Azure SQL DW, Interactive querying using Power BI and AAS.
Guru Charan Bulusu is part of AzureCAT and works in the Microsoft Bengaluru office. His key focus areas are Streaming data, NoSQL, AI and ML.

We hope you are as excited as we are to bring this content to you. We hope you can join us on August 8th at the DPS 10 pre-conference!

Turbo boost data loads from Spark using SQL Spark connector

Denzil Ribeiro — Sat, 12 May 2018 10:40:43 GMT

Reviewed by: Dimitri Furman, Xiaochen Wu Apache Spark is a distributed processing framework commonly found in big data environments. Spark is often used to transform, manipulate, and aggregate data. This data often lands in a database serving layer like SQL Server or Azure SQL Database, where it is consumed by dashboards and other reporting applications. Prior to the release of the SQL Spark connector, access to SQL databases from Spark was implemented using the JDBC connector, which gives the ability to connect to several relational databases. However, compared to the SQL Spark connector, the JDBC connector isn’t optimized for data loading, and this can substantially affect data load throughput. As an example, utilizing the SQLBulkCopy API that the SQL Spark connector uses, dv01, a financial industry customer, was able to achieve 15X performance improvements in their ETL pipeline, loading millions of rows into a columnstore table that is used to provide analytical insights through their application dashboards. In this blog, we will describe several experiments that demonstrate the major performance improvement provided by the SQL Spark connector. You can download the SQL Spark Connector here

Dataset

Dataset size: 117 million rows Source: Azure Blob Storage containing 50 parquet files. Spark Cluster: 8+1 node cluster, each node is a DS3V2 Azure VM (4 cores, 17 GB RAM)

Scenario 1: Loading data into SQL Server

SQL Version: SQL Server 2017 CU 5 on RedHat 7.4 Azure VM Size: DS16sV3 (16 cores, 64 GB RAM) Storage: 8 P30 disks in Azure Blob Storage Database Recovery Model: Simple Performance in SQL on windows v/s SQL on Linux is comparable and for brevity we only depict results on SQL Server on Linux.

Loading into a heap

Using Spark JDBC connector

Here is a snippet of the code to write out the Data Frame when using the Spark JDBC connector. We used the batch size of 200,000 rows. Changing the batch size to 50,000 did not produce a material difference in performance.

[code lang="scala"] dfOrders.write.mode("overwrite").format("jdbc") .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") .option("url", "jdbc:sqlserver://server.westus.cloudapp.azure.com;databaseName=TestDB") .option("dbtable", "TestDB.dbo.orders") .option("user", "myuser") .option("batchsize","200000") .option("password", "MyComplexPassword!001") .save() [/code]

Since the load was taking longer than expected, we examined the sys.dm_exec_requests DMV while load was running, and saw that there was a fair amount of latch contention on various pages, which wouldn’t not be expected if data was being loaded via a bulk API.

Examining the statements being executed, we saw that the JDBC driver uses sp_prepare followed by sp_execute for each inserted row; therefore, the operation is not a bulk insert. One can further example the Spark JDBC connector source code, it builds a batch consisting of singleton insert statements, and then executes the batch via the prep/exec model.

This is an 8-node Spark cluster, each executor with 4 CPU’s and due to sparks default parallelism, there were 32 tasks running simultaneously with multiple insert statements batched together. The primary contention was PAGELATCH_EX and just like any latch contention the more parallel sessions requesting for the same resource, the more the contention.

Using SQL Spark connector

The SQL Spark connector also uses the Microsoft JDBC driver. However, unlike the Spark JDBC connector, it specifically uses the JDBC SQLServerBulkCopy class to efficiently load data into a SQL Server table. Given that in this case the table is a heap, we also use the TABLOCK hint ( "bulkCopyTableLock" -> "true") in the code below to enable parallel streams to be able to bulk load, as discussed here. It is a best practice to use the BulkCopyMetadata class to define the structure of the table. Otherwise, there is additional overhead querying the database to determine table schema.

[code lang="scala"] import com.microsoft.azure.sqldb.spark.config._ import com.microsoft.azure.sqldb.spark.connect._ import com.microsoft.azure.sqldb.spark.query._ import com.microsoft.azure.sqldb.spark._ import com.microsoft.azure.sqldb.spark.bulkcopy._ var bulkCopyMetadata = new BulkCopyMetadata bulkCopyMetadata.addColumnMetadata(1, "o_orderkey", java.sql.Types.INTEGER, 0, 0) bulkCopyMetadata.addColumnMetadata(2, "o_custkey", java.sql.Types.INTEGER, 0, 0) //trimming other columns for brevity… only showing the first 2 columns being added to BulkCopyMetadata val bulkCopyConfig = Config(Map( "url" -> "server.westus.cloudapp.azure.com", "databaseName" -> "testdb", "user" -> "denzilr", "password" -> "MyComplexPassword1!", "dbTable" -> "dbo.orders", "bulkCopyBatchSize" -> "200000", "bulkCopyTableLock" -> "true", "bulkCopyTimeout" -> "600" )) dfOrders.bulkCopyToSqlDB(bulkCopyConfig) [/code]

Looking at the Spark UI, we see a total of 50 tasks that this DataFrame write is broken into, each loading a subset of the data:

Further investigating the statements, we see the familiar INSERT BULK statement, which is an internal statement used by the SQL Server bulk load APIs. This proves that we are indeed using the SQLBulkCopy API.

There is no longer page latch contention, rather now we are waiting on network IO or client fetching the results

Loading into a clustered columnstore table

With JDBC connector

Load performance in the columnstore case will be far worse with the JDBC connector than in the heap case. Given that the JDBC connector emits single row insert statements, all this data lands in the delta store. And we are back with more severe latch contention this time around. You can read about many more details of loading data into columnstore tables in this blog 😊.

If we examine the sys.dm_db_column_store_index_physical_stats DMV, we notice that all rows are going into an OPEN rowgroup that is a delta store until that rowgroup is filled up and closed. These rows will then have to be compressed by tuple mover into compressed segments later.

Using SQL Spark connector

For the bulk load into clustered columnstore table, we adjusted the batch size to 1048576 rows, which is the maximum number of rows per rowgroup, to maximize compression benefits. Having batch size > 102400 rows enables the data to go into a compressed rowgroup directly, bypassing the delta store. Also, you have to set TABLOCK hint to false, else you will be serializing the parallel streams.

When bulk loading in parallel into a columnstore table, there are a few considerations:

Memory grants and RESOURCE_SEMAPHORE waits. Depending on how many parallel streams, you could run into this issue, and it could end up bulk inserting into delta row groups. For more information, see this blog.
Compressed rowgroups could be trimmed due to memory pressure. You would see the trim_reason_description column in the sys.dm_db_column_store_row_group_physical_stats DMV as “MEMORY_LIMITATION”.

In this case, you see that rows land in the compressed rowgroup directly. We had 32 parallel streams going against a DS16sV3 VM (64GB Ram). There are cases where having too many parallel streams, as a result of memory requirements for a bulk insert can cause rowgroups to get trimmed due to memory limitations.

Scenario 2: Loading data into Azure SQL Database

SQL version: Azure SQL Database Database performance level: P11 There are a couple of differences that need to be noted, which make the Azure SQL Database tests fundamentally different than the tests with SQL Server on a VM.

Database recovery model is Full, vs. Simple recovery model that was used with SQL Server in VM. This prevents the use of minimal logging in Azure SQL Database.
P11 is a Premium database, therefore it is in an availability group used to provide built-in HA within the service. This adds an overhead of committing every transaction on multiple replicas.

When loading into Azure SQL Database, depending on the performance level of the database, you may see other wait types such as LOG_RATE_GOVERNOR, which would be an indicator of a bottleneck. There are multiple ways to monitor resource utilization in Azure SQL Database to detect resource bottleneck, e.g. the sys.dm_db_resource_stats DMV. If a resource bottleneck exists, the database can be easily scaled up to a higher performance level to achieve higher throughput during data loads. More on Azure SQL Database monitoring can be found here. Recapping a few considerations relevant to data loading from Spark into SQL Server or Azure SQL Database:

Use the Spark SQL connector. We have just shown that in the bulk insert scenario, we get fundamentally better performance, by an order of magnitude, than with the Spark JDBC connector.
For tables that are heaps, use the TABLOCK hint to allow parallel streams. This is particularly relevant for staging tables, which tend to be heaps.
For bulk loads into columnstore tables, do not use the TABLOCK hint, as that would serialize parallel streams.
For bulk loads into columnstore tables, ensure that batch size is >= 102400 so that row go directly into a compressed rowgroup. Ideally start with the batch size of 1048576
For partitioned tables, see the section on partitioned tables in the Data Loading performance considerations with Clustered Columnstore indexes Depending on the number of rows per partition, they could land in the delta store, which would affect bulk insert performance.

CPU and Memory Allocation on Azure SQL Database Managed Instance

Dimitri Furman — Fri, 04 May 2018 15:44:55 GMT

Reviewed By: Ajay Kalhan, Borko Novakovic, Drazen Sumic, Branislav Uzelac In the current Azure SQL Database Managed Instance (MI) preview, when customers create a new instance, they can allocate a certain number of CPU vCores and a certain amount of disk storage space for the instance. However, there is no explicit configuration option for the amount of memory allocated to the instance, because on MI, memory allocation is proportional to the number of vCores used. How can a customer determine the actual amount of memory their MI instance can use, in GB? The answer is less obvious than it may seem. Using the traditional SQL Server methods will not provide the right answer on MI. In this article, we will go over the technical details of CPU and memory allocation on MI, and describe the correct way to answer this question. The information and behavior described in this article are as of the time of writing (April 2018). Some aspects of MI behavior, including the visibility of certain compute resource allocations, may be temporary and will likely change as MI progresses from the current preview to general availability and beyond. Nevertheless, customers using MI in preview will find that this article answers some of the common questions about MI resource allocation.

First glance at CPU and memory on MI

We will use a MI instance with 8 vCores as an example. On the traditional SQL Server, most customers would look at the Server Properties dialog in SSMS to see the compute resources available to the instance. On our example MI instance, this is what we see: We should right away note that the resource numbers in this dialog, as well as in several other sources (DMVs) described later, can change over the lifetime of a given MI instance. These changes can be relatively frequent. Customers should not take any dependencies, or make any conclusions based on these numbers. Later in the article, we will describe the correct way to determine actual compute resource allocation on MI. An immediate question is why we see 24 processors here, when we have created this instance with only 8 vCores/processors. To determine the actual number of logical processors available to this instance, we can look at the number of VISIBLE ONLINE schedulers in the sys.dm_os_schedulers DMV: [code language="sql"] SELECT COUNT(1) AS SchedulerCount FROM sys.dm_os_schedulers WHERE status = 'VISIBLE ONLINE'; [/code] [code language="text"] SchedulerCount -------------- 8 [/code] This is in line with the number of vCores we have for this MI instance. Then why does SSMS show that 24 processors are available?

CPU and memory resources are managed differently on MI

To answer this question, we need to take a high-level look at the MI architecture. Each MI instance runs in a virtual machine (VM). Each VM may host multiple MI instances of varying sizes, in terms of compute resources allocated to the instance. It is important to note here that all MI instances on a given VM always belong to the same customer; there is no multi-tenancy at the VM level. In effect, the VM hosting MI instances serves as an additional isolation boundary for customer workloads. This does not mean that if a customer creates multiple MI instances, they will necessarily be packed on the same VM. In reality, this does not happen very often. The service intelligently allocates instances to VMs to always provide guaranteed SLAs and ensure good customer experience. What SSMS shows in the Server Properties dialog is the number of processors and the amount of memory at the OS level on the VM that happens to currently host the instance. This works the same way for the traditional SQL Server, where SSMS also shows OS level numbers. SQL Server error log, which is accessible on MI, shows the same information during MI instance startup: [code language="text"] SQL Server detected 2 sockets with 12 cores per socket and 12 logical processors per socket, 24 total logical processors; using 24 logical processors based on SQL Server licensing. This is an informational message; no user action is required. Detected 172031 MB of RAM. This is an informational message; no user action is required. [/code] For our example MI instance, this means that there are 24 processors accessible to the OS on the underlying VM. However, given the number of visible online schedulers, the MI instance can only use 8 of these processors, as expected given its current provisioned size. To reiterate, the number of processors and the amount of memory at the VM level (24 processors and 172031 MB in this example) is not fixed. It can change over time as the MI instance moves across VMs allocated to the customer, for example when it is scaled up or scaled down. These values will, however, always be larger than or equal to the resource values actually allocated to the instance. But what about memory? Does this MI instance have 168 GB of memory, as shown in SSMS and in the error log? Let’s look at some DMVs. [code language="sql"] SELECT cpu_count, physical_memory_kb, committed_target_kb FROM sys.dm_os_sys_info; [/code] [code language="text"] cpu_count physical_memory_kb committed_target_kb ----------- -------------------- -------------------- 8 176,160,308 48,586,752 [/code] [code language="sql"] SELECT cntr_value FROM sys.dm_os_performance_counters WHERE object_name LIKE '%Memory Manager%' AND counter_name = 'Target Server Memory (KB)'; [/code] [code language="text"] cntr_value ---------- 48,586,752 [/code] Both of these show that the server target memory, which is commonly used to measure the amount of memory available to the instance, is about 46 GB, while the total physical memory at the OS level is 168 GB, as seen in SSMS and in the error log. This shows that not all memory available at the VM OS level is allocated to this MI instance. On the traditional SQL Server, the usual reason for the target memory to be much lower than the available OS physical memory is configuring a limit on server maximum memory using sp_configure. Is that the case here? [code language="sql"] SELECT value_in_use FROM sys.configurations WHERE name = 'max server memory (MB)'; [/code] [code language="text"] value_in_use ------------ 2147483647 [/code] This large value shows that the maximum server memory for this instance is not limited, and the instance should be able to allocate all available physical memory. What is causing target memory to be much less than the total physical memory for this instance? Is there some other mechanism that can impose a lower limit? For MI, this mechanism is Job Objects. Running a process such as SQL Server in a job object provides resource governance for the process at the OS level, including CPU, memory, and IO. This resource governance is what allows the service to share the same VM among multiple instances belonging to the same customer, without resource contention and “noisy neighbor” problems. At the same time, this mechanism guarantees a dedicated allocation of vCores and memory for each instance. Memory is allocated according to the GB/vCore ratio for the instance size selected by the customer. There is no overcommit of either vCores or memory across instances on the same VM. In other words, the instance always gets the resources specified during provisioning. Can we see the configuration of the job object that contains our MI instance? Yes, we can: [code language="sql"] SELECT cpu_rate, cpu_affinity_mask, process_memory_limit_mb, non_sos_mem_gap_mb FROM sys.dm_os_job_object; [/code] [code language="text"] cpu_rate cpu_affinity_mask process_memory_limit_mb non_sos_mem_gap_mb ----------- -------------------- ----------------------- -------------------- 100 255 57,176 9,728 [/code] The sys.dm_os_job_object DMV exposes the configuration of the job object that hosts the MI instance. For our current topic, there are four columns in the DMV that are particularly relevant: cpu_rate: this is set to 100%, showing that each vCore can be utilized by the MI instance to its full capacity. cpu_affinity_mask: this is set to 11111111 (in binary), showing that only eight OS level processors can be used by the process hosted in the job object (SQL Server). This is in line with the number of vCores provisioned for this instance. process_memory_limit_mb: this is set to about 56 GB, and is the total amount of memory allocated to the process within the job object. Note that this is larger than the server target memory, which, as we saw earlier, is about 46 GB. The next column provides the explanation. non_sos_mem_gap_mb: this is the amount of memory that is a part of total process memory, but is not available for SQL Server SOS (SQL OS) memory allocations, i.e. is reserved for things like thread stacks and DLLs loaded into the SQL Server process space. The difference between process_memory_limit_mb and non_sos_mem_gap_mb is 46 GB, which is exactly the server target memory that we saw earlier. To elaborate on the last point, even though SQL Server target memory visible in DMVs and in the output of DBCC MEMORYSTATUS is less than the total memory allocated to the instance, this difference, known as the non-SOS memory gap, is still being used by the instance. In fact, a sufficiently large allocation of non-SOS memory is required for the instance to function reliably.

Conclusion

To summarize the technical details above: 1. MI compute resource values shown in SSMS and the instance error log reflect resources at the underlying OS level, not the actual resources available to the MI instance. The resource values at the OS level can change over time, without affecting the resources allocated to the MI instance in any way. 2. MI instance is resource-governed at the OS level using a job object. 3. The sys.dm_os_job_object DMV exposes job object configuration for the MI instance. This is the DMV that should be used to determine the actual compute resources (vCores and memory) allocated to the MI instance, as described above. We hope that the information in this article will help customers using Managed Instance to accurately and confidently determine the amount of compute resources allocated to their instances, and avoid potential confusion in this area due to architectural differences between the traditional SQL Server and Managed Instance.

Azure SQL DB Managed Instance - sp_readmierrorlog

Dimitri Furman — Fri, 04 May 2018 15:18:34 GMT

Reviewed by: Kun Cheng, Borko Novakovic, Arvind Shyamsundar, Mike Weiner Azure SQL Database Managed Instance is a new offering that provides an instance-based SQL PaaS service in Azure. If you are not yet familiar with this new Azure SQL Database capability, you can start with the What is Managed Instance documentation topic. Since early private preview of Managed Instance (MI), SQLCAT has been working with early adopter customers to help them evaluate MI as a new platform for their applications, gather their feedback, and improve the offering for everyone. Azure SQL Database Managed Instance is now available in public preview.

SQL error log is available on MI

The primary goal of MI is close compatibility with the traditional SQL Server, to help facilitate migrations from on-premises environments to the Azure SQL Database PaaS service. In this article, we will discuss one MI capability that it shares with SQL Server, namely the ability to see the instance error log. For SQL Server DBAs, the error log is one of the first things to check when troubleshooting application issues, and that is still the case when using a managed service such as MI. The availability of the instance error log highlights the MI focus on compatibility with SQL Server; by comparison, in Azure SQL Database, diagnostics and error information are exposed in different ways, i.e. using the sys.event_log DMV and diagnostics logging. When you connect to an MI instance, you can right away look at its error log, using either Log File Viewer in SSMS (latest SSMS is strongly recommended), or the sp_readerrorlog stored procedure. If you do that on an instance of SQL Server that has just started, you will typically see 100-200 lines of output, depending on server configuration, the number of databases, etc. But when you look at the MI error log, you may be surprised by the large volume of messages. For example, in the first minute after instance startup, more than 2500 messages are logged. While some of them are the familiar messages you would find in a SQL Server error log, many others might look a bit cryptic, and are not actionable from an end-user standpoint. Why is all this information in the MI error log? This diagnostic data is needed for Microsoft engineers to manage the service and troubleshoot any problems efficiently. As an aside, the MI error log is so detailed, that a curious user with relatively advanced knowledge of SQL Server internals can glean many interesting details about internal workings of MI, even without full knowledge of the MI architecture. An important note is that in current MI preview, error logs are not persisted across instance restarts and failovers. Therefore, when looking at older logs, gaps are possible.

A new way to look at the MI error log

But for most customers, the high volume of debug-level messages in the log just makes it harder to see the messages relevant to their applications/databases, which are lost in the noise, as it were. This is the reason why we wrote sp_readmierrorlog. This is a simple stored procedure that returns the contents of the instance error log, filtering out messages that are unlikely to be useful to an MI customer. sp_readmierrorlog has the same familiar parameters and result set that sp_readerrorlog has. It can be created in the master database of each MI instance you use, making it easy to call the procedure from the context of any database on the instance. As an example, on a mostly idle MI instance, sp_readmierrorlog reduced a 150,000-line error log, generated over one day, to 760 lines. This is much more manageable, while still providing useful diagnostic information to MI customers. For customers requiring a more in-depth look into instance behavior and diagnostics, access to the unfiltered log is still available using the built-in sp_readerrorlog procedure. The procedure is open source and is hosted on GitHub. The filtering logic used by the procedure can be easily customized to include/exclude specific messages from the result set. If you find that the procedure is filtering out too much or too little, we would welcome pull requests with changes that make the stored procedure better for all MI customers.

SQL Server VLDB in Azure: DBA Tasks Made Simple

Dimitri Furman — Mon, 22 Jan 2018 14:29:20 GMT

Reviewed by: Rajesh Setlem, Mike Weiner, Xiaochen Wu With thanks to Joey D’Antoni (blog) for asking a great question that prompted this article. As any experienced DBA knows, supporting a very large database (VLDB) tends to be exponentially more complex than supporting smaller databases. The definition of VLDB has been shifting in recent years, but most DBAs would agree that complexities become significant starting with database size in the 3-5 TB range. A few obvious examples of tasks that are more challenging with a VLDB are backup/restore, consistency checking, HA/DR, partitioning, etc. While these challenges are inherently present in any VLDB, the platform used for hosting the database can sometimes make them easier. In this article, we will show how using Azure as the cloud platform for SQL Server can simplify some of these common challenges. To be clear from the start, we will be talking about databases in the 3-30 TB range running on Azure IaaS VMs. At those sizes, the traditional database operation methods tend to be inefficient, or sometimes break down completely. For smaller databases, the approaches described in this article may be unnecessary (though they would still work).

A VLDB database

In the examples in this article, we will be using a 12 TB database, appropriately named VLDB01. The database was created on a SQL Server VM in Azure by loading data from text files in Azure Blob Storage, using PolyBase. The database was created with files stored directly in Azure blob storage, using a Premium Storage account. The ability to store database files directly in blob storage, rather than on disk, is what enables the scenarios described in this article. Starting with SQL Server 2014, in addition to using files stored on a local disk or on a UNC path, SQL Server also supports placing database files as page blobs in Azure Blob Storage, and accessing them over an http(s) endpoint. This lets SQL Server use certain features of Azure Blob Storage that are not necessarily available in the more traditional storage subsystems. One example of this is described in Instant Log Initialization for SQL Server in Azure. This article will cover additional scenarios, with emphasis on simplifying operational tasks for a VLDB. Database file layout for the VLDB01 database is described by the output from the following query: [code language="sql"] SELECT name, physical_name, SUM(CAST(FILEPROPERTY(name, 'SpaceUsed') AS bigint) * 8192.0/1024/1024/1024/1024) AS space_used_TB, SUM(CAST((size/128.0) AS decimal(15,2))/1024/1024) AS file_size_TB FROM sys.database_files GROUP BY GROUPING SETS ((name, physical_name),()); [/code]

name	physical_name	space_used_TB	file_size_TB
VLDB01	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01.mdf	0.0000068	0.0000687
VLDB01_File1	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File1.ndf	1.5332245	2.0000000
VLDB01_File2	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File2.ndf	1.5351921	2.0000000
VLDB01_File3	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File3.ndf	1.5322309	2.0000000
VLDB01_File4	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File4.ndf	1.5393468	2.0000000
VLDB01_File5	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File5.ndf	1.5361704	2.0000000
VLDB01_File6	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File6.ndf	1.5354084	2.0000000
VLDB01_File7	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File7.ndf	1.5347077	2.0000000
VLDB01_File8	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File8.ndf	1.5317377	2.0000000
VLDB01_log	https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_log.ldf	0.0281003	0.1874998
		12.3061257	16.1875685

Each data file, other than the PRIMARY filegroup data file and the log file, uses a 2 TB Premium Storage page blob, the equivalent of a P40 disk. This provides maximum per-blob storage throughput and IOPS available in Azure Premium Storage today. The number of data files, eight for this particular database, was chosen to allow the database to grow by another ~4 TB without any changes in the storage layout. Even if the data could fit into fewer data files, using more files lets us increase the overall storage throughput and IOPS for the database (subject to VM level limits; more on this in the Limitations section below).

Backup and restore

Let’s see how long it would take to back up this 12 TB database using the traditional streaming backup to URL. Since each SQL Server backup blob cannot exceed ~195 GB (see Backing up a VLDB to Azure Blob Storage for details), we have to stripe the backup over multiple blobs. In this case, we are using the maximum allowed number of stripes, 64, to ensure we do not hit the 195 GB per-stripe limit. That said, for a 12 TB database, we could use fewer stripes, assuming reasonably high backup compression ratio. Here is the abbreviated BACKUP DATABASE command (omitting most stripes), with parameters optimized for backup to URL as described in Backing up a VLDB to Azure Blob Storage: [code language="sql"] BACKUP DATABASE VLDB01 TO URL = 'https://storageaccount.blob.core.windows.net/backup/vldb01_stripe1.bak', ... URL = 'https://storageaccount.blob.core.windows.net/backup/vldb01_stripe64.bak' WITH COMPRESSION, CHECKSUM, MAXTRANSFERSIZE = 4194304, BLOCKSIZE = 65536, STATS = 1; [/code] On a GS5 VM (32 cores, 448 GB RAM, 20000 Mbps network bandwidth), this took nearly 17 hours: [code language="text"] ... 99 percent processed. Processed 848 pages for database 'VLDB01', file 'VLDB01' on file 1. Processed 206278432 pages for database 'VLDB01', file 'VLDB01_File2' on file 1. Processed 206014904 pages for database 'VLDB01', file 'VLDB01_File1' on file 1. Processed 205881752 pages for database 'VLDB01', file 'VLDB01_File3' on file 1. Processed 206835160 pages for database 'VLDB01', file 'VLDB01_File4' on file 1. Processed 206409496 pages for database 'VLDB01', file 'VLDB01_File5' on file 1. Processed 206307512 pages for database 'VLDB01', file 'VLDB01_File6' on file 1. Processed 206213568 pages for database 'VLDB01', file 'VLDB01_File7' on file 1. Processed 205815696 pages for database 'VLDB01', file 'VLDB01_File8' on file 1. 100 percent processed. Processed 2 pages for database 'VLDB01', file 'VLDB01_log' on file 1. BACKUP DATABASE successfully processed 1649757370 pages in 60288.832 seconds (213.783 MB/sec). [/code] While for some applications this might be acceptable (after all, this is a potentially infrequent full backup; more frequent differential and transaction log backups would take less time), a restore of this database would take at least as long as the full backup took, i.e. close to a day. There are few applications that can accept a downtime that long. Let’s see what would happen if instead of a streaming backup, we use a file-snapshot backup, available if database files are stored directly in Azure Blob Storage. [code language="sql"] BACKUP DATABASE VLDB01 TO URL = 'https://standardstorageaccount.blob.core.windows.net/backup/VLDB01_snapshot1.bak' WITH FILE_SNAPSHOT; [/code] [code language="text"] Processed 0 pages for database 'VLDB01', file 'VLDB01' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File1' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File2' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File3' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File4' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File5' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File6' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File7' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File8' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_log' on file 1. BACKUP DATABASE successfully processed 0 pages in 0.151 seconds (0.000 MB/sec). [/code] This backup completed in 16 seconds! This is possible because no data movement has actually occurred, unlike in the streaming backup case. Instead, a storage snapshot was created for each database file, with SQL Server ensuring backup consistency for the entire database by briefly freezing write IO on the database. The backup file referenced in the BACKUP DATABASE statement above (VLDB01_snapshot1.bak) stores pointers to these snapshots, and other backup metadata. We should note that the database was idle during this backup. When taking the same backup on the same GS5 VM running queries with 60% CPU utilization, and reading ~1 GB/s from database files in blob storage, the backup took 1 minute 29 seconds. The backup took 1 minute 55 seconds when a bulk load into a table was running. But even though file-snapshot backups get slower on a busy instance, they are still orders of magnitude faster than streaming backups. Let’s try restoring this backup on another SQL Server instance on a different VM: [code language="sql"] RESTORE DATABASE VLDB01 FROM URL = 'https://standardstorageaccount.blob.core.windows.net/backup/VLDB01_snapshot1.bak' WITH MOVE 'VLDB01' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01.mdf', MOVE 'VLDB01_File1' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File1.ndf', MOVE 'VLDB01_File2' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File2.ndf', MOVE 'VLDB01_File3' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File3.ndf', MOVE 'VLDB01_File4' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File4.ndf', MOVE 'VLDB01_File5' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File5.ndf', MOVE 'VLDB01_File6' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File6.ndf', MOVE 'VLDB01_File7' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File7.ndf', MOVE 'VLDB01_File8' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File8.ndf', MOVE 'VLDB01_log' TO 'https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_log.ldf', RECOVERY, REPLACE; [/code] [code language="text"] Processed 0 pages for database 'VLDB01', file 'VLDB01' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File1' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File2' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File3' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File4' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File5' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File6' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File7' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_File8' on file 1. Processed 0 pages for database 'VLDB01', file 'VLDB01_log' on file 1. RESTORE DATABASE successfully processed 0 pages in 1.530 seconds (0.000 MB/sec). [/code] This restore completed in 8 seconds on an idle instance. For restore, concurrent load does not affect timing as much as it does for backup. On a busy instance with the same CPU and storage utilization as above, restore completed in 10 seconds. During restore, no data movement is required if the same storage account is used for the restored database, even though a new set of database files (blobs) is created. Azure Blob Storage ensures that the initial content of these new blobs is identical to the content of the backup snapshots. You might notice that the output from the backup and restore commands above looks a little unusual, with the page and throughput statistics reported as zeroes. This is because SQL Server does not have them. The backup and restore are actually performed at the Azure Blob Storage level, with SQL Server calling blob storage API to create storage snapshots of all database files as a part of backup, or create blobs from snapshots as a part of restore. You may also notice that the backup and restore times in the output are much shorter compared to the actual duration. This is because they are reported for a specific phase (data copy) of the overall backup/restore process, which for file-snapshot backup/restore is very quick, compared to the overall duration of the operation. See this blog by Nacho Alonso Portillo for details. These examples show that for a VLDB with files stored directly in Azure Blob Storage, backup/restore scenarios can be greatly simplified by using file-snapshot backups, in much the same way as SAN storage snapshots, if used correctly, simplify these scenarios for large on-premises databases.

Consistency checking

What about another task that is challenging with a VLDB, namely consistency checking (DBCC CHECKDB)? For a multi-TB database, the traditional on-premises solutions usually involve some combination of limited (PHYSICAL_ONLY) and/or partial (at the table/filegroup level) checks. This compromise is often required to complete the check within the available maintenance window, and without causing excessive resource contention between the check and the regular database workload. Sometimes, the check is done only on a replica (if using availability groups), or on a restored copy of the database. Neither is ideal: using an AG replica increases resource utilization on the replica, potentially affecting other workloads running on the replica and AG replication itself (not to mention that corruption could exist on some replicas but not on others). Checking a copy of the database restored on a separate instance requires a lengthy (hours, days) restore process, implying that corruption on the source database could potentially occur while this restore is in progress, and remain undetected until the next check. How does this look in Azure? We have just shown that the entire backup/restore process for a 12 TB database with files in Azure blob storage completed in a couple of minutes, even on a busy instance. This provides an elegant solution for checking consistency of a VLDB in Azure: restore a file-snapshot backup on a separate VM, and run consistency check against the restored copy of the database. This VM can be stopped most of the time to save costs, and only started periodically to run consistency checks. The size of this VM can be different from the size of the VM hosting the database, to strike the right balance between the VM cost and the duration of the check, which is determined by available compute resources. Most importantly, the checks will be done on a “fresh” copy of the database, minimizing the likelihood of corruption happening on the source database while the copy is being created.

Reporting and analytics

If we take a step back, we will realize that with file-snapshot backups, we effectively have a way to create a copy of the database in near real-time, within minutes or even seconds. This opens a possibility to use such a copy (or multiple copies, in a scale-out fashion) for reporting and analytics, to offload the part of these workloads that can tolerate some data latency from the primary database. Other scenarios may be possible as well, e.g. quickly creating an “as-of” copy of the VLDB for research and audit purposes.

Limitations

As promising as all this may look, before everyone starts deploying VLDBs with files stored directly in blob storage, we must describe the limitations. 1. Each Premium Storage account is limited to 35 TB, and all files for a database must reside in the same storage account. Using Standard storage with its 500 TB per-account limit, while supported, is usually not recommended for performance reasons. For the restore from file-snapshot backup to be fast, the files of the restored database must reside in the same storage account as the files of the source database. Restoring to a different storage account is possible, but requires a size-of-data operation in blob storage, which for a VLDB could take hours or days. This implies that for practical purposes, there is a limited number of database copies that can be created using this approach. For example, for a 12 TB database used in the examples above, we can create at most one other copy in the same storage account via file-snapshot backup/restore, and for databases larger than ~17 TB, no additional copies may be created. Attempting to restore a file-snapshot backup when there is not enough space in the storage account results in an error: [code language="text"] Msg 5120, Level 16, State 152, Line 127 Unable to open the physical file "https://premiumstorageaccount.blob.core.windows.net/mssqldata/VLDB01_File3.ndf". Operating system error 1450: "1450(Insufficient system resources exist to complete the requested service.)". Msg 3013, Level 16, State 1, Line 127 RESTORE DATABASE is terminating abnormally. [/code] The mitigating factor here is that the snapshots taken for file-snapshot backups are not counted towards the 35 TB limit. As mentioned in the Scalability and Performance section of Premium Storage documentation, the snapshot capacity of a Premium Storage account is 10 TB. The extent to which this additional space is used depends on the volume of changes in the database between each pair of consecutive file-snapshot backups. This is analogous to the way space is consumed by sparse files used in SQL Server database snapshots (not to be confused with Azure Blob Storage snapshots used in file-snapshot backups). 2. Since we create database files directly in a storage account, we cannot take advantage of the Managed Disks capabilities. In particular, for databases using availability groups, we cannot guarantee that the different storage accounts behind each availability replica are on different storage scale units. Using the same storage scale unit for more than one replica creates a single point of failure. While the likelihood of storage scale unit failure is not high, there are known cases when database availability had been negatively affected due to this, before Managed Disks became available to avoid this issue. 3. Because database files are accessed over http, disk IO becomes a part of network traffic for all intents and purposes. This means that unlike with Azure VM disks, there is no local SSD cache to potentially improve storage performance. Additionally, the same network bandwidth is shared between the VM network traffic, and the database storage traffic, which can affect network intensive applications. Further, the VM limits on disk throughput and IOPS that are published in Azure VM documentation do not apply to SQL Server database files stored directly in blob storage, though the network bandwidth limits are still applicable. 4. There are additional considerations and limitations to consider, documented in File-Snapshot Backups for Database Files in Azure. That said, even with these limitations, managing a SQL Server VLDB in Azure can be greatly simplified if files are stored directly in blob storage, allowing the use of techniques described in this article. We strongly recommend considering this option for your next project involving a SQL Server VLDB in Azure.

Recent Updates to setting up SQL Server Availability Groups in Azure VM with AAD Domain Services

Rajesh Setlem — Mon, 30 Oct 2017 16:07:30 GMT

Reviewed by: Dimitri Furman, Kun Cheng This blog is an extension to the one that was published in February 2017 . In these eight months there has been some notable improvements to AAD Domain Services (AAD DS) and Azure Virtual Network (VNET) capabilities. Now, it's even more easier and convenient to leverage AAD DS for setting up SQL Server Availability Groups (AG) in Azure VMs. As a recap, following scenarios were covered in the blog linked above: Scenario 1: Enabling AAD Domain Services in Classic Virtual Network, deploying two SQL Server 2016 Classic VMs (Windows Server 2012), and then setting up AG Scenario 2: Leveraging AAD Domain Services enabled in Classic Virtual Network from an ARM virtual network by adding an ARM based SQL Server 2016 VM as a replica to the existing AG. Summary of recent improvements to AAD DS and new VNET capability:

AAD DS is generally available in many Azure regions now. Check here to find out availability by region (AAD DS is under Security + Identity).
General availability of AAD DS support for Azure Resource Manager (ARM) based VNET. This was eagerly expected by customers. You can read more about it here.
General availability of AAD DS in the new Azure Portal (portal.azure.com).
Public preview for Global VNET Peering. VNET peering is a powerful feature allowing you to peer virtual networks in Azure. Peering VNETs in the same region is generally available, and the ability to peer VNETs across different Azure regions is in preview. You can read more about it here.

With these improvements, scenarios covered in previous blog can be done in a different way. A new scenario is enabled as well. Scenario 1: This can be implemented in ARM only mode with no need to create any classic resources. Good news is it can be done in new Azure Portal (portal.azure.com). Scenario 2: Since AAD DS can be enabled on ARM VNET now, you can peer it with another ARM VNET, or with a Classic VNET if there is a specific need. Scenario 3: This is a new scenario possible with Global VNET Peering to connect an Azure region designated as your DR with your primary Azure region. Asynchronous AG replica in your Azure DR region would connect to primary replica over VNET Peering and provide failover capabilities in the event of disaster. We tried this setup and it works just as expected. Go ahead and explore these new capabilities and scenarios and let us know if you have any questions or comments.

SQLCAT @PASS Summit 2017

Kun Cheng (SQLCAT) — Mon, 30 Oct 2017 08:57:31 GMT

Are you coming to the PASS Summit 2017 in Seattle? SQLCAT will be in full force at the PASS Summit 2017, and we will also bring along our colleagues from the broader AzureCAT team, including the newly formed DataCAT team.

SQLCAT Sessions

SQLCAT sessions are unique. We bring in real customer stories and present their deployments, architectures, challenges and lessons learned. This year at the PASS Summit, we will have 4 sessions – each one filled with rich learnings from real world customer deployments. SQL Server on Linux: DBA Focused Lessons Learned from Early Deployments Building a Graph Database Application with SQL Server 2017 and Azure SQL Database Real-world SQL Server R Services How Microsoft Sales Evolved from a Monolithic On-Prem Solution to a Scaleout Solution in Azure

SQL Clinic

Have a technical question, a troubleshooting challenge, want to have an architecture discussion, want to find ways to move your data solutions to Azure, or want to find best ways to upgrade your SQL Server? SQL Clinic is the place you want to be at. SQL Clinic is the hub of technical experts from SQLCAT, Tiger team, SQL PG, CSS and others. Whether you want a makeover of your deployment or need an open-heart surgery, the experts at SQL Clinic will have the right advice for you. Find all your answers in one place!

Bonus

AzureCAT is hiring for various opportunities across the team, including data experts in the areas of Oracle, Teradata, open source data solutions, big data technologies, data science and related areas. Come talk to a CAT!

And More …

That’s not all. SQLCAT will be involved in many more events and customer conversations during the Summit. If you have a suggestion on how we can make your experience at the PASS Summit more effective and more productive, don’t hesitate to leave a note. Thanks, and see you all at the PASS Summit 2017 in Seattle. You are coming, right?