UK Data Platform Team Blog

DfE and Microsoft bring Open Data to life with Power BI

Nicholas Revell — Thu, 22 Feb 2018 09:23:34 GMT

Microsoft and the Department for Education (DfE) have recently collaborated to showcase how open data published by the UK government can be modelled and visualised with Power BI to bring their data to life and make it more engaging. School Workforce Census Since 2010, the DfE has published their annual School Workforce Census data. This includes statistics on the number and characteristics of teachers, teaching assistants and other non-classroom-based school support staff in all schools in England. The data is published as part of their Open Data Strategy and includes a copy of the raw data as spreadsheets in ODS or Microsoft Excel format. This is accompanied with a static PDF report, prepared by the DfE's statisticians, that summarises and describes the key findings. To understand why we undertook this project and the benefits we wanted to achieve, let's briefly consider the current state of Open Government Data in the UK. What is Open Government Data? The UK government is a strong advocate and practitioner of making public sector information available to the public as open data. This means that it is: accessible to all at no more than the cost of reproduction; available in a digital, machine readable format; and free of restriction on its use or redistribution (under the Open Government Licence terms). A key driver for publishing this data is transparency. This includes making data available to allow members of the public to make informed choices about public services, holding central and local government accountable for spending and outcomes, and enabling the private sector to develop innovative new services. In their 2012 Open Data Whitepaper, The government "committed to making open data an effective engine of economic growth, social wellbeing, political accountability and public service improvement". To date, they have published over 40,000 datasets on the Data.Gov.UK site, adopted the principles of the Open Data Charter, and the UK is now ranked first in the world by The Open Data Barometer for readiness, implementation and impact, and 2^nd in the Global Open Data Index. Who uses Open Government Data? The government provides statistics on the number of views and downloads for published datasets and views for the publishers of that data. This provides us with some insights, but it cannot tell us anything about the profiles of the persons or organisations accessing the data or what they did with it. The government's open data principles focus on the quality, consistency, availability and open access of data. Open data should be in a re-usable format, which means it is structured and stored in a machine-readable form based on open standards. This is typically one or more CSV files depending on the volume of the data, which restricts its usage to those with the tools and skills to analyse the data. Alternatively, it is made available through Applications Programming Interfaces (APIs). It is reasonable to assume, therefore, that the primary users are predominately government departments themselves, researchers, journalists, commercial organisations, or software developers. Most individuals and organisations are dependent on the private sector to develop services and applications on top of open data to make it easy to use and relevant for their needs. There is a growing industry of organisations and start-ups building these solutions and the government will actively promote apps that utilise open data on its open data portal. Transport for London is a great example of the impact that open data can have, while also recognising that this success has been built on a determined effort over 10 years. What were the objectives of this project? In the words of Isi Avbulimen, Statistical Officer at the DfE, who worked on the project… "The School Workforce Power BI dashboard is a joint project between Teachers Analysis Division at the Department for Education and Microsoft. It is an interactive dashboard that will allow users to visualise National, Regional and LA Level data in a colourful, interactive, and graphic form utilising a user-friendly format. The dashboard is based entirely on published data from the School Workforce Census (SWC). The data is currently published within a large, complex non-machine readable Excel file comprising of over 20,000 rows, making the data largely inaccessible to the general public who do not have expertise in data manipulation. This dashboard will allow users to pull out the information they are interested in and perform their own analysis. The dashboard will allow users to examine time series data for a range of different measures at the click of a button, e.g. users can easily examine how the headcount of teachers within the schools of York has varied over time." How did we prepare the data for analysis? We performed several steps in Power BI to get the data into the correct shape before we created the visualisations. These included loading and consolidating multiple year files into single tables, joining it with additional school data from 'Get Information About Schools', conforming the data to ensure consistency, converting the flat files back into a data model with dimension and fact tables, and adding some calculated measures and shape map files. Screenshot of one of the source files downloaded from Data.Gov.UK We also scaled back the scope of the project. A decision was taken to exclude all data for 2010 because there were too many structural differences with other years. We also removed the school level data because it is subject to data suppression to prevent the identification of individuals. This meant that school level aggregations could be slightly different from local authority level aggregations and might lead to confusion. An extension of the project could be to republish this data separately. Screenshot of one of the Power BI report pages What lessons did we learn? The opportunity to analyse trends and changes over time are key insights that would not have been possible without going through the process of consolidating multiple files. Combining data from multiple sources (e.g. Get Information About Schools) would not have been practical without transforming the data into a related data model. Conversely, building data models provides the opportunity to combine data sources from multiple public sector bodies, a key objective of the Government Transformation Strategy. The latest UK assessment by the European Data Portal identified issues with users' awareness of open data and understanding the value of these datasets. This project took a single dataset and reimagined how it could have been presented (alongside the open data) to lower the barriers to its use. There are real opportunities to make open data easier to find, access and use. These include: predesigned reports that allow people to get started immediately and interactively query the data; agile tools that empower analysts to explore the data and build their own reports; and natural language interfaces and AI-powered agents that will query the data on your behalf. The latest generation of self-service visualisation and analytics tools, such as Power BI, will provide all these capabilities. For many requirements this will be a more cost-effective and viable option than directly analysing the open data files or developing an application to do it. Finally… Click here to see the report. Try selecting a region from the map or searching for your local authority.

Usability and Security – an oxymoron?

Kalai Elumalai — Fri, 20 Jan 2017 14:18:32 GMT

by Kalai Elumalai - Data Platform Solution Architect

This article attempts to answer,

Why Enterprise Architects should care about usability and security controls?
How the latest features in SQL Server 2016 (and Azure SQL Database) can help to address those challenges?

Usability and Security:

I recently attended a course (one of the best I've been to) on "People and Security" at Oxford University given by Professor Angela Sasse who has done extensive research in the field of usability and security. Her main research interest is the design and implementation of innovative technologies that are fit for purpose, value for money, and improve productivity and well-being of individuals and society. This course has prompted me to put into context some of the security implementations in Data Warehouse architectures to meet complex security requirements (both internal and external) from a usability perspective.

The process of building a secure system is pivotal for its effectiveness. The process is the methodology that is used to elicit the security needs, assess those needs, define the security policies and implement the countermeasures. While designing the countermeasures, the designers should consider that security is not the primary goal of the users and organisation. For example, a data analyst's primary goal is to extract insight from information assets. The role of security is a supporting function, that is, to protect the assets and the operations the system users perform on those assets that are part of their core business activity. [1]

Zurko and Simon [2] were among the first to point out that current security mechanisms create additional overhead on all stakeholders: system administrators, application developers, and end users. This overhead puts additional onus on these stakeholders with the amount of work and additional complexity involved in securing the systems and in using the systems while adhering to the security controls in place.

The crux of the issue is that the implementation of security methods should not put an impediment on the usability of the system, for e.g. a database administrator looking to implement a security policy should introduce security measures that are resilient enough to allow changes to security policies and also be easy to implement. The ability to technically implement a security measure and the effort required by the users to adhere to the security controls plays a crucial role in the compliance decision. [3] "…The 'pain tolerance' - the amount of extra effort an individual is prepared to make for no personal gain – is what we call the Compliance Budget. The limit of the Compliance Budget is referred to as the compliance threshold; this being the point at which the individual no longer has the will to comply with official requirements."

Also, when you view this from a Data Protection Act perspective, the Information Commissioner's office has defined a set of data protection principles providing guidance to manage the security of the data (personal, sensitive etc.) organisations hold. Principle 7 suggests that organizations must have the right technical and physical security underpinned by information security policies and procedures. With the proposed changes to the EU Directive on Privacy and the introduction of Privacy by Design (Article 23), it is no longer only a responsibility of a siloed team or department in an organisation that needs to care about privacy. It is gaining more focus now than ever before and the report "Privacy and Data Protection by Design" [4] provides guidance to build in privacy in the lifecycle of system development and not as a bolt on. Organisations must, therefore, need to be better equipped with capabilities to effectively implement information security measures and become compliant with the regulations.

In this section, I'm going to discuss about how one of the security requirement was implemented in an Enterprise Data Warehouse project that I've worked on recently. To comply with the data protection act and the organisation's information security policy framework, data was classified at attribute level by different groupings such as highly sensitive, confidential, financially sensitive etc. Based on the data model, data classification and the security policy was defined during the design phase drawing guidance from the organisations information classification standards.

At the time of development, the chosen data platform did not have out of the box capability to protect sensitive data attributes. Examples of such data include customer sensitive attributes such as date of birth and email address; financial information such as invoice amount, debt amount, credit or debit card type etc. This sensitive information was stored in the database and as per the organisation information security and privacy policy, we wanted this information to be accessible only to specific roles on a need-to-know basis. Therefore, we wanted to mask certain attributes partially or sometimes fully based on the users role. We opted to build a bespoke solution.

The bespoke solution was to create views on top of the underlying tables. For example, entities whose attributes fall into multiple data classification types, will be segregated into separate views. In this way, only the data items allowed by a specific role will be able to access it. This solution met our security requirements and was implemented.

Before looking at the challenges of this bespoke solution, let's look at few business scenarios which will be useful for our discussion.

The data management team in the business, which supports the customer insight team and campaign management team, wanted to run some data quality rules on the completeness/conformance of some of the attributes which are currently deemed as sensitive. For instance: from a marketing perspective, we would like to check whether a field has an email address such as abc@xyz.com. The analyst doesn't necessarily need to view the email address and all they care about is whether the column has any value.
The customer segmentation team may not need access to the date of birth of the customer – they only need the year of birth to understand the age band of their customers.

Let's look at the downside of implementing such a bespoke solution.

All or nothing: In the above examples, the end user (campaign data planner, customer insight analysts) either gets full access to the underlying data or gets no access to all.
It becomes very impractical to anticipate at the time of design for all combinations of attributes used by a role, especially when roles are based on the business functions. For instance: In this scenario, after few months of go-live, the Operational Insight team wanted access to some of the financial data to understand the debt position and allocate resources between customer service and credit handling, which was not anticipated during the design phase by the design team.
The rationale and knowledge of those database views are lost when the delivery team and business transformation team move on (either move to a different project or in case of third party system integrators scenario the resources are released to different assignments). The security logic is hidden in multitude of views and are susceptible to manual errors too.
When there is a change to the view to be made, you need a resource who has the knowledge of those views to make the change and DBA team to implement the change in Dev/Test/Pre-Prod and Production. Multiple teams, especially with domain knowledge, involved which means more red tape, time and cost. This will allow the end users to request for access to all data rather than going through the complex process of implementing the change in the view to gain access to the required data.

The security controls that are difficult to change and adhering to those controls means more bureaucratic processes. This means most of the time is wasted in non-productive efforts and all of which would compete against the primary business goal taking up the compliance budget [3].

How does Dynamic Data Masking which is a built-in feature in SQL Server 2016 (and Azure SQL Database) help to secure sensitive data?

In simple terms, this feature masks the data that is returned from the DB engine. This new security programmability feature limits access to sensitive data by obfuscating query results. This feature offers:

Sensitive Data Protection: Protects sensitive data against unauthorized access by built-in or custom masking rules. Privileged users can view the unmasked data e.g. using DDM to mask sensitive information in an HR database and only authorized users can access the unmasked data.
Regulatory Compliance: Enabler for applications to meet regulatory compliance needs such as PCI DSS, Sarbanes-Oxley, HIPAA and other regulations which demand protection of sensitive financial and personal data.
Agility and Transparency: The biggest benefit of this feature is its ease of implementation and ease of use. The data is masked on the fly and the underlying data remains intact. And it is completely transparent to the application.

In the example above, the Email address column of the Employee table has been masked using a simple set of T-SQL statements based on the security policy. The Data Quality analyst can access the email address column and still can perform their primary tasks without compromising on security compliance. Whilst, the privileged user "admin1", which could be used by applications such as a campaign management tool, can still access the unmasked email address and trigger email campaigns.

From implementation perspective, all the DBA or the security admin should do is,

define the security policy for the sensitive attributes in the entities.
apply those policies in T-SQL over the sensitive columns.
these data masking policies will then obfuscate the sensitive data in the query results when accessed by non-privileged users.

If I were doing the project I discussed above, I would be using SQL Server 2016 DDM. It would have not only simplified the implementation but also would have provided an effective governance with a central data masking policy and the agility required to meet the changing business data access needs.

Find out more information on "how to":

References:

[1] Usable security. Why do we need it? How do we get it?; M. Angela Sasse and Ivan Flechais

[2] Mary E. Zurko and Richard T. Simon, "User-Centered Security," New Security Paradigms Workshop 1997[3] A. Beautement, M. A. Sasse & M. Wonham, M. (2008).

[3] The Compliance Budget: Managing security behaviour in organisations. In Proceedings of the 2008 Workshop on New Security Paradigms (2008)

[4] Privacy and Data Protection by Design – from policy to engineering December 2014. European Union Agency for Network and Information Security

Using a ‘User Query Store’ for Exceptions and Timeouts

BenCooper — Thu, 29 Sep 2016 13:59:11 GMT

by Mike Boswell – Data Platform Solution Architect. In the previous post, I mentioned that as a Data Solution Architect we regularly work with clients to pilot new data platforms and with SQL Server 2016 an increasing number of clients are looking to modernise those SQL Platforms. We spoke of how you could create a 'User Query Store' to retain test results. More recently I had to investigate queries which were timing out or had exceptions. These are usually the priority in getting the applications working before we start performance tuning. Within Query Store these are deemed to be "Aborted" or "Exceptions". -- Exception Queries SELECT qt.query_sql_text, q.query_id, qt.query_text_id, p.plan_id, rs.runtime_stats_id, rsi.start_time, rsi.end_time, rs.avg_physical_io_reads, rs.avg_rowcount, rs.count_executions, rs.execution_type_desc, p.query_plan, so.userobjectname, so.userobjecttype FROM dbo.user_query_store_query_text qt JOIN dbo.user_query_store_query q ON qt.query_text_id = q.query_text_id JOIN dbo.user_query_store_plan p ON q.query_id = p.query_id JOIN dbo.user_query_store_runtime_stats rs ON p.plan_id = rs.plan_id JOIN dbo.user_query_store_runtime_stats_interval rsi ON rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id JOIN dbo.user_sys_objects so on so.userobjectid = q.object_id where rs.execution_type=4 order by count_executions desc -- Aborted Queries SELECT qt.query_sql_text, q.query_id, qt.query_text_id, p.plan_id, rs.runtime_stats_id, rsi.start_time, rsi.end_time, rs.avg_physical_io_reads, rs.avg_rowcount, rs.count_executions, rs.execution_type_desc, p.query_plan, so.userobjectname, so.userobjecttype FROM dbo.user_query_store_query_text qt JOIN dbo.user_query_store_query q ON qt.query_text_id = q.query_text_id JOIN dbo.user_query_store_plan p ON q.query_id = p.query_id JOIN dbo.user_query_store_runtime_stats rs ON p.plan_id = rs.plan_id JOIN dbo.user_query_store_runtime_stats_interval rsi ON rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id JOIN dbo.user_sys_objects so on so.userobjectid = q.object_id where rs.execution_type=3 order by count_executions desc The above queries will output the top queries that have exceptions or aborts along with the SHOWPLAN. As we have imported the sys.objects we are able to reference the object which contains the query. An object id of '0' points to an Ad-Hoc query. The only challenge with these queries is that you are unable to get a full count of executions where the query has timed out. The reason is that the query could have a different runtime_stats_id and therefore has a separate entry in the dbo.user_query_store_runtime_stats. Therefore, to get a total count of the number of executions, which timed out, you need to group by query_id and run the script below. -- Exception Queries Grouped by query id. SELECT qt.query_sql_text, q.query_id, sum(rs.count_executions) sum_of_executions, so.userobjectname, so.userobjecttype FROM dbo.user_query_store_query_text qt JOIN dbo.user_query_store_query q ON qt.query_text_id = q.query_text_id JOIN dbo.user_query_store_plan p ON q.query_id = p.query_id JOIN dbo.user_query_store_runtime_stats rs ON p.plan_id = rs.plan_id JOIN dbo.user_query_store_runtime_stats_interval rsi ON rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id JOIN dbo.user_sys_objects so on so.userobjectid = q.object_id where rs.execution_type=4 group by q.query_id, qt.query_sql_text, so.userobjectname, so.userobjecttype order by sum(count_executions) desc -- Aborted Queries Grouped by query id. SELECT qt.query_sql_text, q.query_id, sum(rs.count_executions) sum_of_executions, so.userobjectname, so.userobjecttype FROM dbo.user_query_store_query_text qt JOIN dbo.user_query_store_query q ON qt.query_text_id = q.query_text_id JOIN dbo.user_query_store_plan p ON q.query_id = p.query_id JOIN dbo.user_query_store_runtime_stats rs ON p.plan_id = rs.plan_id JOIN dbo.user_query_store_runtime_stats_interval rsi ON rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id JOIN dbo.user_sys_objects so on so.userobjectid = q.object_id where rs.execution_type=3 group by q.query_id, qt.query_sql_text, so.userobjectname, so.userobjecttype order by sum(count_executions) desc The grouped output now allows you to see which queries you should focus on the most:

Conclusion

Query Store or a 'User Query Store' provides a very quick way to fault find timeouts and exceptions in your application. Previously you would have had to set-up a Profiler or Extended Events Trace to fault find. Query Store runs in the background and collects information for you, giving instant insight to your application performance.

Exporting SQL Server 2016 Query Store

BenCooper — Tue, 06 Sep 2016 07:47:04 GMT

By Mike Boswell – Data Platform Solution Architect. As a Data Solution Architect, we regularly work with clients to pilot new data platforms and with SQL Server 2016 an increasing number of clients are looking to modernise their SQL Server, Oracle and DB2 Platforms. During the piloting phase we will run performance labs and pilot new features. Now one of these new features which gives us a lot of benefit, both in these labs, and in production is the Query Store (https://msdn.microsoft.com/en-GB/library/dn817826.aspx) in SQL Server 2016. However, there is one "query store challenge" with this in both a Dev, Test and Production environments. During Dev/Test the database is often restored after a test and we needed a way to keep the query store data for analysis. Likewise, production would like to provide feedback to development on performance enhancements. Microsoft have yet to produce a utility which will extract query store data, into user database, and this blog explains how you can do this to gain insight into your database performance. To achieve this, it led me down the path of having to look to export the data from Query Store, build Primary Keys and Columnstore Indexes (columnstore indexeshttps://msdn.microsoft.com/en-us/library/gg492088.aspx).. Primary Keys, created as clustered indexes, to support point lookups and Non Clustered Columnstore Indexes to support aggregations. Once we had this, we could back up the database and share the output with between dev and test teams now that the performance metric data is safely captured. Note: The rest of this blog is based on the WorldWideImporters database from https://github.com/Microsoft/sql-server-samples/releases/tag/wide-world-importers-v1.0.

Setting up Query Store

For details on setting up Query Store, in SQL Server 2016, please refer to "Monitoring Performance By Using the Query Store and a Channel 9 vid at https://channel9.msdn.com/Shows/Data-Exposed/Query-Store-in-SQL-Server-2016. Once setup and you have workload running then you should start seeing data returned by running the following DMVs:

and start using Query Store as documented.

Setting up a User Query Store

We need to set up a user database to be able to pull down the data from the Query Stored and the script below:

Create a Database
Select from the WorldWideImporters DB (alter to your DB)
Import sys.objects table
Create Primary Keys
Create ColumnStore Indexes to support aggregation of data

/* DISCLAIMER: This code is not supported under any Microsoft standard support program or service. This code and information are provided "AS IS" without warranty of any kind, either expressed or implied. The entire risk arising out of the use or performance of the script and documentation remains with you. Furthermore, Microsoft or the author shall not be liable for any damages you may sustain by using this information, whether direct, indirect, special, incidental or consequential, including, without limitation, damages for loss of business profits, business interruption, loss of business information or other pecuniary loss even if it has been advised of the possibility of such damages. */ create database [UserQueryStore] GO ALTER DATABASE [UserQueryStore] SET RECOVERY SIMPLE GO USE [UserQueryStore] GO CREATE TABLE [dbo].[user_sys_objects] ( userobjectname nvarchar(128) NOT NULL, userobjectid int NOT NULL, userobjecttype char(2) NOT NULL ) ON [PRIMARY] GO select * into [dbo].[user_query_store_plan] from [WideWorldImporters].[sys].[query_store_plan]; select * into [dbo].[user_query_store_query] from [WideWorldImporters].[sys].[query_store_query]; select * into [dbo].[user_query_store_query_text] from [WideWorldImporters].[sys].[query_store_query_text]; select * into [dbo].[user_query_store_runtime_stats] from [WideWorldImporters].[sys].[query_store_runtime_stats]; select * into [dbo].[user_query_store_runtime_stats_interval] from [WideWorldImporters].[sys].[query_store_runtime_stats_interval]; insert into [dbo].[user_sys_objects]([userobjectname],[userobjectid],[userobjecttype]) select [name], [object_id],[type] from [WideWorldImporters].[sys].[objects]; INSERT INTO [dbo].[user_sys_objects] ([userobjectname] ,[userobjectid] ,[userobjecttype]) VALUES ('Ad-Hoc Query', 0,'AD') GO --Add Primary Keys and ColumnStore Indexes ALTER TABLE dbo.user_query_store_plan ADD CONSTRAINT PK_user_query_store_plan PRIMARY KEY CLUSTERED ( plan_id ) WITH( STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] GO ALTER TABLE dbo.user_query_store_query ADD CONSTRAINT PK_user_query_store_query PRIMARY KEY CLUSTERED ( query_id ) WITH( STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] GO ALTER TABLE dbo.user_query_store_query_text ADD CONSTRAINT PK_user_query_store_query_text PRIMARY KEY CLUSTERED ( query_text_id ) WITH( STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] GO ALTER TABLE dbo.user_query_store_runtime_stats ADD CONSTRAINT PK_user_query_store_runtime_stats PRIMARY KEY CLUSTERED ( runtime_stats_id ) WITH( STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] GO ALTER TABLE dbo.user_query_store_runtime_stats_interval ADD CONSTRAINT PK_user_query_store_runtime_stats_interval PRIMARY KEY CLUSTERED ( runtime_stats_interval_id ) WITH( STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] GO --ColumnStores CREATE NONCLUSTERED COLUMNSTORE INDEX [user_query_store_plan-CS] ON [dbo].[user_query_store_plan] ( [plan_id], [query_id], [plan_group_id], [engine_version], [compatibility_level], [is_online_index_plan], [is_trivial_plan], [is_parallel_plan], [is_forced_plan], [is_natively_compiled], [force_failure_count], [last_force_failure_reason], [last_force_failure_reason_desc], [count_compiles], [initial_compile_start_time], [last_compile_start_time], [last_execution_time], [avg_compile_duration], [last_compile_duration] )WITH (DROP_EXISTING = OFF, COMPRESSION_DELAY = 0) GO CREATE NONCLUSTERED COLUMNSTORE INDEX [user_query_store_query_text-CS] ON [dbo].[user_query_store_query_text] ( [query_text_id], [is_part_of_encrypted_module], [has_restricted_text] )WITH (DROP_EXISTING = OFF, COMPRESSION_DELAY = 0) GO CREATE NONCLUSTERED COLUMNSTORE INDEX [user_query_store_query-cs] ON [dbo].[user_query_store_query] ( [query_id], [query_text_id], [context_settings_id], [object_id], [is_internal_query], [query_parameterization_type], [query_parameterization_type_desc], [initial_compile_start_time], [last_compile_start_time], [last_execution_time], [last_compile_batch_offset_start], [last_compile_batch_offset_end], [count_compiles], [avg_compile_duration], [last_compile_duration], [avg_bind_duration], [last_bind_duration], [avg_bind_cpu_time], [last_bind_cpu_time], [avg_optimize_duration], [last_optimize_duration], [avg_optimize_cpu_time], [last_optimize_cpu_time], [avg_compile_memory_kb], [last_compile_memory_kb], [max_compile_memory_kb], [is_clouddb_internal_query] )WITH (DROP_EXISTING = OFF, COMPRESSION_DELAY = 0) ON [PRIMARY] GO CREATE NONCLUSTERED COLUMNSTORE INDEX [user_query_store_runtime_stats_interval-cs] ON [dbo].[user_query_store_runtime_stats_interval] ( [runtime_stats_interval_id], [start_time], [end_time] )WITH (DROP_EXISTING = OFF, COMPRESSION_DELAY = 0) ON [PRIMARY] GO CREATE NONCLUSTERED COLUMNSTORE INDEX [user_query_store_runtime_stats-cs] ON [dbo].[user_query_store_runtime_stats] ( [runtime_stats_id], [plan_id], [runtime_stats_interval_id], [execution_type], [execution_type_desc], [first_execution_time], [last_execution_time], [count_executions], [avg_duration], [last_duration], [min_duration], [max_duration], [stdev_duration], [avg_cpu_time], [last_cpu_time], [min_cpu_time], [max_cpu_time], [stdev_cpu_time], [avg_logical_io_reads], [last_logical_io_reads], [min_logical_io_reads], [max_logical_io_reads], [stdev_logical_io_reads], [avg_logical_io_writes], [last_logical_io_writes], [min_logical_io_writes], [max_logical_io_writes], [stdev_logical_io_writes], [avg_physical_io_reads], [last_physical_io_reads], [min_physical_io_reads], [max_physical_io_reads], [stdev_physical_io_reads], [avg_clr_time], [last_clr_time], [min_clr_time], [max_clr_time], [stdev_clr_time], [avg_dop], [last_dop], [min_dop], [max_dop], [stdev_dop], [avg_query_max_used_memory], [last_query_max_used_memory], [min_query_max_used_memory], [max_query_max_used_memory], [stdev_query_max_used_memory], [avg_rowcount], [last_rowcount], [min_rowcount], [max_rowcount], [stdev_rowcount] )WITH (DROP_EXISTING = OFF, COMPRESSION_DELAY = 0) GO

Once we have the data in our user database we can take the queries mentioned in "Monitoring Performance By Using the Query Store" and alter them to point to our user tables. The queries include the user_sys_objects table to be able to display the name of the object that the query is part of. If the query is part of an ad-hoc query then the object name will display as ad-hoc.

/* /* DISCLAIMER: This code is not supported under any Microsoft standard support program or service. This code and information are provided "AS IS" without warranty of any kind, either expressed or implied. The entire risk arising out of the use or performance of the script and documentation remains with you. Furthermore, Microsoft or the author shall not be liable for any damages you may sustain by using this information, whether direct, indirect, special, incidental or consequential, including, without limitation, damages for loss of business profits, business interruption, loss of business information or other pecuniary loss even if it has been advised of the possibility of such damages. */ USE [UserQueryStore] GO --Last N queries that were executed on the database SELECT TOP 10 qt.query_sql_text, q.query_id, qt.query_text_id, p.plan_id, rs.last_execution_time, so.userobjectname, so.userobjecttype FROM dbo.user_query_store_query_text qt JOIN dbo.user_query_store_query q ON qt.query_text_id = q.query_text_id JOIN dbo.user_query_store_plan p ON q.query_id = p.query_id JOIN dbo.user_query_store_runtime_stats rs ON p.plan_id = rs.plan_id JOIN dbo.user_sys_objects so on so.userobjectid = q.object_id ORDER BY rs.last_execution_time DESC GO --Count of executions for each query SELECT q.query_id, qt.query_text_id, qt.query_sql_text, SUM(rs.count_executions) AS total_execution_count, so.userobjectname, so.userobjecttype FROM dbo.user_query_store_query_text qt JOIN dbo.user_query_store_query q ON qt.query_text_id = q.query_text_id JOIN dbo.user_query_store_plan p ON q.query_id = p.query_id JOIN dbo.user_query_store_runtime_stats rs ON p.plan_id = rs.plan_id JOIN dbo.user_sys_objects so on so.userobjectid = q.object_id GROUP BY q.query_id, qt.query_text_id, qt.query_sql_text,so.userobjectname, so.userobjecttype ORDER BY total_execution_count DESC GO -- Top N queries with longest average execution time within last hour SELECT TOP 10 qt.query_sql_text, q.query_id, qt.query_text_id, p.plan_id, getutcdate() as CurrentUTCTime, rs.last_execution_time, rs.avg_duration, so.userobjectname, so.userobjecttype FROM dbo.user_query_store_query_text qt JOIN dbo.user_query_store_query q ON qt.query_text_id = q.query_text_id JOIN dbo.user_query_store_plan p ON q.query_id = p.query_id JOIN dbo.user_query_store_runtime_stats rs ON p.plan_id = rs.plan_id JOIN dbo.user_sys_objects so on so.userobjectid = q.object_id --WHERE rs.last_execution_time > dateadd(hour, -1, getutcdate()) ORDER BY rs.avg_duration DESC GO --Top N queries that had the biggest average physical IO reads in last 24 hours, --with corresponding average row count and execution count SELECT TOP 10 qt.query_sql_text, q.query_id, qt.query_text_id, p.plan_id, rs.runtime_stats_id, rsi.start_time, rsi.end_time, rs.avg_physical_io_reads, rs.avg_rowcount, rs.count_executions, so.userobjectname, so.userobjecttype FROM dbo.user_query_store_query_text qt JOIN dbo.user_query_store_query q ON qt.query_text_id = q.query_text_id JOIN dbo.user_query_store_plan p ON q.query_id = p.query_id JOIN dbo.user_query_store_runtime_stats rs ON p.plan_id = rs.plan_id JOIN dbo.user_query_store_runtime_stats_interval rsi ON rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id JOIN dbo.user_sys_objects so on so.userobjectid = q.object_id --WHERE rsi.start_time >= dateadd(hour, -24, getutcdate()) ORDER BY rs.avg_physical_io_reads DESC GO -- Queries that recently regressed in performance -- The following query example returns all queries which execution time was -- doubled in last 48 hours. SELECT qt.query_sql_text, q.query_id, qt.query_text_id, p1.plan_id AS plan1, rsi1.start_time AS runtime_stats_interval_1, rs1.runtime_stats_id AS runtime_stats_id_1, rs1.avg_duration AS avg_duration_1, p2.plan_id AS plan2, rsi2.start_time AS runtime_stats_interval_2, rs2.runtime_stats_id AS runtime_stats_id_2, rs2.avg_duration AS plan2, so.userobjectname, so.userobjecttype FROM dbo.user_query_store_query_text qt JOIN dbo.user_query_store_query q ON qt.query_text_id = q.query_text_id JOIN dbo.user_query_store_plan p1 ON q.query_id = p1.query_id JOIN dbo.user_query_store_runtime_stats rs1 ON p1.plan_id = rs1.plan_id JOIN dbo.user_query_store_runtime_stats_interval rsi1 ON rsi1.runtime_stats_interval_id = rs1.runtime_stats_interval_id JOIN dbo.user_query_store_plan p2 ON q.query_id = p2.query_id JOIN dbo.user_query_store_runtime_stats rs2 ON p2.plan_id = rs2.plan_id JOIN dbo.user_query_store_runtime_stats_interval rsi2 ON rsi2.runtime_stats_interval_id = rs2.runtime_stats_interval_id JOIN dbo.user_sys_objects so on so.userobjectid = q.object_id WHERE rsi1.start_time > dateadd(hour, -1000, getutcdate()) AND rsi2.start_time > rsi1.start_time AND rs2.avg_duration > 2*rs1.avg_duration

Conclusion

Once you have this running you are ready to go to be able to share an insight into dev/test and production whilst keeping a snapshot of where the performance was on a particular date. Combining this with a lightweight performance monitor collection, during testing, takes away all the hassle of running a SQLDiag and then waiting to import/aggregate the data via the ReadTrace utility (Part of RML Utilities).

The case for building a consolidated SSIS environment

BenCooper — Thu, 04 Aug 2016 17:06:22 GMT

By Jonathan Ball – Data Platform Solution Architect. SQL Server Integration Services, SSIS, can be memory intensive, CPU hungry and will generally want to max out the server that it is running on. Your typical DBA is not going to want this running on the same server as the database engine taking up resources and making the monitoring of the database performance harder. Your typical developer is going to want to have SSIS as near as possible to the database for maintenance, possibly performance and to keep a tidy package for deployment and ongoing maintenance. SSIS packages can be heavily used in keeping data up to date and with the consistent approach to development in visual studio and SQL Server data tools creating integrated data solutions with Continuous Integration, CI and Continuous Deployment, CD is more common. Deploying SSIS on the same box as your DB engine has been very common and can have benefits for ease of deployment, ongoing maintenance and logging/auditing. Another option is to deploy a separate SQL instance with your application/solution solely for SSIS. This can get costly and keeping track of multiple ETL servers for moving data around the data estate can be an unwanted overhead. In an estate where there is a large amount of SSIS being used it is worth considering a consolidation approach. So with this in mind what are some of the benefits for consolidating:

Licensing

This will be one of the strongest arguments for consolidating but savings in this area depend on how you deploy SSIS. If you are installing SSIS on a SQL Server instance that has other services on it, it could be running the database engine or SSRS, then you will already be licensing the server. If you consistently separate the SSIS out to other hardware then you will need to license those instances and consolidating will obviously have merit from a cost perspective.

Ease of deployment

The project approach for SSIS has made deployment a much simpler matter. The use of ISPAC in a similar approach to DACPAC means the dev teams can pass across the deployment as a whole including the deployment specific variables, connections, folders within it. Deployment to a separate folder for the different applications/solutions combined with a single deployable pack makes manual deployment much easier. The possibility of integration with deployment management software for a CI/CD, approach is much stronger, but that is another blog! With these benefits, deploying to a single environment means that the dev teams and the teams doing the deployments will have one approach and environment to deal with for all releases.

Logging and auditing.

From an enterprise data architecture perspective this should be high on the list of benefits. Knowing where and when your data is moving is a key part of understanding who is using what, when and for what purpose, with growing regulation identifying this as required. With the SSISDB, SSIS Catalog this information starts to become easier to locate. The out of the box reports available in SSMS querying the SSISDB deliver a good start to viewing and keeping track of the packages that are moving data around. Having this data in one place can massively simplify the job of the data stewards.

Logical separation of applications/logs/reports.

This is more of a showcase of how project deployment makes separation possible within a consolidated environment. In the SQL Server 2012 release, the changes to SSIS around SSISDB for logging and the project deployment really started to support the case for a consolidated SSIS environment. A deployment of SSIS packages in earlier versions would allow for a single storage location but with no logical or role based grouping and minimal built in logging this would not provide any great benefit. Releases to either folders or into the MSDB would generally all sit in one area relying on naming conventions for any form of identification. Setting up different folders within the SSIS Catalog allows you to keep different applications separated and secured. The reporting will show the execution for these different folders/applications. There are clearly benefits to putting together a consolidated environment but there are things that need careful planning. As with any environment you will need to consider an appropriate disaster recovery approach, considering high availability with SQL 2016 offering support for the SSISDB with availability groups. The approach for security for execution, separation of projects, reporting will need planning as well as permissions for CLR, third party installations on the environment and the actual execution method to use. With consolidated environment the upgrade path will be easier to identify but will also require more planning. When there are multiple SSIS installs, the sprawl can make standardising and upgrading difficult but with a shared platform if you upgrade, all projects will need to buy into the upgrade. In addition to these obvious environmental design requirements, a shared platform will need to consider the possibility of setting up execution timetables. The benefit of a shared environment is that it is all located and maintained in one place. What it also means is that the resources are shared so administering the environment and executions will require knowledge and understanding of what the packages are doing and the time windows for execution, you might not want big jobs hitting each other. There are clearly benefits to consolidating a SSIS platform but it needs a level of planning to make it run smoothly. The need for an appropriate design and architecture to specify the security, availability, disaster recovery and execution methods needs to go hand in hand with a consistent development approach by those producing the SSIS packages and the teams supporting the environment. If you are putting together an approach for consolidating SSIS these points can be used for validation and planning.

Live Query Statistics

BenCooper — Thu, 07 Jul 2016 09:36:31 GMT

By Mike Reed – Data Platform Solution Architect. So you work in an organisation who run 'mission critical' applications but don't want to pay for tooling. You know, they want you to keep the lights on but won't invest in any of the great 3^rd party monitoring tools that Microsoft's partners have written to make your life easier. So, you do your best, you write your own suite of monitoring and alerting tools. Sometimes these are pretty sophisticated and if you have the time to invest in them they can be really powerful. Or, you may be at the other end and you just have your 'Top 5 worst performing queries' script that you run when things are going south. Either way you are going to love some of the new features in SQL Server 2016 that have been put in place to help those monitoring and maintaining SQL Server instances. Actually, these are great features for developers too that will help you tune your queries (you do that right?). One of those new features is Live Query Statistics, a really nice little feature to help you see what's going on while a query is running. It gives you a view of the query you are tracking whilst its running. You can turn it on several different ways. The easiest is to hit its icon in Management Studio which sits next to Include Actual Execution Plan: But you can also right click inside the text of a query And you get a new tab alongside Results and Messages with your Live Query Statistics information in like the screenshot below. How cool is that! So you get a running progress of the query both in the top and bottom left. Its useful in both places because if you switch to the Results or Messages pane you can still see how far through you are in the bottom left. However, even better, each step in the execution plan has its own little percentage counter so you can see in real time where the hold-up or the pain is. For example, in the Screenshot below you can see that the hold-up is in the bottom right in the read of the FactResellerSalesXL table. It's also worth noting here that the tool is also available via Activity Monitor. If you right click on a query in the Active Expensive Queries pane you can also activate Live Query Statistics by clicking on Show Live Execution Plan and it will open up in its own tab. Just a final note that there is a warning on the MSDN page which reminds the user that this feature is primarily supposed to be for troubleshooting and that by using this feature it can 'moderately slow the overall query performance'. You have been warned!! Hope you enjoy this new feature and it makes your life a little easier.

Power BI in Your Organisation

BenCooper — Tue, 21 Jun 2016 13:31:06 GMT

By Michael Robson – Technical Solution Professional (Data Platform / BI).

Since the launch of Power BI last July there's been a lot of interest and, it's fair to say, a lot of confusion as to exactly what Power BI is and where it fits.

Is it Microsoft's replacement for SSRS?

Do I need Office365?

This is really cheap; can I replace my Business Objects/Cognos investment with this?

How does it compare with Qlik/Tableau?

And so on.

You may be familiar with terms such as Polyglot Programming and Polyglot Persistence (see http://martinfowler.com/bliki/PolyglotPersistence.html for more information), which refers to the combination of different technologies to solve complex problems, whilst not relying on a single language or storage mechanism. Modern Information architectures generally refer to this concept when dealing with data from multiple sources with varying properties. We can think of this as perhaps fast moving data vs batch data, or structured vs unstructured data.

If we extend this concept of Polyglot Persistence to also include analytical and reporting tools, we could perhaps coin the phrase Polyglot Analysis or Polyglot Reporting? After all, the needs of your organisation are going to be very different depending on who is consuming information. Does the CEO want the same level of analysis from a reporting tool than an Analyst or a Data Scientist?

So let's think about who our users are and where they fit in the organization.

In this diagram, we can think of our users as ether consumers or producers.

Business Users

The majority of our users are going to be simply consumers. For the most part they'll have reporting delivered to them or they'll login to a portal. Consumption will be typically via static/paginated reporting deriving data from an operational system or a data warehouse. They may also demand a little more interaction via dashboards which provide some ability to search and slice data, but still within a guided environment (links and pathways through the data are defined ahead of time). Commonly, mobile reporting is also delivered to satisfy tablet and mobile users within the community.

Analyst/Super-user/Excel-Jockey

This is where we start to hit what Gartner refer to as bi-modal IT (http://www.gartner.com/it-glossary/bimodal). The super user or analyst who wants information now and they don't want to wait for IT to deliver it. We can think of this segment of users in two ways:

Self-Service BI: Users who want to consume existing content but want to create their own reports. To deliver this IT might provide a pre-defined model (such as a Data Warehouse or Cube) and a set of tools to connect to it allowing the super-user to create their own content.
Business Analytics: A step beyond Self-Service BI, where analysts not only want to create their own reports and visuals, but they also want to create their own models. Here, an Analyst might take data from multiple sources to create data 'mashups' in order to provide analysis for 'top-of-mind' business problems. Rather than wait for IT to build a solution to address these questions (which could take months), Business Analytics tools allow user to quickly perform ad-hoc analysis to prove/disprove a hypothesis within days, if not hours.

Data Scientist

The increased demand for Data Science skills means an increased demand for specialised tooling in this discipline. Data Scientists typically will require a set of tools that allow them to experiment with multiple models unconstrained by schemas or existing datasets.

From IT's perspective, reporting tools will be focused on development and deployment tools for report creation, database/cube development, integration, planning, source control and so on.

Now that we've looked at who our users are and what types of workloads they might be running, where does Power BI fit in?

The chevron-shaded area in the enhanced diagram above shows Power BI's key user base, primarily, the Self-Service and Business Analytics community. It also shows some examples of Microsoft tooling that satisfies the other workloads. You may note, however, that the shaded area overlaps all users to some extent. This is simply because there are feature overlaps. Let's explore this a little.

Power BI for Reporting and Dashboards

Just like SSRS, Power BI can be used to create rich dashboards and reports. Unlike SSRS, Power BI tends to be not so granular. With SSRS, the developer can open up detailed scripting and formatting to control the navigation path between reports as well as detailed pagination options such as defining margins and so on. Power BI adopts a more drag-and-drop approach, with the focus on obtaining insight from the data through interactive visualisation, not scripting pixel perfect dashboards.

Power BI for the Data Scientist

I'm not claiming that Power BI can replace a wealth of tools and languages being used by Data Scientists, but there are a couple of key areas where it can significantly accelerate the value Data Scientists can bring to the rest of the organisation. For example, native integration with

R as a data source, allowing you to import the content of data frames, and
R visuals, allowing you to embed native R visuals directly into a Power BI report or dashboard

will allow data scientists to quickly produce and share insights with the business community without the need for complex database integration.

Power BI for IT

A key use case for Power BI for the IT organisation is rapid prototyping. The underlying technology behind Power BI is SQL Server Analysis Server's tabular engine, allowing IT to produce prototypes using familiar enterprise capabilities. These prototypes could be rapidly developed and shared with the business community to gain feedback, and then, if required, they can be imported into SQL Server. As with the Data Science use case, this could be developed without the need for large infrastructure to be deployed or changed. Once value has been proven to the business, more formal models can be deployed using SQL Server Analysis Services, Azure Machine Learning, and R Server, and necessary integration pipelines built.

Conclusion

At a high-level, this illustrates where the various BI and Analytics technologies fit into an organisation and what workloads are best suited. Power BI fits into the Self-Service and Business Analytics field well, spanning the spectrum and providing a broad set of tools for your organisation. However, consider the end user and the outcomes they are trying to achieve. It may well be that in order to achieve the outcomes your organisation needs you need to use a number of different reporting tools.

Moving Schema and Data: SQL Azure Migration Wizard

BenCooper — Mon, 06 Jun 2016 08:59:34 GMT

By Dan Stillwell – Data Platform Solution Architect. In my previous blog I wrote about a scenario where one of my customers wanted to utilise the flexibility of Azure for their SQL Server provisioning. The deployment started as a POC but very quickly became more than that with senior members of the organisation using the functionality. At this point we wanted to demonstrate the flexibility and elasticity of Azure by migrating the POC workload to a platform with HA, and SLA's. At this point we done a feasibility study between using SQL within a VM (IaaS) or Azure SQL DB (PaaS). We decided that Azure SQL DB was the right service offering for the requirements of the customer. Azure strategies can vary and this is one of the great strengths within cloud technologies; whatever the customer requirements, Azure can generally offer a service to meet them. I have worked with customers who want to migrate all databases to Azure, I have worked with some customer who want to use Azure just for disaster recovery and High availability. This particular customer at present - is using Azure for development/ POC work. Once they are happy with the POC in Azure they then deploy and handover the service to their own Data Centre team. So with this in mind we need to migrate the Schema and data from the standalone SQL Server within an Azure VM to Azure SQL DB. I will walk through the process, but as stated above I need a tool that can migrate in any 'Azure direction'. What I mean by this is the first stage will be to migrate the data from IaaS SQL Server to PaaS SQL DB, but the same method can be used to move it from PaaS to IaaS, PaaS to PaaS, on premise to Azure or Azure to on premise. There are a few different ways when migrating data from one environment to another. You can use Generate Script Wizard, Deploy Database Wizard, SQL Server data tools, Export Data Tier, SSIS, BCP, sqlpackage.exe, AZ Copy, Backup/ Restore, Transactional Replication, HA technologies to replicate data (Log Shipping, AlwaysOn) to name a few. The tool I have used for the migration process is 'SQL Database Migration Wizard' the tool can be found from the open source site CodePlex (http://sqlazuremw.codeplex.com/). You can migrate from SQL Server 2005, 2008, 2008R2, 2012, 2014. This is a different tool to the Upgrade Advisor, there is now a SQL Server 2016 Upgrade Advisor available in preview (https://www.microsoft.com/en-us/download/details.aspx?id=48119). One of the issues with migration is the compatibility between the source data/ instance version and the destination. For example, in Azure SQL DB at present CLR, XML, table partitions. FileStream is not supported. Full updated list of Azure SQL DB support (https://azure.microsoft.com/en-gb/documentation/articles/sql-database-transact-sql-information/#features-not-supported-in-sql-database). This tool is a great way to identify in a planned way what may not work or is not supported on the destination environment. The tool uses regular expressions to analyse the phrases, so it can be configurable by yourself. The tool can fix some issues, but some issues can only be identified and not fixed automatically. This tool can be used as an end to end process for migration (analyse, script out schema, BCP the data) or you can use parts of the tool. For example, use the tool to analyse the schema and then use another method to migration the actual data. Once you install the tool, you will be welcomed with the following splash screen. Here you can see the different options. Please note that you can capture a profiler trace and analyse that via this tool: At this stage you will connect to a instance or database of SQL Server: *You will need to create a firewall rule on Azure for the migration tool to access the Azure VM or Azure SQL DB. The IP address from the machine running the tool will require access. You will then need to select your source database Select all objects or select individual objects. Clicking 'Advanced' gives the option to only script schema, data or both. You may wish to only run the schema script first to fix any compatibility issues. You can also select or deselect compatibility checks at this stage. After your selection click next, review and then choose to generate the script. You then get presented with the script which you can then edit within the window, copy out to SSMS or save the script to run manually or edit at another time In the 'Result Summary' tab any errors will be highlighted in red, you can also see the location of the .dat file where the data has been written to ready for the BCP process to import Select Next and you will need to enter the destination server In Azure, you then have an option to select a current SQL Database or create a new one, I am creating a new one when I am selecting a S1 service tier Once the SQL Database is created, you can then select it from the list and execute the script against it. Alternatively, you can create the database manually and then run the script that you had the option of saving earlier against the database As you can see, the table has been created and the data has been migrated: Below are some things to highlight when using the tool:

Please note that this tool is not officially supported by Microsoft. It is available on CodePlex which is Microsoft's open source site. Microsoft internal groups do monitor the feedback and discussions and will send a 'best effort' response.
The tool will fix some compatibility issues. Pay attention to the output report for what has been fixed and what needs additional fixing from you.
You can work with the script within the tool, or saving it and editing it in SQL Server Management Studio or SQL Server Data Tools (Visual Studio).
Within your script you can use 'IF Exist', this allows you to run the script multiple times. However, you may not want to script the data at this stage as it will BCP the data multiple times.
The process should be to allow the tool to fix automatically/ apply any recommendations, fix any other issues, migrate the schema, confirm migration of schema is correct, then import the data via running the BCP Data script.
If you are loading a large amount of data, you can run the script from different machines to utilise more threads and processing power.
DAT files are kept by default in 'C:\SQLAzureMW\BCPData)'.
You can increase or decrease the parallel processing of the data migration by altering the 'NumberofBCPThreads' within the SQLAzureMW.exe.config file:

As you can see the Migration Wizard gives great flexibility to assist in moving the schema and data to Azure whether that is IaaS or PaaS. When I have used the tool with my customers it can help solve a major barrier around complexities, time and resource in creating scripts and even a test process for ensuring compatibility to the different platform.

Transactional Replication Using an Azure SQL Database Subscriber

BenCooper — Mon, 23 May 2016 13:53:30 GMT

By Graeme Scott – Data Platform Solution Architect. For companies that have geographically dispersed operating models which may involve physical stores, outlets, venues or branches; data replication from individual stores to data centres is often a key technical function that enables companies to measure overall business performance. It is not uncommon for geographically dispersed companies to capture a significant amount of transactional and operational data at individual stores or branches; in these scenarios, customers often use transactional replication to transport the data centrally for analysis. With SQL Server 2016 it's now possible to replicate data from an on premise replication publisher to an Azure SQL Database subscriber. This replication configuration provides customers with the following possibilities:

A very easy route to transition to a hybrid – on premise cloud configuration with minimal disruption to business continuity.
Off load reporting workload to separate cloud infrastructure.
Provide fine grain control as to the database objects (tables, views) that are stored within an Azure SQL Database. With replication, individual tables (articles) are added to a publication that are to be replicated to a subscriber.
With data stored within an Azure SQL Database subscriber, customers can enrich the data and gain increased insight by leveraging a range of Azure services, for example Azure Machine Learning and R.
Data recoverability and high availability.

Setting up transactional replication from an on premise publisher to an Azure SQL Database subscriber is practically identical when compared to an on premise setup experience using SQL Server Management Studio. The configuration can also be achieved using TSQL scripts. When I setup transactional replication to use an Azure SQL Database subscriber, the only change I had to make compared to a full on premise configuration was the subscriber connection to suit a Azure SQL Database (Azure SQL Database connection string): SQL Server Replication Monitor also works seamlessly with an Azure SQL Database subscriber, all the replication information and diagnostics data are present. So what are the limitations associated with using an Azure SQL Database as a subscriber to for transactional replication?

Only transactional replication is supported.
The transactional replication needs to be a push subscription.
Any post snapshot scripts would need to be verified to determine suitability for an Azure SQL Database.
A SQL Azure Database can only be a subscriber database for transactional replication.
There will be limitations on publishing database objects that are CLR based.

It's possible to meet the majority of these goals with SQL Server Availability Groups. The main differences with these two options are that Availability Groups replicate an entire database as a secondary replica. With transactional replication it's possible to replicate only a single table from within a publisher database. This may be less of a barrier for customers to consider moving data and workload to an Azure SQL Database. The following article has more details about replicating to SQL Azure: https://azure.microsoft.com/en-gb/blog/transactional-replication-to-azure-sql-db/ The following article has more details about SQL Server replication with SQL Server 2016: https://msdn.microsoft.com/en-us/library/ms152559.aspx It's outside the scope of this article however it is possible to use earlier versions of SQL Server (SQL Server 2012 upwards) to replicate from an on premise publisher to an Azure virtual machine. Some more details can be found here: https://blogs.msdn.microsoft.com/repltalk/2014/07/28/microsoft-sql-replication-between-windows-azure-virtual-machines/

Back on the mat

BenCooper — Fri, 20 May 2016 11:32:13 GMT

By Craig Clayton – Technical Solution Professional. I recently went back to Brazilian jiu-jitsu after a lay off due to injury (due to old age!). BJJ is a ground based martial art that "promotes the concept that a smaller, weaker person can successfully defend against a bigger, stronger, heavier assailant by using proper technique, leverage, and most notably, taking the fight to the ground, and then applying joint-locks and chokeholds to defeat the opponent". For more details, see: https://en.wikipedia.org/wiki/Brazilian_jiu-jitsu. Whilst learning the art, you get lots of opportunities to try the techniques in live "rolling" or sparring, with the objective to try and submit your opponent, or get them to 'tap out'. This can be quite disheartening when you first start and when you subsequently roll with more experienced team members, as you spend a lot time tapping out! However, this forms the rights to passage - where it all adds to the learning and development of your skill, ability and style. This Journey reminds me a lot of SQL Server's journey against Oracle. The years of being submitted by the more experienced player, over and over again. But, instead of the SQL Server product team giving up and accepting the product's level, it only toughened the team's resolve. Focusing on the weaknesses and limitations within the product, coupled with leveraging the natural attributes of Microsoft (e.g. Azure, Windows etc.) has led to a real game changing product with SQL Server 2016. SQL Server 2016 offers outstanding features (these features will be the focus of subsequent blogs by both myself and the rest of the team), all built-in to the product, to provide seamless integration with an attractive price point that negates the need for addition extras. The slide below demonstrates some high-level business benefits underpinned by many new capabilities: Some of the headline features being (but believe me it is much more than these points!):

Always Encrypted – where the data is encrypted by the client, hence is stored encrypted within SQL Server as well as across the network finally to be decrypted at the client. This allows businesses greater piece of mind and more potential to use Azure.
Stretch Database – allowing Customers to archive 'cold' data to Azure, whilst still having the ability to query the data, as if it was still on-premises in the same table. Resulting in the reduction of both storage on-premises and SQL Server maintenance windows.
Real Time Operational Analytics – enabling Customers to report directly from OLTP Source systems in real time, without the fear of disruption to the OLTP capabilities and reducing the need for extensive ETL processes.
R Services – embracing the data scientist open-source language of choice. Integrated into SQL Server 2016 with massive improvements around performance and security.
Extensive in-memory capabilities – enabling businesses to achieve outstanding performance with less cores, potentially reducing the SQL Server License spend!

This has resulted in what we see today: Microsoft - The leader in the Operational Database Management System market as per the Gartner Magic Quadrant (October 2015) for ODBMS. Finally, 'SQL Server Tapping out Oracle!' Subscribe to this blog, to understand more about SQL Server 2016's capabilities across industry, on-premises and the cloud as well as more broad data platform architectural patterns and best practices.