In my last blog entry,I gave you some insights on my project objective. Let’s call this project as SQLSNAP (SQL Snapshot). In  VDI application development, the actual work of writing the backup metadata and taking backups is done by the VDI application. SQL Server would be doing the work as instructed by VDI app. So we have to be careful in our code and logic.

Consider following sequence of logic for SQLSNAP. we will rectify as and when necessary in subsequent posts.

1. On startup, SQLSNAP does certain initializations (for VDI backups)

2. SQLSNAP connect to local SQL Server instance.

3. Just like any other backup, a header file is created (you could do a RESTORE HEADERONLY on it) by SQL Server, the responsibility to write the metadata (and taking actual disk snapshot) lies with SQLSNAP.

4. SQL puts database IO in freeze mode (via VDI calls made by SQLSNAP).

6. SQLSNAP makes the disk level snapshots. This is a storage level work and will not be touched here. We will simply call a batch file that does storage level work.  In my case, I am working on a Windows 7 laptop. so I will make use of the VSS to create “Previous versions” of my C: 

for large production database systems running on SANs, you should use tools specific to your SAN for making snapshots.

7. on completion of the snapshot, SQLSNAP informs SQL Server about backup completion. SQL Server resumes the IO on server.

And Our backup gets completed.

Sounds simple.  Lets get it coded !!

First, Download the SQL Server 2005 Virtual Backup Device Interface (VDI) Specification from Microsoft. Run the downloaded EXE to extract few files onto your development machines.  you will see few folders and a help file extracted. The vbackup.chm provides descriptions of each of the functions. The descriptions include the following information:

• Function purpose
• Function syntax
• Parameter list
• Return values
• Remarks

The other folders are sample code for you to study and develop your custom applications.

so lets open the snapshot sample in more details.

open the snapshot.cpp file in notepad or in your favorite C++ editor (such as Visual Studio). I am using VS2010 for this.  Firstly note the initial comments in the source code. The same are applicable on to the modifications done/shown on this blog. 

// This source code is an intended supplement to the Microsoft SQL
// Server online references and related electronic documentation.
//
// This sample is for instructional purposes only.
// Code contained herein is not intended to be used "as is" in real applications.
// 
// snapshot.cpp
//
// This sample extends the "osimple.cpp" sample to demonstrate BACKUP WITH SNAPSHOT.
// It is not fully functional.  The ability to take/mount snapshots must be provided.
//
// This is a sample program used to demonstrate the Virtual Device Interface
// feature of Microsoft SQL Server together with an ODBC connection.
//
// The program will request backup or restore of a single database
// on some instance of sql server.

Let’s go over the rest comments inside the main function of the code.

    // Check the input parm

    // Initialize COM Library
    // Note: _WIN32_DCOM must be defined during the compile.

    // Get an interface to the device set.
    // Notice how we use a single IID for both the class and interface
    // identifiers.

    // Setup the VDI configuration we want to use.
    // This program doesn't use any fancy features, so the
    // only field to setup is the deviceCount.
    //
    // The server will treat the virtual device just like a pipe:
    // I/O will be strictly sequential with only the basic commands.

    // Create a GUID to use for a unique virtual device name

    // Create the virtual device set
    // Notice that we only support unicode interfaces

    // Send the SQL command, by starting a thread to handle the ODBC

    // Wait for the server to connect, completing the configuration.

    // Open the single device in the set.

now Logic moves to performTransfer function. following comments from performTransfer function.

// VDI data transfer handler.
//
// This routine reads commands from the server until a 'Close' status is received.
// It simply reads or writes a file 'superbak.dmp' in the current directory.

and few more inside a while loop. I am not pasting them here.

A typical database backup command has following syntax

BACKUP DATABASE DBNAME TO DISK=’FILENAME’

Just doing a simple search on the file about word “DATABASE “, I found following line inside a function named  “SQLRoutine”

// Generate the command to execute
//
sprintf (sqlCommand, "%s DATABASE [%s] %s VIRTUAL_DEVICE='%s' WITH SNAPSHOT",
    parms->doBackup ? "BACKUP": "RESTORE",
    parms->pDbName,
    parms->doBackup ? "TO": "FROM",
    aVdsName);

So as we see, the majority of work that we intend to do is done in this 3 functions. main, performTransfer and SQLRoutine.  There are few other functions as well.

In my next blog post, I will go deeper into the sample code and modifications.

Recently, I was working with one of my customers to help them design their backup scheme. This customer has their IT team very comfortable with their SAN infrastructure. Their current infrastructure and backup solution for non-Microsoft product stack is all based on SAN.

The way they work is they mirror their disks (or volumes) to other disks. Once mirror is in Sync, they break the mirror, mount the mirrored disk on to another server and take file system level backups. They call it as BCV copy.

This customer was interested to know if he could do the same with Windows/SQL.

As it sounds independent of the OS/DB, customer gave it a shot. It worked. They were able to make their BCVs to tape without a problem. (Disk Mirror -> Sync -> Break Mirror -> mount it on new server -> take tape backup).

Now, no backup test is complete without a restore. So they tried to restore. It worked as expected as well. However, there was a issue which needed to be addressed.

Customer couldn’t restore BCV data files and then apply additional transaction log files to bring database up to date. Why? As soon as you attach these BCV’ed files, SQL will recover them and database will be operational.

In comes SQL Server VDI (Virtual Device Interface).

Next : What I did to SQL Server VDI Samples to facilitate subsequent transaction log restores. Stay tuned for more.

Meanwhile see these articles for more knowledge.

A Guide for SQL Server Backup Application Vendors

How It Works: SQL Server - VDI (VSS) Backup Resources

I made following procedure to make their life a little easier. hope this helps others as well.

No explanations on that this procedure does. Run it on SQL 2005 or SQL 2008 and see it for yourself.

CREATE proc sp_what ( @session int = null, @plan bit = 0 ) 
as 
begin
    select session_id ,command ,
    blocking_session_id as blocked, wait_time, wait_resource,
    open_transaction_count as trans , percent_complete as [%], 
    cpu_time , 
    convert(varchar, total_elapsed_time /60000 ) + ':' + right('0' + convert(varchar(2), (total_elapsed_time /1000) % 60 ),2 ) as StartedSince,
    reads , 
    writes ,logical_reads, row_count, nest_level, granted_query_memory as mem ,
    object_name( s.objectid) as obj,
    SUBSTRING(s.text, statement_start_offset / 2, ( (CASE WHEN statement_end_offset = -1 THEN (LEN(CONVERT(nvarchar(max),s.text)) * 2) ELSE statement_end_offset END) - statement_start_offset) / 2) 
    from sys.dm_Exec_requests r cross apply sys.dm_exec_sql_text (sql_handle) s
    where command is not null and session_id <> @@spid

    and ( session_id = @session or @session is null )

    if ( @plan = 1 )
    begin
        select session_id , query_plan from sys.dm_Exec_requests r
        cross apply sys.dm_exec_query_plan (plan_handle) s
        where command is not null and session_id <> @@spid
        and ( session_id = @session or @session is null )
 
    end
end 

Lets’ call this function as Split1. The code is as follows:

CREATE FUNCTION [dbo].Split1(@input AS Varchar(4000) )

RETURNS

      @Result TABLE(Value BIGINT)

AS

BEGIN

      DECLARE @str VARCHAR(20)

      DECLARE @ind Int

      IF(@input is not null)

      BEGIN

            SET @ind = CharIndex(',',@input)

            WHILE @ind > 0

            BEGIN

                  SET @str = SUBSTRING(@input,1,@ind-1)

                  SET @input = SUBSTRING(@input,@ind+1,LEN(@input)-@ind)

                  INSERT INTO @Result values (@str)

                  SET @ind = CharIndex(',',@input)

            END

            SET @str = @input

            INSERT INTO @Result values (@str)

      END

      RETURN

END

This is a very old fashioned (but still effective enough) script which does a loop over a string to cut out all possible string values that are separated by a comma.

Let’s see now, how the same could be achieved in modern day TSQL languages (such as SQL 2005 or SQL 2008).

Approach 1:  Common Table Expression (CTE)

Lets call this function as Split2.  here we are using

CREATE FUNCTION dbo.Split2 ( @strString varchar(4000))

RETURNS  @Result TABLE(Value BIGINT)

AS

begin

    WITH StrCTE(start, stop) AS

    (

      SELECT  1, CHARINDEX(',' , @strString )

      UNION ALL

      SELECT  stop + 1, CHARINDEX(',' ,@strString  , stop + 1)

      FROM StrCTE

      WHERE stop > 0

    )

   

    insert into @Result

    SELECT   SUBSTRING(@strString , start, CASE WHEN stop > 0 THEN stop-start ELSE 4000 END) AS stringValue

    FROM StrCTE

   

    return

end   

 

GO

Approach 2:  XML (surprise)

XML could be applied to do some type of string parsing (see this) Let’s call this function as Split3.

CREATE FUNCTION dbo.Split3 ( @strString varchar(4000))

RETURNS  @Result TABLE(Value BIGINT)

AS

BEGIN

      DECLARE @x XML 

      SELECT @x = CAST('<A>'+ REPLACE(@strString,',','</A><A>')+ '</A>' AS XML)

      INSERT INTO @Result            

      SELECT t.value('.', 'int') AS inVal

      FROM @x.nodes('/A') AS x(t)

    RETURN

END   

GO

Approach 4: Classic TSQL Way

I got this approach from SQL Server Central site. This approach is slightly unusual but very effective. this needs you to create a table of sequential numbers called a Tally Table.

SELECT TOP 11000 --equates to more than 30 years of dates       

IDENTITY(INT,1,1) AS N  

INTO dbo.Tally  

FROM Master.dbo.SysColumns sc1,       

Master.dbo.SysColumns sc2

Lets index the table for better performance.

ALTER TABLE dbo.Tally   

ADD CONSTRAINT PK_Tally_N PRIMARY KEY CLUSTERED (N)

WITH FILLFACTOR = 100

Finally out Split4 function.

CREATE FUNCTION dbo.Split4 ( @strString varchar(4000))

RETURNS  @Result TABLE(Value BIGINT)

AS

BEGIN

 

      SET @strString = ','+@strString +','

     

      INSERT INTO @t  (Value)

      SELECT SUBSTRING(@strString,N+1,CHARINDEX(',',@strString,N+1)-N-1)  

      FROM dbo.Tally 

      WHERE N < LEN(@strString)   

      AND SUBSTRING(@strString,N,1) = ',' --Notice how we find the comma

 

      RETURN

END   

GO

 

Now, what about the most crucial question: Performance.  so lets put all 4 functions to test.

please note that I am running this on SQL Server 2008. you may need to modify it for SQL 2005.

DECLARE @str VARCHAR(4000) = '6,7,7,8,10,12,13,14,16,44,46,47,394,396,417,488,714,717,718,719,722,725,811,818,832,833,836,837,846,913,914,919,922,923,924,925,926,927,927,928,929,929,930,931,932,934,935,1029,1072,1187,1188,1192,1196,1197,1199,1199,1199,1199,1200,1201,1202,1203,1204,1205,1206,1207,1208,1209,1366,1367,1387,1388,1666,1759,1870,2042,2045,2163,2261,2374,2445,2550,2676,2879,2880,2881,2892,2893,2894'

SELECT * FROM dbo.split1 ( @str )  

SELECT * FROM dbo.split2 ( @str )  

SELECT * FROM dbo.split3 ( @str )  

SELECT * FROM dbo.split3 ( @str )  

On my laptop, I saw following numbers. I expected the XML to be the fastest.

so I changed the code slightly to test out XML function directly.

DECLARE @str VARCHAR(4000)

= '6,7,7,8,10,12,13,14,16,44,46,47,394,396,417,488,714,717,718,719,722,725,811,818,832,833,836,837,846,913,914,919,922,923,924,925,926,927,927,928,929,929,930,931,932,934,935,1029,1072,1187,1188,1192,1196,1197,1199,1199,1199,1199,1200,1201,1202,1203,1204,1205,1206,1207,1208,1209,1366,1367,1387,1388,1666,1759,1870,2042,2045,2163,2261,2374,2445,2550,2676,2879,2880,2881,2892,2893,2894'

Declare @x XML 

select @x = cast('<A>'+ replace(@str,',','</A><A>')+ '</A>' as xml)

select t.value('.', 'int') as inVal

from @x.nodes('/A') as x(t)

Hmm. so just as I thought, XML is faster on its’ own.

there are some more options that I did not consider such as CLR functions. but we will get to those some other time.

During my interactions with them, I came across following case studies.

Quanta Computer Gains 80 Percent Data Compression for SAP with SQL Server 2008

With its SAP database supporting its global operations growing by more than half a terabyte a month, Quanta needed to either replace its SAN infrastructure or find a compression solution. The company upgraded to Microsoft® SQL Server® 2008 Enterprise (64-bit) to take advantage of new SQL Server compression technology. SQL Server 2008 Database Compression has reduced some tables by as much as 90 percent, while SQL Server 2008 Backup Compression has provided compression of 85 percent

Moving SAP Off Mainframe to SQL Server Nets Australia’s QR Limited U.S.$100,000 a Month

Having been in the business since the 1860s, Australia’s QR Limited knows how to run a railroad. As the company prepared to upgrade its SAP R/3 enterprise resource planning (ERP) software it decided to move its SAP applications and databases from a mainframe and DB2 environment to the Microsoft® Application Platform, including Microsoft SQL Server® 2005 database software running on a geocluster supported by the Windows Server® 2003 Enterprise Edition operating system. The company estimates it is saving about U.S.$100,000 per month in mainframe costs, since moving its SAP infrastructure to a cluster of commodity servers running the Microsoft Application Platform. QR Limited reports 20 to 40 percent faster SAP transactional response times and up to 8 times faster batch processing since moving SAP to SQL Server 2005.

Australia’s CSR Lowers Costs by Migrating SAP from Oracle to SQL Server 2005

CSR Limited (CSR), which traces its roots to 1855, is one of the largest manufacturers in Australia. It is the sixth largest sugar producer in the world and a major manufacturer of building products including plasterboard, roofing material, insulation, and glass. CSR is also a joint venture participant in the Tomago aluminium smelter. The company uses SAP software to manage its multinational enterprise. CSR originally deployed SAP on a UNIX/Oracle stack, but later migrated to the Microsoft® Application Platform including Microsoft SQL Server® 2005 database software. Running SAP on SQL Server has proven so effective that after the company recently purchased Australia and New Zealand’s Pilkington Glass, it again migrated SAP from the UNIX/Oracle stack to the Microsoft Application Platform. CSR is enjoying a lower total cost of ownership, enhanced performance, and high availability.

Isn’t that great? Running SQL Server gives you lower total cost of ownership, enhanced performance, high availability & with SQL 2008, you also get a great storage size reduction!!!  A truly valuable solution in these recession days ….

Also, you would find following blogs as very useful resources for SAP on SQL Server.

Running SAP Applications on SQL Server

This Blog will provide information about running SAP applications on SQL Server and Windows. The Blog is written by folks of Microsoft who are working with SAP and SQL Server for more than a decade or who are running Microsoft's SAP landscape.

SAP Global Alliance Technology Team Blog: find various information that relates to Microsoft Windows platform and SAP. It contains information on Whitepapers, books, various software releases related to SAP & Windows.!!!

-- Amit

One of the solutions of this kind of issue was “Lock Pages in Memory” which was an Enterprise Edition only feature.

On Popular Demand (& Noise), This might be available for Standard Editions as well !!!

Read following blog entry from Bob ward

https://blogs.msdn.com/psssql/archive/2009/04/24/sql-server-locked-pages-and-standard-sku.aspx

and for now ….  just wait for those CU to be released !!!

My favorite of them all is

Troubleshooting Performance Problems in SQL Server 2008 Whitepaper

This white paper provides step-by-step guidelines for diagnosing and troubleshooting common performance problems by using publicly available tools such as:
▪ SQL Server Profiler
▪ System Monitor (in the Windows Server 2003 operating system) or
▪ Performance Monitor (in the Windows Vista operating system and Windows Server 2008), also known as Perfmon
▪ Dynamic management views (sometimes referred to as DMVs)
▪ SQL Server Extended Events (Extended Events) and the data collector, which are new in SQL Server 2008

Happy reading and troubleshooting!

Following is an excerpt from SQL Server Best Practices Article published on Technet.

https://technet.microsoft.com/en-us/library/cc966412.aspx 

{

Volume alignment, commonly referred to as sector alignment, should be performed on the file system (NTFS) whenever a volume is created on a RAID device. Failure to do so can lead to significant performance degradation; these are most commonly the result of partition misalignment with stripe unit boundaries. This can also lead to hardware cache misalignment, resulting in inefficient utilization of the array cache. For more information on this, see Disk performance may be slower than expected when you use multiple disks in Windows Server 2003, in Windows XP, and in Windows 2000.

Volume alignment on NTFS volumes can be performed by using diskpar.exe or DiskPart.exe. There is no advantage of using one over the other. However, DiskPart.exe is the preferred method since it is now included as part of Microsoft Windows® and the functionality will continue to be enhanced. Diskpar.exe is a Windows 2000 Resource Kit tool and is most commonly used in versions of Windows prior to Windows 2003 SP1. DiskPart.exe is a disk configuration utility that is built into Windows; in Windows 2003 version SP1 or greater DiskPart.exe contains an ALIGN option that can be used to align volumes.

When aligning the file system keep in mind the following: 

• Ideally, the offset used to align the volume should be supplied by the OEM of your particular storage array.

• If the OEM does not provide an offset, the only way to determine the optimal offset is through testing.

• A 64-KB offset (128 sectors) is a common value that works on many storage arrays and has been used with success in the SQL Server test labs (at the time of writing this paper). Vista™ and Windows Longhorn Server remove the need to manually establish correct alignment and use 1024 KB (2048 sectors) as a starting offset, which will probably work well for almost any array.

• Diskpar.exe works only on MBR partitions. DiskPart.exe supports both MBR and GPT partitions. Alignment of dynamic volumes is not currently supported as of Windows 2003 SP1.

• The value of the offset for diskpar.exe is expressed in sectors (512 bytes) and differs from the ALIGN option of DiskPart.exe, which uses a value defined in KB. For example “create partition primary align=64” creates a partition using a 64-KB offset.

To determine the current starting offset of a volume (or partition), you can either use Diskpar.exe with the –i option or WMI (as demonstrated in the following script) if Diskpar.exe is not available.

strComputer = "." 
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\CIMV2") 
Set colItems = objWMIService.ExecQuery( _     "SELECT * FROM Win32_DiskPartition",,48) 
For Each objItem in colItems     Wscript.Echo "Disk: " & objItem.DiskIndex & "  Partition: " & objItem.Index & "  StartingOffset: " & objItem.StartingOffset/1024 & "KB"     Wscript.Echo 
Next

For more information on DiskPart.exe, see DiskPart on Microsoft TechNet, specifically the "create partition primary" section.

}

I have been setting up this disk partition offset at all of my customer’s wherever a new SQL box or a new SAN deployment was planned out. As this is a task that could only be done while formatting hard drives.

I was following this at a customer place where some new clustered disk(s) were to introduced. the only challenge out there was that this customer was still running their systems on Windows 2000 Advanced Server.  (I know it’s a bit outdated, but still running their mission critical database system) and I never used DiskPart.EXE on windows 2000.

So, I downloaded the DiskPart.EXE from Windows 2000 Resource Kit and ran the usual command.

CREATE PRIMARY PARTITION ALIGN=64

and I could not believe on the result. The tool diskPart.exe did not liked my syntax for this command. and I could not do this task on windows 2000.

I went through a lo of reading and then came to know that  I should use DiskPar.EXE (no typo – It’s DiskPar.exe - Without the ‘t’ ).

then I began my quest to download the tool. After a long search, I managed to get the tool.

you could get it from My SkyDrive folder here

The usage is simple.

DiskPar.Exe [-?] [ -i | -s ] DriveNumber

Where:

-? displays command-line help.

-i DriveNumber   :queries drive layout and partition information.

-s DriveNumber  :sets new partition information.

you would get the Drive number in the Disk Management GUI.

also note that you would need to restart two services for the disks to be used again as needed.

See Technet  for more information

Hope this helps.

This is a nice tool for someone who needs to see whats inside a 8k SQL Data Page.

check it out yourself: https://internalsviewer.codeplex.com

Features:

Features

• Integration with SSMS (SQL Server Management Studio) 2005 and 2008
  • The application is installed as a SSMS add-in
  • Internals information integrated into the Object Explorer
  • Transaction Log viewer integrated into the Query Results

• Allocation Map
  • Displays the physical layout of tables and indexes
  • Displays PFS status
  • Overlay pages in the Buffer Pool

• Page Viewer
  • Displays Data pages including forwarding records and sparse columns
  • Displays Index pages
  • Displays allocation pages (IAM, GAM, SGAM, DCM, and BCM pages)
  • Displays pages with SQL Server 2008 row and page compression

SQL Server 2008 Upgrade Technical Reference Guide

https://www.microsoft.com/downloads/details.aspx?FamilyID=66D3E6F5-6902-4FDD-AF75-9975AEA5BEA7&displaylang=en 

Also One more must download is SQL Server 2008 Upgrade Advisor.  It is a separate download. The most recent downloadable version is available as part of the Microsoft SQL Server 2008 Feature Pack.

If you are (still) running your systems on SQL Server 7.0, then upgrade to SQL Server 2000 or 2005 first and then to SQL Server 2008.