I searched for out-sql Powershell cmdlet or function which allows to save the pipeline data to SQL Server, and I could not find one. So I wrote one.

It did the job for me, but obviously it's not perfect. Specifically:

• I use plain old INSERT statements - it's not the fastest method for inserting large amounts of data. Furthermore, I send each individual statement as a batch and an individual transaction. But I would not worry about it unless I insert hundreds of thousands or rows.
• No explicit error handling. SQL will throw the errors if it has a problem. The drawback is that script execution won't stop on say connection open error, and will continue - printing an additional error for every row.
• Short of dates, I save everything as NVARCHAR(MAX). I chose not to mess with integers - it was not needed for my project.

If you do end up using this, drop me a note on your experiences. If you do make enhancements, feel free to share with me!

Here's the function:

##############################################################################

##

## out-sql.ps1

##

## by Alexey Yeltsov, Microsoft Corp.

##

## Export pipeline contents into a new SQL table

##

## Parameters:

##    $SqlServer        - SQL Server

##    $Database      - Database name

##    $Table         - Table name

##    $DropExisting  - Drop $Table if it already exists and recreate it

##                    (default $false)

##    $RowId         - Add identity column named $RowId and make it a primary key.

##                    (default "RowID". Can pass $null if identity is not needed)

##

##

## Examples:

##   

##    #First, load the function

##    . .\out-sql.ps1

##

##    #Export processes to table Process in database Scratch on local sql server

##    get-process | out-sql -SqlServer . -database Scratch -table Process -dropexisting $true

##

##    #Export volume details from 4 servers into a table

##    @("Server1","Server2","Server3","Server4") `

##    | % {$Server = $_ ; Get-WMIObject Win32_Volume -computer $Server } `

##    | Select-Object `

##        SystemName, `

##        Name, `

##        @{Name="CapacityGb";Expression={[math]::truncate($_.Capacity / 1Gb)}}, `

##        @{Name="FreeGb";Expression={[math]::truncate($_.FreeSpace / 1Gb)}} `

##    | out-sql -sqlserver . -database Scratch -table DiskVolume -dropexisting $true

##

##

##

##############################################################################

function Out-Sql($SqlServer=$null,$Database=$null,$Table=$null,$DropExisting=$false,$RowId="RowID") {

  begin

  {

       $Line = 0

       [string]$CreateTable = ""

       if(-not $SqlServer) { throw 'Out-Sql expects $SqlServer parameter' }

       if(-not $Database) { throw 'Out-Sql expects $Database parameter' }

       if(-not $Table) { throw 'Out-Sql expects $Table parameter' }

       if($DropExisting) { write-debug "Note: If the table exists, it WILL be dropped and re-created."}

       $SqlConnectionString = "  Provider=sqloledb;" +

                           "  Data Source=$SqlServer;" +

                    "  Initial Catalog=$Database;" +

                    "  Integrated Security=SSPI;"

       write-debug "Will open connection to SQL server ""$SqlServer"" and will populate table ""$Table."""

       write-debug "Connection string: `n$SqlConnectionString"

       $SqlConnection = New-Object System.Data.OleDb.OleDbConnection $SqlConnectionString

       $SqlCommand = New-Object System.Data.OleDb.OleDbCommand "",$SqlConnection

       $SqlConnection.Open()

  }

  process

  {

       $Line ++

       $Properties = $_.PSObject.Properties

       if (-not $Properties)

       {

         throw "Out-Sql expects object to be passed on the pipeline. The object must have .PSObject.Properties collection."

       }

       #if we're at the first line, initialize the table

       if ($Line -eq 1)

       {

              #initialize SQL connection and create table

              if($DropExisting) { $CreateTable += "IF OBJECT_ID('$Table') IS NOT NULL DROP TABLE $Table;`n"}

              $CreateTable +="CREATE TABLE $Table ( `n"

              $col = 0

              if ($RowId)

              {

                     $col++;

                     $CreateTable +="$RowId INT NOT NULL IDENTITY(1,1) PRIMARY KEY CLUSTERED `n"

              }

              foreach($Property in $Properties)

              {

                     $col++;

                     if ($col -gt 1) { $CreateTable +="," }

                     # In below, why not use "if ($Property.Value -is [datetime])"?

                     # Because access can be denied to the value, but Property.TypeNameOfValue would still be accessible!

                     if ($Property.TypeNameOfValue -eq "System.DateTime")

                     {

                           $CreateTable +="$($Property.Name) DATETIME NULL `n"

                     }

                     else

                     {

                           $CreateTable +="$($Property.Name) NVARCHAR(MAX) NULL `n"

                     }

              }

              $CreateTable +=")"

              write-debug "Will execute SQL to create table: `n$CreateTable"

              $SqlCommand.CommandText = $CreateTable

              $rows = $SqlCommand.ExecuteNonQuery()

       }

       #Prepare SQL insert statement and execute it

       $InsertStatement = "INSERT $Table VALUES("

       $col = 0

       foreach($Property in $Properties)

       {

              $col++;

              if ($col -gt 1) { $InsertStatement += "," }

              #In the INSERT statement, do speacial tratment for Nulls, Dates and XML. Other special cases can be added as needed.

              if (-not $Property.Value)

              {

                     $InsertStatement += "null `n"

              }

              elseif ($Property.Value -is [datetime])

              {

                     $InsertStatement += "'" + $Property.Value.ToString("yyyy-MM-dd HH:mm:ss.fff") + "'`n"

              }

              elseif ($Property.Value -is [System.Xml.XmlNode] -or $Property.Value -is [System.Xml.XmlElement])

              {

                     $InsertStatement += "'" + ([string]$($Property.Value.Get_OuterXml())).Replace("'","''") + "'`n"

              }

              else

              {

                     $InsertStatement += "'" + ([string]$($Property.Value)).Replace("'","''") + "'`n"

              }

       }

       $InsertStatement +=")"

       write-debug "Running insert statement: `n $InsertStatement"

       $SqlCommand.CommandText = $InsertStatement

       $rows = $SqlCommand.ExecuteNonQuery()

  }

  end

  {

       write-debug "closing SQL connection..."

       $SqlConnection.Close()

  }

}

Enjoy,
Alexey

In recent months I had a few chances to use MERGE statement, and I realized that frequently input data applies only to a “slice” or "window" (could say “partition”, but let’s not reuse that term) of the target table. Let me illustrate on example.

Suppose database tracks occupancy of offices in a campus with multiple buildings. Suppose that updates always come in per-building batches. Below shows how to handle such cases in optimal way.

Let’s define our table structure and get some sample data:

--USE <YourDatabase>
DROP TABLE BuildingOccupancy
DROP TABLE ImportBuildingData
GO

-- Define tables

-- This is our target table
CREATE TABLE BuildingOccupancy(BuildingId int, OfficeNumber int, OccupiedBy int,
     CONSTRAINT BuildingOccupancy_PK PRIMARY KEY(BuildingId, OfficeNumber));

-- This is our source data.
-- Note that it does not have BuildingId because business rules dictate that import is done per-building
-- The corresponding BuildingId to be imported will come from a variable
CREATE TABLE ImportBuildingData(OfficeNumber int, OccupiedBy int,
     CONSTRAINT ImportBuildingData_PK PRIMARY KEY(OfficeNumber));

-- Populate tables with mock-up data

-- Create target table's data.
-- Say we have 40 buildings, each with 5 floors and 200 offices on each floor.
-- That is 40K rows
INSERT BuildingOccupancy(BuildingId,OfficeNumber,OccupiedBy)
SELECT Building.BuildingID,Office.OfficeNumber, OccupiedBy = ROW_NUMBER() OVER (ORDER BY Office.OfficeNumber,Building.BuildingID)
FROM
  (
    -- this just returns numbers 1..40
    SELECT TOP 40 BuildingID = ROW_NUMBER() OVER (ORDER BY object_id) FROM msdb.sys.objects
  ) AS Building
  CROSS JOIN
  (
    -- this just returns numbers 1..1000
    SELECT TOP 1000 OfficeNumber = ROW_NUMBER() OVER (ORDER BY object_id) FROM msdb.sys.objects
  ) AS Office

-- Create some import data.
-- Suppose the new building data is 90% unchanged, the rest 10% are split between
-- 5% deletes (people "moved out" of the offices) and 5% updates (someone else moved into the office)
-- Let's say import data happens to be about building 12

INSERT ImportBuildingData(OfficeNumber, OccupiedBy)
SELECT OfficeNumber, OccupiedBy
FROM BuildingOccupancy WHERE BuildingId = 12

-- Delete 5% of the data
DELETE ImportBuildingData
WHERE OfficeNumber % 20 = 1

-- Update 5% of the data
UPDATE ImportBuildingData
SET OccupiedBy = OccupiedBy+30000
WHERE OfficeNumber % 20 = 5

Now, let’s import the data using the MERGE statement. We’ll compare two approaches which produce same result:

-- Approach 1:

DECLARE @InputBuilding INT = 12;

MERGE BuildingOccupancy AS t
  USING ImportBuildingData AS s
  ON t.OfficeNumber = s.OfficeNumber AND t.BuildingId = @InputBuilding
WHEN MATCHED AND t.OccupiedBy <> s.OccupiedBy THEN
  UPDATE SET OccupiedBy = s.OccupiedBy
WHEN NOT MATCHED BY TARGET THEN
  INSERT VALUES(@InputBuilding, OfficeNumber, OccupiedBy)
WHEN NOT MATCHED BY SOURCE AND t.BuildingId = @InputBuilding THEN
  DELETE;

-- Approach 2:

DECLARE @InputBuilding INT = 12;

;WITH FilteredBuilding AS (
       SELECT *
       FROM BuildingOccupancy
       WHERE BuildingId = @InputBuilding
)
MERGE FilteredBuilding AS t
  USING ImportBuildingData AS s
  ON t.OfficeNumber = s.OfficeNumber AND t.BuildingId = @InputBuilding
WHEN MATCHED AND t.OccupiedBy <> s.OccupiedBy THEN
  UPDATE SET OccupiedBy = s.OccupiedBy
WHEN NOT MATCHED BY TARGET THEN
  INSERT VALUES(@InputBuilding, OfficeNumber, OccupiedBy)
WHEN NOT MATCHED BY SOURCE THEN
       DELETE;

Approach 1 uses AND t.BuildingId = @InputBuilding in the delete section to make sure we don’t delete data that belongs to other buildings.
Approach 2 uses CTE (Common Table Expression – refresher) to restrict MERGE operation to only building of interest. Note that there’s no longer need for additional condition in the delete section.

While both approaches achieve same result, performance is quite different. Approach 1 does 500+ reads and Approach 2 does less than 10. This is because the whole target table (BuildingOccupancy) is scanned in the first approach, and in the second approach index seek is used (please feel free to inspect this in the query plan).

You may have noted that approach 2 is using a redundant AND condition in the join (e.g. why did I add AND t.BuildingId = @InputBuilding if it’s already filtered out in the CTE). This is actually needed to help optimizer a bit. If we were to use literal value, the additional clause would not be necessary. SQL team might optimize this in the next version, but adding this extra clause does not hurt.

MERGE also allows use of view or TVF as a source, but I believe using CTE is most readable choice.

Acknowledgements: Thanks to Sara Tahir for the CTE trick and to Andrew Richardson for explaing the need of redundant join clause.

P.S. If you have SQL Server 2008 older than RC0, above MERGE statements will fail with syntax error. There was a slight wording change in merge clauses between CTP6 and RC:

Old Syntax:	New Syntax (RC0 and all subsequent releases):
WHEN TARGET NOT MATCHED	WHEN NOT MATCHED BY TARGET
WHEN SOURCE NOT MATCHED	WHEN NOT MATCHED BY SOURCE

• Speed. Once the process is established, it's quick to follow.
• Lesser chance of making mistake while navigating the setup GUI.
• Guarantees that instances in your project are installed exactly the same way.
• If setup runs into problems, it's very easy to escalate, because you know exactly what you did

This is how typical SQL command-line setup command looks like:

setup.exe /q /PID="XXXXX-XXXXX-XXXXX-XXXXX-XXXXX" /ACTION=install /FEATURES=SQL,IS,Tools,BIDS /InstanceName=MSSQLSERVER /SQLSVCACCOUNT="DOMAIN\svcaccount" /AGTSVCACCOUNT="DOMAIN\svcaccount" /SQLSYSADMINACCOUNTS="BUILTIN\ADMINISTRATORS" /ISSVCAccount="NT AUTHORITY\Network Service" /AGTSVCSTARTUPTYPE=2 /SQLSVCSTARTUPTYPE=2/ISSVCSTARTUPTYPE=2 /InstanceDir=D:\ /ERRORREPORTING=1 /SQMREPORTING=1 /SQLSVCPASSWORD="****" /AGTSVCPASSWORD="****" /SQLSYSADMINACCOUNTS="BUILTIN\ADMINISTRATORS" "DOMAIN\svcaccount"

A few comments:

• The complete syntax and all the options can be found in Books Online (How to: Install SQL Server 2008 from the Command Prompt)
• The edition is inferred from the /PID. If /PID is not specified, evaluation edition (180-day expiration) is installed.
• There are some pre-requisites to SQL Setup (Windows Installer 4.5, Powershell). If they are not installed, the setup will install them first and may ask you to reboot and re-run the setup.
• At least with SQL Server 2008 RC With Vista/Win2008 it's better to run the setup from elevated (Administrator) command prompt. If ran from non-elevated command prompt, the setup will not be able to print errors or messages - it will just silently end, and you will have to look at the setup logs to see what happened.
• Even with successful setup, the logs should be inspected. The summary log is here: "C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Log\Summary.txt". On successful install, there's a "Final result:                  Passed" line at the top of the log.
• For couple of reasons, I don't like SQL installed into C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\... - I prefer that SQL is installed in the root of a different drive. Hence I specified /InstanceDir=D:\ (SQL will be installed into D:\MSSQL10.MSSQLSERVER\MSSQL), but if you like default just don't specify this parameter.

Now, the fresh install of SQL will not allow connections from outside. Continuing command-line theme, here is how one can enable Named Pipes and TCP/IP (SQL needs to be cycled for changes to take effect):

WMIC /NAMESPACE:\\root\Microsoft\SqlServer\ComputerManagement10 PATH ServerNetworkProtocol WHERE "ProtocolName='Tcp' OR ProtocolName='Np'" CALL SetEnable

Open up port 1433 for TCP/IP connections:

netsh firewall add portopening TCP 1433 SQLServerDefaultInstance

You will need to tweak these commands for your environment, but I'm sure that small investment into getting your command-line install steps documented will speed up your installs and bring better predictability.

As a DBA, I found especially useful to provide a guide to Developers and Testers for the SQL setups. Now when I log into their server to troubleshoot something, I know how exactly SQL was installed.