Project Description

Often there is a need to move data from an application database to an archive database and if it was as simple as copying the entire table, it would be no problem and the wizard would provide what you need.  But as the tables get larger, the idea of copying the entire table would present far too much load and delays on the source application database.  So in this example we will create an SSIS project which copies data from the source database to the archive database based on a key column based on the max key value found in the archive database so only new records added since the prior run are added. 

(While this code will be close to complete there are some things which require additional consideration prior to using this code in a true production environment.)

Prerequisites

This blog assumes you followed and completed the first blog in this series "SSIS 2012 EzAPI - Basic Intro".  If you have not completed that post you may have trouble following this post since this post assumes you are starting with the project as it is left in the prior post.

Database Setup

In order to use the code in this blog you will need to create a table called TablesToArchive where we will get the list of tables to build into the SSIS packages and what the key column for each table will be.  Scripts to create these examples can be found in the "Database and Table create scripts.sql" file included in the Zip for this blog.

New C# Code

In the last blog we just added all the code to the "Main" function, but since we will actually make real SSIS packages this time we will add new functions to make sure we code that is correct and readable.  But we have a problem...In order to get only new data from the source database we need to look at the destination database to find the max prior key that we loaded.  In order to do this, we need to use an ExecSQL task in SSIS and return the results to a variable so it can be used in a "Data Flow" task to filter the selected data.  But the EzAPI code does not have the functions needed to return results from an ExecSQL task so we are going to need to add that first.

On line 608 of "EzExecutables.cs" you need to add the following code so you can include result set bindings when dealing with EzExecSQL tasks.

  publicIDTSResultBindings ResultSetBindings

   {

     get { return (host.InnerObject asExecuteSQLTask).ResultSetBindings;}

   }  

And on Line 690 of the same file.

    publicbool UseProjectReference

   {

        get { return (bool)host.Properties["UseProjectReference"].GetValue(host); }

         set { host.Properties["UseProjectReference"].SetValue(host, value); }

   }

Finally on line 1023 of "EzComponents.cs" you need to add the following code so you can use a variable as a source sql string for an OLEDBSource in a Dataflow task.

  publicstring SqlCommandVar

   {

     get { return (string)m_meta.CustomPropertyCollection["SqlCommand"].Value; }

     set 

        {

             if (AccessMode != AccessMode.AM_SQLCOMMAND && AccessMode != AccessMode.AM_SQLCOMMAND_VARIABLE)
                    m_comp.SetComponentProperty("AccessMode", AccessMode.AM_SQLCOMMAND_VARIABLE.GetHashCode());
             m_comp.SetComponentProperty("SqlCommandVariable", value); ReinitializeMetaData();

        }

   }

Actual Console Project code

Next are the code changes required for the Console Project so we can create SSIS projects via EzAPI.

First, since I am starting with the project code from the first blog post I wanted to change a few things to make sure we don't keep building a project with the name "FirstEzAPISSISPackage.Exe" so I needed to right click on the Project and change a few names in the project properties menu.

Next for some of the code we are adding you will need to make sure you add a reference to "Microsoft.SqlServer.SQLTask".  This is done just like the step from the first Blog when we added the other references.

using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 using System.Threading.Tasks;
 
 using Microsoft.SqlServer.Dts;
 using Microsoft.SqlServer.SSIS.EzAPI;
 
 
 namespace ArchiveEzAPIPackage
 {
 class ArchiveSSIS
 {
 static void Main(string[] args)
 {
 
 //Unlike in 2008R2 and earlier we start with a Project in SSIS now
 EzProject myEzAPIProject = new EzProject();
 
 //With EzAPI YOU ALWAYS WANT TO NAME EVERYTHING (if not you get a GUID for the name)
 myEzAPIProject.Name = "ArchiveDataProject";
 
 //now lets create a package which can be used to run all the other packages as one start point
 EzPackage MasterPackage = new EzPackage();
 
 //Dont forget to name the master package...
 MasterPackage.Name = "Archive Data Master Package";
 
 //Hard coding the source and Archive DB connections for now. This should be a parameter in a true package
 string SourceConnString = "Data Source=.\\SQL2k12;Initial Catalog=DemoSource;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;";
 string ArchiveConnString = "Data Source=.\\SQL2k12;Initial Catalog=DemoArchive;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;";
 
 
 //Create the connection and command to get the list of tables and loop through them
 string ListSourceConnString = "Data Source=.\\sql2k12;Initial Catalog=dba;Integrated Security=True;";
 string query = "SELECT [TableName], [KeyColumn], [KeyType] FROM [dbo].[TablesToArchive]";
 System.Data.SqlClient.SqlConnection conn = new System.Data.SqlClient.SqlConnection(ListSourceConnString);
 System.Data.SqlClient.SqlCommand cmd = new System.Data.SqlClient.SqlCommand(query, conn);
 
 //Open the connection
 conn.Open();
 
 //create data adapter
 System.Data.SqlClient.SqlDataReader dr = cmd.ExecuteReader();
 
 //Loop through all the rows 
 while (dr.Read())
 {
 //get the columns of data from the source list
 string TableName = dr["TableName"].ToString();
 string KeyColumn = dr["KeyColumn"].ToString();
 string KeyType = dr["KeyType"].ToString();
 
 //get a package reference with the archive package
 EzPackage ArchivePackage = CreateTransferPackage(TableName, KeyColumn, KeyType, SourceConnString, ArchiveConnString);
 
 //add that archive package to the Project
 myEzAPIProject.AddPackage(ArchivePackage);
 
 //now that the archive package is part of the project we need the master package to have an exec package task for it
 EzExecPackage exPack = new EzExecPackage(MasterPackage);
 exPack.PackageName = TableName.Replace("[", "").Replace("]", "").Replace(".", "").ToString() + ".dtsx";
 exPack.UseProjectReference = true;
 exPack.Name = TableName.Replace("[", "").Replace("]", "").Replace(".", "").ToString();
 
 }
 
 //Close the connection
 conn.Close();
 
 
 //Finally add your master package to the project
 myEzAPIProject.AddPackage(MasterPackage);
 
 //When you save the Project it saves the entire project in a single ISPAC file
 myEzAPIProject.SaveTo("c:\\SSIS\\ArchiveDataSSISProject.ispac");
 
 }
 
 static EzPackage CreateTransferPackage(string _tblName, string _KeyColumn, string _KeyColumnType, string _SourceConnString, string _ArchiveConnString)
 {
 EzPackage CTP = new EzPackage();
 
 //create a name for the package but in case 
 // any "[" "]" "." characters are in the table name replace them
 // with an empty string.
 string SafeTableName = _tblName.Replace("[", "").Replace("]", "").Replace(".", "").ToString();
 CTP.Name = SafeTableName;
 
 
 //Add a connection to access the Source Database
 EzSqlOleDbCM srcConn = Activator.CreateInstance(typeof(EzSqlOleDbCM), new object[] { CTP }) as EzSqlOleDbCM;
 srcConn.ConnectionString = _SourceConnString;
 srcConn.Name = "SourceDB";
 
 //Add a connection to access the Archive Database
 EzSqlOleDbCM ArchiveConn = Activator.CreateInstance(typeof(EzSqlOleDbCM), new object[] { CTP }) as EzSqlOleDbCM;
 ArchiveConn.ConnectionString = _ArchiveConnString;
 ArchiveConn.Name = "ArchiveDB";
 
 //Based on the Type of Column the Key is create a Variable of that Type so it can store the max Key from the Archive
 if (_KeyColumnType.ToUpper() == "DATETIME")
 {
 CTP.Variables.Add("_MaxKey", false, "User", "1900-01-01 00:00:00:000");
 }
 else //we are only supporting Datetime and Int so far but this can easily be extended
 {
 CTP.Variables.Add("_MaxKey", false, "User", 0);
 }
 
 EzExecSqlTask GetMaxVal = new EzExecSqlTask(CTP);
 GetMaxVal.Connection = ArchiveConn;
 //make the string based on the KeyField Value from the Config Table because a few of the tables call RecoveryKey something else...
 //Note that this process does not support a composite key, only when a single key exists in a single column with this work without error or manual update
 
 if (_KeyColumnType.ToUpper() == "DATETIME")
 {
 GetMaxVal.SqlStatementSource = "Select convert(varchar, convert(datetime, isnull(Max(" + _KeyColumn + "), '1/1/1900 00:00:00:000')), 121) as recKey from " + _tblName;
 }
 else
 {
 GetMaxVal.SqlStatementSource = "Select isnull(Max(" + _KeyColumn + "), -1) as recKey from " + _tblName;
 }
 
 //Dont forget to name the ExecSQL task
 GetMaxVal.Name = "Get Max Key Value";
 
 //get the max recoveryKey value
 GetMaxVal.ResultSetType = Microsoft.SqlServer.Dts.Tasks.ExecuteSQLTask.ResultSetType.ResultSetType_SingleRow;
 GetMaxVal.ResultSetBindings.Add();
 Microsoft.SqlServer.Dts.Tasks.ExecuteSQLTask.IDTSResultBinding resBin = GetMaxVal.ResultSetBindings.GetBinding(0);
 resBin.DtsVariableName = "User::_MaxKey";
 resBin.ResultName = "recKey";
 
 //Create a variable which will be the source of the DataFlow task
 Microsoft.SqlServer.Dts.Runtime.Variable sqlStr = CTP.Variables.Add("_SourceSQL", false, "User", "select * from " + _tblName + " with (nolock) where " + _KeyColumn + " >= 0 ORDER BY " + _KeyColumn);
 sqlStr.EvaluateAsExpression = true;
 if (_KeyColumnType.ToUpper() == "DATETIME")
 {
 sqlStr.Expression = "\"select * from " + _tblName + " with (nolock) where " + _KeyColumn + " > convert(datetime, '\" + (DT_WSTR, 50) @[User::_MaxKey] + \"') ORDER BY " + _KeyColumn + " \"";
 }
 else
 {
 sqlStr.Expression = "\"select * from " + _tblName + " with (nolock) where " + _KeyColumn + " > \" + (DT_WSTR, 30) @[User::_MaxKey] + \" ORDER BY " + _KeyColumn + " \"";
 }
 
 //Create the dataflow task
 EzDataFlow dataFlow = new EzDataFlow(CTP);
 dataFlow.Name = "Move New Records";
 
 //Connect the dataflow task to the execSQL task
 dataFlow.AttachTo(GetMaxVal);
 
 //Create an OLEDbSource Connection which will use the Variable with max key 
 EzOleDbSource source = Activator.CreateInstance(typeof(EzOleDbSource), new object[] { dataFlow }) as EzOleDbSource;
 source.Connection = srcConn;
 
 source.SqlCommandVar = "User::_SourceSQL";
 source.Name = "Source Data for " + SafeTableName;
 
 EzOleDbDestination dest = Activator.CreateInstance(typeof(EzOleDbDestination), new object[] { dataFlow }) as EzOleDbDestination;
 dest.Connection = ArchiveConn;
 dest.Table = _tblName;
 dest.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD;
 dest.FastLoadMaxInsertCommitSize = 10000;
 dest.FastLoadOptions = "ROWS_PER_BATCH = 10000";
 dest.Name = "Dest for " + SafeTableName;
 
 
 //Connect the source and Dest in the DataFlow task
 dest.AttachTo(source);
 
 
 //Return the Package to be added to the Project
 return CTP;
 }
 }
 }
 

Don't worry, since there is a lot more code this time around, I am going to also include a Zip file with all the code samples zipped up to make life a little easier.

Testing and Results

Once again when you build this project you will end up with an EXE file you need to run to get the IsPac project file but once you do, assuming you used the examples I built for this package you should have a project file that has 3 different packages in it.  Since this is a more advanced package which may be worth taking a look at I am going to walk though how to get this ispac file in SSDT. 

Step 1 is to Open Visual Studio
(I am sorry for switching versions here but I had 2010 installed before 2012 was released so my 2012 SSIS packages are edited in 2010 which will be almost identical to 2012 and should be easy to jump back and forth).

Select New Project then open the SSIS Project Type menu (Note this is not the SSIS option on the main Business Intelligence menu.  You need to open it and select Integration Services to get the option we need).

On the Integration Services menu you have 2 options (the import option is not on the main BI screen).

Again you need to select the Import option.

Once you select the import option you will get a menu much like what you see if you were going to deploy an ISPAC file.  Select the ispac file created when you run the EXE file created by building the package then select next.

Finally if there is no problem you will have a screen that confirms your import of the package into a new project.

Here is an example of the solution you will see if you used the scripts from this demo.

Finally, if you check any of the Archive table packages you will see identical layouts and the "Master" package a list of ExecPackage tasks for each archive package created by this process.  Here is an example of one of each type

(Master is on top and one of the archive packages on the bottom).

DemoCode.zip

Setup (Download and Build EzAPI)

First thing you are going to need to do is download the Source Code for EzAPI (SSIS 2012 version).

Instead of downloading the installer I prefer to download just the source code so I can include it as a project reference when setting up my project to create packages.  So we will start by going to http://sqlsrvintegrationsrv.codeplex.com/SourceControl/latest

From here you can click on the Main section in the lower left menu to expand the grouping.

While “main” is selected, click on the “Download” link.

This should download “sqlsrvintegrationsrv-100610.zip”
(Note that if a new version has been released for a later version of SQL Server you need to make sure you download the history version for 2012 to follow this guide).

Extract all the files in the Zip file so you can open a project to begin working with EzAPI.

Inside the Zip file is 2 folders that look like this:

Open the “main” folder, then “EzAPI” folder and finally the “src” folder.

You should now be in the “src” folder and it should look like this:

Now you can open the “EzAPI.csproj” file starting a new project in Visual Studio 2012.

Before moving any farther you should make sure you can “Build” the solution to make sure you have the references required for the project to compile. 

Creating your first EzAPI based package/SSIS project

Once the project has been successfully built EzAPI you can move on to creating your first SSIS Project.

In this case “Right Click” on the Solution and select Add -> New Project

Then select “Visual C#” -> “Windows” -> “Console Application”

(Don’t forget to give the project a name.  I used “FirstEzAPIPackage” for this example.)

While the new project is selected you want to make sure you add references to the following DLLs from the .Net tab.

Microsoft.SqlServer.DTSRuntimeWrap

Microsoft.SqlServer.ManagedDTS

Now so we can use EzAPI but also add new features in later blog posts we will add a Reference to the copy in our Solution.

Now we add the Code to the program file in the project.

 using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;
 using System.Threading.Tasks;
 
 using Microsoft.SqlServer.Dts;
 using Microsoft.SqlServer.SSIS.EzAPI;
 
 
 namespace FirstEzAPIPackage
 {
 class Program
 {
 static void Main(string[] args)
 {
 
 //Unlike in 2008R2 and earlier we start with a Project in SSIS now
 EzProject myEzAPIProject = new EzProject();
 
 //With EzAPI YOU ALWAYS WANT TO NAME EVERYTHING (if not you get a GUID for the name)
 myEzAPIProject.Name = "MyFirstEzAPISSISProject";
 
 //now lets create 2 empty packages just to see an example of the process
 EzPackage FirstEzAPIPackage = new EzPackage();
 EzPackage SecondEzAPIPackage = new EzPackage();
 
 //Dont forget to name them...
 FirstEzAPIPackage.Name = "FirstPackage";
 SecondEzAPIPackage.Name = "SecondPackage";
 
 
 //One of the last steps you take is to add your packages to your project
 myEzAPIProject.AddPackage(FirstEzAPIPackage);
 myEzAPIProject.AddPackage(SecondEzAPIPackage);
 
 
 //When you save the Project it saves the entire project in a single ISPAC file
 myEzAPIProject.SaveTo("c:\\SSIS\\FirstEzAPISSISProject.ispac");
 
 }
 }
 }
 

Final steps – Compile and Test

Save your changes, then Build the new project.

In the last line of the code we setup the project to save to “C:\SSIS\” folder.  So you need to make sure you have an SSIS folder off the C drive (or change that path in the code) prior to compiling and running your code.

Once your project is built you need to remember that you don’t have the IsPac project file created yet.  The Build compiles the Console Application you just created into a .EXE file.  If you look at the second to last line of the “Output” screen you can copy the full path to your .EXE file so you can run the code to test it out.

In this case we didn’t add any parameters, so just open the command prompts and run the file.

We also didn’t add any output yet so a few seconds later it should complete without printing anything.

Now if we open the C:\SSIS folder we should have a new .IsPac file named “FirstEzAPISSISProject.ispac”

 
If you want to check the contents you can either load the package into SSDT, deploy the code to an SSIS server or take the shortcut and just check the contents.  If you change the IsPac file type to a .Zip instead you can actually open the file and see that it has 2 packages and some config/parameter files inside just like any other compiled SSIS project should.

Next Steps

In the next blog entry I will be actually making the packages from this example do something then in future versions look into adding Project level parameters to show configuration options added with SSIS 2012.  If there are specific features you would like to see in a future blog please leave a comment here.

Paul Rizza’s T-SQL, SSIS and SSRS Blog

The blog is actually the intro to a class I teach for SQL Saturdays for PASS (Professional Association for SQL Server) called “Shortcuts to SSIS via Dot Net”.  In that class I cover the basics of getting started building your first package via the SSIS API Libraries.  In the Blog I am going to go into the detail I can’t cover on the screen in that class.    

Background:

Each time someone new to SSIS starts to work with the tool the first thing they notice is that the SSIS packages are really just a complex XML document.  Then they start to think how easily they could use a quick program to pump out the packages they need with the minor metadata changes covered by the script.  Then they may even go the one step farther and find that all of the SSIS development tools that Microsoft provides are actually just interfacing with the SSIS API libraries, so if their code can just interface with those libraries they are one step closer to home. 

Then they start actually trying to develop this code, or extend a package using the API libraries.  Depending on the background and abilities of the developer going down this path, the next few days to 3 weeks normally has an almost identical story.  They try to find documentation and examples of how to build even a simple package, and they learn it doesn’t really exist.  Then they start to try to work with the code directly and run into roadblock after roadblock.  After the normal amount of time has passed and they have exactly ZERO work to show for the efforts they admit that the development tools from Microsoft are actually doing a lot of steps behind the scenes.  From the GUI you draw a line for your source object to your destination object and a few seconds later you have source to destination mapped in a transform.  What may not have been clear was when they drew that simple line, SQL Server Data Tools [SSDT] (the new replacement for BIDS in SQL Server 2012) actually just got the metadata for all the columns (Source and Destination), tested them to see they can be mapped (not putting a VARCHAR data string into a DateTime), and setting all the default transformation properties. 

This problem of needing to do all the steps that SSDT did for you in the code is exactly where EzAPI can step in and shine. 

What is EzAPI:

EzAPI started as a support DLL used by members of the SSIS development team to test new API’s as they are coded without the need to spend time looping them into SSDT just to test the one new feature.  Once they were using it all the time, someone had the great idea that if it was helping the SSIS development team, then maybe it could help actual SSIS users.  As a codeplex Project EzAPI is not officially supported by Microsoft, but I will say that for now the SSIS development team still seems to be releasing updates to the code. 

EzAPI’s base DLL and the source code (C#) that generates that code can be found at http://sqlsrvintegrationsrv.codeplex.com/

Please note that if you are looking to download the code and samples, but do not want to use SQL Server 2012 (or whatever version is the latest as you read this document) you want to check the release notes carefully.  The 2008R2 version for example was last stable in the release tagged with Change Set 65381.  You can find the notes I am referring to by following the steps in the image below:

Wait, my client does not want custom code or DLLs on the Production SQL or SSIS Servers

Here is where EzAPI can really shine.  If used correctly, the results of using the tool is an SSIS package, with no need to install the EzAPI DLL anywhere except your own DEV machine so it can spit out more SSIS packages.  There is only one minor difference between the package made with EzAPI and the ones made with SSDT.  Specifically, packages that are made programmatically (either with EzAPI, the root SSIS DLLs or any other tool) all lack a package layout definition.  The layout is what tells SSDTs that TaskA needs to be 113 pixels to the left and 64 pixels below TaskB.  At runtime the layout is ALMOST useless.  I say almost because there are some 3^rd party logging tools that use the layout definition to help mock a SSIS package as though it was being run in debug mode.  One of those tools is called BiExpress. 

The good news is that if you made your packages programmatically, but now need a Package Layout added to the SSIS package, you can open the SSIS package in SSDTs, move any single object 1 pixel to the left/right/top/bottom and it will then generate a package layout for the next time you save the package.  If you open the package in a text editor, you can normally see the Layout information because it does not have the same use of XML (the XML is broken into code since it is code within code). 

Layout Code in the Text Editor:

SSIS Code in the Text Editor:

As you can see the SSIS code is cleaner, the Layout uses a lot of break out characters.

Getting EzAPI Started:

First thing you need to note.  This is C# programming, not SSIS, so if you do not have the C# project type, you are not going to proceed past this point without getting visual studio and the C# project type. 
For this example I am going to work with SSIS on SQL 2008R2, using Visual Studio 2010. 

I am going to start by opening the Solution in the PaulRizza-BlogEntry2.Zip file from this blog.  This example Solution has the 3 projects that I cover in my basic 1 hour PASS class.  Specifically, this zip file contains the 3 project:
 

1.       A simple project that generates a very simple first package so you can get the hang of using EzAPI

2.       The EzAPI source code that complies into the EzAPI DLL so it can be using and referenced in the other project types

3.       A complex package starting to show all the functionality that EzAPI has to offer

Adding RowCount as an EzAPI Component:

Since EzAPI extends the default SSIS API library, any of the default components can be inherited and used by the EzAPI Class.  In this case, if you search the EzAPI project for the string “//DEMO” you will example code where I extended EzAPI to include the SSIS RowCount Component.  In this case, the code that I am showing below was added to the EzComponents.cs file so the SSIS package generation programs were able to also build out RowCount components in the SSIS packages.

//DEMO

    [CompID("{150E6007-7C6A-4CC3-8FF3-FC73783A972E}")]

    public class EzRowCount : EzComponent

    {

        public EzRowCount(EzDataFlow dataFlow) : base(dataFlow) { }

        public EzRowCount(EzDataFlow parent, IDTSComponentMetaData100 meta) : base(parent, meta) { }

        public string VariableName

        {

            get { return (string)m_meta.CustomPropertyCollection["VariableName"].Value; }

            set { m_comp.SetComponentProperty("VariableName", value); }

        }

    }

The hardest part is finding the CompID which is a 16 byte GUID that correctly References the SSIS API Component ID.  In my case, I selected the GUI from a component that was already there, then used BING.Com to find a larger list that included most of the Components that add not be set.  In my case, it was "{150E6007-7C6A-4CC3-8FF3-FC73783A972E}" that I was looking for. 

In addition, the Row Count control only has 1 setting, which is the Variable that stores the current count so it can be used later in the package. 

Building that first package (SimplePackage Project from the zip file):

I am going to walk through each section of the code and example what that section does.

        static void Main(string[] args)

        {

            //handle command line items

            string _CL_PackageName = " ";

            string _CL_SourceServerName = " ";

            string _CL_SourceDBName = " ";

            string _CL_DestServerName = " ";

            string _CL_DestDBName = " ";

            string _CL_TableName = " ";

            if (args.Count() == 0)

            {

                _CL_PackageName = "SimplePackageExample";

                _CL_SourceServerName = "Localhost\\sql2k8r2";

                _CL_SourceDBName = "DemoSource";

                _CL_DestServerName = "Localhost\\sql2k8r2";

                _CL_DestDBName = "DemoDest";

                _CL_TableName = "Delta.Example1";

            }

            else if (args.Count() != 6)

            {

                Console.WriteLine("Invalid Command Line Format");

                Console.WriteLine("Usage: " + System.AppDomain.CurrentDomain.FriendlyName.ToString() + " {PackageName} {SourceServerName} {SourceDatabaseName} {DestServerName} {DestDatabaseName} {TableName}");

                System.Environment.Exit(-1);

            }

            else

            {

                //handle comand line items

                _CL_PackageName = args[0];

                _CL_SourceServerName = args[1];

                _CL_SourceDBName = args[2];

                _CL_DestServerName = args[3];

                _CL_DestDBName = args[4];

                _CL_TableName = args[5];

            }

This class is setup so that when it is run from the command line, different parameters can be passed which result in different packages being produced.  Each parameter is fairly self-explanatory but just in case here is the quick rundown:

1.       _CL_PackageName = the name you want to use for the package at runtime, as well as the filename.  Note that the filename will also include the TableName (As defined in a later command line variable) being transferred.

2.       _CL_SourceServerName = the Server name where the data is coming from.

3.       _CL_SourceDBName = the database name where the data is coming from.

4.       _CL_DestServerName = The Server name where the data is going too.

5.       _CL_DestDBName = The database name where the data is going too.

6.       _CL_TableName = The name of the table where the data is coming from, as well as the name of the destination table. 

As I am going through the code you may also notice that each time a new control is added, one of the very first steps is to give that component a name since this makes sure the name is not defaulted to the GUID ID of that component.  The code also defaults the parameters in the event nothing is passed.  This is done to make it easy if you wanted to run a quick test without doing a compile then typing out the command line.

Creating a new package then giving it a package name.  Setting the name is very important since without the name the log file will just so the GUID and will be very hard to read correctly.

            // Creating the package and the Data Flow Task:

            EzPackage package = new EzPackage();

            package.Name = _CL_PackageName;

This next section creates the 2 connections in the connection manager and sets them up based on the command line parameters.

            // Creating a connection to the database:

            EzSqlOleDbCM srcConn = Activator.CreateInstance(typeof(EzSqlOleDbCM), new object[] { package }) as EzSqlOleDbCM;

            EzSqlOleDbCM destConn = Activator.CreateInstance(typeof(EzSqlOleDbCM), new object[] { package }) as EzSqlOleDbCM;

            srcConn.SetConnectionString(_CL_SourceServerName, _CL_SourceDBName);

            srcConn.Name = "SourceDB";

            destConn.SetConnectionString(_CL_DestServerName, _CL_DestDBName);

            destConn.Name = "DestDB";

This section creates a ExecuteSQL task that deletes data from the table Delta.Example1

            // need to add a step to delete from the DEST before we load the data

            EzExecSqlTask DelDest = new EzExecSqlTask(package);

            DelDest.Name = "Delete Dest Data";

            DelDest.Connection = destConn;

            DelDest.SqlStatementSource = "Delete from Delta.Example1";

Next we add a data flow. 

            EzDataFlow dataFlow = new EzDataFlow(package);

            dataFlow.Name = "SimplePackageDataTransformStep";

The AttachTo() function (outside of a dataflow) defaults to a On Success Flow connection.

            dataFlow.AttachTo(DelDest);

This next section adds a Source connection to the Dataflow and uses the command line table as the data source and destination.

            //add conns to dataflow

            EzOleDbSource source = Activator.CreateInstance(typeof(EzOleDbSource), new object[] { dataFlow }) as EzOleDbSource;

            source.Connection = srcConn;

            source.Name = "Source";

            source.Table = _CL_TableName;

If instead of a table as the source, you wanted to use a SQL statement, Source.Table above would be replaced by Source.SQLCommand example below.

            //source.SqlCommand = "select * from table1";

In this code the Destination is added to the Dataflow and again based on the command line table name. 

            EzOleDbDestination Dest = Activator.CreateInstance(typeof(EzOleDbDestination), new object[] { dataFlow }) as EzOleDbDestination;

            Dest.Connection = destConn;

            Dest.Name = "Destination";

AttachTo also has the option to support multiple input sources (like would be used on a Union ALL), or multiple outbound sources (like in a multicast).  AttachTo({SourceToAttachTo, IndexOfOutbound, IndexOfInbound)

            Dest.AttachTo(source, 0, 0);

            Dest.Table = _CL_TableName;

            Dest.AccessMode = AccessMode.AM_OPENROWSET_FASTLOAD;

            Dest.SetComponentProperty("FastLoadMaxInsertCommitSize", 100000);

This next to last section is used to configure the package configurations which would be used to inherit configuration settings from the parent package calling this child package.

            //Package Configs

            package.EnableConfigurations = true;

            //new config to edit

            Configuration MyConfigSource = package.Configurations.Add();

            MyConfigSource.Name = "SourceDBConfig";

            MyConfigSource.ConfigurationType = DTSConfigurationType.ParentVariable;

            MyConfigSource.ConfigurationString = "Copy_SourceDB";

            MyConfigSource.PackagePath = "\\Package.Connections[SourceDB].Properties[ConnectionString]";

            Configuration MyConfigDest = package.Configurations.Add();

            MyConfigDest.Name = "DestDBConfig";

            MyConfigDest.ConfigurationType = DTSConfigurationType.ParentVariable;

            MyConfigDest.ConfigurationString = "Copy_DestDB";

            MyConfigDest.PackagePath = "\\Package.Connections[DestDB].Properties[ConnectionString]";

Finally Save the package as a .DTSX file, or executed if this was a runtime package only

            //save the package

            package.SaveToFile(".\\PackageOutput\\" + _CL_PackageName + "_" + tableNameWithoutSchema + ".dtsx");

            //package.Execute();

Next Time…

In the next SSIS Blog, I will review the details of the Complex Package just like I did for the Simple Package today. 

Comments, Questions and Requests:

I know this is a very simple SSIS package for the Blog entry, but it is a basis which will become the foundation for much more complex packages.  Please feel free to post comments, leave questions or add requests, and I will do my best to address them as time permits.  

EzAPI - DOTNet Source.zip

Many of my clients and the students in my classes for SQL Pass SQLSaturday sessions often ask if I have a blog where I post more information, so I decided I should go ahead and start one. 

Since many of my future blog posts will be on T-SQL I decided to make this first blog a very simple one, where I can store the setup for the database I will run future code,  as well as store some of the common items I will need for those future entries.

My background:

I have been working with SQL Server since version 6.5.  For the last 13 years I have been working as a Consultant, most recently for Microsoft Corporation.  I also started with DTS in SQL 2000, and moved to SSIS in SQL 2005.  Many of my SSIS based postings will also include C# code that makes use of a set of DLLs called EzAPI.  EzAPI is a code wrapper for the SSIS DLLs that helps to produce SSIS packages in C# code without as much effort (saving 100s of hours in many cases).

Topic requests:

While I will do my best to post Blog entries on the topics I get feedback requesting, I am limited to the topics I have time to fully research.  In many cases that limitation of time is driven by my current assignments for my current clients at Microsoft.  If there is something you would like to see in more detail please leave a comment and I will do my best to cover it in a later post, but I cannot promise to cover everything. 

Getting Started:

This first script will create the database where I will keep and run all of my future blog posts:

USE [master]

GO

/****** Object:  Database [BlogDB]    Script Date: 5/27/2013 2:10:12 PM ******/

CREATE DATABASE [BlogDB] ON  PRIMARY

( NAME = N'BlogDB', FILENAME = N'C:\MSSQL\Data\BlogDB.mdf' , SIZE = 524288KB , MAXSIZE = UNLIMITED, FILEGROWTH = 131072KB )

 LOG ON

( NAME = N'BlogDB_log', FILENAME = N'C:\MSSQL\Logs\BlogDB_log.ldf' , SIZE = 131072KB , MAXSIZE = 2048GB , FILEGROWTH = 131072KB )

GO

ALTER DATABASE [BlogDB] SET COMPATIBILITY_LEVEL = 100

GO

ALTER DATABASE [BlogDB] SET RECOVERY SIMPLE

GO

This next script will create and populate a table called Tally which has a single column (N type INT) that holds the numbers from 1 to 1 Million.  You would not think that having a table with 1 million numbers in it would be that important but once you have read through a few of my blogs you are going to find that you may not be able to live without it.

USE [BlogDB]

go

-- A Tally table comes in very handy for lots of different functions (but you should make sure you keep track of how many rows you load

CREATE TABLE Tally(N INT NOT NULL PRIMARY KEY)

-- I am going to keep 1 million rows because that has worked very well for me in the past

INSERT INTO Tally

(N)

SELECT TOP 1000000 ROW_NUMBER() OVER (ORDER BY A.Object_ID) AS N

FROM

       --Just some random tables so I know my result set will have at least 1 million rows

       sys.all_objects a, sys.objects b, sys.objects c

Comments, Questions and Requests:

I know this is a very simple Blog for the first entry.  I understand that there is not much value add, but the main reason for this one is so I can reference it in the future.  As new permanent items are added to the BlogDB database I will update this entry and add them to the code for this blog.  Please feel free to post comments, leave questions or add requests, and I will do my best to address them as time permits.  

Initial setup.sql