A commonly used method achieve this is to use the following powershell script:

Import-module “C:\Program Files (x86)\Microsoft BizTalk Server 2013 R2\SDK\Utilities\PowerShell\BizTalkFactory.PowerShell.Extensions.dll”

Get-ChildItem –Path ‘BizTalk:\Health and Activity\Service Instances’ | % { $_.Terminate() }

This method is very convenient compared to the standard method of deleting all the artifacts in the BizTalk Server Administration Console.

In automated test scenarios it is often necessary to reset the test environment to its initial state. You can use the following code to create a test step to achieve this.

string scope = @"root\MicrosoftBizTalkServer";
/*
  ServiceClass possible values:
    Orchestration 1
    Tracking 2
    Messaging 4
    MSMQT 8
    Other 16
    Isolated adapter 32
    Routing failure report 64
*/
SelectQuery query = new SelectQuery("SELECT * FROM MSBTS_ServiceInstance Where ServiceClass = 1 or ServiceClass = 4 or ServiceClass = 64");

ManagementScope mScope = new ManagementScope(scope);

ManagementObjectSearcher searcher = new ManagementObjectSearcher(mScope, query);

ManagementObjectCollection btsArtifacts = searcher.Get();

foreach (ManagementObject mobj in btsArtifacts)
{
  mobj.InvokeMethod("Terminate", null);
}

Another very useful code snippet is the following. It bounces all BizTalk host instances on the local machine. We use this in our automated tests scenarios where it is necessary to restart all instances after a configuration change.

string scope = @"root\MicrosoftBizTalkServer";
/*
                ServiceState = 4 (= running)
*/

var query = new SelectQuery("SELECT * FROM MSBTS_HostInstance WHERE ServiceState = 4");
ManagementScope mScope = new ManagementScope(scope);
ManagementObjectSearcher searcher = new ManagementObjectSearcher(mScope, query);
ManagementObjectCollection btsArtifacts = searcher.Get();
foreach (var mobj in btsArtifacts.Cast<ManagementObject>())
{
   mobj.InvokeMethod("Stop", null);
   mobj.InvokeMethod("Start", null);
}

As you can see in the picture below, we decided to use a combination of clustering and BizTalk Groups to get the most flexible architecture possible with only two BizTalk nodes. Basically, the two nodes are in a Windows Server Cluster. Only those BizTalk hosts which runs FTP, MSMQ, MLLP or ordered delivery file adapters run within a cluster. All other hosts use the standard BizTalk Server load balancing mechanism to maximize throughput.

To be able to distribute the workload as flexible as possible, we planned a cluster group for each type of adapter (FTP, MSMQ, MLLP, ordered delivery file). This allows us to distribute the incoming workload on both BizTalk server nodes. Each cluster group must also hosts MSDTC so that a failure in MSDTC will also cause a failover of the corresponding BizTalk resource. MSDTC must also run locally on each server, so that all BizTalk hosts not running within a cluster group can use this local running version.

Please be aware that it is not possible to use Windows Network Load Balancing, when the BizTalk Servers are clustered. If you have a scenario where the BizTalk hosts services which should be accessed from outside (web services, http endpoints), you have to use an external load balancer. Windows load balancing is not supported when the servers are in a Windows cluster.

The BizTalk databases are hosted on a SQL Server cluster. We also use this cluster to host Enterprise Single Sign-On and MSDTC. There are multiple cluster groups in this cluster. One is dedicated to the BizTalk databases. MSDTC and SSO runs in one cluster group.

In the first chapter I am writing about the data sources and data source formats. My experience was, that it is a very important part to define the sources carefully, precisely and completely.

After the examination of the structure of the source data, I will have a look at the data content and the possible parts which are important for ETL processes.

The third chapter is an accumulation of functional requirements one may face when designing ETL processes.

Don’t consider this document as a comprehensive guide for designing ETL processes. My intent is to maintain this document by extending and adding content.

Source data

The nature of a Data Warehouse project is that data is collected from various sources and also in various formats. So one of the most essential things is to define strict contracts for the interfaces thru which the data is collected.

• Here are some questions which should be answered by the contract
• Where does the data come from?
• Which data formats are used?
• How is the data formatted?
• Which Information is necessary to handle the data correctly?
• How are different data types formatted? (number, date, time, ...)
• Content
• Is there one record in the source for one Business Case? (for example one record for one customer)
• How are deleted records marked?
• Is it a complete set of data? (e.g. all customers or just a set of changes)
• Are trailing whitespaces part of the data?

Structure

• Data source
• E.g. data files in directory or on a share, database tables or views, web services, …
• Data format
• Comma separated file, EBCDIC encoded files, etc.
• Encoding
• How are the files encoded (Code page
• Database Collatio
• Language settings
• Date Formats
• Number Formats

My recommendation is to build up a repository where you store the mentioned contracts. This repository can be an Excel workbook or even better a SQL Server database. If the repository contains a comprehensive view of all the contracts and data structures of your sources, you can use the repository as your meta-data for your ETL processes. For example in the data warehouse project I mentioned at the beginning of the article, we generated some of our ETL processes automatically with the information from such a meta-data store.

Historiography

Data validity period

In many cases it is important for the ETL Processes to know how to interpret the data. For example if the source systems provide information about the time spans within the data is valid. If this data is important for the target system, detailed information about the meaning of the data fields defining the validity period is required.

Here are some questions which should be answerable:

• How is the validity information provided (TimeFrom – TimeTo, Timestamp)
• How is the time information interpretable? E.g. what does a value for ValidTo means? (data is guilty < ValidTo or data is guilty <= ValidTo)

Historical data in a data warehouse

Data in a Data Warehouse is stored in one of two forms, historically or not historically. Historically stored data can be subdivided into time point related data or time span related data.

There are three types how data can be stored in the matter of historiography.

• Type 1: No history information is stored. Existing data is overridden by new values
• Type 2: The history of data changes is preserved. Every data row has a valid from and valid to date indicating the time period of the data’s validity.
• Type 3: This type is used for data that is valid for a specific point in time. Every record has a timestamp column indicating the point in time where it is valid.

The decision what type you use for your data should be made in the design phase of your target database. The important information for the ETL developer is where the history information for Type 2 und Type 3 data comes from.

• What time information can be used to made up the validity period information
• How to identify a deleted/invalid data record
• How to handle validity period information from the source system
• What does the validity period information mean (e.g. exact definition what a ValidTo date means: <, <=)
• What granularity is used to define validity periods?

What time information can be used to made up the validity period information

To build up a validity period you need information on what point in time a data change take effect. This point - for example - can be the timestamp of the time when the import process is running. But in most cases this time point is determined by the source system. So you have to find a way to hand this information over from the source system to your ETL process.

How to identify a deleted/invalid data record

The question is how to identify records which are no longer valid in the source system. This can be accomplished by sending the record with a flag indication the record status. If the source system always sends a complete list of valid records, another method can be to match the records from the source system with the records in the target system and set all the records which are not in the source, but in the target to invalid.

Setting a record to invalid can mean one of the following things:

• Delete the record in the target system (not the best choice)
• Mark the record as invalid (e.g. deleted timestamp in type 1 data, or set ValidToDate in type 2 data)

How to handle validity period information from the source system

Maybe the system where your data comes from already uses validity periods for their data and this information is handed over to the new system. In this case you have to define what this information means to the new system.

What does the validity period information mean?

Define exactly how validity periods are defined. What does a specific ValidTo date means? In terms of an SQL statement, does ValidTo means ‘<’ or ‘<=’.

What granularity is used to define validity periods?

In some cases the validity of data is specified on a low granularity, for example on a weekly or a monthly basis. This leaves a lot of space for interpretation. For example a data record which is defined as guilty for January 2010. What does this mean? Is the data guilty starting from the first of February or is the data guilty only for January?  All this questions has to be clarified before the development of the ETL processes can start.

Functional Requirements for the ETL processes

My recommendation when planning your ETL is to think of all the functional requirements you packages must meet and build up some prototype or template packages.

Examples:

• Transactional behavior / restart ability
• Logging
• Batch processing

Transactional behavior

One important thing to consider is the behavior of your ETL processes in the case of an error, and how you can recover your system into a consistent state after a process abortion.
So the first task to complete is to identify the objects affected by your processes. In a DWH project where the task of the processes is to move data from a source to a destination, usually the only affected system is the target database. Fortunately there is a very good, well known mechanism to ensure the consistency of a database: transactions. So to ensure your processes do not leave your destination database in an inconsistent state is to assure all of the work done in the destination runs within one transaction.

Logging

Integration Services includes logging features that writes information off all kinds of events during the package execution to a customizable location. This logging information is very technically and it is hard to acquire the required business information out of this logs. So, one thing you should think about, when you are designing your packages is what additional information is useful or required and how/where to log this data.

• Package Start and Stop time
• The Batch Name or ID the package execution was part of
• Package parameters
• Names of the input files the packages has processed
• Number of records imported/declined/…

To acquire the logging requirements

• How to analyze package execution errors in the production system
• How to analyze data errors
• How to analyze the loading process (plausibility checks)

• Is the amount of loaded data comparable to the last load
• Is the load performance stable
• Is there an increase in the count of erroneous input data

Interview the business specialists and the IT guys which will operate the system to get an comprehensive list of all the questions which should be answerable by the logged data.

Batch processing

Usually, data delivered from a source system is provided in multiple parts (e.g. data files or data tables), and this parts are related to each other, and it is necessary to process this parts in one batch. Considering this requirement is very important in the design phase of your ETL processes, because the single processing units (ETL packages) must be aware of the batch they are part of.

For example a source system which provides customers data, and the data is split into multiple parts. One part (source file) contains information about the customers personal data (name, age, …). A second part contains the details about the customers contact information (email addresses, phone numbers, …). Assume that we have 2 SSIS packages to import the customer data in this simple example. It is obvious that the packages belong to each other and that the package which imports the customers personal data must be run before the package importing the customers personal data. This is a very simplified example where it is necessary to process the packages in a batch, which distinct dependencies between the members of the batch.

• Members of a batch are related to each other
• Batches should be as small as possible to minimize dependencies.
• All related parts must be in one batch
• Package executions should be assigned to a distinct batch (batch identifier)
• Logging information should include information about the batch (e.g. a batch identifier)

Bulk load

A rule of thumb is that you should use bulk inserts whenever possible. The performance improvement can be tremendous compared to non-bulk inserts when you are inserting more than just a few hundreds of records.

• Use table looks
• Parallel writes into one target table must be avoided
• If your target tables are connected with foreign keys, you have to pay special attention to the dependencies of your ETL processes (details in the next chapter).

DWH and Foreign-Key constraints

This chapter is not a comprehensive essay about foreign key constraints. I just want to write down some of my experiences with this topic.

The scenario is form a project I was involved. My task was to load data from data files into a database with a normalized data schema. The nature of a normalized database schema is that there are many relatively small tables (in the sense of number of columns) which are heavily related to each other. These relationships must be addressed when the database is loaded with data. For example the customer base table must be loaded before the customer details table and so on. The rule of thumb is that if there is a dependency between to tables in the data schema, this dependency must be addressed when loading the data into the database. In addition to this rule there might be some more dependencies to consider. For example if you are using bulk inserts to write data into the target tables.

Here is an example how bulk inserts can affect the ordering of your ETL processes.

In the example there are three tables in the target database. One ETL process – SSIS package – exits to load each of the tables. The tables are related to each other, as you can see in the picture.

When you use a bulk insert operation to insert data into the CustomerCountry table you should not bulk insert data into CustomerAddress table at the same time. Inserting in both tables at the same time could end up in an erroneous behavior in one of the insert tasks. Bulk insert operations require that the schema of the involved tables and objects stay the same during the whole operation. But bulk inserting into a table with a constraint will change the meta-data of the constraint. This change is threaded as a schema change of the constraint in the referenced table. So if a second bulk insert operation is running even on a different table, this operation is may be ended with an error message.

As you can see in the example you have to consider a lot of dependencies, when you design your package execution plan. So, not using foreign key constraint can make your work easier and your load process faster because you can load your warehouse with more parallelism.

DataFlow Components have input or/and output column collections depending on the type of component. Components like DataSources do not have an input collection whilst destination component do not have a destination collection.

If you already have built your data flow you can examine the components columns and how they are connected to other components by querying the components input and output column collection.

As you may know data flow components are interconnected through paths. A path describes the flow of data from the sources through transformations to the destinations. Programmatically a path is represented by a IDTSPath100 object. Unless a component isn’t connected by a path it is not possible to add any columns to its input column collection. So the first step after creating the data flow components is to interconnect them with paths.

Here is an example how you can connect two components with an IDTSPath100 object:

// Create the source component. 
IDTSComponentMetaData100 sourceComponent =
dataFlowTask.ComponentMetaDataCollection.New();
sourceComponent.ComponentClassID = "DTSAdapter.OleDbSource";
CManagedComponentWrapper srcDesignTime = source.Instantiate();
srcDesignTime.ProvideComponentProperties();

// Create the destination component.
IDTSComponentMetaData100 destination =
dataFlowTask.ComponentMetaDataCollection.New();
destination.ComponentClassID = "DTSAdapter.OleDbDestination";
CManagedComponentWrapper destDesignTime = destination.Instantiate();
destDesignTime.ProvideComponentProperties();

// Create the path.
IDTSPath100 path = dataFlowTask.PathCollection.New();
path.AttachPathAndPropagateNotifications(source.OutputCollection[0],
destination.InputCollection[0]);

To retrieve the possible input columns for a component, one can use the GetVirtualInput method of the InputColumnCollection object.

IDTSInput100 input = destination.InputCollection[0];
IDTSVirtualInput100 virtualInput = input.GetVirtualInput(); 

The virtual input column collection contains all the possible columns from an upstream component.

This example code shows how to iterate over all input columns and connect them to the destination component.

      foreach (IDTSVirtualInputColumn100 column in virtualInput.VirtualInputColumnCollection)
      {
        destDesignTime.SetUsageType(
           input.ID, virtualInput, column.LineageID, DTSUsageType.UT_READONLY);
      }

1.  Add a cloud service project to your existing solution
   1. Solution Explorer right click your solution and select ‘Add -> New project …’
   2. In the Add New Project dialog select Installed Templates –> Cloud -> Windows Azure Project template
   3. Choose name and press the OK button.
   4. Don’t add a role in the New Windows Azure dialog. The existing ASP.NET web application will be used as a web role.
2. Associate the ASP.NET project to the cloud project
   1. Right click Windows Azure Project Role folder -> Add -> Web Role Project in solution ...
   2. In the Associate with Role Project dialog, select your web project and click OK.
3. Add references to the required assemblies.
   1. Right click web application project and select add reference. Add references to the following assemblies:
      1. Microsoft.WindowsAzure.Diagnostics
      2. Microsoft.WindowsAzure.ServiceRuntime
      3. Microsoft.WindowsAzure.StorageClient
4. Enable diagnostic logging by adding the following section to your Web.config.
   <system.diagnostics>
     <trace autoflush="false" indentsize="4">
           <listeners>
         <add name="AzureDiagnostics" type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
       </listeners>
      </trace>
   </system.diagnostics>
5. Modify Global.asax
   1. Set configuration settings publisher. Add the following code snippet to the Application_Start procedure:
      Microsoft.WindowsAzure.CloudStorageAccount.SetConfigurationSettingPublisher((configName, configSetter) =>
      {
          configSetter(RoleEnvironment.GetConfigurationSettingValue(configName));
        }); 
      In addition to this, you have to add using statements for the following namespaces:
        
         using Microsoft.WindowsAzure;
         using Microsoft.WindowsAzure.ServiceRuntime;
6. Deploy your solution to the cloud.

In this part of my series of blogs about dynamically modifying SSIS Packages I describe how to programmatically add or modify an Aggregate component.

Sample

The following code snippet shows how to add and modify an aggregate component by implementing the following steps

1. Add a new aggregate component to a data flow tasks.
2. Link the newly created component to another component in your data flow (this step is not included in the code example)
3. Delete the input and output column collection. This step is not necessary if you created a new aggregate component as mentioned in step 1. Do this step only in the case you modify an existing data flow, with an existing
ggregate component where you want to change the behavior by exchanging the columns in the aggregate.
4. Add input columns to the input column collection from the virtual input collection
5. Create output columns.
6. Modify the output columns properties

// Step 1: create aggregate task
IDTSComponentMetaData100 aggregate =
        dataFlowTask.ComponentMetaDataCollection.New();
      aggregate.ComponentClassID = "DTSTransform.Aggregate.2";

// Step 2:
// TODO:
// link the component to the upstream component here
CManagedComponentWrapper componentWrapper = component.Instantiate();
componentWrapper.AcquireConnections(null);

// Step 3:
// do this if you modify an existing aggregate task. You don't have
// to delete the input and output collection if you just created the
// component with the code above.
component.InputCollection[0].InputColumnCollection.RemoveAll();
component.OutputCollection[0].OutputColumnCollection.RemoveAll();

// Step 4:
// get the input/output and virutal column collection of the component
IDTSVirtualInput100 virtualInput = component.InputCollection[0].GetVirtualInput();
IDTSOutput100 outputCollection = component.OutputCollection[0];
IDTSInput100 inputCollection = component.InputCollection[0];
 

// add all the columns witch are involved in the aggregation to the
// component by retrieving the columns linage id from the virtual input
// collection and adding the column to the input column collection by calling
// the SetUsageType method.

int sourceColumnLineageId = virtualInput.VirtualInputColumnCollection[name].LineageID; 

componentWrapper.SetUsageType(
              inputCollection.ID,
              virtualInput,
              sourceColumnLineageId,
              DTSUsageType.UT_READONLY);

// Step 5:
// create a new output column
IDTSOutputColumn100 newOutputColumn = componentWrapper.InsertOutputColumnAt(outputCollection.ID, 0, "ColumnName", string.Empty);        

// set the data type porperties to the same values as these of the input column   newOutputColumn.SetDataTypeProperties(virtualInput.VirtualInputColumnCollection[name].DataType, virtualInput.VirtualInputColumnCollection["ColumnName"].Length, 0, 0, virtualInput.VirtualInputColumnCollection["ColumnName"].CodePage);

newOutputColumn.MappedColumnID = 0;

// Step 6
// modify the properties of the newly created column
for (int i = 0; i < newOutputColumn.CustomPropertyCollection.Count; i++)
{
IDTSCustomProperty100 property = newOutputColumn.CustomPropertyCollection[i];
       switch (property.Name)
       {
             case "AggregationColumnId":
                    property.Value = sourceColumnLineageId;
                    break;
             case "AggregationType":
                    property.Value = 0;
                    break;
             case "IsBig":
                    property.Value = 0;
                    break;
             case "AggregationComparisonFlags":
                    property.Value = 0;
                    break;
       }
}
 

After adding all the columns to the component which are involved in the aggregation, you may also want to add a column which holds the number of items in each group. This is done by adding another output column to the aggregate.

IDTSOutputColumn100 newOutputColumn = componentWrapper.InsertOutputColumnAt(outputCollection.ID, 0, "Count all", string.Empty);
newOutputColumn.SetDataTypeProperties(DataType.DT_UI4, 0, 0, 0, 0);
newOutputColumn.MappedColumnID = 0;
 
for (int i = 0; i < newOutputColumn.CustomPropertyCollection.Count; i++)
{
       IDTSCustomProperty100 property = newOutputColumn.CustomPropertyCollection[i];
       switch (property.Name)
       {
             case "AggregationColumnId":
                    property.Value = 0;
                    break;
             case "AggregationType":
                    property.Value = 2;
                    break;
             case "IsBig":
                    property.Value = 0;
                    break;
             case "AggregationComparisonFlags":
                    property.Value = 0;
                    break;
       }
}

Aggregate column properties

These are the possible values of the aggregate properties:

In Silverlight for Windows Phone 7 this can be done with the use of the XamlReader Class. This class can be found in the System.Windows.Markup namespace.

With the use of this class it’s very easy to create in memory representations of XAML object you created in tools like expression blend. In my example I created an ellipsis with a gradient fill and a gradient stroke.

<Ellipse Grid.Row="1" Height="40" VerticalAlignment="Top" HorizontalAlignment="Left" Width="40">
<Ellipse.Stroke>
             <LinearGradientBrush EndPoint="0.5,1" StartPoint="0.5,0">
                    <GradientStop Color="#FF03063D" Offset="0"/>
                    <GradientStop Color="#FF333CD8" Offset="1"/>
             </LinearGradientBrush>
       </Ellipse.Stroke>
       <Ellipse.Fill>
             <RadialGradientBrush>
                    <GradientStop Color="#FF06114D" Offset="1"/>
                    <GradientStop Color="#FF6F72FF"/>
             </RadialGradientBrush>
       </Ellipse.Fill>
</Ellipse>

If you try to load this xaml snippet with XamlReader.Load you will receive an error message. There are a view things you have to do with the snippet first.

1. Remove the Grid.Row attribute.
2. Add the required namespaces to the root (Ellipse) element
3. Remove or ajust position information

After the modifications the XAML looks like this:

<Ellipse xmlns=\"http://schemas.microsoft.com/winfx/2006/xaml/presentation\" xmlns:x=\"http://schemas.microsoft.com/winfx/2006/xaml\" Height=\"40\" VerticalAlignment=\"Top\" HorizontalAlignment=\"Left\" Width=\"40\">
             <Ellipse.Stroke>
             <LinearGradientBrush EndPoint=\"0,1\" StartPoint=\"0.5,0\">
                    <GradientStop Color=\"#FF02076E\" Offset=\"0\"/>
                    <GradientStop Color=\"#FF272FBA\" Offset=\"1\"/></LinearGradientBrush>
              </Ellipse.Stroke>
              <Ellipse.Fill>
             <RadialGradientBrush  GradientOrigin=\"0.35,0.65\">
                    <GradientStop Color=\"#FF06114D\" Offset=\"1\"/>
                    <GradientStop Color=\"#FF6F72FF\"/></RadialGradientBrush>
              </Ellipse.Fill>
</Ellipse>

 Now the XAML is in the right shape and we can create the Ellipse objects with the following statement:

Ellipse ellipse = (Ellipse)XamlReader.Load("<Ellipse xmlns=\"http://schemas.microsoft.com/winfx/2006/xaml/presentation\" xmlns:x=\"http://schemas.microsoft.com/winfx/2006/xaml\" Height=\"40\" VerticalAlignment=\"Top\" HorizontalAlignment=\"Left\" Width=\"40\"><Ellipse.Stroke><LinearGradientBrush EndPoint=\"0,1\" StartPoint=\"0.5,0\"><GradientStop Color=\"#FF02076E\" Offset=\"0\"/><GradientStop Color=\"#FF272FBA\" Offset=\"1\"/></LinearGradientBrush></Ellipse.Stroke><Ellipse.Fill><RadialGradientBrush  GradientOrigin=\"0.35,0.65\"><GradientStop Color=\"#FF06114D\" Offset=\"1\"/><GradientStop Color=\"#FF6F72FF\"/></RadialGradientBrush></Ellipse.Fill></Ellipse>");

 In the example the XAML for the ellipse object is hard coded but in an real project you can store it in the project resources and load it from there.

string ellipseXAML;
System.IO.Stream st = this.GetType().Assembly.GetManifestResourceStream("Ellipse.xaml");
using(StreamReader reader = new StreamReader(st) )
{
        ellipseXAML = reader.ReadToEnd();
}

Be careful, when you use it! All data of all referenced tables will be deleted.

create procedure temp.RecursiveDelete
(
     @tablename nvarchar(1000)
     ,@schemaname nvarchar(1000))
as

declare @oid as int
declare @parentSchemaName as varchar(max)
declare @parentObjectName as varchar(max)

select @oid = t.object_id from sys.tables t join sys.schemas s on t.schema_id = s.schema_id where t.name = @tablename
and s.name = @schemaname

declare @parentOid int

declare fkCursor cursor LOCAL for select parent_object_id, OBJECT_NAME(parent_object_id) from sys.foreign_keys where referenced_object_id = @oid

open fkCursor

fetch next from fkCursor into @parentOid, @parentObjectName

while @@FETCH_STATUS = 0
Begin
   select @parentSchemaName = s.name from sys.schemas s join sys.tables t
      on s.schema_id = t.schema_id
   where t.object_id =@parentOid

   exec temp.RecursiveDelete @parentObjectName,
   @parentSchemaName

   fetch next from fkCursor into @parentOid, @parentObjectName
End

-- delete data
declare @delStmt nvarchar(max)

set @delStmt = 'delete from [' + @schemaname + '].[' + @tablename + ']'

exec sp_executesql @delStmt

close fkCursor
deallocate fkCursor

go

In the first article of this series I painted the high level picture and provided an overview of the package generation framework. In this and the following articles I want to show how to programmatically generate or modify specific SSIS components.

FlatFile Source Component

The first step is to find the connection manager component to modify or to add a new connection manager to the package.

All package connection managers can be found in the Connections collection of the Microsoft.SqlServer.Dts.Runtime.Package class.

So the first thing to do is to find the Connection Manager you want to edit.

foreach (ConnectionManager connectionManager in _Package.Connections)

            {

                if (connectionManager.Name.Equals("SourceFlatFileCM"))

                {
                    …

In the case that you want to add a Connection Manager programmatically versus editing an existing one you can use the following code.

ConnectionManager conManager = _Package.Connections.Add("FLATFILE");
IDTSConnectionManagerFlatFile100 connectionFlatFile = (IDTSConnectionManagerFlatFile100)conManager.InnerObject;

A list with all the viable parameters for the Add procedure can be found on this page: http://msdn.microsoft.com/en-us/library/ms136093.aspx

Setting the necessary configurations on the Connection Manager is the next step. There are some generally valid properties and a lot of Connection Manager specific properties. The general properties are members of Connection Manger class whilst the specific properties can be set by acquiring an actual instance of the specific Connection Manager object (oleDbConnectionManager in the above example).

In my simple example I set some hardcoded properties just do demonstrate the use of the Connection Manager.

connectionManager.Name = "SourceFlatFileCM";

connectionManager.ConnectionString = "C:\Data\TestData.CSV";

             IDTSConnectionManagerFlatFile100 connectionFlatFile

         = (IDTSConnectionManagerFlatFile100)connectionManager.InnerObject;

             connectionFlatFile.RowDelimiter = "\n";

       connectionFlatFile.Format = "Delimited";

connectionFlatFile.ColumnNamesInFirstDataRow = true;

       connectionFlatFile.LocaleID = 1033;

       connectionFlatFile.CodePage = 1252;

       connectionFlatFile.RowDelimiter = "\n";

The next step is just needed if you are modifying an existing package and you want set up the column definition from scratch.

// remove existing columns

      foreach (IDTSConnectionManagerFlatFileColumn100 column in connectionFlatFile.Columns)

{

connectionFlatFile.Columns.Remove(column);

}

The final step is to add the column definitions to the Connection Managers Columns collection.

The demo code shows how to add a FlatFileColumn to the collection.

IDTSConnectionManagerFlatFileColumn100 flatFileColumn

= (IDTSConnectionManagerFlatFileColumn100)connectionFlatFile.Columns.Add();

flatFileColumn.ColumnType = "Delimited";

flatFileColumn.ColumnWidth = 0;

flatFileColumn.MaximumWidth = 100;

flatFileColumn.DataPrecision = 0;

flatFileColumn.DataScale = 0;

flatFileColumn.TextQualified = false;

IDTSName100 columnName = (IDTSName100) flatFileColumn;
columnName.Name = "CustomerName";

flatFileColumn.ColumnDelimiter = "\n"; // for the last column in the row

In a real world example the above step will be done in a loop to add all the desired columns. An important point is to remember to set the ColumnDelimiter property of the last column of the row to the row delimiter.

The following script can be used to retrieve the filegroup name for a given table and a partition range.  It can be very useful in an automated sliding window process.

declare @tablename nvarchar(128)

declare @range sql_variant

set @tablename = ‚myPartitionedTable‘

set @range = 1209

declare @tableId int

declare @FGName nvarchar(128)

set @tableId = object_id(@tablename)

select

@FGName = main.name

from

       (

             select

                    a.partition_number,

                    c.name,

                    e.function_id

             from

                    sys.partitions a

                    inner join sys.allocation_units b

                           on a.hobt_id = b.container_id

                    inner join sys.data_spaces c

                           on b.data_space_id = c.data_space_id

                    inner join sys.destination_data_spaces d

                           on c.data_space_id = d.data_space_id

                    inner join sys.partition_schemes e

                           on d.partition_scheme_id = e.data_space_id

             where a.object_id = @tableId

       ) main

       inner join

             (

             select

                    a.function_id,

                    b.value,

                    case when a.boundary_value_on_right = 0 then b.boundary_id else b.boundary_id + 1 end partition_id

             from

                    sys.partition_functions a

                    inner join sys.partition_range_values b

                           on a.function_id = b.function_id

             where

                    b.value = @range

       ) part

on main.function_id = part.function_id and main.partition_number = part.partition_id