Charles's Host Integration Server Ramblings

What is the DB2 DRDA Resync Service?

Charles Ezzell — Fri, 18 Sep 2009 13:37:06 GMT

Technorati Tags: Host Integration Server,DB2,DRDA,DTC,Transaction

This is something I knew a bit about, but not the whole story, so I had a talk with one of our developers, Vitaliy Golub, who very kindly explained to me (sometimes giving me a few kicks) how things work. Much of the below I gathered from him.

The DB2 DRDA Resync Service is installed when you install Host Integration Server 2009. The service shows up as the “HIS TCP/IP Resync Service”, but we prefer to call it the DRDA Resync Service. This service maintains

Host Integration Server, DB2, DRDA, DTC, Transaction

Technorati Tags: Host Integration Server,DB2,DRDA,DTC,Transaction

the transaction record for each DTC conversation. If you are not running distributed transactions, you can just ignore it, and for the rest of you, you may never even need it.

After Host Integration Server (or the ‘BizTalk Adapters for Host Systems’ is installed), another tool is then run to configure services, SQL databases, etc. One of the options is for configuring Data Integration, and this is where the DRDA Resync Service is enabled when you select support for distributed transactions. The dialog appears as such:

The Resync Port defaults to 50002. This is the port that the DRDA Resync Service will be listening on for incoming requests from the DB2 Servers. The next 2 settings, while adjustable, probably never need to be touched. . In fact, they could easily be lowered. If you only have 2 DB2 Servers, there is no need to have 50 connections and 4 worker threads sitting around doing nothing most of the time.

Next is the “Partner DB2 Resource Managers”. Here is where you enter in the names (or IP addresses) of your DB2 Servers which will be participating in distributed transactions. So, what happens if you don’t enter anything? Well, nothing 99.99998% of the time. This is the ‘rough’ estimate we have that this will never be used. The only time the resource manager will be contacted is IF the host has a problem with a transaction and need to know if to commit or rollback. This would happen very seldom. So, what happens if you enter nothing? Well, if the host ‘did’ need to find out what to do, it would attempt to contact the DRDA Resource manager. But, since it is not listed as a partner, the request would be rejected. At that point the host system would need to determine to commit or rollback on its own. Depending on the host configuration, it could send this request indefinitely, until the worker threads on the host were actually ‘killed’. So, it is a good idea to enter in your DB2 servers here if you do plan to use distributed transactions.

The next option is the location of the SQL database where we will store information for the transactions.

So, what happens during a distributed transaction from say a DQP process such as SQL Query?

Say you are running a statement similar to this:

SET XACT_ABORT ON
go
 BEGIN DISTRIBUTED TRAN
Go
INSERT INTO OPENQUERY (CHARLIEEW2K3DB2_PSS_CHARLIEE, 
    'SELECT * from CHARLIEE.MYAREA') 
    (AREAID, AREADESC,REGIONID) 
    values ('11111', 'Bellevue', 101)
COMMIT TRAN
GO
SET XACT_ABORT OFF

What happens here is that we will log an entry in the database to be used by the resource manager at the beginning of the transaction. We will pass a message to the DB2 host in a SYNCLOG message containing the IP address and port number of the HIS Server hosting the DRDA Resync Service, along with a transaction ID. If, for some reason, the connection between HIS and the DB2 server was closed without a commit being sent, the host will communicate to our DRDA Resync Service to reconcile this in-doubt transaction.

After the commit, everything is ‘removed’. If everything flows properly, there should never be an entry here but for a few seconds (normally).

In a nutshell, the DRDA Resync Service is like the airbag in your car. It’s there, waiting, and you hope it is never utilized. But, if it is needed, it could save your life (transaction).

Replicating NTEXT and IMAGE data to DB2

Charles Ezzell — Wed, 05 Aug 2009 13:23:34 GMT

One issue that came to us recently was on how to replicate SQL data containing NTEXT and IMAGE columns to DB2. Replicating most other data is straightforward, but NTEXT and IMAGE is a bit of a problem. Currently our data provider does not support LOBs (Large objects) such as CLOB or BLOB (character/binary large objects). So the only way to move the data is by using either VARCHAR or VARCHAR FOR BIT DATA. However, there are a few limitations if using VARCHAR with DB2.

DB2 does not allow you to have multiple VARCHAR(MAX) columns in the same table.
CREATE TABLE Categories (description1 VARCHAR(32703),
Picture1 VARCHAR(32703) FOR BIT DATA)
will return error " SQL statement too long or Complex".
NTEXT, TEXT, and IMAGE data types in SQL can hold much larger values than VARCHAR or VARCHAR() for BIT DATA in DB2. So there is a VERY good chance of truncation if the SQL value happens to go beyond 32703. Especially for image data. Also VARCHAR limits on DB2 depend on page size of DB2. It can range from 4045 to 32703 for page sizes ranging from 4KB to 32KB. The maximum VARCHAR value you can have is 32703.
AS400 object names are case sensitive.

MSDN documents the following about DB2 Subscribers with SQL Server (at https://msdn.microsoft.com/en-us/library/ms152492.aspx):

DB2 can support VARCHAR columns as large as 32 kilobytes (KB); therefore it is possible that some SQL Server large object columns can be appropriately mapped to DB2 VARCHAR columns. However, the OLE DB provider that replication uses for DB2 does not support mapping SQL Server large objects to DB2 large objects. For this reason, SQL Server text, varchar(max), ntext, and nvarchar(max) columns are mapped to VARCHAR(0) in the generated create scripts. The length value of 0 must be changed to an appropriate value prior to applying the script to the Subscriber. If the data type length is not changed, DB2 will raise error 604 when the table create is attempted at the DB2 Subscriber (error 604 indicates that the precision or length attribute of a data type is not valid).

In order to change the data type of VARCHAR(0) to something meaningful, you need to change the publication to use a custom creation script. This is discussed in the same link as above (see item #4 on the page), and the purpose of this blog entry is to show how to use sp_changearticle and a custom creation script to replicate NTEXT and IMAGE data from SQL Server to DB2.

For my test purposes, I used the Northwind database, specifically the Categories table, which contains 4 columns (int, nvarchar, ntext, image) with 8 rows of data.

CREATE TABLE [dbo].[Categories](
    [CategoryID] [int] NOT NULL,
    [CategoryName] [nvarchar](15) NOT NULL,
    [Description] [ntext] NULL,
    [Picture] [image] NULL)

First, to ensure there would be no truncation, I checked the length of the columns as follows:

use Northwind
select DATALENGTH([CategoryID]) as CategoryID_Length,
    DATALENGTH([CategoryName]) as CategoryName_Length,
    DATALENGTH([Description]) as Description_Length,
    DATALENGTH ([Picture]) as Picture_Length
from dbo.Categories

This returned the following to me:

CategoryID_Length	CategoryName_Length	Description_Length	Picture_Length
4	18	86	10746
4	20	116	10746
4	22	70	10746
4	28	14	10746
4	28	70	10746
4	24	28	10746
4	14	50	10746
4	14	32	10746

Based on the above, the Maximum length of the Destination column is 116, while the Picture column is 10,746 bytes in length, so moving this data to DB2 should be easy. Since the IMAGE data type is binary, you would want to use VARCHAR() FOR BIT DATA in DB2.

Now that we know the lengths, let’s ensure that we can create the table properly.

For my purposes, I am using DB2/400 v5r4, but this should be similar for DB2 on other platforms. Also, I will be creating the table in my own personal schema (CHARLIEE2). Using iSeries Navigator, I ran the following script:

/* Creating table CHARLIEE2.CATEGORIES */
CREATE TABLE CHARLIEE2.CATEGORIES (
      CATEGORYID INTEGER NOT NULL
    , CATEGORYNAME VARCHAR(15) NOT NULL
    , DESCRIPTION VARCHAR(116) DEFAULT NULL
    , PICTURE VARCHAR(10746) FOR BIT DATA DEFAULT NULL )   ;
CREATE UNIQUE INDEX PK_CATEGORIES ON CHARLIEE2.CATEGORIES(CATEGORYID);

This returned successful. It is important to test this. Also, do NOT delete the table you just created, as it will break things later on for this example as the table needs to be created 1^st for the creation script used by the publication to work properly.

Next, create the snapshot publication using the “New Publication Wizard”. But, there are a few things you need to do while creating the publication:

After selecting the Categories table, click “Article Properties” then “Set Property of Highlighted Table Article.” This dialog should appear:

Change this to appear as such:

The AS400 is case sensitive, so we want to ensure everything is correct, and we don’t need the Destination Object owner here.

Do not set the snapshot agent to run immediately. We will be running some scripts later that will invalidate the snapshot.

I named my publication “CATEGORIES”. You will need to know the name of your publication later for the scripts we will run.

After creating the publication, right click on it and select properties. Under Subscription Options, change “Allow non SQL Server Subscribers” to True.

Now that the publication is created, we need to create the subscription. Run the “New Subscription Wizard”. For my connection string, I used the following (created by the Data Access Tool):

Provider=DB2OLEDB;User ID=;Password=;Initial Catalog=S105FRDM;Network Transport Library=TCP;Host CCSID=37;PC Code Page=1252;Network Address=S105FRDM;Network Port=446;Package Collection=CHARLIEE2;Default Schema=CHARLIEE2;Process Binary as Character=False;Units of Work=RUW;Default Qualifier=CHARLIEE2;DBMS Platform=DB2/AS400;Defer Prepare=False;Rowset Cache Size=0;Persist Security Info=True;Connection Pooling=False;Derive Parameters=False;

I also set the synchronization schedule to “Run on demand only” for testing.

After the subscription is created, we need to create our custom creation script. Begin by creating a new SQL Script file on your system. For my test, I created c:\temp\replciation\CUSTOMREPL.SQL.

In this file, I added the following:

bypass_translation

DROP TABLE CHARLIEE2.CATEGORIES
GO

CREATE TABLE  CHARLIEE2.CATEGORIES (
CATEGORYID   INT NOT NULL,
CATEGORYNAME VARCHAR(15) NOT NULL,
DESCRIPTION  VARCHAR(116) DEFAULT NULL,
PICTURE VARCHAR(10746) FOR BIT DATA DEFAULT NULL )
GO

CREATE UNIQUE INDEX PK_CATEGORIES ON CHARLIEE2.CATEGORIES(CATEGORYID)
GO

Notice that the Create Table is the same as I used when creating the file with iSeries.

Now we need to tell the publication to use this file, and make a couple of other changes. To do this, run the following scripts:

-- set the creation_script property of the Article to point to custom Create Script
sp_changearticle @publication = CATEGORIES,
                @article = Categories,
                @property = 'creation_script',
                @value = "C:\temp\Replication\CUSTOMREPL.sql",
                @force_invalidate_snapshot = 1,
                @force_reinit_subscription = 1

-- then change the Schema option.
DECLARE @option AS int;
SET @option = (SELECT CAST(0x00 AS int));
EXEC     sp_changearticle @publication = CATEGORIES,
            @article = Categories,
            @property = N'schema_option', 
            @value = @option,
                @force_invalidate_snapshot = 1,
                @force_reinit_subscription = 1

-- make sure to set the pre creation command to nothing 
exec sp_changearticle 
    @publication = CATEGORIES,
    @article = Categories,
    @property = pre_creation_cmd,
    @value = 'none',
    @force_invalidate_snapshot = 1,
    @force_reinit_subscription = 1

At this point you are done. You should be able to successfully replicate the data from SQL Server to DB2 without any problems.

WMI and Host Integration Server…

Charles Ezzell — Sat, 06 Jun 2009 10:52:39 GMT

Internally and externally we get customer inquiries wanting to know how to get configuration and status information from Host Integration Server without using the manager interface.

Well, this is fairly simple, but complicated if you are not familiar with using WMI. Most users are familiar with using snacfg.exe, which is documented fairly well. However, for many things, WMI is your friend.

Now, to be honest, I cheat. When we first added in WMI into the product, I was totally unfamiliar with it. I started playing, using wbemtest (which comes with the operating system), and looking at samples. Then, I discovered WMI Code Creator. This is a great tool for ‘simple’ things, and what I really like is it will create, besides VBScript code, C# and VB.NET code. Using this tool gave me a greater understanding in how to create queries to gather information I needed (in conjunction with using wbemtest).

So, one question that came across our internal alias recently was “I'm wondering if SNACFG.EXE is able to display all LUs assigned for each pool.” Well, yes, but not cleanly, and what if you needed a bit more information, like is the LU actually being used, etc?

Here is where WMI comes in handy.

Using WMI, the below query will display every LU in a pool:

   1: OPTION EXPLICIT   2: Dim strDomainComputer, strPoolTargeted   3: Dim WmiLocator, WmiNameSpace   4: Dim objLuInThisPool   5: Dim xLuInPool   6: Dim objPool, xPool   7:     8: Set WmiLocator = CreateObject("WbemScripting.SWbemLocator")   9: strDomainComputer = "."  10:    11: BigLoop()  12:    13: Private Sub DoPool()  14:        Wscript.Echo "Checking POOL " & xPool.Name  15:        strPoolTargeted   = xPool.Name  16:        Set objLuInThisPool = WmiNameSpace.ExecQuery("Select Name from MSSna_LuDisplay where PoolName='" & strPoolTargeted & "'")  17:        if objLuInThisPool.Count > 0 then  18:               for each xLuInPool in objLuInThisPool  19:                      wscript.echo "     " & xLuInPool.Name  20:               next  21:        end if  22:        Wscript.Echo "Total LU count for the pool " & strPoolTargeted & " is " & objLuInThisPool.Count  23:        Wscript.Echo “”  24: End Sub  25:    26: Private Sub BigLoop()  27:        Set WmiNameSpace = WmiLocator.ConnectServer(strDomainComputer,"root\MicrosoftHIS")  28:        Set objPool = WmiNameSpace.ExecQuery("Select Name from MsSna_PoolDisplay")  29:        For Each xPool In objPool  30:               DoPool()  31:        next  32: End Sub

What this code does is 1st makes a query to get each pool defined on the machine, then makes a query based on the pool name to get the LUs using that pool.

But say you need a bit more? As in wanting to find out what sessions were actually in use?

Well, that can be done fairly easily as follows:

   1: rem - this will dump out all InSession or SSCP 3270 LUs   2: rem - Also, the script will display as follows   3: rem - Connection name    4: rem - 3270 LU name   5: rem - Status the LU is in at the time (either SSCP or InSession)   6: rem - User Account accessing the LU   8: strComputer = "."    9: quote = chr(34)   7:    10: Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\MicrosoftHIS")   11: Set colItems = objWMIService.ExecQuery( "SELECT * FROM MsSnaStatus_Lu3270 WHERE StatusText = 'SSCP' OR StatusText = 'InSession'",,48)   12: Wscript.Echo "---------------------------------------------------------------------------------------------------------"  13: For Each objItem in colItems  14:       Messagestr = "ConnectionName: " & objItem.ConnectionName & vbtab   15:       Messagestr = Messagestr & "Name: " & objItem.Name & vbtab   16:       Messagestr = Messagestr & "StatusText: " & objItem.StatusText & vbtab  17:     rem map the 'active' 3270 LU to the user account accessing it  18:     qrystring = "references of {MsSnaStatus_Lu3270.Name=" & quote & objItem.Name & quote & "}"  19:     set colitems1 = objWMIService.ExecQuery( qrystring,,48)   20:       For Each objItem1 in colitems1   21:       rem - strip out MsSnaStatus_ClientConnection.Name= from the item....  22:             Messagestr = Messagestr & "User" & Mid (objItem1.PathToUser,34)  23:       Next  24:       Wscript.Echo Messagestr  25: Wscript.Echo "---------------------------------------------------------------------------------------------------------"  26: Next  27: WScript.Echo ""

The output from this script would be similar to this:

   1: ---------------------------------------------------------------------------------------------------------   2: ConnectionName: D3270   Name: D3270002  StatusText: SSCP        User: W2KS2/ Administrator"   3: ConnectionName: D3270   Name: D3270003  StatusText: InSession   User: W2KS2/ Bob"   4: ConnectionName: D3270   Name: D3270004  StatusText: SSCP        User: W2KS2/ Jane"   5: ConnectionName: D3270   Name: D3270005  StatusText: SSCP        User: W2KS2/ Ted"   6: ConnectionName: D3270   Name: D3270005  StatusText: InSession   User: W2KS2/ Alice"   7: ---------------------------------------------------------------------------------------------------------

Please note this last query only queries against the local Host Integration Server. If you have multiple servers, things can be more complicated, as status information has to be queried against each Host Integration Server in the ‘HIS sub domain’.

I hope this is helpful to many of you out there. WMI can be most helpful when gathering information that is not as easily available in the manager interface.

Performance Testing the Microsoft OLE DB Provider for DB2 (Part 3)

Charles Ezzell — Wed, 03 Jun 2009 12:37:36 GMT

In Part 1 of this series I covered testing SQL queries using the Microsoft OLE DB Provider for DB2 using a linked server. In Part 2 I showed how you can test performance with SSIS. However, in many cases, customers may be using C# or VB.NET, writing web services or stand-alone applications to query DB2 for data. So, let’s begin.

First I wanted to again test different settings for the Rowset Cache Size option on the parameter. But, since these queries were going to be smaller, I was wondering if using Connection Pooling would help. The Microsoft OLE DB Provider that comes with Host Integration Server (and the SQL Feature Pack) implements its own connection pooling internally.

When Connection Pooling is enabled, the data provider does not immediately issue a close to the DB2 database when it receives a close from the consumer (application). It waits before closing the connection (this is controlled by the “Connection Timeout” property, which defaults to 30 seconds). When a new Open() is issued, for the SAME connection string, this helps speed up reconnecting as some frames do not have to be repeated to the host system (security for instance). This is most beneficial when you are making many small queries to the host, one behind the other. It may not be as beneficial when queries take many seconds or minutes to complete.

So, to begin with, I decided to limit my query to 5,000 rows. I also wanted to make the DB2 server do a bit of work, so I decided to use the following query (with the fetch first it would return over 2 million rows):

const string DB2QueryString = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 FETCH FIRST 5000 ROWS ONLY";

I created my initial connection string with the Data Access Tool, and then copied this into my C# application, after striping out the “Connection Pooling” and “Rowset Cache Size” options, and I ended up with this:

const string cnStringDB2OLEDB = @"Provider=DB2OLEDB;User ID=USERNAME;Password=password;Initial Catalog=DB2400;Network Transport Library=TCP;Host CCSID=37;PC Code Page=1252;Network Address=DB2400;Network Port=446;Package Collection=SCHEMA;Default Schema=SCHEMA;Process Binary as Character=False;Units of Work=RUW;Default Qualifier=SCHEMA;DBMS Platform=DB2/AS400;Persist Security Info=True;Defer Prepare=False;Derive Parameters=False;";

At this point I was almost ready, but I needed to decide how many times I wanted my query to run, and how many different settings of Rowset I wanted to use. I decided I wanted to the query to run 10 times for each connection, and increment the Rowset by 30 each time, starting at 0 and finishing at 300.

// number of times to run each query

const int iQueryIteration = 10;

// rowset cache size increments from 0 to 300 using this value

const int iCacheInc = 30;

From various other tests, I know that simulating working with the incoming data makes a difference, so I setup a variable that can be adjusted as necessary, which would cause the data to actually be read into a variable later. For my purposes, I left it at true.

// set bReadValues to true to simulate working with the data coming in

bool bReadValues = true;

Now, at this point, everything is setup to begin. The main procedure controls building the connection string, and then calling another procedure to make the calls.

The 1^st thing that the Main procedure does is calls a procedure to create a database in SQL Server. Then it builds the connection string as follows:

// multiple tests at this point

// test with pooling off, then on

// each test to be done with different settings of rowset

// to gauge performance

int iParamTest = 1;

while (iParamTest < 3)

{

string poolString = "";

switch (iParamTest)

{

case 1:

poolString = "Connection Pooling=False;";

break;

case 2:

poolString = "Connection Pooling=True;";

break;

}

iRowSet = 0;

while (iRowSet < 301)

{

myconstr = "";

rowString = "Rowset Cache Size=" + iRowSet.ToString() + ";";

// build final connection string here

myconstr = cnStringDB2OLEDB + poolString + rowString;

After the connection string is built, it runs the query, 10 times as follows:

for (int iquery = 1; iquery < iQueryIteration + 1; iquery++)

{

testno++;

// start timer watch

watch.Start();

rows = RunDbConnectionTest(myconstr, DB2QueryString);

watch.Stop();

Console.WriteLine("Test # {0}, ParamString: {1}, Rowset: {2}, Elapsed Time {3}", testno, poolString, iRowSet, (int)watch.Elapsed.TotalMilliseconds);

UpdateSQLPerfTable(testno, poolString, iRowSet, /*(int)*/watch.Elapsed.TotalMilliseconds);

watch.Reset();

if (rows == -1) // error happened, exit

{

Environment.Exit(rows);

}

iRowSet = iRowSet + iCacheInc;

The UpdateSQLPerfTable() call logs the times into SQL Server for each query. Then the watch is reset. If an error was to happen (rows returns = -1), the program will exit. The loops will then repeat until all tests are run.

The RunDbConnectionTest() procedure builds the OleDbConnection object, creates a command object, and a data reader, and the command is executed. At this point the data is read in (by assigning each column to a string value). After all columns and rows are read in, the objects created are closed and disposed of accordingly. Any errors are caught in Try/Catch statements, which will return a -1 to the main procedure.

Once the data is in SQL Server, you can query against the table to see the differences at each setting. I ran a pivot query against the table, with the following results (times in seconds):

rowset

Connection Pooling=False;

Connection Pooling=True;

2.05654372

1.28185158

1.41009238

0.87079308

1.44409547

0.87172373

1.46562513

1.10562903

120

1.3914555

1.05005861

150

1.38745682

0.88520811

180

1.44412404

0.84566784

210

1.41111907

0.87019778

240

1.41200047

0.83981306

270

1.42906141

0.82953586

300

1.44249608

1.1492816

So, just looking at the averages, you can see there is a difference when using connection pooling. But looking at the raw numbers gives a better idea. My first query was to look at rowset 0:

TestNumber

Pooling

rowset

Seconds

Connection Pooling=False;

2.65

Connection Pooling=False;

2.44

Connection Pooling=False;

1.78

Connection Pooling=False;

2.50

Connection Pooling=False;

1.94

Connection Pooling=False;

1.91

Connection Pooling=False;

1.83

Connection Pooling=False;

1.83

Connection Pooling=False;

1.84

Connection Pooling=False;

1.84

111

Connection Pooling=True;

1.89

112

Connection Pooling=True;

1.31

113

Connection Pooling=True;

1.21

114

Connection Pooling=True;

1.20

115

Connection Pooling=True;

1.19

116

Connection Pooling=True;

1.20

117

Connection Pooling=True;

1.20

118

Connection Pooling=True;

1.20

119

Connection Pooling=True;

1.20

120

Connection Pooling=True;

1.21

Now, looking at the above, you can see that times without connection pooling were higher than when using connection pooling.

Next, I wanted to see if setting the Rowset Cache Size made any difference with and without connection pooling. Using a query to get the minimum of the duration across all the results, I was able to determine that in my test it was test #208, with a rowset of 270. I then issued a query to pull out the tests results using “rowset = 270”:

SELECT TestNumber, ParamString AS Pooling, rowset, CAST(durationMs / 1000 AS decimal(6, 2)) AS Seconds

FROM CSPerfTrials

WHERE (rowset = 270)

ORDER BY TestNumber, Pooling

This query gives me the following results:

TestNumber

Pooling

rowset

Seconds

Connection Pooling=False;

270

1.39

Connection Pooling=False;

270

1.41

Connection Pooling=False;

270

1.41

Connection Pooling=False;

270

1.40

Connection Pooling=False;

270

1.40

Connection Pooling=False;

270

1.41

Connection Pooling=False;

270

1.46

Connection Pooling=False;

270

1.52

Connection Pooling=False;

270

1.41

100

Connection Pooling=False;

270

1.48

201

Connection Pooling=True;

270

1.30

202

Connection Pooling=True;

270

0.87

203

Connection Pooling=True;

270

0.78

204

Connection Pooling=True;

270

0.77

205

Connection Pooling=True;

270

0.76

206

Connection Pooling=True;

270

0.77

207

Connection Pooling=True;

270

0.76

208

Connection Pooling=True;

270

0.75

209

Connection Pooling=True;

270

0.76

210

Connection Pooling=True;

270

0.78

In the above, with connection pooling turned off, the times for each query were between 1.39 and 1.52 seconds. However, with connection pooling, the initial query (TestNumber 201) took 1.30 seconds, and subsequent queries were significantly faster, and in most cases below .8 seconds. The savings in time (about 0.4 to 0.5 seconds) would be a benefit as more and more queries were being ran, and with this particular query, you can run 2 queries with connection pooling (after the initial connection) almost as fast as 1 query without connection pooling.

I ran another pivot query which summed all the times over 30 iterations of the query, and came back with this chart (in seconds), which shows the times savings a bit better:

rowset

Connection Pooling=False;

Connection Pooling=True;

55.7799701

36.5997885

42.7464447

23.7765142

42.1101308

24.9684609

42.5139897

23.7936823

120

41.9289347

24.7713371

150

42.5943495

24.1104076

180

42.2610654

23.8675952

210

42.2377875

24.3038118

240

43.31585

24.105081

270

42.7937661

25.1228602

300

42.2332467

23.6615868

And again, with 100 iterations of the query at each setting, using this SQL pivot query:

USE [DB2TestDatabases]

declare @comstr nvarchar(4000) -- command string

declare @success int -- variable for executing SQL commands

declare @params nvarchar(255) --

declare @tempstr nvarchar(1500)

declare param_cursor CURSOR FOR

select distinct ParamString

from [dbo].CSPerfTrials

set @tempstr = ''

OPEN param_cursor

FETCH NEXT FROM param_cursor into @params

WHILE @@FETCH_STATUS = 0

BEGIN

set @tempstr = @tempstr + '[' + @params + ']'

FETCH NEXT FROM param_cursor into @params

if @@FETCH_STATUS = 0

begin

set @tempstr = @tempstr + ', '

end

END

CLOSE param_cursor

DEALLOCATE param_cursor

set @comstr = 'select rowset, ' + @tempstr + ' from (select rowset, ParamString, durationms/1000 as durationms '

set @comstr = @comstr + 'from dbo.CSPerfTrials) AS rf PIVOT ( sum(durationms) FOR ParamString IN ('

set @comstr = @comstr + @tempstr + ')) AS p'

exec @success=sp_executesql @comstr

rowset

Connection Pooling=True;

Connection Pooling=False;

121.6649042

190.0007958

150

78.565167

144.2463663

210

78.1296505

141.0669099

77.8703857

142.5785544

270

78.3527013

142.036522

78.0573694

141.4086889

120

77.8508053

145.7663853

300

78.8497313

141.7899574

240

77.4381023

146.7418084

78.2401951

141.6710446

180

78.3692842

142.6525498

Executing 100 queries with connection pooling takes almost half the time it takes without, when used in conjunction with RowSet Cache Size, with this particular query.

For my final test, I wanted to see what impact this would have on a large table, where the results were not limited. I used this query (select count(*) from schema.table where amount BETWEEN 100.00 AND 110.00), takes 51 seconds from SQL to return the count of 1296 rows. So a good test to pull in those 1296 rows and read the data, 100 times for each connection string (100 * 11 * 2) for 2,200 reads against the table. I thought this would more accurately demonstrate trying to pull in a small amount of data across a large table.

After the tests were run, I ran a query to sum up the times for each setting, with these results (times are in seconds):

rowset

Connection Pooling=True;

Connection Pooling=False;

3721.2146623

4052.1730781

150

3720.9639454

3976.7106773

210

3712.2296273

4068.358878

3724.0935948

4026.1043348

270

3675.8706398

4084.5275822

3718.429541

3970.0594041

120

3689.3583401

4035.6269915

300

3785.033983

4010.1079362

240

3703.0019921

4007.4356928

3689.1961373

3973.2904402

180

3685.1553973

3944.8446177

So, I was thinking wow. Some savings, 200 – 300 seconds total. But, before you get excited, remember this was only across 100 queries. So, in reality, the savings is only 2 to 3 seconds for each query. Changing the query to show the average confirms this:

rowset

Connection Pooling=True;

Connection Pooling=False;

37.212146623

40.521730781

150

37.209639454

39.767106773

210

37.122296273

40.68358878

37.240935948

40.261043348

270

36.758706398

40.845275822

37.18429541

39.700594041

120

36.893583401

40.356269915

300

37.85033983

40.101079362

240

37.030019921

40.074356928

36.891961373

39.732904402

180

36.851553973

39.448446177

So, the savings, on average, is only 2 – 3 seconds for each query when accessing a large table, in my particular case.

As the above charts show, using Rowset by itself helps, but when you combine that with pooling, on small queries, a significant time savings can be achieved. Since the numbers are somewhat close once Rowset is implemented here, you should see significant improvement with a setting of at least 30 for the RowSet Cache Size. Also, when querying against a large table, implementing pooling can save a few seconds, but you may not see as much of a performance impact as you would against ‘smaller’ queries where data can be retrieved much faster.

One thing to note here – all the times above were generated on a system that had little overhead against a DB2 server with minimal resources being used. Your results will vary, depending on hardware, network speed and utilization, loads on DB2, loads on the servers running the tests. But, this test and the previous 2 should give an indication of which settings may be most beneficial in your environment.

The test hardware being used in all 3 tests were identical. Windows 2003 SP2 + SQL Server 2005 (with all service packs and fixes applied for both), 2GB of memory on a 100MB network. The queries were run against DB2/400 V5R4, which was a couple of hops away from the test servers.

I hope this series of articles will help in your performance testing. If you have any questions, please do not hesitate to ask.

The code for this particular test is below.

using System;

using System.Collections;

using System.Text;

using System.Data;

using System.Data.OleDb;

using System.Data.Common;

using System.Data.SqlClient;

using System.Diagnostics;

namespace DB2PerfTest

{

class Program

{

// set bReadValues to true to simulate working with the data coming in

static bool bReadValues = true;

// connection string is missing rowset cache size and connection pooling options

// number of times to run each query

const int iQueryIteration = 10;

// rowset cache size increments from 0 to 300 using this value

const int iCacheInc = 30;

// below query returns 3,581,029 rows without the fetch first statement

// limiting to 5000 as it is not realistic that many C# applications will be handling

// millions of rows at one time

const string DB2QueryString = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 FETCH FIRST 5000 ROWS ONLY";

static void Main(string[] args)

{

int rows = 0;

string rowString = "";

string myconstr = "";

int iRowSet = 0;

CreateSQLPerfTable();

Stopwatch watch = new Stopwatch();

int testno = 0;

// multiple tests at this point

// test with pooling off, then on

// each test to be done with different settings of rowset

// to gauge performance

int iParamTest = 1;

while (iParamTest < 3)

{

string poolString = "";

switch (iParamTest)

{

case 1:

poolString = "Connection Pooling=False;";

break;

case 2:

poolString = "Connection Pooling=True;";

break;

}

iRowSet = 0;

while (iRowSet < 301)

{

myconstr = "";

rowString = "Rowset Cache Size=" + iRowSet.ToString() + ";";

// build final connection string here

myconstr = cnStringDB2OLEDB + poolString + rowString;

for (int iquery = 1; iquery < iQueryIteration + 1; iquery++)

{

testno++;

// start timer watch

watch.Start();

rows = RunDbConnectionTest(myconstr, DB2QueryString);

watch.Stop();

Console.WriteLine("Test # {0}, ParamString: {1}, Rowset: {2}, Elapsed Time {3}", testno, poolString, iRowSet, (int)watch.Elapsed.TotalMilliseconds);

UpdateSQLPerfTable(testno, poolString, iRowSet, /*(int)*/watch.Elapsed.TotalMilliseconds);

watch.Reset();

if (rows == -1) // error happened, exit

{

Environment.Exit(rows);

}

iRowSet = iRowSet + iCacheInc;

}

iParamTest++;

} // iparamtest loop

} // end main

static int RunDbConnectionTest(string cnString, string cmString)

{

int result = 0;

try // outer try

{

OleDbConnection cn = new OleDbConnection(cnString);

cn.Open();

try

{ // inner try

OleDbCommand cm = cn.CreateCommand();

cm.Connection = cn;

cm.CommandText = cmString;

if (bReadValues)

{

OleDbDataReader dr = cm.ExecuteReader();

try

{

while (dr.Read())

{

int cnt = dr.FieldCount;

for (int i = 0; i < cnt; i++)

{

// simulate actually doing something with the data so it is 'read'

string msg = dr.GetValue(i).ToString();

}

finally

{

dr.Close();

cm.Dispose();

}

} // bReadValues

else // not reading data, just execute

{

try

{

result = cm.ExecuteNonQuery();

}

catch (Exception ex)

{

System.Console.WriteLine("");

System.Console.WriteLine(ex.Message);

result = -1;

}

finally

{

cm.Dispose();

}

} // !bReadValues

} // inner try

finally

{

cn.Close();

cn.Dispose();

}

} // outer try

catch (Exception e)

{

System.Console.WriteLine("");

System.Console.WriteLine(e.Message);

result = -1;

}

finally

{

}

return result;

}

static void CreateSQLPerfTable()

{

SqlConnection mySqlConnection = new SqlConnection("Persist Security Info=True;Integrated Security=SSPI;database=DB2TestDatabases;server=(local);pooling=false");

SqlCommand mySqlCommand = mySqlConnection.CreateCommand();

mySqlConnection.Open();

// drop perf table

Console.WriteLine("Dropping CSPerfTrials table");

mySqlCommand.CommandText =

"IF EXISTS (SELECT * FROM sys.objects " +

"WHERE object_id = OBJECT_ID(N'[dbo].[CSPerfTrials]') " +

"AND type in (N'U')) " +

"DROP TABLE [dbo].[CSPerfTrials]";

int result = mySqlCommand.ExecuteNonQuery();

Console.WriteLine("mySqlCommand.ExecuteNonQuery() = " + result);

// create new table

mySqlCommand.CommandText =

"CREATE TABLE dbo.CSPerfTrials" +

"( TestNumber int NULL" +

", ParamString Nvarchar(2000) NULL" +

", rowset int NULL" +

", durationMs float NULL" +

")";

Console.WriteLine("Creating dbo.CSPerfTrials table");

result = mySqlCommand.ExecuteNonQuery();

Console.WriteLine("mySqlCommand.ExecuteNonQuery() = " + result);

mySqlConnection.Close();

}

static void UpdateSQLPerfTable(int testno, string ParamString, int rowset, double durationMS)

{

// added pooling=false to the sql connection string to resolve an error condition

// may not be needed when in all cases but on 'fast' reads this appears to be needed

SqlConnection mySqlConnection = new SqlConnection("Persist Security Info=True;Integrated Security=SSPI;database=DB2TestDatabases;server=(local);pooling=false");

SqlCommand mySqlCommand = mySqlConnection.CreateCommand();

mySqlConnection.Open();

Console.WriteLine("Connection state is " +

mySqlConnection.State.ToString());

Console.WriteLine("inserting data into SQL table dbo.CSPerfTrials");

mySqlCommand.CommandText = @"insert into dbo.CSPerfTrials " +

"(TestNumber" +

",ParamString" +

",rowset" +

",durationMS"

+ ") values (" +

testno.ToString() + ",'" +

ParamString + "'," +

rowset.ToString() + "," +

durationMS.ToString() + ")";

int result = mySqlCommand.ExecuteNonQuery();

Console.WriteLine("mySqlCommand.ExecuteNonQuery() = " + result);

mySqlConnection.Close();

Console.WriteLine("Connection state is " +

mySqlConnection.State.ToString());

}

The SQL Queries I ran to give me the pivot charts is below. Modifying the “avg(durationms” to “sum(durationms” will give you the total times for each test run:

-- SSISPerf Pivot report

-- ***********************

-- Builds a Pivot report

USE [DB2TestDatabases]

declare @comstr nvarchar(4000) -- command string

declare @success int -- variable for executing SQL commands

declare @params nvarchar(255) --

declare @tempstr nvarchar(1500)

declare param_cursor CURSOR FOR

select distinct ParamString

from [dbo].CSPerfTrials

set @tempstr = ''

OPEN param_cursor

FETCH NEXT FROM param_cursor into @params

WHILE @@FETCH_STATUS = 0

BEGIN

set @tempstr = @tempstr + '[' + @params + ']'

FETCH NEXT FROM param_cursor into @params

if @@FETCH_STATUS = 0

begin

set @tempstr = @tempstr + ', '

end

END

CLOSE param_cursor

DEALLOCATE param_cursor

set @comstr = 'select rowset, ' + @tempstr + ' from (select rowset, ParamString, durationms/1000 as durationms '

set @comstr = @comstr + 'from dbo.CSPerfTrials) AS rf PIVOT ( avg(durationms) FOR ParamString IN ('

set @comstr = @comstr + @tempstr + ')) AS p'

exec @success=sp_executesql @comstr

Performance Testing the Microsoft OLE DB Provider for DB2 (Part 2)

Charles Ezzell — Wed, 03 Jun 2009 12:03:22 GMT

Technorati Tags: Data Providers,DB2,Host Integration Server,Microsoft OLE DB Provider for DB2,OLEDB,Performance,SSIS,SQL Server

In my previous entry, I covered testing SQL queries using the Microsoft OLE DB Provider for DB2 using a linked server, but many customers really need to test performance over SSIS, when moving data from DB2 to SQL Server.

Initially I did my test ‘the hard way’, creating multiple packages with different parameters for the RowSet Cache Size setting (which greatly increases performance), then running them, and looking at the final numbers.

This was crude, but not automated.

So, I wrote a C# application which recorded the start and stop time of the package execution, and ran this for each package.

watch.Start(); // start watch

pkgResults = pkg.Execute(null, null, eventListener, null, null);

watch.Stop();

This was better, but the only problem was still having to creating multiple packages. Next, I looked at creating one package, but having multiple configuration files. Again, workable, but if you wanted to test 10 settings, that would mean 10 configuration files.

So I did some research (and asking other Engineers more experienced with SSIS) that it is possible to change a package on the fly (so to speak), and execute it. So I began work on that.

First I had to decide on what needed to be changed. I knew I wanted to try out various settings for the “Rowset Cache Size”, and I also wanted to test a different type query than in the previous test, but also limit the number of returned rows. The table I decided to use contains 8.3 million rows, and the query I decided to use against this table was this: “select * from schema.table where amount BETWEEN 5.00 AND 10.00”. This particular query would return 3,581,029 rows from the table.

Once I decided on what needed changing, I created a new Integration Services Project in Visual Studio (or BIDS). To make things easier (and simpler) I ran the SSIS Import Export Wizard to create the standard package I wanted to use. Before making any changes to the package, I ran the package to ensure it functioned properly. (Note: When creating the package, I set it to drop and recreate the destination tables in SQL Server)

Once I was sure that the package ran correctly, I had to decide on how to modify the things. After doing a bit of research, I decided to use a variable in the package to hold the query string, and to modify the actual connect string from within my visual studio application.

On the control flow I created a variable called “varQueryString”, with a value type of string, and set its initial value to the query above. Then on the source data flow I changed the data access mode to “SQL command from variable” and set the Variable name to point to my varQueryString. I then tested the package again to ensure everything was correct.

At this point things were ready to start making creating a control application that would automate multiple tests.

I used Visual Studio 2008 and created a new C# console application. I then added a reference to Microsoft.SQLServer.ManagedDTS. Also, I had decided I wanted each ‘run’ to be clean, by stopping and starting the SQL and SSIS services, and was going to be logging all my results to SQL Server.

So, in my C# file, I setup the following using directives.

using System;

using System.Text;

using System.Diagnostics;

using Microsoft.SqlServer.Dts.Runtime;

using System.Data.SqlClient;

using System.ServiceProcess;

using System.Threading;

using System.Data;

System.Threading I added after I found the application was trying to start the packages before the SQL and SSIS services were fully running, which would cause issues. My final application sleeps in a few areas to ensure the 2 services are fully operational. This you may need to modify on your particular test machines (and may not be necessary in all cases).

The application itself contains 2 global variables:

const string pkgFullPath = @"C:\projects\SQLPKG\PerfTest.dtsx";

static int TestCounter = 0; // counter to hold total test runs

The 1^stcontains the path information to the package, the 2^nd a counter which is used to hold the # of packages that have been executed.

The main procedure is simple:

static void Main(string[] args)

{

CreateSQLPerfTable();

for (int x = 0; x < 2; x++)

{

RunSSISTests();

}

The first call to CreateSQLPerfTable () creates a table in SQL Server to hold the test results. The “for” loop will run the test twice (in this case), which would allow an average of the test runs to be gathered. The procedure “RunSSISTests()” begins by setting up some local variables, then sets up an event listener to catch any errors that may be generated when the package is run:

MyEventListener eventListener = new MyEventListener();

Next the procedure creates a timer for timing the package execution time, and loads the package in.

Stopwatch watch = new Stopwatch();

app = new Application();

pkg = app.LoadPackage(pkgFullPath, null);

After this, we need to access the source connection object in the package, which is handled as such:

// need to access the source connection object in the package

Connections myConns = pkg.Connections;

ConnectionManager connMgr = myConns["SourceConnectionOLEDB"];

Note: if your Source Connection is named differently, you will need to adjust the above appropriately.

Next, an outer loop is started, which will run 4 times, in each test modifying the package variable varQueryString. We begin by querying for 10,000 rows, increasing each run until the entire data is pulled in.

for (int iQuery = 1; iQuery < 5; iQuery++)

{

switch (iQuery)

{

// default query string for variable varQueryString

// select * from schema.table where amount BETWEEN 5.00 AND 10.00

case 1: // 10,000 rows

QueryCommand = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 FETCH FIRST 10000 ROWS ONLY";

break;

case 2: // 100,000 rows

QueryCommand = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 FETCH FIRST 100000 ROWS ONLY";

break;

case 3: // 1,000,000 rows

QueryCommand = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 FETCH FIRST 1000000 ROWS ONLY";

break;

case 4: // all rows (this table has ~8.3 million rows, where clause pulls in 3581029)

QueryCommand = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 ";

break;

}

pkg.Variables["varQueryString"].Value = QueryCommand;

After the package variable has been modified, the connection string needs to be changed. To get the initial value, from VS I opened up the package, went to the “SourceConnectionOLEDB” properties, and copied out the “ConnectionString”. I then cut the “Rowset Cache Size = 0” value from this, and added “Rowset Cache Size=” to the end of the string. Then, in my next ‘inner’ loop, I append the value of the rowset values I want to test. In my case, I want to test 8 separate values to see which one would give me the best performance.

for (int i = 1; i < 9; i++)

{

// setup connection string

switch (i)

{

case 1: // rowset cache size = 0

rowset = 0;

break;

case 2: // rowset cache size = 30

rowset = 30;

break;

case 3: // rowset cache size = 50

rowset = 50;

break;

case 4: // rowset cache size = 100

rowset = 100;

break;

case 5: // rowset cache size = 150

rowset = 150;

break;

case 6: // rowset cache size = 200

rowset = 200;

break;

case 7: // rowset cache size = 250

rowset = 250;

break;

case 8: // rowset cache size = 300

rowset = 300;

break;

}

connMgr.ConnectionString =

"Data Source=DB2400;User ID=USER;Initial Catalog=DB2400;Provider=DB2OLEDB;Persist Security Info=True;Network Address=DB2400;Package Collection=SCHEMA;Default Schema=SCHEMA;Default Qualifier=SCHEMA;DBMS Platform=DB2/AS400;Rowset Cache Size="

+ rowset.ToString() + ";";

Initially, at this point, I thought everything was good. But I discovered it was not. Anyone that is familiar with editing SSIS packages knows that after opening the Source or Destination objects that in many cases saved passwords are ‘lost’. Well, this apparently happens when using code also. When I changed the ConnectionString, the password was ‘lost’, so I had to go back into the code, and hardcode the password. This could be set in a configuration file, if you wish, but for my testing purposes, I did not care if the password is hard coded.

// now we need to setup the password, as this get's lost for some reason

DtsProperties connProperties = connMgr.Properties;

DtsProperty myconnProp = connProperties["Password"];

myconnProp.SetValue(connMgr, "password");

Once the password is set, the SQL and SSIS services are stopped and then restarted. The package is ran, incrementing the testcounter value. Results are sent to the SQL Server, and the next iteration is run:

TestCounter++;

Console.WriteLine("Starting test #{0}", TestCounter);

watch.Start(); // start watch

pkgResults = pkg.Execute(null, null, eventListener, null, null);

watch.Stop();

Console.WriteLine("Finished Test #{0}, Query: {1}, Rowset: {2}, Duration: {3}", TestCounter, QueryCommand, rowset, watch.Elapsed.TotalMilliseconds);

UpdateSQLPerfTable(TestCounter, QueryCommand, rowset, (int)watch.Elapsed.TotalMilliseconds);

watch.Reset();

pkg.Dispose();

After the tests are complete, the following SQL Script can be used to create a pivot chart which shows the output of the test runs, averaging the duration times to get a better idea of which setting may be best in your environment:

-- SSISPerf Pivot report

-- ***********************

-- Builds a Pivot report

USE [DB2TestDatabases]

declare @comstr nvarchar(4000) -- command string

declare @success int -- variable for executing SQL commands

declare @commands nvarchar(255) --

declare @tempstr nvarchar(500)

declare command_cursor CURSOR FOR

select distinct QueryCommand

from [dbo].SSISPerfTrials

set @tempstr = ''

OPEN command_cursor

FETCH NEXT FROM command_cursor into @commands

WHILE @@FETCH_STATUS = 0

BEGIN

set @tempstr = @tempstr + '[' + @commands + ']'

FETCH NEXT FROM command_cursor into @commands

if @@FETCH_STATUS = 0

begin

set @tempstr = @tempstr + ', '

end

END

CLOSE command_cursor

DEALLOCATE command_cursor

set @comstr = 'select rowset, ' + @tempstr + ' from (select rowset, QueryCommand, durationms '

set @comstr = @comstr + 'from dbo.SSISPerfTrials) AS rf PIVOT ( avg(durationms) FOR QueryCommand IN ('

set @comstr = @comstr + @tempstr + ')) AS p'

exec @success=sp_executesql @comstr

In my next entry, I will demonstrate a C# test application that reads data from DB2, which also records the performance data to SQL Server.

Below is the full C# code for this test application. It contains some additional statements not discussed above, mainly around creating and updating the SQL tables, and the stopping and starting of the services.

using System;

using System.Text;

using System.Diagnostics;

using Microsoft.SqlServer.Dts.Runtime;

using System.Data.SqlClient;

using System.ServiceProcess;

using System.Threading;

using System.Data;

namespace VS2008SSISPackageRun

{

class MyEventListener : DefaultEvents

{

public override bool OnError(DtsObject source, int errorCode, string subComponent,

string description, string helpFile, int helpContext, string idofInterfaceWithError)

{

// Add application-specific diagnostics here.

Console.WriteLine("Error in {0}/{1} : {2}", source, subComponent, description);

return false;

}

class Program

{

// globals

const string pkgFullPath = @"C:\projects\SSIS_EMIS_IMPORT\SSIS_EMIS_IMPORT\TableVariables.dtsx";

static int TestCounter = 0; // counter to hold total test runs

static void Main(string[] args)

{

CreateSQLPerfTable();

for (int x = 0; x < 2; x++)

{

RunSSISTests();

}

static void CreateSQLPerfTable()

{

SqlConnection mySqlConnection = new SqlConnection("Persist Security Info=True;Integrated Security=SSPI;database=DB2TestDatabases;server=(local);pooling=false");

SqlCommand mySqlCommand = mySqlConnection.CreateCommand();

mySqlConnection.Open();

// drop perf table

Console.WriteLine("Dropping SSISPerfTrials table");

mySqlCommand.CommandText =

"IF EXISTS (SELECT * FROM sys.objects " +

"WHERE object_id = OBJECT_ID(N'[dbo].[SSISPerfTrials]') " +

"AND type in (N'U')) " +

"DROP TABLE [dbo].[SSISPerfTrials]";

int result = mySqlCommand.ExecuteNonQuery();

Console.WriteLine("mySqlCommand.ExecuteNonQuery() = " + result);

// create new table

mySqlCommand.CommandText =

"CREATE TABLE dbo.SSISPerfTrials(" +

"TestNumber int NULL," +

"QueryCommand Nvarchar(2000) NULL," +

"rowset int NULL," +

"durationMs int NULL" +

")";

Console.WriteLine("Creating dbo.SSISPerfTrials table");

result = mySqlCommand.ExecuteNonQuery();

Console.WriteLine("mySqlCommand.ExecuteNonQuery() = " + result);

mySqlConnection.Close();

}

static void UpdateSQLPerfTable(int testno, string QueryCommand, int rowset, int durationMS)

{

// added pooling=false to the connection string to resolve an error condition

// may not be needed when SSIS takes a while, but on 'fast' imports appears to be needed

SqlConnection mySqlConnection = new SqlConnection("Persist Security Info=True;Integrated Security=SSPI;database=DB2TestDatabases;server=(local);pooling=false");

SqlCommand mySqlCommand = mySqlConnection.CreateCommand();

mySqlConnection.Open();

Console.WriteLine("Connection state is " +

mySqlConnection.State.ToString());

Console.WriteLine("inserting data into SQL");

mySqlCommand.CommandText = @"insert into dbo.SSISPerfTrials " +

"(TestNumber,QueryCommand,rowset,durationMS) values (" +

testno.ToString() + ",'" +

QueryCommand + "'," +

rowset.ToString() + "," +

durationMS.ToString() + ")";

int result = mySqlCommand.ExecuteNonQuery();

Console.WriteLine("mySqlCommand.ExecuteNonQuery() = " + result);

mySqlConnection.Close();

}

static void RunSSISTests()

{

bool bgood = false;

Package pkg;

Application app;

DTSExecResult pkgResults;

int rowset = 0;

string QueryCommand = "";

MyEventListener eventListener = new MyEventListener();

Stopwatch watch = new Stopwatch();

app = new Application();

pkg = app.LoadPackage(pkgFullPath, null);

// need to access the source connection object in the package

Connections myConns = pkg.Connections;

ConnectionManager connMgr = myConns["SourceConnectionOLEDB"];

for (int iQuery = 1; iQuery < 5; iQuery++)

{

switch (iQuery)

{

// default query string for variable varQueryString

// select * from schema.table where amount BETWEEN 5.00 AND 10.00

case 1: // 10,000 rows

QueryCommand = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 FETCH FIRST 10000 ROWS ONLY";

break;

case 2: // 100,000 rows

QueryCommand = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 FETCH FIRST 100000 ROWS ONLY";

break;

case 3: // 1,000,000 rows

QueryCommand = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 FETCH FIRST 1000000 ROWS ONLY";

break;

case 4: // all rows (this table has ~8.3 million rows, where clause pulls in 3581029)

QueryCommand = "select * from schema.table where amount BETWEEN 5.00 AND 10.00 ";

break;

}

pkg.Variables["varQueryString"].Value = QueryCommand;

for (int i = 1; i < 9; i++)

{

// setup connection string

switch (i)

{

case 1: // rowset cache size = 0

rowset = 0;

break;

case 2: // rowset cache size = 30

rowset = 30;

break;

case 3: // rowset cache size = 50

rowset = 50;

break;

case 4: // rowset cache size = 100

rowset = 100;

break;

case 5: // rowset cache size = 150

rowset = 150;

break;

case 6: // rowset cache size = 200

rowset = 200;

break;

case 7: // rowset cache size = 250

rowset = 250;

break;

case 8: // rowset cache size = 300

rowset = 300;

break;

}

connMgr.ConnectionString =

+ rowset.ToString() + ";";

// now we need to setup the password, as this get's lost for some reason

DtsProperties connProperties = connMgr.Properties;

DtsProperty myconnProp = connProperties["Password"];

myconnProp.SetValue(connMgr, "password");

// now stop and start the sql servers so things are 'clean'

bgood = StopSql();

if (bgood)

{

// hard coded timing delay to allow services

// to clean up and stop properly

Thread.Sleep(15000);

bgood = StartSQL();

}

if (bgood)

{

// wait an additional 15 seconds just in case

// the services have not started completly

Thread.Sleep(15000);

//Console.WriteLine("Starting Test {0}", pkgFullPath);

TestCounter++;

Console.WriteLine("Starting test #{0}", TestCounter);

watch.Start(); // start watch

pkgResults = pkg.Execute(null, null, eventListener, null, null);

watch.Stop();

Console.WriteLine("Finished Test #{0}, Query: {1}, Rowset: {2}, Duration: {3}", TestCounter, QueryCommand, rowset, watch.Elapsed.TotalMilliseconds);

UpdateSQLPerfTable(TestCounter, QueryCommand, rowset, (int)watch.Elapsed.TotalMilliseconds);

watch.Reset();

pkg.Dispose();

}

else

{

break; // exit for loop, services stop/start failed

}

//Console.WriteLine(pkgResults.ToString());

} // for loop

}

pkg.Dispose();

}

static bool StopSql()

{

bool bgood = true;

ServiceController sqlsvc = new ServiceController("MSSQLSERVER");

ServiceController dtssvc = new ServiceController("MsDtsServer");

try

{

TimeSpan timeout = TimeSpan.FromMilliseconds(15000);

if (sqlsvc.Status.Equals(ServiceControllerStatus.Running))

{

Console.WriteLine("Stopping MSSQLSERVER");

sqlsvc.Stop();

sqlsvc.WaitForStatus(ServiceControllerStatus.Stopped, timeout);

}

if (dtssvc.Status.Equals(ServiceControllerStatus.Running))

{

Console.WriteLine("Stopping MsDtsServer");

dtssvc.Stop();

dtssvc.WaitForStatus(ServiceControllerStatus.Stopped, timeout);

}

catch (Exception err)

{

Console.WriteLine("An error occurred while trying to stop service: " + err.Message);

bgood = false;

}

return bgood;

} // end StopSql()

static bool StartSQL()

{

bool bgood = true;

ServiceController sqlsvc = new ServiceController("MSSQLSERVER");

ServiceController dtssvc = new ServiceController("MsDtsServer");

try

{

TimeSpan timeout = TimeSpan.FromMilliseconds(15000);

Console.WriteLine("Starting MSSQLSERVER");

sqlsvc.Start();

sqlsvc.WaitForStatus(ServiceControllerStatus.Running, timeout);

Thread.Sleep(15000);

Console.WriteLine("Starting MsDtsServer");

dtssvc.Start();

dtssvc.WaitForStatus(ServiceControllerStatus.Running, timeout);

}

catch (Exception err)

{

Console.WriteLine("An error occurred while trying to start service: " + err.Message);

bgood = false;

}

return bgood;

} // end StartSQL()

}

Performance Testing the Microsoft OLE DB Provider for DB2 (Part 1)

Charles Ezzell — Mon, 18 May 2009 11:03:00 GMT

Technorati Tags: DB2,SQL Server,SSIS,Microsoft OLE DB Provider for DB2,Host Integration Server,Data Providers,Performance,OLEDB

There are many factors that can affect pulling data from DB2 into SQL Server (or other applications). These include the network speed, hard drive; machine CPU speed, memory, # of CPUs, loads (on DB2 or SQL). There are way too many factors to consider.

One thing that we offer is the ability to pre-fetch data from the DB2 server before the consumer asks for this data. This parameter is normally only used for SSIS or SSAS, but can be used by other consumers. It should ONLY be used when moving large amounts of data (we recommend millions of rows), but there are times when it can be useful in DQP applications (such as SQL Server’s Query Analyzer). The pre-fetch is controlled by a parameter on the connection string “Rowset Cache Size”. The default for this setting is 0 (off), but can be set to almost any number (we do not recommend going over 300 at this point). What the parameter allows is for us to pre-fetch data from DB2 before the consumer actually requests it. IE – if using SSIS, if SSIS asks for 10,000 rows of data, we will ask DB2 to send us 300,000 rows (if this parameter is set to 30), and stored this internally, thus avoiding delays in constantly asking DB2 for smaller chunks of data.

But, how do you know what to set this to? Well, normally we recommend 30 as a starting point, but this can be cumbersome to test.

Recently I worked a case which involved testing multiple queries across multiple linked servers, each with a separate setting for the Rowset Cache Size. I did not care about the actual data, but was concerned with performance at different settings. Manually creating queries, linked servers, gathering timing numbers, was very cumbersome at best, and I decided there had to be a better way.

So, I came up with 4 SQL Scripts that do all the work for me. They can easily be used by anyone, as being very generic, and only 1 script file needs modification. Since the scripts are generic in nature, I named these Generic Part1.sql, GenericPart2.sql, GenericPart3.sql, and GenericPart4.sql. These scripts assume you have a Database called DB2TestDatabases.

So, what do these do?

GenericPart1.sql

This script file creates 3 tables. 2 of the tables are used to create the 3^rd one.

To begin with, you need to make a few changes.

Ignoring the declare section, skip down to this section:

-- ********************* BEGIN USER DEFINED VARIABLES *********************

-- ***** first, need your connection string, WITHOUT ROWSET CACHE SIZE

set @constr = N'Provider=DB2OLEDB;User ID=USERNAME;Password=password;Initial Catalog=DB2400;Network Transport Library=TCP;Host CCSID=37;PC Code Page=1252;Network Address=DB2400;Network Port=446;Package Collection=SCHEMA;Default Schema=SCHEMA;Process Binary as Character=False;Units of Work=RUW;Default Qualifier=SCHEMA;DBMS Platform=DB2/AS400;Defer Prepare=False;Persist Security Info=True;Connection Pooling=False;Derive Parameters=False;'

-- ***** supply your servername prefix, I use the name of system I'm connecting to

set @srvprefix = N'DB2400_'

-- ***** supply your SQL Querys

set @sqlquery1 = N'select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY'

set @sqlquery2 = N'select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY'

set @sqlquery3 = N'select * from SCHEMA.TABLE FETCH FIRST 1000000 ROWS ONLY'

set @sqlquery4 = N'select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY'

set @sqlquery5 = N'select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY'

set @sqlquery6 = N'select * from SCHEMA.TABLE FETCH FIRST 1000000 ROWS ONLY'

set @sqlquery7 = N'select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY'

set @sqlquery8 = N'select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY'

set @sqlquery9 = N'select * from SCHEMA.TABLE FETCH FIRST 1000000 ROWS ONLY'

-- ********************* END USER DEFINED VARIABLES *********************

-- create QueryTable

-- this table is used for looping purposes

-- when creating the trials table later

-- it can contain as many queries as you wish to test with

create table [dbo].[QueryTable] (query nVarChar(2000))

insert into [dbo].[QueryTable] values (@sqlquery1)

insert into [dbo].[QueryTable] values (@sqlquery2)

insert into [dbo].[QueryTable] values (@sqlquery3)

insert into [dbo].[QueryTable] values (@sqlquery4)

insert into [dbo].[QueryTable] values (@sqlquery5)

insert into [dbo].[QueryTable] values (@sqlquery6)

insert into [dbo].[QueryTable] values (@sqlquery7)

insert into [dbo].[QueryTable] values (@sqlquery8)

insert into [dbo].[QueryTable] values (@sqlquery9)

Only a few changes are needed here to make this usable for your needs.

For the variable @constr, this is set to your connection string, but without the “RowSet Cache Size=x” parameter. This will be appended on later, when the linked servers are created.

For the variable @srvprefix, this can be anything you want. I use the name of the DB2 server I want to connect to. This will become the name of the linked server, in incremental values (more on that later).

Next, enter the SQL Queries you want to run. In my case, I wanted to run 3 queries, with each of them pulling in 10,000, 100,000 and 1,000,000 rows of data. Nice and simple, but you can use any query you want here. I also wanted them to run 3 times each, so I could average the times to get a better idea of performance over time.

Once you have the queries, make sure the insert statements match up. In my case, 9 inserts. This piece of code creates a table that is read in later, in a loop, for building the test to be run later.

After the above is completed, you are done with any modifications you need to make to the script files.

In the next part of the script, we created another table which contains the link server names and the final connection string for each linked server:

-- Create Linked Server names in a table

-- tables contains the name of the linked server

-- we also construct the data provider connection string here

-- appending the RowSet Cache Size setting it here from 0 to 300 in increments of 30

create table [dbo].[LinkServerNames] (servername Nvarchar(255), constr Nvarchar(2000))

set @i = 0

while @i < 301 -- creating linked server names for rowset 0, 30, 60, 90, 120, 150, 180, 210, 270,300

begin

set @istr = cast(@i as nvarchar(3))

-- set connection string proproperty now

set @realconstr = @constr + 'Rowset Cache Size=' + @istr + ';'

set @j = len(@istr)

-- and now build istr up to 3 characters

-- for linked server name

set @rowstr = '000'

set @rowstr = stuff(@rowstr,4-@j,@j,@istr)

-- insert data

insert into [dbo].[LinkServerNames] values(@srvprefix + @rowstr, @realconstr )

set @i = @i + 10 -- increment value (was 30)

end

As I mentioned, the table contains 2 columns, the linked server name, and the connection string. These are built in a loop, which, in this case, will create 31 rows. The linked server names will use the prefix you put in earlier, plus a 3 column number. In my case, I would have DB2400_000, DB2400_010 ….DB2400_300. Also, the final connection string is setup here, appending the “Rowset Cache Size” parameter to the connection string entered earlier.

After the above is run, the next part of this script creates the linked servers. I will leave out that part of the script here, but it creates and executes SQL Commands similar to what you would do if creating the linked server manually (the commands are similar to what you would get if you scripted the linked server from SSMS to a query window).

Finally, this script creates the last table. This table will be used by both GenericPart2.sql and GenericPart4.sql.

IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[TRIALS]') AND type in (N'U'))

DROP TABLE [dbo].[TRIALS]

CREATE table [dbo].[TRIALS]

(trialname Nvarchar(255) -- test name

,servername Nvarchar(255) -- linked server name

,command Nvarchar(2000) -- command to run

,StartTime DateTime -- start time

,EndTime DateTime -- end time

,durationMs int -- run time in millisecnds

)

The table contains a test name (built on loop numbers), the linked server name, and the command which will be run. It also contains columns for the start and end times of the test, and the duration (in milliseconds) of the test.

Here are a few sample columns created in the table:

Trial #: 001-001	DB2400_000	select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY	NULL	NULL
Trial #: 001-002	DB2400_010	select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY	NULL	NULL
Trial #: 001-003	DB2400_020	select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY	NULL	NULL

In the above, you can see that the 1^st ‘run’ will be using a query to pull in 10,000 rows, against multiple linked servers. The linked servers in the above have Rowset Cache Size set to 0, 10 and 20. In my case, I have 279 separate rows. 3 separate queries repeated 3 times with 31 linked servers.

GenericPart2.sql

This is the workhorse of the test scripts provided here.

This script opens a cursor against the TRIALS table (created above), reads in the linked server name and query, and builds an exec statement, then executes it as follows:

DECLARE trial_cursor CURSOR FOR

SELECT trialname, servername, command

FROM [dbo].[TRIALS]

Open trial_cursor

FETCH NEXT FROM trial_cursor into @trial, @server, @command

WHILE @@FETCH_STATUS = 0

BEGIN

PRINT @trial + N' ' + @server

-- append server location to the command

SET @command= N'exec (''' +@command+ ''') at ' + @server

-- run it

SET @starttime = getdate()

exec @success=sp_executesql @command

Once the exec statement returns, the script updates the TRIALS table with the start and end times + the duration of the test, then loops back, until all tests in the table are completed.

-- NULL the end time only if it failed, otherwise set the end time

If @success=0 SET @endtime = getdate()

-- save the times and durations

Update [dbo].[TRIALS]

Set StartTime = @starttime, EndTime=@endtime, DurationMs=DateDiff(MS, @starttime, @endtime)

WHERE trialname =@trial

-- just in case of failure

SET @starttime =NULL

SET @endtime =NULL

FETCH NEXT FROM trial_cursor into @trial,@server, @command

END

CLOSE trial_cursor

DEALLOCATE trial_cursor

If an error happens, the script will stop running. There is no ‘error’ checking, as I could not find a way to catch DB2 errors if something happened, but it is possible to catch SQL Errors using a TRY-CATCH, but that requires a bit more work.

After the script runs, the TRIALS table will contain data similar to the below:

Trial #: 004-123	DB2400_290	select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY	5/11/2009 10:47:17 PM	5/11/2009 10:47:27 PM	9736
Trial #: 004-124	DB2400_300	select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY	5/11/2009 10:47:27 PM	5/11/2009 10:47:37 PM	9743
Trial #: 005-125	DB2400_000	select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY	5/11/2009 10:47:37 PM	5/11/2009 10:49:10 PM	92610
Trial #: 005-126	DB2400_010	select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY	5/11/2009 10:49:10 PM	5/11/2009 10:50:58 PM	108460
Trial #: 005-127	DB2400_020	select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY	5/11/2009 10:50:58 PM	5/11/2009 10:52:45 PM	107063
Trial #: 005-128	DB2400_030	select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY	5/11/2009 10:52:45 PM	5/11/2009 10:54:29 PM	104143

In the above table, you can see where test 4 (reading 10,000 rows) switched to test 5 (reading 100,000 rows). The actual test# being here were the 123^rd through the 128^th query being executed. The last 3 columns contain the start and stop time, plus the duration in milliseconds. Also, by looking at the linked server name, you can see that the tests were using rowsets of 290, 300, 0, 10, 20, and 30.

I would also highly recommend running this script from a command line, piping the output to nul

GenericPart3.sql

This is a simple script that cleans up most of part1 tables. It does not delete the TRIALS tables, but removes the linked servers that were created, and drops the LinkServerNames and Query tables.

GenericPart4.sql

Now, after running GenericPart2.sql, you have your table full of performance data. But, the data is intermingled. You know you should have 3 queries for 10,000 rows against each of the linked server, but how do you get that out? And, even better, can you average the 3 test times? Well, yes you can, with this script.

This script builds a pivot sql query against the trials database, using the actual query. To only get the 3 queries, it first opens up a cursor against the TRIALS database, reading in the DISTINCT commands that were issued. These are saved in a string to be used by the pivot query (the command strings become the column headers).

Next, the script builds the query. The query also takes the average of the durationMS column from the TRIALS table. The final query (in my case) would look similar to this:

Select servername

, [select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY]

, [select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY]

, [select * from SCHEMA.TABLE FETCH FIRST 1000000 ROWS ONLY]

from (select servername

, command

, durationms

from dbo.trials) AS rf

PIVOT ( avg(durationms)

FOR command IN (

[select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY]

, [select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY]

, [select * from SCHEMA.TABLE FETCH FIRST 1000000 ROWS ONLY])

) AS p

Once the script is executed, you have a nice table that can easily be reviewed to see which setting may work best for a particular query:

servername	select * from SCHEMA.TABLE FETCH FIRST 10000 ROWS ONLY	select * from SCHEMA.TABLE FETCH FIRST 100000 ROWS ONLY	select * from SCHEMA.TABLE FETCH FIRST 1000000 ROWS ONLY
DB2400_000	10798	103306	940229
DB2400_010	12305	113839	1175765
DB2400_100	10297	93475	925285
DB2400_110	10418	93156	941389
DB2400_120	10374	93995	938454
DB2400_130	10635	95111	940769
DB2400_140	10527	93925	936573
DB2400_150	10566	93308	944626
DB2400_160	10516	94326	941008
DB2400_170	10760	94126	938562
DB2400_180	10503	93899	935958
DB2400_190	10623	93555	922817
DB2400_020	11847	110398	1080601
DB2400_200	10894	93490	904607
DB2400_210	10658	93555	905882
DB2400_220	10799	93133	919812
DB2400_230	10687	95160	923053
DB2400_240	10848	96003	919006
DB2400_250	10623	96033	931469
DB2400_260	10486	98468	927744
DB2400_270	10663	96406	941534
DB2400_280	11876	96829	936327
DB2400_290	11311	98304	931216
DB2400_030	11903	105999	1068744
DB2400_300	11307	98747	927451
DB2400_040	11594	103962	1053893
DB2400_050	11363	103201	1032705
DB2400_060	11171	100775	1012864
DB2400_070	10931	98275	994035
DB2400_080	10623	96421	981286
DB2400_090	10338	94936	953186

I hope this helps. In part 2 and part 3 I will address performance testing SSIS and C# queries using the Microsoft OLE DB Provider for DB2.

DB2 Code Page Fun

Charles Ezzell — Fri, 31 Oct 2008 10:24:00 GMT

A while back I had a customer who was importing data from AS/400 DB2 tables into SQL Server, using SSIS, and we came upon an interesting error. The below describes how to actually reproduce the issue, and the fix. The fix is currently in all versions of our DB2 Data Provider at this point (assuming you have the latest of the stand-alone versions, or latest QFEs for HIS 2004). In the customers case, they were using the standalone provider that is part of the SQL Server Feature Pack.

The customer has very large tables, with all their CHAR() columns using CCSID 65535. A CCSID of 65535 means the data is stored on the AS400 in binary format, and requires ‘no conversion’. Almost true, if you only want the actual hex values.

Also, about half of the customer’s data was binary English, but the rest was binary Japanese, which complicates things a bit.

So, let’s begin. First, we need to create a table to show how things behave. To do so, you need 2 columns using CCSID 939 (Japanese), 2 columns using CCSID 65535 (binary), and one column using CCSID 37 (English).

Create a linked server to the AS400 using English (CCSID 37, PC Code page 1252). Call this linked server CPENGLISH.

Now, we are going to create the table. Run the following 2 scripts (be sure to change the schema name to yours).

exec('

CREATE TABLE .CODEPAGEFUN (

JCHAR939 CHAR(40) CCSID 939 DEFAULT NULL ,

JCHAR65535 CHAR(40) CCSID 65535 DEFAULT NULL ,

ECHAR37 CHAR(40) CCSID 37 DEFAULT NULL ,

ECHAR939 CHAR(40) CCSID 939 DEFAULT NULL ,

ECHAR65535 CHAR(40) CCSID 65535 DEFAULT NULL )

') at CPENGLISH

Exec('

LABEL ON COLUMN .CODEPAGEFUN

( JCHAR939 IS ''Japanese 939 CP '' ,

JCHAR65535 IS ''Japanese 65535 CP '' ,

ECHAR37 IS ''English 37 CP '' ,

ECHAR939 IS ''English 939 CP '' ,

ECHAR65535 IS ''English 65535 CP '' )

') at CPENGLISH

Now we need to insert some data into the database. Since we are dealing with Japanese data, you can not use English to insert Japanese data (humor me) so create a new linked server using CCSID 939, PC Code Page 932. Call this one CPJAPANESE.

Then, run the following script:

DECLARE @var1 nchar(40);

DECLARE @var2 nchar(40);

set @var1 = N'チャールズ';

set @var2 = N'Charles';

EXEC('Insert into .CODEPAGEFUN

(JCHAR939, JCHAR65535, ECHAR37, ECHAR939, ECHAR65535)

Values (?,?,?,?,?)',@var1,@var1,@var2,@var2,@var2)

at CPJAPANESE

The above script inserted Japanese into the first 2 columns, then English into the last 3 columns.

The JCHAR65535 column contains the hex code for “チャールズ”, which translated to English is Charles (hopefully :) ).

Now, what is the issue you ask? This table contains multiple columns, with each column using different character sets. This is not common, but I’ve seen this in a few customer tables, and 2 of the columns are using CCSID 65535.

So, what happens in SQL if you try to run a select statement against this table?

First, let’s use the English code page (the host is English, the Windows system is English, so why not, right?). Since we already have our CPENGLISH linked server setup, let’s start there.

EXEC('SELECT * FROM .CODEPAGEFUN') at CPENGLISH

This query returns:

JCHAR939 JCHAR65535 ECHAR37 ECHAR939 ECHAR65535

0x0E43924353435843AF43C70F40404040404040404040404040404040404040404040404040404040 Charles 0xC38881999385A2404040404040404040404040404040404040404040404040404040404040404040

Well it appears only 3 columns seem to have come back with data – JCHAR65535, ECHAR37, ECHAR65535. What’s up with that?

Well, the codepages for 1252/37 do not contain the Japanese character set, so those can’t be translated. This seems simple enough, right. So, let’s change the provider string to use the Japanese linked server we setup earlier:

EXEC('SELECT * FROM .CODEPAGEFUN') at CPJAPANESE

This query returns:

JCHAR939 JCHAR65535 ECHAR37 ECHAR939 ECHAR65535

ƒ`ƒƒ[ƒ‹ƒY 0x0E43924353435843AF43C70F40404040404040404040404040404040404040404040404040404040 Charles Charles 0xC38881999385A2404040404040404040404040404040404040404040404040404040404040404040

Well, better, all the columns came back this time. Also note that columns ECHAR37 and ECHAR939 both return “Charles ”. It just so happens that the 939/932 code pages are a MBCS (multi-byte character set), and contain both DBCS for Japanese, and the SBCS code page for English. Thus, you can use this combination to convert Code Page 37 to Code Page 932 and not lose any data.

But, we still can’t actually ‘read’ the 65535 data. Our data provider has an option to convert Binary to Character, so let us try this and see what happens. Create a new linked server, with “Process Binary as Character=True” set (default is false). Call this one CPJAPBIN.

New Query using the new linked server would be:

EXEC('SELECT * FROM .CODEPAGEFUN') at CPJAPBIN

And the output:

JCHAR939 JCHAR65535 ECHAR37 ECHAR939 ECHAR65535

ƒ`ƒƒ[ƒ‹ƒY Charles Charles

Well. That seems to have made things worse. The binary columns did not come out. You can actually verify this by running the following query, which would return 2 ‘empty’ columns:

EXEC('SELECT JCHAR65535, ECHAR65535 FROM .CODEPAGEFUN') at CPJAPBIN

JCHAR65535 ECHAR65535

Nothing appears to have come down. Those 2 columns ‘should’ have been converted to character data but were not. To make matters worse no errors are reported back.

So, let’s try something different.

If you were attempting to import this table via SSIS into SQL with settings similar to the CPJAPBIN linked server, you WOULD get errors with Binary as Character set to true.

[Source - CODEPAGEFUN [1]] Error: There was an error with output column "JCHAR65535" (23) on output "OLE DB Source Output" (11). The column status returned was: "The value could not be converted because of a potential loss of data.”

[Source - CODEPAGEFUN [1]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "output column "JCHAR65535" (23)" failed because error code 0xC0209072 occurred, and the error row disposition on "output column "JCHAR65535" (23)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.

Basically, SQL can not transform the column properly, because it is DBCS data. Apparently after the transformed data is converted from Binary to Char, and from EBCDIC to ASCII, there is no guarantee that the you would have 40 characters, so it gives up.

There is a ‘reason’ for this, in a sense, and you have to be very careful when using some DBCS languages. In some character sets, an EBCDIC DBCS value may be represented by one ‘character’. When converted from EBCDIC to ASCII, the ASCII representation may be 2 characters in length. The reverse is also true. I’ve seen this with Arabic, and it can happen for other BIDI languages:

· The Arabic Lam Alef is one character in 20420 while it is two in 1256

· Final Seen, Sheen, Sadh and Dhadh are two characters in 20420 while they are one in 1256

With MBCS, you also have SO/SI characters that are part of the data that will be transferred.

So, after talking with the development team about this, we came up with a solution to help convert the columns as character. A new parameter was added to the data source string which ‘forces’ the conversion to take place in a separate manner than previous done (this is mainly a problem for DBCS languages). This additinoal parameter is not available via the “Data Access Tool”, but can be accessed via the ALL tab on the “Data Link Properties”. The property is “Binary Codepage”, and only works when “Process Binary as Character” is set to “True”.

In the data provider string, we need to add this new parameter in. The easiest way to do this is as follows.

1) Using the DAT (Data Access Tool), create a new UDL similar to the linked server CPJAPBIN, and save this as CPJAPBINSET.

2) The DAT‘s default directory for storing UDLS is “C:\Documents and Settings\\My Documents\Host Integration Projects\Data Sources”, so open this directory.

3) Double click the CPJAPBINSET.udl file. This will bring up the “Data Link Properties” window.

4) Under the “Advanced” tab, verify that the Host CCSID is 939, the PC code page is 932, and Process Binary as character is checked. Then go to the “All” tab.

5) Find the “Binary Codepage” setting. Double click this and set this to 939. Click “OK” and save the UDL file.

6) Now open the UDL file up in notepad, and copy the init string. It should be similar to this:

Provider=DB2OLEDB;Password=password;Persist Security Info=True;User ID=USERID;Initial Catalog=S105FRDM;Defer Prepare=False;Binary Codepage=939;Derive Parameters=False;Rowset Cache Size=0;Network Transport Library=TCP;Host CCSID=939;PC Code Page=932;Network Address=S105FRDM;Network Port=446;Package Collection=CHARLIEE2;Default Schema=CHARLIEE2;DBMS Platform=DB2/AS400;Process Binary as Character=True;Connection Pooling=False;Units of Work=RUW

Now, create a new linked server using the init string from above, and name this linked server CPJAPBINTRUE.

Now, using the new linked server, run the same query again:

EXEC('SELECT * FROM .CODEPAGEFUN') at CPJAPBINTRUE

This should return:

JCHAR939 JCHAR65535 ECHAR37 ECHAR939 ECHAR65535

ƒ`ƒƒ[ƒ‹ƒY ƒ`ƒƒ[ƒ‹ƒY Charles Charles Charles

Success!!!

And SSIS should now work without errors when “Binary Codepage =939” is set.

In this particular customer’s case, their tables columns contained mixed Japanese and English.

For an example, run the following script:

declare @counter int

DECLARE @var1 nchar(40)

DECLARE @var2 nchar(40)

set @counter = 0

while @counter < 5

begin

set @counter = @counter + 1

print @counter

if (@counter = 1)

begin

set @var1 = N'マイクロソフト'

set @var2 = N'Microsoft'

end

if @counter = 2

begin

set @var1 = N'ホスト統合サーバー'

set @var2 = N'Host Integration Server'

end

if @counter = 3

begin

set @var1 = N'ノースカロライナ州'

set @var2 = N'North Carolina'

end

if @counter = 4

begin

set @var1 = N'Washington'

set @var2 = N'Washington'

end

if @counter = 5

begin

set @var1 = N'Texas'

set @var2 = N'Texas'

end

begin

print @var1

print @var2

EXEC('Insert into .CODEPAGEFUN

(JCHAR939, JCHAR65535, ECHAR37, ECHAR939, ECHAR65535)

Values (?,?,?,?,?)',@var1,@var1,@var2,@var2,@var2)

at CPJAPANESE

end

Now, after running the following select query:

EXEC('SELECT JCHAR939, JCHAR65535 from .CODEPAGEFUN') at CPJAPBINTRUE

You should see the following returned (again note that Query Analyzer's output does not display the Japanese Character set properly here, but the binary bits are correct):

JCHAR939 JCHAR65535

ƒ`ƒƒ[ƒ‹ƒY ƒ`ƒƒ[ƒ‹ƒY

ƒ}ƒCƒNƒƒ\ƒtƒg ƒ}ƒCƒNƒƒ\ƒtƒg

ƒzƒXƒg“‡ƒT[ƒo[ ƒzƒXƒg“‡ƒT[ƒo[

ƒm[ƒXƒJƒƒ‰ƒCƒiB ƒm[ƒXƒJƒƒ‰ƒCƒiB

Washington Washington

Texas Texas

This also shows that it is possible for a customer to have mixed languages in the same column, depending on the actual codepages they are using.

Note: Another solution to ‘force’ conversion of the binary strings is by telling DB2 to do the conversion by doing a CAST against the column. So, using the ‘Japanese” linked server we created originally, the following query would return everything as character data:

EXEC('SELECT

JCHAR939,

cast(JCHAR65535 as CHAR(40) CCSID 939) as JCHAR65535,

ECHAR37,

ECHAR939,

cast (ECHAR65535 as CHAR(40) CCSID 939) as ECHAR65535

from .CODEPAGEFUN') at CPJAPANESE

JCHAR939 JCHAR65535 ECHAR37 ECHAR939 ECHAR65535

ƒ`ƒƒ[ƒ‹ƒY ƒ`ƒƒ[ƒ‹ƒY Charles Charles Charles

ƒ}ƒCƒNƒƒ\ƒtƒg ƒ}ƒCƒNƒƒ\ƒtƒg Microsoft Microsoft Microsoft

ƒzƒXƒg“‡ƒT[ƒo[ ƒzƒXƒg“‡ƒT[ƒo[ Host Integration Server Host Integration Server Host Integration Server

ƒm[ƒXƒJƒƒ‰ƒCƒiB ƒm[ƒXƒJƒƒ‰ƒCƒiB North Carolina North Carolina North Carolina

Washington Washington Washington Washington Washington

Texas Texas Texas Texas Texas

Using PowerShell to Access DB2

Charles Ezzell — Fri, 31 Oct 2008 07:44:00 GMT

Yes, PowerShell can be used to access DB2 using The Microsoft Host Integration Server Data Providers.

Not a tutorial, but here is the script, ran using HIS 2004’s data provider:

$cn = new-object system.data.OleDb.OleDbConnection("Provider=DB2OLEDB;User ID=;Password=;Initial Catalog=;Network Transport Library=TCP;Host CCSID=37;PC Code Page=1252;Network Address=DB2V82;Network Port=50000;Package Collection=;Default Schema=;Process Binary as Character=False;Units of Work=RUW;DBMS Platform=DB2/NT;Defer Prepare=False;Persist Security Info=True;Connection Pooling=False;");

$ds = new-object "System.Data.DataSet" "dsTest"

$q = "SELECT ID_NUMBER"

$q = $q + " ,FIRST_NAME"

$q = $q + " ,LAST_NAME"

$q = $q + " ,CITY"

$q = $q + " ,STATE"

$q = $q + " FROM .ADELINS"

$da = new-object "System.Data.OleDb.OleDbDataAdapter" ($q, $cn)

$da.Fill($ds)

$dtPerson = new-object "System.Data.DataTable" "dtPersonData"

$dtPerson = $ds.Tables[0]

$dtPerson | FOREACH-OBJECT { " " + $_.ID_NUMBER + ": " + $_.FIRST_NAME + ", " + $_.LAST_NAME + ", " + $_.CITY + ", " + $_.STATE }

Output (the values are dummy ones in the table and don’t match the column names, but everything is correct):

PS C:\scripts> c:\scripts\db2query.ps1

1 : 1, 2, 3, 4

1 : 2, 3, 4, 5

1 : 3, 4, 5, 6

1 : 4, 5, 6, 7

1 : 5, 6, 7, 8

x : 6, 7, 8, 9

x : 7, 8, 9, 10

x : 8, 9, 10, 11

x : 9, 10, 11, 12

x : 10, 11, 12, 13

1 : 6, 7, 8, 9

1 : 7, 8, 9, 10

1 : 8, 9, 10, 11

1 : 9, 10, 11, 12

1 : 10, 11, 12, 13

A more useful script is this one, that pings an AS400 (in this case), then tries to open a socket to the DDM port (446) to see if it’s listening:

$ip ="172.29.136.200"

$ping = new-object System.Net.NetworkInformation.Ping

$rslt = $ping.send($ip)

if ($rslt.status.tostring() -eq "Success")

{

write-host "ping worked"

# if the ping works, try opening a socket to the DDM port

$port = 446

$socket = new-object System.Net.Sockets.TcpClient($ip, $port)

if ($socket -eq $null)

{

write-host "could not open DDM socket"

}

else

{

write-host "got socket to DDM"

$socket = $null

}

else

{

write-host "ping failed"

}

$ping = $null

Output:

C:\scripts>powershell c:\scripts\portping.ps1

ping worked

got socket to DDM

Importing CHAR() fields containing null values from DB2 to SQL Server via SSIS

Charles Ezzell — Thu, 30 Oct 2008 11:22:00 GMT

Earlier this year, I was working with a customer who was doing a mainframe migration from DB2 to SQL Server, and they were using SSIS to import the tables.

Several of their columns were CHAR data types, with a null byte as the first byte of the field. The columns, when imported, all contained empty character strings. Also, when using t-sql from a linked server, the import was also failing.

For testing purposes, I setup a simple table as such, in DB2 (all columns are CHAR(10) in this table:

Row 1:
Column 1 = “Bob”
Column 2 = “[null]Bob”
Column 3 = “Bob[null]”
Row 2:
Column 1 = “Bob”
Column 2 = “[null][null]Bob”
Column 3 = “Bob[null] [null]”

Now, as we all know, NULL is a 'string terminator'. So when SSIS or our data provider would see the null, the data was truncated at that point. We were able to provide a fix to correct the issue with the data provider truncating the null, so that t-sql scripts similar to the below would work:

insert into localtable (select * from linkedserver.database.schema.table)

The above would be fine if you only had a few rows, but if you had millions of rows, a performance hit. And when using SSIS to import the data, even though we passed the correct string to SSIS, it was still terminating the string at the null.

Through some creative scripting with the help of some SSIS Developers, we were able to come up with a workaround that was acceptable to the customer for their migration project.

First, you would need to create a temporary database in SQL, using the TEXT data type for the columns containing NULLs, similar to this:

CREATE TABLE [DB2TestDatabases].[dbo].[TEMPDATABASE]
([C1] char(10),
[C2] text,
[C3] text)

Then, in your source query for SSIS, use a query similar to this:

Select
C1,
cast(c2 as char(10) for bit data) as C2,
cast(c3 as char(10) for bit data) as C3
from schema .table

The above query tells DB2 to cast columns C2 and C3 as binary data.

Next, you need to create a scripting component in the SSIS package.

In the script component, select C2 and C3 as the input columns. No need, in this case, to use C1 as it does not contain nulls, and can be passed directly through.

Next, create 2 output columns. I called mine (not being very creative) newcolumn2 and newcolumn3. Set these columns to be DT_TEXT data types, since this is what we are going to be writing to the SQL Server Database.

Then, in the script editor, add code similar to this:

Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
'
' Add your code here
'
Dim i As Integer
Dim C2bytearray(UBound(Row.C2)) As Byte
For i = 0 To UBound(Row.C2)
C2bytearray(i) = Row.C2(i)
Next
Row.NewColumn2.AddBlobData(C2bytearray)
Dim C3Bytearray(UBound(Row.C3)) As Byte
For i = 0 To UBound(Row.C3)
C3Bytearray(i) = Row.C3(i)
Next
Row.NewColumn3.AddBlobData(C3Bytearray)
End Sub

The above was not the original code, but proved to be much faster. It takes the binary data and converts it to a format acceptable for the TEXT data type. The original code I uses was a bit slower, converting each byte from binary to CHR, inserting into a STRING, then assigning the string to the new columns.

After running the package, you can verify the import by running the following t-sql script (you have to double cast c2 and c3 to read the text data as binary to actually see that the nulls are present.

select cast(c1 as varbinary(10)) as C1,
cast(cast(c2 as char(10)) as varbinary(10)) as C2,
cast(cast(c3 as char(10)) as varbinary(10)) as C3
from dbo.TEMPDATABASE

This returns:

0x426F6220202020202020 0x00426F62202020202020 0x426F6200202020202020
0x426F6220202020202020 0x0000426F622020202020 0x426F6200002020202020

Notice in the 2nd columns the nulls(00) before the ‘42’ and in the 3rd column the nulls after the ‘62’.
Now, the only issue left here is that the data in the temp database is a TEXT data type, not char. So, create the final table:

CREATE TABLE [dbo].[newTable](
[C1] [char](10) NULL,
[C2] [char](10) NULL,
[C3] [char](10) NULL
)

Then run the following t-sql:

insert into dbo.newTable
select * from dbo.TEMPDATABASE

Once this completes, the data in the newTable contains the null char values as was originally needed:

select cast(c1 as varbinary(10)) as C1,
cast(c2 as varbinary(10)) as C2,
cast(c3 as varbinary(10)) as C3
from dbo. newTable

This returns:

0x426F6220202020202020 0x00426F62202020202020 0x426F6200202020202020
0x426F6220202020202020 0x0000426F622020202020 0x426F6200002020202020

This is not the optimal solution, but I currently do not have a better method.

Finally, if the data types in the original DB2 database had been CHAR() FOR BIT DATA (binary), this would not have been an issue, as you could have directly mapped the columns to VarBinary() in SQL and the import would have worked properly.

If anyone knows of another method, I would love to hear about it! And hope this helps someone else in a similar situation.