Darren, Ewan and myself are really close to getting the book, Professional BizTalk Server 2006 out of the door, it's been hard work but hopefully it'll be worth while. The book offers insight in to how BizTalk works under the hood which will enable you to really exploit the architecture, this is blended with the lessons learned designing, developing, and operating some of the most mission critical BizTalk solutions that have been deployed around the world by Microsoft Consulting Services. I knew it would be a massive amount of work before we started, and it has definitely been that!!

As you may have seen GotDotNet is being phased out by Microsoft, because of this I've moved BizUnit to its new home at CodePlex, I have some great ideas for new features which, now the book is done I might actually get time to implement!! The latest version includes a bunch of work that we did on the last performance lab in Redmond that I was involved in, including some new test steps to enable it to be used to automate performance tests.

Aside form the book, a lots been going on for me. I left Microsoft last summer to work for a leading european investment bank, leaving after 8+ incredible years was not easy I can tell you. I had an amazing time working for Microsoft, a lot of great times. Since leaving I've been playing with a lot of different but very interesting technology, so I should get back to some blogging soon!

I’ve finally got around to finishing off BizUnit 2006 - v2.1 just in time for the BizTalk 2006 GA, this version is targeted at BizTalk 2006 and Visual Studio 2005 and is built using .Net 2.0. I’ve been using the new unit testing capability in Visual Studio 2005 to drive BizUnit for a while now instead of NUnit and have to say I’m pretty pleased with it.

• FactBasedRuleEngineStep
• PerfmonCountersStep
• DBExecuteNonQueryStep
• DBQueryReturnXmlStep
• ExportDBDataToDataSetStep
• HostConductorStep
• ImportDatasetToDBStep

And some new Validation Steps:

• CodeValidationStep
• XmlValidationStepEx

I’ve also added wild card support for reading configuration, the following wild cards are supported, let me know if there are others you’d like to see supported:

• %DateTime% - will replace the wild card with the current date time in the format HHmmss-ddMMyyyy
• %ServerName% - will replace the wild card with the name of the server BizUnit is being executed on
• %Guid% - will be replaced by a new Guid

For example, for the test step configuration below:

<TestStep assemblyPath="" typeName="Microsoft.Services.BizTalkApplicationFramework.BizUnit.FileCreateStep">

      <SourcePath>..\..\..\TestData\InDoc1.xml</SourcePath>        

      <CreationPath>..\..\..\Rec_03\TransactionId_%Guid%_%ServerName%.xml</CreationPath>

</TestStep>

CreationPath becomes "..\..\..\Rec_03\TransactionId_12345678-D6AB-4aa9-A772-938972E3FD51_ZEUS001.xml"

The diagram below illustrates the format of a test case.

In addition to test steps, BizUnit has the notion of validation steps and context loader steps. These can be thought of as sub-steps and can in general be independantly executed from any test step. For example, an MSMQ-read step might be used to read and validate both Xml and Flat File data from a queue, the same step can be used with both the RegExValidationStep and the XmlValidationStep to validate the data read.

A test step within a test case can be marked with the attribute - runConcurrently which causes subsequent test steps to be started before it has completed. In addition test steps maybe marked with the attribute - failOnError, setting it to false cause BizUnit to ignore a failure of that test step, this is particularly useful for the setup and cleanup stages of test cases.

Lets look at an Example Scenario...

BizUnit takes a black box approach to testing solutions, if you look at the scenario below, a BizTalk solution receives a request-response message over HTTP, the message is routed to an Orchestration which, sends a message to MSMQ and another to a FILE drop, the Orchestration waits for a FILE to be received, after which the Orchestration sends the response back to the waiting HTTP client. The solution also uses BAM, writing business data to the BAM database.

In order to test this scenario, a BizUnit test case is defined that has 5 test steps:

1. The HttpRequestResponseStep sends the request to the two-way receive port and waits for the response. This step is executed concurrently so that the other test steps may execute whilst it waiting for the response
2. The MSMQReadStep waits for a message to appear on an MSMQ queue, when it reads the message it uses the XmlValidationStep to perform schema validation and also execute a number of XPath expression to ensure the message contains the correct data
3. The FileValidateStep waits for a FILE to be written to a given directory, when it reads the FILE it validates the data using the RegExValidationStep validation step since the FILE picked up was a flat file format
4. The FileCreateStep creates a new FILE in the specified directory containing the data that the backend system would typically create. This allows the Orchestration to complete and send the response back to the waiting HttpRequestResponseStep step
5. Finally, DBQueryStep is used to check that all of the BAM data has been successfully written to the BAMPrimaryImportDB

Finally, I'd like to give a big thanks to the following who have contributed new steps, ideas, etc to this version of BizUnit, my apologies if I have missed off anyone, it’s not intentional just my disorganisation!, please let me know if that is the case and I’ll add you:

Jon Fancey

Mike Becker

Tanveer Rashid

Young Jun Hong

If you find bugs, have new steps or ideas/requirements for new features that you'd like incorporated let me know.

Enjoy!

• Right click the BizTalk map and select “Validate Map”
• In the output window you’ll see a link to the .xsl, click on the link to bring it up in VS
• Right click the .xsl and select “View Source”
• At that point you can set break points in your .xsl
• Then select”XML / Debug XSLT” – at which point you’ll be able to step thru the XSLT

I've been tracking the CCR internally within Microsoft for a while now, and now there's an interesting channel 9 interview with the CCR guys: https://channel9.msdn.com/Showpost.aspx?postid=143582 if you've ever done any amount of concurrent programming you may find it interesting. As we move towards multi-core CPU architectures concurrent programming will become essential in order to make apps scale. Enjoy!

If you're interested or want to find out more about the opportunities email Kerry Van Voris your resume: kerryv@microsoft.com

The first problem was getting the message on the MQ dead letter queue, John and Anil gave some great pointers here, it turns out to send a message to the dead letter queue the message needs to be pre-pended with the dead letter header (DLH) and the MQMD_Format property needs to be set to MQ "MQDEAD  " so that MQ knows to expect the DLH.

Serializing the Dead Letter Header

To achieve this I wrote a DLH utilities component that allows the various fields of the DLH header to be set, the component also serializes the DLH into a byte[] so that it may be pre-pended to the message. The component was called from a custom send pipeline component which allows the key fields to be set at design time, the pipeline component also sets the MQMD_Format property to the message context.

The format of the MQ DLH struct is as follows:

char[] strucId = new char[4]; // Structure identifier - MQCHAR4 StrucId

int version = 1; // Structure version number

int reason; // Reason message arrived on dead-letter

char[] destQName = new char[48];// Name of original destination queue

char[] destQMgrName  = new char[48];// Name of orig dest queue manager

int encoding; // Numeric encoding of data that follows MQDLH

int codedCharSetId; // Char set identifier of data that follows MQDLH

char[] format = new char[8]; // Format name of data that follows MQDLH

int putApplType; // Type of app that put message on DLH

char[] putApplName = new char[28]; // Name of app that put msg on DLH

char[] putDate = new char[8]; // Date when message was put the DLH

char[] putTime = new char[8]; // Time when message was put the DLH

This struct needs to be serialised into a byte[] and pre-pended to the message:

public byte[] SerializeHeader()

{

      byte[] header = new byte[172];

      int index = 0;

      byte[] tmp = null;

      // strucId

      int written = System.Text.Encoding.UTF8.GetBytes(strucId, 0, strucId.Length, header, index);

      index += 4;

      // version

      tmp = BitConverter.GetBytes(version);

      tmp.CopyTo(header, index);

      index += 4;                  

      // reason

      tmp = BitConverter.GetBytes(reason);

      tmp.CopyTo(header, index);

      index += 4;

      // destQName

      written = System.Text.Encoding.UTF8.GetBytes(destQName, 0, destQName.Length, header, index);

      index += 48;

      // destQMgrName

      written = System.Text.Encoding.UTF8.GetBytes(destQMgrName, 0, destQMgrName.Length, header, index);

      index += 48;

      // encoding

      tmp = BitConverter.GetBytes(encoding);

      tmp.CopyTo(header, index);

      index += 4;

      // codedCharSetId

      tmp = BitConverter.GetBytes(codedCharSetId);

      tmp.CopyTo(header, index);

      index += 4;

      // format

      written = System.Text.Encoding.UTF8.GetBytes(format, 0, format.Length, header, index);

      index += 8;

      // putApplType

      tmp = BitConverter.GetBytes(putApplType);

      tmp.CopyTo(header, index);

      index += 4;

      // putApplName

      written = System.Text.Encoding.UTF8.GetBytes(putApplName, 0, putApplName.Length, header, index);

      index += 28;

      // putDate - Format: yyyymmdd

      written = System.Text.Encoding.UTF8.GetBytes(putDate, 0, putDate.Length, header, index);

      index += 8;

      // putTime - Format: hhmmss00

      written = System.Text.Encoding.UTF8.GetBytes(putTime, 0, putTime.Length, header, index);

      index += 8;

      return header;

}

Pipeline Component

The MQ adapter does not directly support setting this property, so the custom send pipeline component is responsible for pre-pending the message data stream with the DLH and setting the message context property which the MQ adapter will then set on the MQ message:

private static PropertyBase mqmd_Format = new MQSeries.MQMD_Format();

private const string MQFMT_DEAD_LETTER_HEADER = "MQDEAD  ";

// Set MQMD_Format property to MQFMT_DEAD_LETTER_HEADER - to indicate the message

// is prepended with the dead letter header...

inmsg.Context.Write(    mqmd_Format.Name.Name,

mqmd_Format.Name.Namespace,

MQFMT_DEAD_LETTER_HEADER );

// Pre-pend the message body with the MQS dead letter header...

inmsg.BodyPart.Data = DeadLetterHelper.BuildDeadLetterMessage(

inmsg.BodyPart.GetOriginalDataStream(),

_DestinationQueue,

_QueueManager,

_ApplicationName );

All that is required then is for the send port to be configured to send the message to the SYSTEM.DEAD.LETTER.QUEUE.

Dead Letter Handler

Once the message is on the dead letter queue the dead handler (runmqdlq.exe) can be configured to take the message off the queue and put it on the destination queue defined in the DLH. This proved to be a little tricky to get working, thanks to Jason for his help in getting the dead letter handler working. 

runmqdlq.exe SYSTEM.DEAD.LETTER.QUEUE QM_demoappserver < qrule.rul

The handler may be fed a rule (qrule.rul), the example here removes the DLH and puts the message on the queue specified in the DLH, one thing to watch out for is the rule below needs to have a CRLF at the end!!:

INPUTQM(QM_demoappserver) INPUTQ('SYSTEM.DEAD.LETTER.QUEUE') WAIT(NO)

ACTION(FWD) FWDQ(&DESTQ) HEADER(NO)

BizUnit defines test cases as Xml documents, which, enables test cases to be rapidly written and also enables the same test cases to moved between different environments by fixing up the Url’s for the endpoints, for example the endpoint Url’s for a development environment will be different to a testing, and pre-production environments.

Also, by having the test cases defined as Xml, it allows them to be easily generated, some of the customers that I have worked with define their test matrixes in Excel, and then auto-generate the BizUnit test cases from Excel.

Test Case Format

A test case is made up of three stages, test setup, test execution and test cleanup, the cleanup stage is always executed and intended to leave the platform in the same state that it started.

Each stage may consist of zero or more test steps, test steps are in general autonomous, state can be flowed between them if required using the ‘context’ object that is passed to each test step.

How does it Work?

BizUnit takes a black box approach to testing BizTalk solutions, if you look at the scenario below, a BizTalk solution receives a request-response message over HTTP, the message is routed to an Orchestration which, sends a message to MSMQ and another to a FILE drop, the Orchestration waits for a FILE to be received, after which the Orchestration sends the response back to the waiting HTTP client. The solution also uses BAM, writing business data to the BAM database.

In order to test this scenario, a BizUnit test case is defined that has 5 test steps:

1. The HttpRequestResponseStep sends the request to the two-way receive port and waits for the response. This step is executed concurrently so that the other test steps may execute whilst it waiting for the response
2. The MSMQReadStep waits for a message to appear on an MSMQ queue, when it reads the message it uses the XmlValidationStep to perform schema validation and also execute a number of XPath expression to ensure the message contains the correct data
3. The FileValidateStep waits for a FILE to be written to a given directory, when it reads the FILE it validates the data using the RegExValidationStep validation step since the FILE picked up was a flat file format
4. The FileCreateStep creates a new FILE in the specified directory containing the data that the backend system would typically create. This allows the Orchestration to complete and send the response back to the waiting HttpRequestResponseStep step
5. Finally, DBQueryStep is used to check that all of the BAM data has been successfully written to the BAMPrimaryImportDB 

BizUnit 2.0 Enhancements

A number of enhancements have been added to BizUnit 2.0, firstly, there are now some 37 test steps, which is vastly improved from the previous version. A lot of people have contributed test steps which I’ve included in this version. Briefly, here’s some of the enhancements:

• Concurrent test step execution option – allows multiple test steps to be executed at once
• Context – allows state to be flowed between test steps, think of the scenario where a FILE is written the content of which specifies where the response should be written
• Test group setup / tear down stages  - allows group wide setup and cleanup
• Help – all of the test steps have help in a .chm (accessed from Program Files\Microsoft Services BizTalkApplicationFramework\BizUnit 2.0\Documentation. The help contains Xml configuration examples for each test step that can be copy/pasted into a test case, along which a description of each configuration option etc
• FailOnError – flag to allow failed test cases to be ignored, this is useful during the cleanup stage 

Here’s a list of all of the test steps in BizUnit build 2.0.1062.0, the documentation provides more details around what these steps do and how to use them:

BAMDeploymentStep

CheckPop3MailStep

ContextManipulatorStep

CrossReferenceDataVerificationStep

CrossReferenceSeedClearStep

CrossReferenceSeedLoadStep

DatabaseDeleteStep

DatabaseRowCountStep

DBQueryStep

DelayStep

DotNetObjectInvokerStep

EventLogCheckStep

EventLogClearStep

ExecuteCommandStep

FileCreateStep

FileDeleteMultipleStep

FileDeleteStep

FileMoveStep

FileMultiValidateStep

FilesExistStep

FilesMoveStep

FileValidateStep

HttpPostStep

HttpRequestResponseStep

MSMQCreateQueueStep

MSMQDeleteQueueStep

MSMQQueuePurgeStep

MSMQReadStep

MSMQWriteStep

OrchestrationConductorStep

ReceiveLocationEnabledStep

ReceivePortConductorStep

RenameDirectoryStep

RuleEngineStep

SMTPReadStep

SOAPHTTPRequestResponseStep

WaitOnFileStep

MQSeriesClearQueueStep

MQSeriesGetStep

MQSeriesPutStep

OutlookReadStep

Validation Steps:

BinaryValidationStep

ContextValidationStep

RegExValidationStep

XmlValidationStep

Context Loader Steps:

RegExContextLoader

TextContextLoader

XmlContextLoader

Test Coverage:

So, you’ve developed all of your BVT’s (Build Verification Tests) for your BizTalk solution using BizUnit, how do you know whether you’ve done a good job, and how much test coverage you have? Enter Jason Births “Orchestration Profiler”. This is a great tool that you can use in conjunction with BizUnit to ensure that you have adequate test coverage, it produces a detailed graphical report of what the test coverage is like, I highly recommend using it with BizUnit.

Finally, I’d like to give my special thanks to a number of people that have contributed test steps, requirements and ideas for BizUnit 2.0, my apologies if I have missed off anyone, it’s not intentional just my disorganisation!, please let me know if that is the case:

Dave Regan

Tanveer Rashid

Daren Jefford

Kevin Purcell

Karina Apostolides

Jon Bonnick

Brian Milburn

Rahmatullah Khan

Future work:

A couple of the guys above have done some great work building a Visual Studio plugin for BizUnit and a command line driver. I didn’t have time to get it in this release, but it’ll be following soon. If you find any bugs, or have test steps you'd like to contribute please sned them along to me and I'll get them into the next drop.

Enjoy!