Event Trace Management Application Landscape

The Event Tracing for Windows (ETW) infrastructure provides the ability to start and stop event tracing sessions, monitor their status, configure a variety of logging settings such as buffer size, flush interval, stop conditions and more. The majority of the administrative tasks are available from both the GUI and the command-line interface to suit the needs and preferences of different IT persona types.

In addition to using the Reliability and Performance Monitor, the application developers and technical teams have access to other system tools and scripts helping automate many aspects of event trace management:

• Logman creates and manages Event Trace Session and Performance logs and supports many functions of Performance Monitor from the command line.
• WEvtUtil enables to retrieve information about event logs and publishers, install and uninstall event manifests, run queries, export, archive, and clear logs.
• Tracelog starts, stops, or enables trace logging.
• Tracefmt formats and displays trace messages from an event trace log file or a real-time trace session.

The BizTalk CAT Instrumentation Framework simplifies the use of the above tools by providing 2 easy-to-use scripts with a minimum of command-line parameters. These scripts can be found in the TracingTools folder and are intended to address the following requirements:

• StartTrace.cmd provides the ability to start an ETW event trace session either for the entire instrumented BizTalk solution, its selected component types or individually instrumented classes. This script is nothing more than just a wrapper for the Tracelog tool from the Windows Resource Kit.
• StopTrace.cmd carries the responsibility for stopping the ETW event trace sessions and converting a binary trace log file into human-readable format. This script relies on Tracelog and Tracefmt tools to flush, stop and apply formatting to the trace logs.

The next section is intended to clarify how the above administrative scripts map to various management scenarios.

Event Trace Management Tasks

Once deployed and launched, a well instrumented BizTalk application will be ready to emit a number of events related to the application’s behavior, internal state, execution scope durations, runtime exceptions and everything else that the application developers decided to include into the instrumentation context. The top management tasks involved into capturing, filtering and persisting the instrumented events can be summarized as follows.

Starting Event Trace Sessions

To start an event tracing session for the entire instrumented BizTalk application, run StartTrace.cmd using one of following commands depending on the trace level required (all/high/medium/low). Note that the log file name specified after -log parameter can be any name but should not contain whitespaces.

: All trace events (Info, Details, Warning, Error, In, Out, Start Scope, End Scope)
StartTrace -log BtsAppAllEvents -level all
: Limited trace events (Info, Warning, Error)
StartTrace -log BtsAppCoreEvents -level high
: Trace events indicating unexpected behavior (Warning, Error)
StartTrace -log BtsAppUnexpectedEvents -level medium
: Trace events related to runtime exceptions (Error)
StartTrace -log BtsAppExceptions -level low

To start an event tracing session for a specific type of application components, run StartTrace.cmd using one of following commands depending on the component type and trace level required:

: Start trace for all instrumented pipeline components
StartTrace -log PipelineComponentsAllEvents -level all -component Pipeline
: Start trace for all instrumented orchestrations
StartTrace -log OrchestrationsAllEvents -level all -component Workflow
: Start trace for all instrumented data access components
StartTrace -log DataAccessComponentsAllEvents -level all -component DataAccess
: Start trace for all instrumented maps or custom transform code
StartTrace -log MapsAllEvents -level all -component Transform
: Start trace for all instrumented Web/WCF services or service proxies
StartTrace -log WCFServicesAllEvents -level all -component Service
: Start trace for all instrumented business rules
StartTrace -log BusinessRulesAllEvents -level all -component Rules
: Start trace for all instrumented BAM activities
StartTrace -log BAMActivitiesAllEvents -level all -component Tracking
: Start trace for all instrumented custom .NET components
StartTrace -log CustomComponentsAllEvents -level all -component Custom

To start an event tracing session for individually instrumented application components, the component’s Guid attribute value must be provided in the command line as per the following examples:

: Start trace to capture all events for instrumented HL7Disassembler component
StartTrace -log HL7DasmFullTrace -level all -component Custom -guid 0EC4A54D-6B97-47C1-9118-A2BF8B4E7595
: Start trace to capture errors in instrumented HL7Disassembler component
StartTrace -log HL7DasmErrorTrace -level low -component Custom -guid 0EC4A54D-6B97-47C1-9118-A2BF8B4E7595

Monitoring Event Trace Sessions

After an event tracing session is started, it will remain running until it is manually stopped, a stop condition is encountered or host is rebooted. To query the present status of the event tracing sessions, one of the following approaches can be used.

To find out the event tracing session status from the command line, run the Logman utility using the following syntax:

: Lists all event tracing sessions configured on the local machine
Logman query -ets
: Lists all event tracing sessions with a name matching the specified pattern
Logman query -ets | find "AllEvents"

To find out the event tracing session status from the GUI, open Reliability and Performance Monitor and navigate to the Event Trace Sessions section.

Stopping Event Trace Sessions

To stop a running event trace session, the target tools differ depending on whether or not the trace session must be temporarily suspended or terminated with no intent to resume. Consequently, one of the following approaches can be used.

To stop an event tracing session from the command line, run StopTrace.cmd passing the original name of the trace log without file extension:

: Stop trace for all instrumented pipeline components
StopTrace -log PipelineComponentsAllEvents
: Stop trace for all instrumented orchestrations
StopTrace -log OrchestrationsAllEvents
: Stop trace for all instrumented data access components
StopTrace -log DataAccessComponentsAllEvents
: Stop trace for all instrumented maps or custom transform code
StopTrace -log MapsAllEvents
: Stop trace for all instrumented Web/WCF services or service proxies
StopTrace -log WCFServicesAllEvents
: Stop trace for all instrumented business rules
StopTrace -log BusinessRulesAllEvents
: Stop trace for all instrumented BAM activities
StopTrace -log BAMActivitiesAllEvents
: Stop trace for all instrumented custom .NET components
StopTrace -log CustomComponentsAllEvents

Note that StopTrace.cmd will also convert the binary trace log into text-based format using the Tracefmt tool. Depending on the log size and disk IO performance, this operation can take a few minutes.

To temporarily suspend an event tracing session from the command line without producing a text-based log file, run the Logman utility using the following syntax:

: Stop event tracing sessions all instrumented pipeline components, do not convert log into text file
Logman stop PipelineComponentsAllEvents -ets

To temporarily suspend the event tracing session from the GUI, open Reliability and Performance Monitor, navigate to the Event Trace Sessions section, right-click on the target event tracing session and select Stop from the context menu:

Configuring Event Trace Sessions

The default configuration of the ETW event trace sessions created by StartTrace.cmd can be attributed as follows:

• Buffer Size: 128K
• Maximum Buffers: 100
• Log Mode: Circular
• Maximum Log Size: 1000MB
• Flush Time: Not Set
• Clock Type: Performance
• Stream Mode: File
• Pre-allocate File Space: No

Should the default configuration be found insufficient and need modifications, the script can be updated to include the desired configuration settings. The new settings must be specified in the following line in StartTrace.cmd:

"%TraceLogTool%" -cir 1000 -b 128 -max 100 -start %TraceLogName% -flags %TraceLevel% -f %TraceLogFileName% -guid #%TraceComponentGUID%

Note that all commands shown in blue will not require any modification. Below are the summary of command line parameters supported by the Tracelog tool:

Usage: tracelog [actions] [options] | [-h | -help | -?]
actions:
 -start   [LoggerName] Starts up the [LoggerName] trace session
 -stop    [LoggerName] Stops the [LoggerName] trace session
 -update  [LoggerName] Updates the [LoggerName] trace session
 -enable  [LoggerName] Enables providers for the [LoggerName] session
 -disable [LoggerName] Disables providers for the [LoggerName] session
 -flush   [LoggerName] Flushes the [LoggerName] active buffers
 -remove  GlobalLogger Removes registry keys that activate GlobalLogger
 -enumguid             Enumerate Registered Trace Guids
 -q       [LoggerName] Query status of [LoggerName] trace session
 -l                    List all trace sessions
 -x                    Stops all active trace sessions
options:
 -b   <n>              Sets buffer size to <n> Kbytes
 -min <n>              Sets minimum buffers
 -max <n>              Sets maximum buffers
 -f <name>             Log to file <name>
 -append               Append to file
 -prealloc             Pre-allocate
 -seq <n>              Sequential logfile of up to n Mbytes
 -cir <n>              Circular logfile of n Mbytes
 -newfile <n>          Log to a new file after every n Mbytes
 -ft <n>               Set flush timer to n seconds
 -paged                Use pageable memory for buffers
 -noprocess            Disable Process Start/End tracing
 -nothread             Disable Thread Start/End tracing
 -nodisk               Disable Disk I/O tracing
 -nonet                Disable Network TCP/IP tracing
 -fio                  Enable file I/O tracing
 -pf                   Enable page faults tracing
 -hf                   Enable hard faults tracing
 -img                  Enable image load tracing
 -cm                   Enable registry calls tracing
 -um                   Enable Process Private tracing
 -guid <file>          Start tracing for providers in file
 -rt                   Enable tracing in real time mode
 -kd                   Enable tracing in kernel debugger
 -age <n>              Modify aging decay time to n minutes
 -level <n>            Enable Level passed to the providers
 -flag <n>             Enable Flags passed to the providers
 -eflag <n> <flag...>  Enable flags (several) to providers
 -ls                   Generate Local Sequence Numbers
 -gs                   Generate Global Squence Numbers
 -heap                 Use this for Heap Guid
 -critsec              Use this for CritSec Guid
 -pids <n> <pid1... >  Tracing for Heap and CritSec for different process

For more information on the related topic, please visit the following resources:

• “Logman Utility Command-line Reference” on TechNet
• “Tracelog Utility Command-line Reference”article in the MSDN Library
• “Controlling Event Tracing Sessions” article in the MSDN Library
• “Tools for Software Tracing“ article in the MSDN Library

Scenario

A BizTalk solution running in a production environment manifests a noticeable performance degradation. The operations team investigates the problem and arrives to a conclusion that it might be related to a bottleneck in a custom component used by a number of orchestrations in the BizTalk solution. The operations team would like to collect detailed diagnostic information to assert the validity of the above assumption. They want to be able to understand the component behavior as it relates to functions being invoked, measuring accurate durations of the method calls as well as tracing parameter values being used for execution.

Luckily, the BizTalk solution was fully instrumented using the ETW instrumentation framework by following the technical guidance and best practices from the BizTalk Customer Advisory Team (CAT). The operations team has full access and all the required privileges to be able to enable ETW tracing in the production environment. However, they might face with a new challenge.

Challenge

The complexity of the BizTalk solution in question has led to implementing a number of custom components spreading across the end-to-end architecture. Virtually all custom components have been enriched with relevant tracing and instrumentation events. When enabled, the ETW trace log contains thousands of events generated by these custom components. This makes it time consuming to locate the events produced by the suspected component causing a performance degradation. In addition, many events are being written by multiple components running simultaneously on separate threads and this increases the complexity in performing trace log data analysis.

The operations team would like to streamline the troubleshooting process by capturing only those events which are directly related to the component being investigated. Ideally, they want to be able to have these events written into a separate trace log so that it can also be shared with the developers to get their input. The operations staff opens a dialogue with the development team and finds that there is a solution provided by the developers and is already available in the production codebase.

Solution

When implementing core custom components for the BizTalk solution, the development team has made a decision to support the extra level of granularity and add instrumentation at the individual component level. The advantage of this approach is that trace events produced by individual .NET components can be captured in isolation from the others. This approach opens the opportunity for collecting the detailed behavioral and telemetry data related to a specific component whilst ensuring that any events produced by other custom components will not interfere with the trace log content.

The underlying implementation has rather been very straightforward.

First, the TraceManager class from the Microsoft.BizTalk.CAT.BestPractices.Framework.Instrumentation namespace was enriched with a new public static method:

// Returns an instance of a trace provider for the specified type. This requires that the type supplies its Guid 
// which will be used for registering it with the ETW infrastructure.
public static IComponentTraceProvider Create(Type componentType)
{
    GuidAttribute guidAttribute = FrameworkUtility.GetDeclarativeAttribute<GuidAttribute>(componentType);

    if (guidAttribute != default(GuidAttribute))
    {
        return new ComponentTraceProvider(componentType.FullName, new Guid(guidAttribute.Value));
    }
    else
    {
        throw new MissingMemberException(componentType.FullName, typeof(GuidAttribute).FullName);
    }
}

Secondly, every .NET component which required to be individually instrumented was decorated with a Guid attribute available in the System.Runtime.InteropServices namespace.

Next, a new protected readonly static field was added into each non-sealed instrumented .NET component. For sealed classes, the private readonly static modifier was used. This class member is statically initialized with an instance of the framework component implementing the IComponentTraceProvider interface which provides a rich set of tracing and instrumentation methods.

Lastly, all important aspects of the custom components’ behavior were wisely instrumented by calling the relevant tracing methods provided by the above static class member as opposed to using TraceManager.CustomComponent:

Summary

This article was intended to demonstrate the simple and very powerful approach to supporting the  code instrumentation in the custom .NET components to be able to capture trace events for individual components. The benefits of this technique lies in the ability to collect only those events which are necessary to perform troubleshooting of specific components in the BizTalk solutions, eliminate  “noisy events” which may come from other components, reduce the amount of events in the trace log and greatly reduce the time required to analyze the trace log data.

As an administrator you may want to create scripts for specific core tasks. This saves time with duties that are done on a repetitive basis. It also makes the process less error prone since each time you will be running the same script file time through that task.

A commonly scripted task that you may want to manage through cmdlets is the AppFabric configuration process. We will now examine some core configuration operations that you can script for AppFabric to function correctly after the installation.   Before running these cmdlets, the following steps in the installation process must complete in order to properly configure AppFabric using scripting.

• Databases have been prepared using the Initialize-ASMonitoringDatabase and Initialize-ASPersistenceSqlDatabase cmdlets. There may be permissions issues that need to be addressed before this will work that are particular to the security requirements at your installation.
• The users of the Administrator role must have logon as a service privileges because they will be used to run the Event Collection and Workflow Management services.
• The identity of the application pool used to run AppFabric applications must be a member of the logical AppFabric Users role.
• The application pool must be a .NET Framework 4 application pool.

Note that in this article we will not discuss installing, configuring, and securing AppFabric on a domain or across a Web farm.

CMDLET/CMD OPERATION	ACTION
NET LOCALGROUP	Add the AppFabric Administrator user to the local Administrators group. NET LocalGroup is a command line executable. One example is: net localgroup Administrators /Add testuser
Get-WmiObject	Set the Event Collection Service and Workflow Management Service identities. Get-WmiObject is a command line executable.
AppCmd	Add connection strings to the connection strings section in root web.config. AppCmd is a command line executable and used for configuring AppPool Identity with AppCmd. appcmd set config /section:applicationPools /[name='Marketing'].processModel.identityType:SpecificUser /[name='Marketing'].processModel.userName: Marketer /[name='Marketing'].processModel.password: M@dr1d$P
AppCmd	Set the default AppPool identity.
Initialize-ASPersistenceSqlDatabase	Creates and initializes a SQL Server persistence database on the specified database server if the specified database does not already exist on the server. The initialization process creates persistence-related artifacts (tables, views, stored procedures, and so on). If the database already exists on the server, the cmdlet initializes the database with the persistence artifacts. The cmdlet also adds any specified Windows groups to corresponding SQL Server roles.
Initialize-ASMonitoringSqlDatabase	Creates and initializes a SQL Server monitoring database on the specified database server if the specified database does not already exist on the server. The initialization process creates monitoring-related artifacts (tables, views, stored procedures, and so on). If the database already exists on the server, the cmdlet initializes the database with the monitoring artifacts. The cmdlet also adds any specified Windows groups to corresponding SQL Server roles.
Set-ASAppMonitoring	Enables monitoring and specifies the connection string that will be used to connect to the SQL Server monitoring database,
Set-ASAppSqlServicePersistence	Enables SQL Server persistence, specifies the connection string that will be used to connect to the SQL Server persistence database, and sets advanced persistence related parameters at the specified level.

These cmdlets can be run individually, or put inside a script file with a .ps1 extension to run as a Windows PowerShell Script. Here is an example of a script to assist in configuration of AppFabric.  The script has four functions and three sections.  Its main goals of this script are to get the user to enter a user name and password, change the logon user and password for a specified service, generate a SQL connection string to a specific database, and to add that connection string to the root Web.config file. To run these commands, load PowerShell and then load the ApplicationServer module using command (in PowerShell) Import-Module ApplicationServer.

   1: ############################################################

   2: # PowerShell script to configure

   3: # Application Server Extensions for .NET 4

   4: # Copyright (c) Microsoft Corporation.  All rights reserved.

   5: ############################################################

   6: if ((Get-Command -Module ApplicationServer) -eq $null)

   7: {

   8:     Import-Module ApplicationServer # Application Server Extensions for .NET 4

   9: }

  10:  

  11: ############################################################

  12: function GetUserCredential([string]$serviceName, [string]$serviceDescription, [string]$serviceUser)

  13: {

  14:     $credential = $host.ui.PromptForCredential($serviceName, $serviceDescription, $serviceUser, "")

  15:     return $credential # PSCredential object

  16: }

  17:  

  18: function SetServiceCredential([string]$serviceName, $credential)

  19: {

  20:     $domainAndUserName = $credential.GetNetworkCredential().Domain + "\" + $credential.GetNetworkCredential().UserName;

  21:     $service = Get-WmiObject Win32_Service -filter "name='$serviceName'"

  22:     $service.Change($null, $null, $null, $null, $null, $null, $domainAndUserName, $credential.GetNetworkCredential().Password) |out-null

  23:     Restart-Service -name "$serviceName"

  24: }

  25:  

  26: #### Gets a SQL connection string to the specified server and database.

  27: function GetSqlConnectionString([string]$server, [string]$database)

  28: {

  29:     $builder = New-Object System.Data.SqlClient.SqlConnectionStringBuilder

  30:     $builder.PSBase.DataSource = $server

  31:     $builder.PSBase.InitialCatalog = $database

  32:     $builder.PSBase.IntegratedSecurity = $true

  33:     return [string] $builder.ToString()

  34: }

  35:  

  36: #### Adds or updates the specified connection string setting in the specified .NET configuration file.

  37: function UpdateConnectionString([string]$name, [string]$connectionString, [string]$providerName)

  38: {  

  39:     &; $appcmd set config /clr:4 /commit:WEBROOT /section:connectionStrings /+"[connectionString='$connectionString',name='$name',providerName='$providerName']" |out-null

  40: }

  41:  

  42: ##############################

  43: ### User defined variables ###

  44: ### List of variables used in the script. You can modify these variables to customize the script to the specifics of the environment.

  45: ##############################

  46:  

  47: $computer = "."

  48: $appcmd = "$env:SystemRoot\system32\inetsrv\appcmd.exe"

  49:  

  50: ### Event Collector Service (ECS) and Workflow Management Service (WMS) service names.

  51: $ECS_ServiceName = "AppFabricEventCollectionService"

  52: $WMS_ServiceName = "AppFabricWorkflowManagementService"

  53:  

  54: ### Domain user, member of Application Server Administrators group.

  55: ### ECS and WMS will run under this identity.

  56: $systemService_Domain ="corp"

  57: $systemService_UserName = "dubAdmin"

  58:  

  59: ### Monitoring database

  60: $Monitoring_ConnectionStringName = "monitoringDB"

  61: $Monitoring_ConnectionString = GetSqlConnectionString "AD-SQL" "monitoringDB"

  62:  

  63: $Monitoring_MonitoringLevel = "HealthMonitoring" # Used by the monitoring behavior.

  64:  

  65: ### Persistence database

  66: $Persistence_ConnectionStringName = "persistenceDB"

  67: $Persistence_InstanceStoreName = "sqlPersistence"

  68: $Persistence_ConnectionString = GetSqlConnectionString "AD-SQL" "persistenceDB"

  69:  

  70: ###########################

  71: ### Collect credentials ###

  72: ### Calls the GetUserCredential function to obtain credentials from the user for use in the configuration of the system services (Workflow Management service and Event Collection service).

  73: ###########################

  74:  

  75: $systemService_Credentials = GetUserCredential "System Services" "Provide the credentials for the System Services user:" "$systemService_Domain\$systemService_UserName"

  76: $systemService_Domain = $systemService_Credentials.GetNetworkCredential().Domain

  77: $systemService_UserName = $systemService_Credentials.GetNetworkCredential().UserName

  78:  

  79: ############################

  80: ### Update configuration ###

  81: ### Apply the configuration based on the variables and the collected information

  82: ############################

  83:  

  84: ####Add the AppFabric Administrator user to the local Administrators group.

  85: Write-Output "Adding the Administrator user to the local Administrators group..."

  86: $oGroup = [ADSI]"WinNT://$computer/Administrators"

  87: trap { continue } #'Administrator user already a member of the local Administrators group...'; 

  88: & { $oGroup.Add("WinNT://$systemService_Domain/$systemService_UserName") }

  89:  

  90: ####Set Event Collection service configuration.

  91: Write-Output "Updating Event Collection service..."

  92: SetServiceCredential $ECS_ServiceName $systemService_Credentials

  93:  

  94: ####Set Workflow Management service configuration.

  95: Write-Output "Updating Workflow Management service..."

  96: SetServiceCredential $WMS_ServiceName $systemService_Credentials

  97:  

  98: ####Add connection strings to the connection strings section in root web.config.Write-Output "Creating connection strings..."

  99: UpdateConnectionString $Monitoring_ConnectionStringName $Monitoring_ConnectionString "System.Data.SqlClient" |out-null

 100: UpdateConnectionString $Persistence_ConnectionStringName $Persistence_ConnectionString "System.Data.SqlClient" |out-null

 101:  

 102: ####Create an Instance Store entry for the persistence connection string.

 103: Write-Output "Creating Persistence Instance Store..."

 104: Add-ASAppSqlInstanceStore -Name $Persistence_InstanceStoreName -ConnectionStringName $Persistence_ConnectionStringName -Root |out-null

 105:  

 106: ####Set the persistence and monitoring behavior.

 107: Write-Output "Creating default behaviors..." 

 108: Set-ASAppMonitoring -ConnectionStringName $Monitoring_ConnectionStringName -MonitoringLevel $Monitoring_MonitoringLevel -Root |out-null

 109: Set-ASAppSqlServicePersistence -ConnectionStringName $Persistence_ConnectionStringName -Root -HostLockRenewalPeriod "00:00:20" -InstanceEncodingOption "GZip" -InstanceCompletionAction "DeleteNothing" -InstanceLockedExceptionAction "BasicRetry" |out-null

Problem: While loading a new Designer view in the rehosted Workflow Designer, the method workflowDesigner.Load (new Sequence ()) by default loads an empty Sequence Activity in the Activity Builder designer grid (Figure 1).  In some ISV scenarios, it may be preferred to load a ‘blank’ Activity Builder Designer Grid, so that the ITPro user is not restricted to using the Sequence Activity. Changing the parameters to (null) in the method (as workflowDesigner.Load (null)); does not help as the Workflow Designer Activity Builder is not loaded.

Figure 1: Rehosted Designer while using Method workflowDesigner.Load (new Sequence ())

Solution: To mimic the Visual Studio Workflow Designer experience use the ActivityBuilder Method as: LoadWorkflowDesigner(new System.Activities.ActivityBuilder()).

Figure 2: Rehosted Designer ActivityBuilder Method as: LoadWorkflowDesigner(new System.Activities.ActivityBuilder())

Background

Code instrumentation has always been crucially important in complex software solutions. Rich tracing and logging enables collecting detailed telemetry and diagnostic information to understand application behavior and facilitate troubleshooting. This is especially important for server applications and BizTalk Server based-applications are surely among them.

We have seen many instances where BizTalk developers often add tracing and logging capabilities to their applications using inefficient methods and this can have adverse effect on the application performance. From the other side, BizTalk developers appear to be unaware of high-performance tracing capabilities which are provided directly by BizTalk. This article will help bridge the gap.

Challenges

The ISV partner has initially implemented component-level tracing by taking dependency on the System.Diagnostics.Trace component in the .NET Framework. They also considered using a third-party logging/tracing component log4net. Both options didn’t deliver the desired level of agility and performance.

The main observations can be summarized as follows:

• The CPU utilization was 85-90% on average when developers were running the DebugView tool to monitor the BizTalk application behavior. The tool was consuming the vast majority of the CPU time due to many events being logged by the BizTalk application.
   
• Running DebugView during stress testing was significantly impacting application performance. DebugView acts as a debugger and hooks on the OUTPUT_DEBUG_STRING_EVENT event. As a result, the BizTalk application was experiencing a performance impact as the application threads will be suspended whilst the debug information is written to the trace listener.
• Poor tracing performance. The .NET Framework’s standard System.Diagnostics.Trace component which is often being used for instrumenting .NET applications was able to handle anywhere between 2000 and 8500 trace events per second when event capture was enabled in the DebugView tool.
   
• Any changes in the tracing configuration such as enabling, disabling or changing the trace level required a restart of the BizTalk host instances. Although this doesn’t impose any critical issues in the development or test environments, the need for application restart may be highly undesirable or even unacceptable in the production environment.

Solution

Our proposed implementation leveraged the ETW infrastructure which enabled us to take advantage of the general purpose, high-speed tracing facility provided by the OS kernel. It turned out that we didn’t have to write our own custom providers. During the analysis phase, we discovered what can be generally described as a hidden gem inside the BizTalk runtime engine called TraceProvider from the Microsoft.BizTalk.Diagnostics namespace in Microsoft.BizTalk.Tracing.dll. As things stand at the moment, this class has not as yet been fully documented despite that it is being extensively used virtually by all major infrastructure components inside the BizTalk runtime, including EPM, transport adapters, Message Agent, etc.

The Microsoft.BizTalk.Diagnostics.TraceManager class implements an ETW provider and exposes the following key properties and methods which have been leveraged in our prototype:

namespace Microsoft.BizTalk.Diagnostics
{
    [ComVisible(true)]
    [Guid("748004CA-4959-409a-887C-6546438CF48E")]
    public sealed class TraceProvider
    {
        public TraceProvider(string applicationName, Guid controlGuid);

        public uint Flags { get; }
        public bool IsEnabled { get; }

        public void TraceMessage(uint traceFlags, object format);
        public void TraceMessage(uint traceFlags, object format, object data1);
        // + a few more overloads of TraceMessage accepting extra data items.
    }
}

Below is the class diagram for the tracing framework:

The key player in the tracing framework is the ComponentTraceProvider class which is simply acting as a wrapper for TraceProvider and provides a richer choice of tracing methods for various types of events such as informational, warnings, errors and exceptions. We also added the support for tracing method calls, measuring durations using high-resolution timers and ability to correlate events using Guid-based tokens.

The ComponentTraceProvider class implements the following methods:

// Writes an information message to the trace.
void TraceInfo(string format, params object[] parameters);

// Writes an information message to the trace. This method is provided for optimal performance when
// tracing simple messages which don't require a format string.
void TraceInfo(string message);

// Writes an information message to the trace. This method is intended to be used when the data that needs to be
// written to the trace is expensive to be fetched. The method represented by the Func<T> delegate will only be invoked if
// tracing is enabled.
void TraceInfo(Func<string> expensiveDataProvider);

// Writes a warning message to the trace.
void TraceWarning(string format, params object[] parameters);

// Writes a warning message to the trace. This method is provided for optimal performance when
// tracing simple messages which don't require a format string.
void TraceWarning(string message);

// Writes a message to the trace. This method can be used to trace detailed information
// which is only required in particular cases.
void TraceDetails(string format, params object[] parameters);

// Writes an error message to the trace.
void TraceError(string format, params object[] parameters);

// Writes an error message to the trace. This method is provided for optimal performance when
// tracing simple messages which don't require a format string.
void TraceError(string message);

// Writes the exception details to the trace.
void TraceError(Exception ex);

// Writes the exception details to the trace.
void TraceError(Exception ex, bool includeStackTrace);

// Writes the exception details to the trace.
void TraceError(Exception ex, bool includeStackTrace, Guid callToken);

// Writes an informational event into the trace log indicating that a method is invoked. This can be useful for tracing method calls to help analyze the 
// code execution flow. The method will also write the same event into default System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceIn method would typically be at the very beginning of an instrumented code.
Guid TraceIn(params object[] inParameters);

// Writes an informational event into the trace log indicating that a method is invoked. This can be useful for tracing method calls to help analyze the 
// code execution flow. The method will also write the same event into default System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceIn method would typically be at the very beginning of an instrumented code.
// This method is provided to ensure optimal performance when no parameters are required to be traced.
Guid TraceIn();

// Writes an informational event into the trace log indicating that a method is about to complete. This can be useful for tracing method calls to help analyze the 
// code execution flow. The method will also write the same event into default System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceOut method would typically be at the very end of an instrumented code, before the code returns its result (if any).
void TraceOut(params object[] outParameters);

// Writes an informational event into the trace log indicating that a method is about to complete. This can be useful for tracing method calls to help analyze the 
// code execution flow. The method will also write the same event into default System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceOut method would typically be at the very end of an instrumented code, before the code returns its result (if any).
// This method is provided to ensure optimal performance when no parameters are required to be traced.
void TraceOut();

// Writes an informational event into the trace log indicating that a method is about to complete. This can be useful for tracing method calls to help analyze the 
// code execution flow. The method will also write the same event into default System.Diagnostics trace listener, however this will only occur in the DEBUG code.
// A call to the TraceOut method would typically be at the very end of an instrumented code, before the code returns its result (if any).
void TraceOut(Guid callToken, params object[] outParameters);

// Writes an informational event into the trace log indicating a start of a scope for which duration will be measured.
long TraceStartScope(string scope, params object[] parameters);

// Writes an informational event into the trace log indicating the start of a scope for which duration will be measured.
// This method is provided in order to ensure optimal performance when no parameters are available for tracing.
long TraceStartScope(string scope);

// Writes an informational event into the trace log indicating the start of a scope for which duration will be measured.
// This method is provided in order to ensure optimal performance when only 1 parameter of type Guid is available for tracing.
long TraceStartScope(string scope, Guid callToken);

// Writes an informational event into the trace log indicating the end of a scope for which duration will be measured.
void TraceEndScope(string scope, long startTicks);

// Writes an informational event into the trace log indicating the end of a scope for which duration will be measured.
void TraceEndScope(string scope, long startTicks, Guid callToken);

We implemented the ability to support instrumentation by a component type so that tracing can be enabled and disabled separately for custom pipeline components, orchestrations, maps, rules and so on. The TraceManager class exposes a static instance of ComponentTraceProvider for each individual type of components as follows:

/// The main tracing component which is intended to be invoked from user code.
public static class TraceManager
{
    #region Private members
    private static IComponentTraceProvider pipelineComponentTracer = new ComponentTraceProvider("PipelineComponent", new Guid("691CB4CB-D20C-408e-8CFF-FD8A01CD2F75"));
    private static IComponentTraceProvider workflowComponentTracer = new ComponentTraceProvider("WorkflowComponent", new Guid("D2316AFB-414B-42e4-BB7F-3AA92B96A98A"));
    private static IComponentTraceProvider dataAccessComponentTracer = new ComponentTraceProvider("DataAccessComponent", new Guid("2E5D65D8-71F9-43e9-B477-733EF6212895"));
    private static IComponentTraceProvider transformComponentTracer = new ComponentTraceProvider("TransformComponent", new Guid("226445A8-5AF3-4dbe-86D2-73E9B965378E"));
    private static IComponentTraceProvider serviceComponentTracer = new ComponentTraceProvider("ServiceComponent", new Guid("E67E8346-90F1-408b-AF40-222B6E3C5ED6"));
    private static IComponentTraceProvider customComponentTracer = new ComponentTraceProvider("CustomComponent", new Guid("6A223DEA-F806-4523-BAD0-312DCC4F63F9"));
    private static IComponentTraceProvider rulesComponentTracer = new ComponentTraceProvider("RulesComponent", new Guid("78E2D466-590F-4991-9287-3F00BA62793D"));
    private static IComponentTraceProvider trackingComponentTracer = new ComponentTraceProvider("TrackingComponent", new Guid("5CBD8BA0-60F8-401b-8FF5-C7F3D5FABE41"));
    #endregion

    #region Public properties
    // The trace provider for user code in the custom pipeline components.
    public static IComponentTraceProvider PipelineComponent
    {
        get { return pipelineComponentTracer; }
    }

    // The trace provider for user code in workflows (such as expression shapes in the BizTalk orchestrations).
    public static IComponentTraceProvider WorkflowComponent
    {
        get { return workflowComponentTracer; }
    }

    // The trace provider for user code in the custom components responsible for data access operations.
    public static IComponentTraceProvider DataAccessComponent
    {
        get { return dataAccessComponentTracer; }
    }

    // The trace provider for user code in the transformation components (such as scripting functoids in the BizTalk maps).
    public static IComponentTraceProvider TransformComponent
    {
        get { return transformComponentTracer; }
    }

    // The trace provider for user code in the service components (such as Web Service, WCF Service or service proxy components).
    public static IComponentTraceProvider ServiceComponent
    {
        get { return serviceComponentTracer; }
    }

    // The trace provider for user code in the Business Rules components (such as custom fact retrievers, policy executors).
    public static IComponentTraceProvider RulesComponent
    {
        get { return rulesComponentTracer; }
    }

    // The trace provider for user code in the business activity tracking components (such as BAM activities).
    public static IComponentTraceProvider TrackingComponent
    {
        get { return trackingComponentTracer; }
    }

    // The trace provider for user code in any other custom components which don't fall into any of the standard categories such as Pipeline, Workflow, DataAccess, Transform, Service or Rules.
    public static IComponentTraceProvider CustomComponent
    {
        get { return customComponentTracer; }
    }
    #endregion
}

The new tracing framework has delivered the following benefits:

• Highest possible performance – our tests on a 3.2GHz Quad Core machine have proven that ETW-based tracing component is able to write about 1.4 million trace events per second;
• No high CPU utilization observed whenever trace events are persisted into a log file. ETW is using a buffering and logging mechanism implemented in the kernel. The logging mechanism uses per-processor buffers that are written to disk by an asynchronous writer thread. This significantly reduces the impact of the log write operations on the application and system performance.
• Full operational flexibility enables switching the tracing on and off as well as changing the trace level dynamically, making it easy to perform detailed tracing in production environments without requiring reboots or application restarts.
• Tracing can be safely enabled to run continuously in a production environment using the circular logging option which ensures that log files will not outgrow the available disk space.

Performance Considerations

When compared to the other 3 popular rivals (System.Diagnostic.Trace in the .NET Framework, log4net and Enterprise Library), the ETW-based implementation has demonstrated a significantly better performance. In order to measure the true extent of performance benefits delivered by the ETW-based component, we have created a fairly simple test scenario.

In our test case, we executed 1 instrumented method 100,000 times writing 1 trace event in each iteration. To make it fair for everyone, we configured all 4 benchmarked instrumentation frameworks to produce a text-based trace log file at the end of the test run.

Below are the results from our benchmarking exercise:

The code samples include a test tool which was used for benchmarking all 4 instrumentation frameworks.

Usage Patterns

The examples below illustrate the usage pattern for the tracing framework. This code fragment demonstrates the use of the TraceManager component for capturing 2 events indicating when method was invoked (TraceIn) and when it was completed (TraceOut).

public void BeginAndCompleteActivity(ActivityBase activity)
{
    Guard.ArgumentNotNull(activity, "activity");

    var callToken = TraceManager.TrackingComponent.TraceIn(activity.ActivityName, activity.ActivityID);

    this.eventStream.BeginActivity(activity.ActivityName, activity.ActivityID);
    this.eventStream.UpdateActivity(activity.ActivityName, activity.ActivityID, ActivityTrackingUtility.GetActivityData(activity));
    this.eventStream.EndActivity(activity.ActivityName, activity.ActivityID);

    TraceManager.TrackingComponent.TraceOut(callToken);
}

When enabled, the trace log will contain both events written with a timestamp along with a call correlation token (callToken) which enables to match both TraceIn and TraceOut events together:

The following sections drill down into specific instrumentation scenarios across the entire family of artifacts and building blocks in a BizTalk solution.

Instrumentation of Custom Pipeline Components

The custom pipeline components can be instrumented using TraceManager.PipelineComponent which is dedicated for this type of BizTalk artifacts. Some of the useful events to be captured through the code instrumentation may include:

• Tracing calls to the core methods such as Execute, Disassemble, GetNext, etc. (using TraceIn and TraceOut);
• Measuring duration of the above methods (using TraceStartScope and TraceEndScope);
• Tracing the internal state of pipeline components which could in turn assist with troubleshooting (using TraceInfo);
• Writing detailed information about a runtime exception (using TraceError).

Below is an example of some of the techniques highlighted above:

Instrumentation of BizTalk Maps

The BizTalk maps can be instrumented using TransformTraceManager component which can be invoked from inside a custom scripting functoid associated with an external assembly. Some useful events that can be captured from the maps when they are being executed by the BizTalk runtime engine may include:

• Tracing the invocation of XSLT templates implementing the BizTalk maps (using TraceIn and TraceOut);
• Tracing the internal state of maps, e.g. the node values which are being transformed, which could in turn assist with troubleshooting (using TraceInfo).

In the example below, the instrumented map is using the TraceInfo method to report the values of the GSxx nodes in the input document:

Note that BizTalk maps cannot invoke static methods from an external assembly by design. Consequently, there is a separate non-static class (TransformTraceManager) that is responsible for relaying method calls between a map and the statically initialized singleton TraceManager.TransformComponent object.

When adding instrumentation into BizTalk maps, it’s important to have the custom scripting functoids linked to appropriate nodes in the output document, otherwise the scripting code will be omitted from the auto-generated XSLT template. In the above example, all scripting functoids are connected to the root node in the output document via a helper Logical OR functoid. This ensures that tracing takes place right before creating the root XML element and results in generating the following XSLT code by the BizTalk mapper:

<!-- There is some XSLT code before this element -->
<xsl:template match="/s2:CSC_837D">
    <xsl:variable name="var:v1" select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS01/text()))" />
    <xsl:variable name="var:v2" select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS02/text()))" />
    <xsl:variable name="var:v3" select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS03/text()))" />
    <xsl:variable name="var:v4" select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS04/text()))" />
    <xsl:variable name="var:v5" select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS05/text()))" />
    <xsl:variable name="var:v6" select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS06/text()))" />
    <xsl:variable name="var:v7" select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0:GS07/text()))" />
    <xsl:variable name="var:v8" select="ScriptNS0:TraceInfo(string(s0:Headers/s0:GS/s0: GS08/text()))" />
    <xsl:variable name="var:v9" select="userCSharp:LogicalOr(string($var:v1), ’true’, string($var:v2), string($var:v3), string($var:v4), string($var:v5), string($var:v6), string($var:v7), string($var:v8))" />
    <xsl:if test="$var:v9">
      <ns0:W4880200-COMMON-EDIT-MED-CLAIM>
      <!-- There is plenty of other XSLT code after this element -->

Instrumentation of BizTalk Orchestrations

As BizTalk orchestrations become increasingly complex, the instrumentation is becoming a key factor in helping to diagnose and troubleshoot behavioral problems, performance-related issues and other bottlenecks that were not foreseen or manifested themselves during development.

A well instrumented orchestration can be described as follows:

• The entry point into an orchestration is recorded (using TraceIn) immediately or at least at the first available opportunity (e.g. after the very first Receive shape);
• The internal state of orchestrations (e.g. variables, results from method calls, non-sensitive message payload) is wisely traced (using TraceInfo);
• Unexpected behavior is reported as soon as it is detected (using TraceWarning or TraceError);
• The detailed information about a runtime exception is traced from inside the exception handling block (using TraceError);
• The duration of individual scopes or/and entire orchestration is measured and traced (using TraceStartScope and TraceEndScope);
• The exit point from an orchestration is recorded (using TraceOut) either right before the Terminate shape or at the very last step in the orchestration.

Below are some example of the instrumentation techniques highlighted above:

Instrumentation of Business Rules

The instrumentation of business rules can be extremely valuable when there is a requirement to understand how rule conditions are being evaluated, what rules are being fired as well as how long it takes to execute a rule set (policy).

A well instrumented business rule policy can be defined as follows:

• All calls to a helper class invoking the BRE APIs (such as Policy objects) are traced (using TraceIn and TraceOut);
• The duration of policy execution is recorded (using TraceStartScope and TraceEndScope);
• The BRE policies are invoked passing an instance of a class implementing the IRuleSetTrackingInterceptor interface - this class should report on all significant events which occur during policy execution;
• Each rule in the policy informs the instrumentation framework that the rules was fired by the BRE engine (using either TraceInfo, TraceWarning or TraceError).

The business rules instrumentation is powered by 3 components in our framework:

• TraceManager.RulesComponent provides general instrumentation methods for all helper .NET components accessing the BRE APIs;
• RuleTraceManager residing in the Microsoft.BizTalk.CAT.BestPractices.Framework.RulesEngine namespace implements a static class which is intended to be used on the Action pane inside the Business Rule Composer. Its main purpose is to expose a set of tracing methods with a fixed parameter list. Internally, the RuleTraceManager is simply proxying all calls to TraceManager.RulesComponent. A requirement for a separate class has arisen from the fact that the Business Rules Engine doesn’t currently support the invocation of methods with a variable number of parameters (also known as parameter arrays);
• TracingRuleTrackingInterceptor which is also available in the Microsoft.BizTalk.CAT.BestPractices.Framework.RulesEngine namespace provides a custom implementation of the IRuleSetTrackingInterceptor interface. All events which occur during policy execution will be traced using TraceManager.RulesComponent.

The following recommendations would apply in the context of business rules instrumentation:

• When calling any static methods from inside the business rules, make sure that StaticSupport parameter is present either in the application’s configuration file (e.g. BTSNTSvc.exe.config)  or system registry. Otherwise, the following behavior may be encountered;
• Consider adding a TraceInfo instrumentation event into each individual rule to facilitate the analysis of rule execution without having to analyze all events reported by the Rules Engine through TracingRuleTrackingInterceptor;
• In addition to TraceInfo, TraceWarning and TraceError can be wisely applied at individual rule level in order to report on unexpected fact values or rules that are not expected to have been executed for a given set of facts.

Below are some example of the approaches highlighted above:

Instrumentation of Custom Components

The custom .NET components can be instrumented by invoking the trace methods provided by TraceManager.CustomComponent which enables:

• Method-level instrumentation whereby calls to individual methods are recorded in the trace log with zero or more optional parameters (using TraceIn and TraceOut);
• Measuring duration to be able to capture accurate execution timings (using TraceStartScope and TraceEndScope);
• Tracing the internal state of custom components which could in turn assist with troubleshooting (using TraceInfo or TraceWarning);
• Writing detailed information about a runtime exception (using TraceError).

Below is an example of some of the techniques highlighted above:

 
Show me the code! You can find the source code in the attachment below. Please share your feedback to help improve the instrumentation samples.

Best Practices for Instrumenting High Performance BizTalk Solutions.zip 

For more information on the related topic, please visit the following resources:

• “Improve Debugging And Performance Tuning With ETW” article in the MSDN Magazine
• “Intro to Event Tracing For Windows” post on Matt Pietrek’s blog

The ability to report Workflow Process tracking status to end users is a very important feature of any Business Process Management (BPM) product, especially ones that run tens of thousands of concurrent processes. Many ISV partners have created their own tracking service, at great cost, that tracks metadata about the Workflow Process and each activity within it. This data is then presented via an UI status dialog for a selected process instance. A typical use of the Workflow Process tracking information is presented in the graphic below.

A key feature of this tracking data is that it also includes some elements of the business data relating to the Workflow Instance. This allows ISV and Enterprise developers to provide some business contextual information along with the Workflow Process metadata. As you can see in the above status dialog the UI displays the client name and code as well as the user who initiated the process/Workflow and the users associated with completed manual tasks.

This blog post provides guidance on using Promoted Properties to generate reports that are of interest to business analysts and ITAdmins. The Promoted Properties (new in .NET4) are extensions supported by .NET4 Workflow Services hosted within AppFabric.

Scenario

Consider the typical scenario where an ISV or Enterprise IT Group develops or extends a dashboard to display the metrics around active business processes which are manifested as Workflow Services executing as an AppFabric application. In our scenario, the Workflow Service is implemented as an application that handles document format (e.g., *.docx to *.pdf) conversion. While the IT Group is interested in monitoring bottlenecks based off average processing time of the processes; the Business Analysts could be interested in the types and counts thereof. The graphic below is a typical representation of the data that addresses the needs of this scenario.  

While data relating to metrics is super useful from the Business and ITAdmin perspective, it’s not available via the tracked events emitted from Workflow Services. The challenge here is to generate data relating to the state of each running Workflow Instance, capture it in the AppFabric persistence database and generate a report of it. The next few sections of this blog post will discuss a few options to capture this data and describe one preferred approach in detail. 

Traditional Methods

The traditional methods to capture and report the state and other such data relating to the state of the Workflow Instances areeither Custom Tracking Records or Extracted Variables.

In the Custom Tracking Record approach, the ‘interesting’ data/record is written to the monitoring database when a Workflow Activity executes and consequently emits the custom record. In the Extracted Variable approach, the value is captured in the Monitoring database when a Workflow Activity enters a certain pre-configured state. While both are valid options, they have a common limitation – the values are emitted as part of the Workflow Service Monitoring infrastructure and therefore can be out of sync with the persisted state of a Workflow Service instance. To elaborate this limitation, consider the use case wherein after executing the Workflow Activity, the data is emitted (as a custom tracking record or triggered variable extraction). Next, the in-flight Workflow Service Instance is abandoned or aborted due to an exception which in turn causes a rollback (of the Workflow Service Instance) to the previous persistence point. Meanwhile, the state and other data relating to the ‘rolled back’ Workflow Service Instance are already written to the Monitoring database and are out of synch with the actual state of the Workflow Service Instance. Solutions using the monitoring store for state information relating to the Workflow Service Instance and will not yield accurate reports – a limitation in .NET3.5 version.

New in .NET4/Windows Server AppFabric: Promoted Properties

Promoted Properties eliminates the consistency and sync issues we encounter in the traditional approaches, because the data is written or updated directly in the persistence database when the state of the Workflow Service Instance is updated. Promoted Properties are a set of properties that are maintained at the persistence points of a Workflow Service. Via this feature- Promoted Properties, as the name implies, the Workflow Service Host can “promote” a set of data - up to 64 variables, that is of interest to the scenario. The values for these Promoted Properties are written to the System.Activities.DurableInstancing.InstancePromotedPropertiesTable and can be queried from the InstancePromotedProperties view in the AppFabric Persistence Database.

The graphic below is an example of the query results from the InstancePromotedProperties view:

The Promoted Properties structure is not normalized and has columns: Value1 through Value64, corresponding to the InstanceId and the PromotionName that labels the set of properties. No column names are stored for these values, so it is up to the consuming/reporting application to correctly interpret these columns. Furthermore, there is only one row of Promoted Properties for each Workflow Service Instance. This means that the values are inserted once and thereafter updated for every ‘persistence’ of the associated Workflow Service Instance - this is a very important point to consider while designing the reporting application, especially if data relating to previous states is important. One shortcoming of the Promoted Properties feature is that they have no representation in the extensions to the IIS Manager User Interface that Windows Server AppFabric provides – a custom application is the only way to leverage this very useful feature.

Final consideration is that the Promoted Properties are a feature of the SQL Workflow Instance Store, and so may not be available if the AppFabric installation is using a custom provider for Persistence.

Using Promoted Properties

Using Promoted Properties is quite straightforward and this section elaborates on using this very interesting feature. 

Configuring the SqlWorkflowInstanceStoreBehavior

The SqlWorkflowInstanceStoreBehavior defines a Promote method through which you define the schema of the values you wish to track. This behavior must be configured when a Workflow Service Host is being loaded for the Service (e.g., before it is opened). In order to accomplish this, you need to define a custom behavior whose sole purpose is to register your Promoted Properties. The following class shows an example of such a behavior that can be modified for the specific projects needs:

   1: public class PromotePropertiesBehavior : IServiceBehavior

   2: {

   3:     XNamespace xNS = XNamespace.Get("https://samples.com/CustomProperties");

   4: 

   5:     public virtual void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters)

   6:     {

   7:     }

   8:  

   9:     public virtual void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)

  10:     {

  11:         WorkflowServiceHost host = serviceHostBase as WorkflowServiceHost;

  12:         if (host != null)

  13:         {

  14:             SqlWorkflowInstanceStoreBehavior store = host.Description.Behaviors[typeof(SqlWorkflowInstanceStoreBehavior)] as SqlWorkflowInstanceStoreBehavior;

  15: 

  16:             if (store != null)

  17:             {

  18:                 List<;XName> variantProperties = new List<XName>()

  19:                 {   

  20:                     xNS.GetName("ConvertFrom"),

  21:                     xNS.GetName("ConvertTo"),

  22:                     xNS.GetName("StartOrComplete"),

  23:                     xNS.GetName("LastPropUpdateDateTime"),

  24:                     xNS.GetName("Duration")

  25:                 };

  26: 

  27:                 store.Promote("ConversionStats", variantProperties, null);

  28:  

  29:                 host.WorkflowExtensions.Add<;CustomPropertiesParticipant>(() => new CustomPropertiesParticipant());

  30:             }

  31:         }

  32:     }

  33:     

  34:     public virtual void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)

  35:     {

  36:     }

  37:  

  38: }

In order to register this behavior with a Service hosted in AppFabric, you will want to be able to declare it in the web.config by defining the BehaviorExtensionElement. The following code snippet shows how you can do this for the previously supplied Service behavior:

   1: public class PromotePropertiesBehaviorElement : BehaviorExtensionElement

   2: {

   3:     private ConfigurationPropertyCollection properties;

   4: 

   5:     protected override object CreateBehavior()

   6:     {

   7:         PromotePropertiesBehavior behavior = new PromotePropertiesBehavior();

   8:         return behavior;

   9:     }

  10:  

  11:     public override Type BehaviorType

  12:     {

  13:         get

  14:         {

  15:             return typeof(PromotePropertiesBehavior);

  16:         }

  17:     }

  18:  

  19:     protected override ConfigurationPropertyCollection Properties

  20:     {

  21:         get

  22:         {

  23:             if (this.properties == null)

  24:             {

  25:                 ConfigurationPropertyCollection props = new ConfigurationPropertyCollection();

  26:                 this.properties = props;

  27:             }

  28:             return this.properties;

  29:         }

  30:     }

  31: }

  32:  

Define Persistence Participant

To ensure Promoted Properties are updated when the state of the Workflow Service Instance is updated in the Persistence Store, we need to define a PersistenceParicipant that can correctly map values supplied from a Workflow Service Instance to the schema defined in the Service behavior. The following class shows how we might do this for this blog’s scenario. Of course, this needs to be appropriately modified to be used in your scenario.

   1: public class CustomPropertiesParticipant: PersistenceParticipant

   2: {

   3:     XNamespace xNS = XNamespace.Get("https://samples.com/CustomProperties");

   4: 

   5:     public string ConvertFrom;

   6:     public string ConvertTo;

   7:     public string StartOrComplete;

   8:     public DateTime LastPropUpdateDateTime;

   9:     public double Duration;

  10:  

  11:     protected override void CollectValues(out IDictionary<;System.Xml.Linq.XName, object> readWriteValues, out IDictionary<System.Xml.Linq.XName, object> writeOnlyValues)

  12:     {

  13:         base.CollectValues(out readWriteValues, out writeOnlyValues);

  14: 

  15:         readWriteValues = new Dictionary<XName, object>();

  16:         readWriteValues.Add(xNS.GetName("ConvertFrom"), this.ConvertFrom);

  17:         readWriteValues.Add(xNS.GetName("ConvertTo"), this.ConvertTo);

  18:         readWriteValues.Add(xNS.GetName("StartOrComplete"), this.StartOrComplete);

  19:         readWriteValues.Add(xNS.GetName("LastPropUpdateDateTime"), this.LastPropUpdateDateTime);

  20:         readWriteValues.Add(xNS.GetName("Duration"), this.Duration);

  21: 

  22:         writeOnlyValues = null;

  23:     

  24:     }

  25:  

  26:     protected override void PublishValues(IDictionary<;System.Xml.Linq.XName, object> readWriteValues)

  27:     {

  28:         base.PublishValues(readWriteValues);

  29: 

  30:         try

  31:         {

  32:  

  33:             this.ConvertFrom = readWriteValues[xNS.GetName("ConvertFrom")] as string;

  34:             this.ConvertTo = readWriteValues[xNS.GetName("ConvertTo")] as string;

  35:             this.StartOrComplete = readWriteValues[xNS.GetName("StartOrComplete")] as string;

  36: 

  37:             if (readWriteValues[xNS.GetName("LastPropUpdateDateTime")] != null)

  38:             {

  39:                 this.LastPropUpdateDateTime = (DateTime)readWriteValues[xNS.GetName("LastPropUpdateDateTime")];

  40:             }

  41:  

  42:             if (readWriteValues[xNS.GetName("Duration")] != null)

  43:             {

  44:                 this.Duration = (double)readWriteValues[xNS.GetName("Duration")];

  45:             }

  46:  

  47:         }

  48:         catch (Exception ex)

  49:         {

  50:  

  51:         }

  52:  

  53:     }

  54:  

  55:     protected override IDictionary<;System.Xml.Linq.XName, object> MapValues(IDictionary<System.Xml.Linq.XName, object> readWriteValues, IDictionary<System.Xml.Linq.XName, object> writeOnlyValues)

  56:     {

  57:         return base.MapValues(readWriteValues, writeOnlyValues);

  58:     }

  59:  

  60: }

  61:  

In the above code snippet, the CollectValues method is what maps the values supplied from the Workflow Service Instance to the schema when the Workflow is persisted. The PublishValues method loads the values read from Persistence and makes them available to your instance for reading and writing.

Web.config for the Scenario

With the appropriate classes in place, you finally need to add the behavior to the web.config of the Service. Below is the complete web.config for the scenario.

   1: <?xml version="1.0" encoding="utf-8"?>

   2: <configuration>

   3:   <system.web>

   4:     <compilation debug="true" targetFramework="4.0" />

   5:   </system.web>

   6:   <system.serviceModel>

   7:     <behaviors>

   8:       <serviceBehaviors>

   9:         <behavior>

  10:           <serviceMetadata httpGetEnabled="true" />

  11:           <serviceDebug includeExceptionDetailInFaults="true" />

  12:           <promoteProperties />

  13:         </behavior>

  14:       </serviceBehaviors>

  15:     </behaviors>

  16:     <extensions>

  17:       <behaviorExtensions>

  18:         <add name="promoteProperties" type="ServiceExtensions.PromotePropertiesBehaviorElement, ServiceExtensions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" />

  19:       </behaviorExtensions>

  20:     </extensions>

  21:   </system.serviceModel>

  22:   <system.webServer>

  23:     <modules runAllManagedModulesForAllRequests="true" />

  24:   </system.webServer>

  25: </configuration>

MWA Schema to include promotedProperties Element

First step is to register the behavior extension which adds the type of the Custom Behavior Element and provides a name for how the element will appear in the config file. Second, add the Custom Behavior to the Service’s collection of Service behaviors. Both are included in the config file above.

While we have defined a custom schema element in the config file, the Microsoft.Web.Administration (MWA) classes that AppFabric uses to read web.config will not recognize the Custom Service Behavior tag (e.g., in this case they will not recognize the promoteProperties element). This will result in an error on the IIS Manager dashboard. This is resolved by updating the MWA the schema to include the custom element. The updated schema is loaded in folder <drive>:\windows\system32\inetsrv\config\schema.

This schema (ServiceExtensions_Schema.xml) includes the <promotedProperties> element within the serviceBehaviors element in system.serviceModel/behavior node. The appropriate snippet of the schema is provided below.

   1: <;configSchema>

   2:   <;sectionSchema name="system.serviceModel/behaviors">

   3:     <;element name="serviceBehaviors">

   4:       <;collection addElement="behavior" removeElement="remove" clearElement="clear" allowDuplicates="true">

   5:         <;element name="promoteProperties"/>

   6:       <;/collection>

   7:     <;/element>

   8:   <;/sectionSchema>

   9: <;/configSchema>

Using PromotedProperties in Workflow Services

With the infrastructure in place, the Workflow Service Instance is ready to generate the Promoted Properties. Technically, the Workflow Activities are responsible for updating the values as a part of their execution. For convenience, below is a code snippet of a Custom Workflow Activity whose sole purpose is to update these Promoted Property values when it executes.

   1: public sealed class CaptureAnalyticProperty : CodeActivity

   2: {

   3:     public InArgument<;string> ConvertFrom { get; set; }

   4:     public InArgument<;string> ConvertTo { get; set; }

   5:     public InArgument<;string> StartOrComplete { get; set; }

   6:       

   7:     protected override void Execute(CodeActivityContext context)

   8:     {

   9:         string valConvertFrom = context.GetValue(this.ConvertFrom);

  10:         string valConvertTo = context.GetValue(this.ConvertTo);

  11:         string valStartOrComplete = context.GetValue(this.StartOrComplete);

  12: 

  13:         try

  14:         {

  15:             context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().ConvertFrom = valConvertFrom;

  16:             context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().ConvertTo = valConvertTo;

  17:             context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().StartOrComplete = valStartOrComplete;

  18:  

  19:             DateTime currentTime = DateTime.UtcNow;

  20: 

  21:             if (valStartOrComplete.ToLower().Equals("complete"))

  22:             {

  23:                 context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().Duration =

  24:                     (currentTime - context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().LastPropUpdateDateTime).TotalSeconds;

  25:             }

  26:  

  27:             context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().LastPropUpdateDateTime = currentTime;

  28:         }

  29:         catch (Exception ex)

  30:         {

  31:             CustomTrackingRecord rec = new System.Activities.Tracking.CustomTrackingRecord("CaptureAnalyticPropertyError", System.Diagnostics.TraceLevel.Error);

  32:             rec.Data.Add("Suggested Fix:", "Verify that the PromotePropertiesBehavior has been added in config.");

  33:             rec.Data.Add("Error:", ex.ToString());

  34:             context.Track(rec);

  35:         }

  36:  

  37:     }

  38: }

To generate Promoted Properties we simply drag this Custom Workflow Activity into the Workflow design and provide its arguments the appropriate values. During the Workflow execution and persistence, the values are updated by this activity in the InstancePromotedPropertiesTable and its corresponding view, InstancePromotedProperties.

Real world implementations

In this section i reference a real world solution implemented by an ISV using this Promoted Properties feature. The ISV Solution at any give time have several instances of the same long running Workflow, processing a financial transaction for example waiting for payment authorization. The Promoted Properties feature prevented the ISV from having to store metadata along with the WorkflowInstanceId that would be needed to query against. This functionality also provided the  additional extensibility option to use the eventing model to resume a particular WorkflowInstance based upon an event scenario and selecting the proper WorkflowInstance to resume based on querying the metadata.

Trending the metadata is also an interesting by product. The ISV archive the data rather than just deleting it. The ISV aligns the metadata with performance monitoring data, and looks at this historically to determine the user preference and how often poor performing Workflows are being invoked or particular parameter/metadata sets that are causing typically well performing workflows to have a dramatic change.

Additional Resources

Windows Server AppFabric: https://msdn.microsoft.com/en-us/windowsserver/ee695849.aspx

MSDN Documentation on Promoted Properties - https://msdn.microsoft.com/en-us/library/ee364726(v=VS.100).aspx

As things stand at the moment, a substantial amount of repetitive coding may be involved when building custom pipeline components. Every custom pipeline component requires repetitive implementation of a minimum of 3 standard interfaces (IBaseComponent, IPersistPropertyBag, IComponentUI) and one extra interface depending on the component type (IComponent, IAssemblerComponent, IDisassemblerComponent).

One of the challenges is that implementation logic of the 3 standard interfaces is often duplicated leading to extra code to be maintained. The amount of duplicate logic can be greatly minimized by introducing a set of base classes from which BizTalk developers can derive their implementations of custom pipeline components. It will help eliminate the majority of repetitive tasks typically involved in the development of pipeline components such as writing plumbing code for persisting and reading configuration properties, returning user-friendly property names, descriptions and design-time metadata elements.

Moreover, by just adding a bit of intellect into the base class library, it will be possible to enable the developers to apply declarative attributes in order to rapidly customize their pipeline components with localizable titles, descriptions, design-time icons and friendly property names without writing a single line of plumbing code.

In order to demonstrate the benefits of the reusable base class library, we have built a prototype which quickly turned itself into a first-class member of the solution framework in the customer project. The new base classes helped the developers ensure that all custom pipeline components share common capabilities such as instrumentation, exception handling, loading and saving component configuration, BAM tracking and more.

We have provided 3 abstract base classes (CustomComponentBase, AssemblerComponentBase and DisassemblerComponentBase) from which the developers derive their end implementations. The class diagram can be depicted as follows:
 
The key benefit of this design was that the amount of plumbing code required for building a custom pipeline component was greatly reduced. The BizTalk developers were able to take advantage of the declarative programming model and add support for fully localizable UI elements such as property names, descriptions and toolbox icons without writing a single line of code. They were also able to stay focused on the core logic whereas the responsibility for exception handling, instrumentation and toolbox interoperability was offloaded to the base class library. This reduced the overall implementation time and delivered better quality code.

The example below illustrates the difference between traditional approach and the new concepts in the pipeline component development:

//
// Pipeline component configuration property management using the traditional approach (equal to 46 lines of code).
//
private string customProperty1 = "default value";

[Description("Static description for customProperty1 embedded into code")]
public string СustomProperty1
{
      get { return this.customProperty1; }
      set { this.customProperty1 = value; }
}

private bool customProperty2 = false;

[Description("Static description for customProperty2 embedded into code")]
public bool СustomProperty2
{
    get { return this.customProperty2; }
    set { this.customProperty2 = value; }
}

private int customProperty3 = true;

[Description("Static description for customProperty3 embedded into code")]
public int СustomProperty3
{
    get { return this.customProperty3; }
    set { this.customProperty3 = value; }
}

public virtual void Load(IPropertyBag pb, int errlog)
{
      object val = null;

      val = this.ReadPropertyBag(pb, "СustomProperty1");

      if (val != null)
      {
            this.СustomProperty1 = (String)(val);
      }

      val = this.ReadPropertyBag(pb, "СustomProperty2");

      if (val != null)
      {
          this.СustomProperty2 = (bool)(val);
      }

      val = this.ReadPropertyBag(pb, "СustomProperty3");

      if (val != null)
      {
          this.СustomProperty3 = (int)(val);
      }
}

public virtual void Save(IPropertyBag pb, bool fClearDirty, bool fSaveAllProperties)
{
      this.WritePropertyBag(pb, "СustomProperty1", this.СustomProperty1);
      this.WritePropertyBag(pb, "СustomProperty2", this.СustomProperty2);
      this.WritePropertyBag(pb, "СustomProperty3", this.СustomProperty3);
}

By contrast, the new base class library delivered greater simplicity which enabled replacing the above code fragment with the following:

//
// Pipeline component configuration property management using the new approach (equal to 12 lines of code).
//
// PropNameСustomProperty[n] is a resource key, the actual name will be loaded from a locale-specific resource file.
// PropDescСustomProperty[n] is a resource key, the actual description will be loaded from a locale-specific resource file.
//
[Browsable(true)]
[BtsPropertyName("PropNameСustomProperty1")]
[BtsDescription("PropDescСustomProperty1")] 
public string СustomProperty1 { get; set; }

[Browsable(true)]
[BtsPropertyName("PropNameСustomProperty2")]
[BtsDescription("PropDescСustomProperty2")]
public bool СustomProperty2 { get; set; }
 
[Browsable(true)]
[BtsPropertyName("PropNameСustomProperty3")]
[BtsDescription("PropDescСustomProperty3")]
public int СustomProperty3 { get; set; }

In summary, the base class library helped encapsulate most common tasks typically associated with custom pipeline component development such as configuration property management. In the above example, the amount of plumbing code was reduced from 46 lines to 12 lines (that’s 75% reduction). The new codebase is easier to maintain and it helps avoid common pitfalls such as forgetting to persist a property value into the property bag.

The reduction in the volume of redundant plumbing code is of course not the only benefit. The real power of a base class library for custom pipeline components comes with the ability to centralize all other common functionality which needs to be shared across all derived implementations:

• Common configuration properties – any common configuration properties which are required for all child implementations of pipeline components can be defined in the base class. This is very handy when there is a need to enrich the entire suite of pipeline components with a common set of properties and avoid duplicating them in each individual component.
• Tracing and instrumentation – all derived implementations of pipeline components can be transparently instrumented with tracing performed by the base classes. This will enable all derived components to output key telemetry events into the trace log, for example, to indicate when the Execute method has been invoked, when it has finished and now long it has taken to execute.
• Exception handling and reporting – the base class is a good place to consolidate the exception management logic and provide a common mechanism for catching, handling and reporting on run-time exceptions.

Now, let’s put everything that was being said about base class library for pipeline components into a visual context. For sake of simplicity, we will title all further sections as “This is how…”.

This is how class inheritance helps derive the end implementations based on particular type of pipeline component:

 
This is how class inheritance helps derive the end implementations based on particular type of pipeline component:
 
This is how a base class for disassembling pipeline components implements the IDisassemblerComponent interface:
  
This is how the explicit implementation of the Disassemble method allows wrapping its user-defined counterpart into a rich instrumented context:
 

It is worth sharing some key notes and observations made during the development phase:

• When compiling a BizTalk pipeline containing a custom pipeline component implemented using the base class library, you will get the following warning message:
  
  Warning BTP0006: Component 'FullComponentTypeName' does not implement IComponentUI interface.
  
  This warning is the result of a missing interface implementation directly by the pipeline component. Since the IComponentUI interface is implemented by the PipelineComponentBase class, the compiler is not currently able to intelligently treat such a case. The warning message can be ignored (we found no way of suppressing this warning in the current release of BizTalk project system). We will see if this can be improved when we ship an update or service pack for the current release.
• The derived implementation of a custom pipeline component will still have to be decorated with a ComponentCategory attribute, otherwise the component will not show up on the Choose Items dialog in Visual Studio IDE. This is due to the fact that we are not currently reflecting the custom .NET attributes across the entire inheritance chain. Again, this appears to be an unfair limitation on which the Product Group was advised and asked to consider improving this behavior.

Show me the money code! You can find the source code in the attachment below. Please share your feedback to help improve the base class library.

Streamlining Custom Pipeline Component Development Using Base Class Library.zip

For more information on the related topic, please visit the following resources:

• “Developing Custom Pipeline Components” article in the MSDN Documentation
• “Custom Pipeline Components Development Best Practices” post on Johan Hedberg’s blog
• “Explicit Interface Implementation” article in the C# Programming Guide

While the blog provides designs to build and consume the package, it does not elaborate on the Workflow Visualizer which is widely covered in technology blogs and MSDN (https://msdn.microsoft.com/en-us/library/cc835242.aspx).

Scenario

Consider a situation where an ISV has created a re-hosted Workflow Designer and Visualizer that their customers use to create, deploy and also debug their solutions.  When the customer encounters a problem that they are unable to debug/address; they turn to the ISV support team to help with troubleshooting and finding a solution to the issue. An obvious challenge is that the ISV support team is not on site or does not have direct access to the AppFabric machines to debug the problem. So it’s imperative that the Support Team should be able to reconstruct the erroneous scenario from the customer and log data.

The grahic below (executed within the Visualizer Application) demonstrates an exception being thrown. The customer offline tests and deploys a Workflow; however this Workflow throws an exception in a specific uses case and aborts. The customer is unable to debug and provide a fix. This finally escalates into a call to the ISV Support team.

Per instructions from the ISV Support team, the customer reopens the erroneous project; loads the specific instance of its execution and creates a snapshot using the appropriate interfaces in the Visualizer. This Snapshot is a package of the project, its supporting files (XAMLX and assemblies) and the event data AppFabric collected from the instance in question via the instrumentation elaborated in previous sections.

This ‘package’ will be loaded by the ISV Support team in the Visualizer to ‘reproduce’ the error on the ‘specific’ Workflow along with the related instance data.

Solution

Leveraging the Windows Server AppFabric ability to track Workflow and WCF event data; the ISV's Workflow Designer created a great support feature that allows the customer to quickly and reliably build a ‘package’ and share their meta and instance data with the ‘offline’ ISV Support team.

Instrumentation

In order to keep this ‘execution’ data separate from other event data that AppFabric monitors on the customer's machine, a separate monitoring database is installed that is only for the use of the ISV's Workflow Designer. Setting up this database is quite simple as elaborated in the PowerShell Scripts below.

   1: import-module ApplicationServer

   2: Initialize-ASMonitoringSqlDatabase -Database WorkflowDesignerMonitoringDB -Server .\SqlExpress

Design and Implementation

Let's take a closer look at the design to build this solution.

Querying for the instance

The customer can pull up a list of the Workflow instances and select the erroneous one with a menu we populate using the AppFabric tables ASWfInstances and ASEventSources.

ASWfInstancesTable

Table ASWfInstances above holds information about each instance that has been executed. The fields ExceptionCount, LastAbortedTime and LastEventStatus should all help the user in finding the erroneous instance. However, this table does not contain information about the application or .XAMLX files involved. For this we use LastEventSourceId to combine this data with the information contained in table ASEventSources. With the extra detail that ASEventSources provides it should be possible for the user to identify the instance being sought.

ASEventSourcesTable

Here is a query used for combining the info from these two tables:

   1: SELECT        wfSvcs.VirtualPath, wfInst.LastModifiedTime, wfInst.LastEventStatus, wfInst.WorkflowInstanceId, wfSvcs.Site, wfSvcs.Name, wfInst.StartTime, wfInst.CurrentDuration, 

   2:                          wfInst.ExceptionCount, wfInst.LastAbortedTime

   3: FROM            ASWfInstances AS wfInst INNER JOIN

   4:                          ASEventSources AS wfSvcs ON wfSvcs.Id = wfInst.LastEventSourceId

Typical results demonstrating this query:

Creating the package

With the instance now selected, the ISV Application gathers the associated event data to be stored in its own monitoring database in preparation for transport. Using this instance’s WorkflowInstanceId the event data is gathered in the following manner:

The ASWfEventsTable is queried using the WorkflowInstanceId as shown below. This table gives us all of the events that occurred during the execution of this instance.

ASWfEventsTable

   1: SELECT        Id, EventTypeId, EventSourceId, ProcessId, TraceLevelId, WorkflowInstanceId, RecordNumber, TrackingProfileId, AnnotationSetId, TimeCreated, Data1Str, Data2Str, 

   2:                          Data3Str, Data4Str, Data5Str, Data6Str, Data7Str, Data8Str, Data1MaxStr, Data1Int

   3: FROM            ASWfEventsTable

   4: WHERE        (WorkflowInstanceId = @InstanceGuid)

Significant fields listed below.

Field	Description	Example Data
EventTypeId	Integer representing type of event (workflow events, wcf events, etc.)	103
EventSourceId	Maps to service this event came from	63
TraceLevelId	Indicates the level of this trace event (e.g., verbose, info, warn, error).	4
WorkflowInstanceId	All events in this instance have this same id.	0f43ee0a-d661-4d14-8657-67003
Data1Str	Context specific event descriptions	Faulted
Data2Str	Context specific event descriptions	Assign
Data5Str	Context specific event descriptions	System.Activities.Statements.Assign

From the result of this query we use the EventSourceId to get information we need from the ASEventSourcesTable as shown in the query that follows. As was stated earlier, this gives us descriptive information to use such as the website, application and source service. The ASEventSourcesTable’s data is described in more detail in the previous section, ‘Querying for the Instance’.

   1: int eventSourceId = asWfEventsTable.First().EventSourceId;

   2: 

   3: SELECT [Id]

   4:       ,[Name]

   5:       ,[Computer]

   6:       ,[Site]

   7:       ,[VirtualPath]

   8:       ,[ApplicationVirtualPath]

   9:       ,[ServiceVirtualPath]

  10:   FROM [AppFabricMonitoring].[dbo].[ASEventSourcesTable]

  11:   WHERE [Id] = @eventSourceId

ASWfEventPropertiesTable

This next table queried for information is the ASWfEventPropertiesTable. This table provides detailed information on the status of each event

   1: ASWfEventPropertiesTable

   2: SELECT                [EventId]

   3:            ,[WfDataSourceId]

   4:            ,[Name]

   5:            ,[Type]

   6:            ,[Value]

   7:            ,[ValueBlob]

   8:            ,[TimeCreated]

   9: FROM ASWfEventPropertiesTable

  10: WHERE [EventId] in (SELECT EventId FROM ASWfEventsTable WHERE [WorkflowInstanceId] = @InstanceGuid)

Significant fields listed below.

Field	Description	Example Data
EventId	The numeric id of the event type.	103
WfDataSourceId	The source of the event property which can be argument, variable or custom data.	0
Name	The name of the event property.	E2EActivityId
Type	The type of the property.	System.Guid
Value	The value of the property.	b82dbde2-d3b0-4ef5-be00-2713820a4128
ValueBlob	The value property when the type is a complex type.	NULL
TimeCreated	Timestamp when the property was emitted.	-

Finally, the ASWfPropertyNamesTable and ASWfTrackingProfilesTable tables are queried. ASWfTrackingProfilesTable is used to capture the tracking profile in use.

   1: ASWfPropertyNamesTable

   2: SELECT [EventSourceId]

   3:       ,[Name]

   4:       ,[Type]

   5: FROM [ASWfPropertyNamesTable]

   6: WHERE [EventSourceId] = @eventSourceId

   7:  

   8: ASWfTrackingProfilesTable

   9: SELECT        Id, Name

  10: FROM            ASWfTrackingProfilesTable

Once the event monitoring information is queried and stored in the ISV’s monitoring database, this data and the project’s files can be written to the package file that is to be sent to ISV Support. This data is visualized on the user’s workflow for visualization of the problem.

Selected instance is visualized on the workflow

With the desired instance visualized, the user chooses to create the package. In this application the package is referred to as a Snapshot. This file allows anyone to see exactly what this workflow was doing at a specific point in time, hence the term snapshot.

The user chooses where to save this Snapshot file, it is created and can then be sent to the ISV Support Team

Loading the package

The ISV Support team will load the package to begin the troubleshooting process. Loading the package requires opening the project and injecting the customer event data into the Monitoring Database used by the ISV Workflow Designer and Visualizer Application. This monitoring database is a separate copy of the monitoring database used specifically for storing the records of a single instance.

AppFabric serializes all of the data it stores; meaning that the customer’s event data in the package (which is serialized) needs to be un-serialized before it can be written to the database.

   1: BinaryFormatter bin = new BinaryFormatter();

   2: FileStream fs = new FileStream(selectedSnapshot, FileMode.Open);

   3: DataSet ds = bin.Deserialize(fs) as DataSet;

   4: fs.Close();

 With the data unpacked into a dataset, it is ready to be placed into the ISV Application’s monitoring database.

   1: AppFabricMonitoringDataSet.ASWfEventsTableDataTable asWfEventsTable =

   2: ds.Tables["ASWfEventsTable"] as AppFabricMonitoringDataSet.ASWfEventsTableDataTable;

   3: AppFabricMonitoringDataSet.ASEventSourcesTableDataTable asEventSourcesTable =

   4: ds.Tables["ASEventSourcesTable"]as AppFabricMonitoringDataSet.ASEventSourcesTableDataTable;

   5: AppFabricMonitoringDataSet.ASWfEventPropertiesTableDataTable asWfEventPropertiesTable =

   6: ds.Tables["ASWfEventPropertiesTable"] as AppFabricMonitoringDataSet.ASWfEventPropertiesTableDataTable;

   7: AppFabricMonitoringDataSet.ASWfPropertyNamesTableDataTable asWfPropertyNamesTable =

   8: ds.Tables["ASWfPropertyNamesTable"] as AppFabricMonitoringDataSet.ASWfPropertyNamesTableDataTable;

   9: AppFabricMonitoringDataSet.ASWfTrackingProfilesTableDataTable asWFTrackingProfilesTable =

  10: ds.Tables["ASWFTrackingProfilesTable"] as AppFabricMonitoringDataSet.ASWfTrackingProfilesTableDataTable;

Before loading the data, we clear the temporary monitoring database.

   1: string conString = System.Configuration.ConfigurationManager.ConnectionStrings["WorkflowDesignerMonitoringDB ConnectionString"].ConnectionString;

   2:                 System.Data.SqlClient.SqlConnection sqlCon = new System.Data.SqlClient.SqlConnection(conString);

   3:  

   4: try

   5: {

   6: sqlCon.Open();

   7: System.Data.SqlClient.SqlCommand cmd = new System.Data.SqlClient.SqlCommand();

   8: cmd.CommandText = "TRUNCATE TABLE [ASWfTrackingProfilesTable]; " +

   9: "TRUNCATE TABLE [ASWfEventPropertiesTable];" +

  10: "TRUNCATE TABLE [ASWfPropertyNamesTable];" +

  11: "TRUNCATE TABLE [ASWfInstancesTable];" +

  12: "TRUNCATE TABLE [ASWfEventsTable];" +

  13: "TRUNCATE TABLE [ASEventSourcesTable];";

  14: cmd.Connection = sqlCon;

  15: cmd.ExecuteNonQuery();

  16: }

  17: finally

  18: {

  19: sqlCon.Close();

  20: }

We then set up a bulk insertion.

   1: System.Data.SqlClient.SqlBulkCopy sbc = new System.Data.SqlClient.SqlBulkCopy(conString,

   2: System.Data.SqlClient.SqlBulkCopyOptions.KeepIdentity | System.Data.SqlClient.SqlBulkCopyOptions.FireTriggers);

Then we insert the records into the appropriate tables.

   1: BulkInsertTable(asWfEventsTable, sbc);

   2: BulkInsertTable(asEventSourcesTable, sbc);

   3: BulkInsertTable(asWfEventPropertiesTable, sbc);

   4: BulkInsertTable(asWfPropertyNamesTable, sbc);

   5: BulkInsertTable(asWFTrackingProfilesTable, sbc);

Once the project and event data is loaded, the ISV Support Team can instantly reproduce the error scenario and troubleshoot the customer’s erroneous Workflow. The Support team has an exact copy of the customer's project with the state reproduced.

Result

Typically, .NET4 Workflows are difficult to debug since the event information relating to these Workflows is ‘hidden’ amongst OS and Server related events. With the advent of Windows Server AppFabric, this event data is now easily accessible from a simple database. Even the ISV's rudimentary Workflow Designer and Visualizer Applications can be designed to package this data for remote debugging.

The error example used in this blog post relates to a variable with the wrong format. While in the real world there will be many more complex use cases, this solution provides a reasonable platform to troubleshoot Workflows, especially in an offsite scenario.

Once the error is ‘fixed’ the Workflow instance is flawlessly executed and can be verified on the Visualizer.

Summary

Windows Server AppFabric provides adequate extensibility for the ISV to build applications that can efficiently manage the lifecycle of .NET4 Workflow Services. This blog post effectively demonstrates the extensibility around remote troubleshooting.

Caveat

The Workflow instances could contain sensitive and confidential data, event info etc. The ISV Application should include designs to obfuscate such information.

As part of the BRE adoption exercise, the ISV implemented a policy which included a static method call as depicted below.

Being new to the Business Rules Engine, the developer didn’t specify any BRE configuration settings in the app.config file, meaning that the entire <Microsoft.RuleEngine> section has been absent from the application configuration. When executing the above policy, the developer has not seen any exceptions and was led to believe that invocation was successful. However, the policy didn’t return any results. The developer has enabled the detailed tracing which confirmed that BRE policy completes successfully but none of the rules were fired.

The developer had to spend some time to troubleshoot the above problem before concluding that it was related to a missing StaticSupport parameter. The Business Rules Engine requires the StaticSupport parameter to be defined in order to be able to invoke static members of a class without the need to assert an object of that class. If StaticSupport is not present either in application configuration file or registry, the BRE policy containing a call to a static class member will be successfully executed but will not return any result. Unfortunately, there is currently no diagnostic information available to the developer to help identify why a policy doesn’t not return any results.

In order to resolve the issue, the developer modified the app.config file and added the following content:

After the above configuration parameter was added, the trace log confirmed that rules are being fired and the correct results are being returned:

Although the product documentation covers the static member invocation, our observations suggest that newbie BizTalk developers are likely to spend their valuable time troubleshooting the policy execution issues if they didn’t read the documentation from the beginning. This may negatively affect the developer experience which is what we would very much like to prevent.

In light of the above experience with BRE, we have advised the Product Group on the need for a better diagnostic of policy execution. This should make it easier for the developers to identify and resolve the BRE policy invocation issues when static methods are being used.

For more information on the related topic, please visit the following resources:

• "Invoking Static Members of a Class" article in the MSDN Documentation
• "MS BRE: Controlling rule side effects" post on Charles Young's blog
• "BizTalk Server 2006 Business Rules and Static Methods" article on Richard Seroter's blog

Please share any feedback that you may have. Thanks.