• When you open Terminator, click the Help menu at the top left and you will see the Update Task Repository menu item.
• Clicking this does the following:
  • Renames the existing MaintenanceRep.dll (located in the same folder as Terminator) to MaintenanceRep[DATETIME].dll
  • Downloads the current MaintenanceRep.dll from Azure
• This feature requires external internet access as well as .NET 4.0 or above for Azure connectivity.  If you don't have either of those on your BizTalk machine, you will need to run Terminator on a box that does and then just copy over the new MaintenanceRep.dll onto your BizTalk machine.  The box where you do this does not need to have access to BizTalk.
• The Update Task Repository menu item is disabled once Terminator connects to a BizTalk environment.  You will need to close and re-open Terminator for it to be re-enabled.
• I would recommend updating the task repository every month or so.

Issue:

BizTalk 2013 R2 Cumulative Update 1 (CU1) install fails with “Package does not contain compatible branch patch” error:

Explanation:

This issue is happening in environments where a version of BizTalk Health Monitor (BHM) from the Microsoft Download Center has been placed within the BizTalk installation folder.

There are 2 branches of BHM:

1. OOB BHM – shipped out-of-the-box with BTS2013 R2
2. DLC BHM – shipped via Microsoft Download Center

OOB BHM has had 2 released versions so far.  Version 2.0.0.0 shipped with BizTalk 2013 R2 RTM and version 3.1.5632.20684 shipped with BizTalk 2013 R2 CU1.

OOB BHM is located at \SDK\Utilities\Support Tools\BizTalkHealthMonitor within the BizTalk installation folder (default installation folder is C:\Program Files (x86)\Microsoft BizTalk Server 2013 R2)

DLC BHM has had many versions and is currently at 3.1.5654.28025.

The BTS2013R2 CU1 installer throws the above mentioned error if it finds any version of BHM within the BizTalk installation folder that is not a part of the OOB BHM branch.

We have seen this issue in environments where the user did one of the following:

1. Replaced the OOB BHM binaries in the \SDK\Utilities\Support Tools\BizTalkHealthMonitor folder with a version of DLC BHM
2. Created a new folder with a version of DLC BHM anywhere within the BizTalk installation folder

Solution:

To allow CU1 to install, simply move the folder containing DLC BHM out of the BizTalk installation folder and run the installer again.

Moving forward, never place any DLC BHM version within the BizTalk installation folder.

Since then, I've come across a caching cleanup scenario that even the hard termination task can't handle.  Basically, if there are cache message refs (in <HOST>Q or <HOST>Q_Suspended tables) that are missing their associated caching instance (in the Instances table), then even the hard termination task won't be able to clean them up.  The reason is that the Terminate Multiple Instances (Hard Termination) task finds the instances for termination by looking through the Instances table - so in this case it never sees the orphaned cache message refs.

If you run into this scenario, what you will see is that that once Hard Termination has successfully cleaned up all caching instances, BHM or MBV will continue to complain about cache messages and if you run the View Count of Cache Messages in All Host Queues task, it will show Cache Messages and Message Refs existing even though there are no Associated Cache Instances:

The workaround so far has been to to first run the DELETE Orphaned Messages In All Hosts task (to clean up the orphaned message refs) and then run the Terminate Multiple Instances (Hard Termination) task as mentioned in the previous blog (to clean up the cache instances).

I have also created a new cache cleanup task called Terminate Caching Items (Hard Termination).  This task checks all the <HOST>Q, <HOST>Q_Suspended, and Instances tables for caching items associated with any host and in any state and cleans them all up.  The purpose of this new task is to handle all cache item cleanup scenarios with one step - so the user doesn't have to dig into the state of the cache items in their environment to know how to clean them up.  So if BHM or MBV complain about cache items, simply running this task on the correct MessageBox should take care of it. 

This new task is available in BHM v3.1 and Terminator v2.2.

If you were not already aware that Terminator has been integrated into the BizTalk Health Monitor (BHM) tool, see here.

• You must be on BHM v3 or above to get the Terminator functionality
• Get BHM from https://www.microsoft.com/en-us/download/details.aspx?id=43716
• See here for a walkthrough of BHM v3 functionality - including Terminator integration.
• See the BHM blog at https://blogs.msdn.com/b/biztalkhealthmonitor for more information on BHM.
• So far, I've included the most popular Terminator tasks in BHM and plan to add more over time
• BHM officially supports BizTalk 2010, 2013, 2013R2, and 2016.
• I'm going to continue updating standalone Terminator for now and it is still available here. 
• At some point, I'll probably end up creating a final build of the standalone version and move all new development to BHM.  I'll update this blog once that decision is made.
• Standalone Terminator tool is no longer being updated. The final version of standalone Terminator is v2.5.  All new development (including BTS2016 support) is happening in BHM Terminator. 
• MBV is no longer being updated and the last update to it was in July 2014.  To make sure you are using the latest MBV repository with the latest rules/queries, be sure to move to BHM.

Problem

Periodically, your BizTalk host process crashes with the following errors in the eventlogs.  Note the error code is 80131544.  If you see the same error with a different code, you’re likely running into a different issue.

Also, notice none of the errors come from event source BizTalk Server.  The ones in the Application event logs come from .NET Runtime and Application Error and the one in the System event logs comes from Service Control Manager.

Log Name:      Application
Source:        .NET Runtime
Date:          9/20/2013 3:47:42 PM
Event ID:      1023
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      servername
Description:
Application: BTSNTSvc64.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 000007FDED170BC1 (000007FDECE00000) with exit code 80131544.

Log Name:      Application
Source:        Application Error
Date:          9/20/2013 3:47:42 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      servername
Description:
Faulting application name: BTSNTSvc64.exe, version: 3.10.229.0, time stamp: 0x50fe567a
Faulting module name: clr.dll, version: 4.0.30319.19106, time stamp: 0x51a512d4
Exception code: 0x80131544
Fault offset: 0x0000000000370bc1
Faulting process id: 0xca8
Faulting application start time: 0x01ceb6394f1dd32a
Faulting application path: C:\Program Files (x86)\Microsoft BizTalk Server 2013\BTSNTSvc64.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 830374f6-222d-11e3-93f8-00155d4683a2
Faulting package full name:
Faulting package-relative application ID:  

Log Name:      System
Source:        Service Control Manager
Date:          9/20/2013 3:47:43 PM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      servername
Description:
The BizTalk Service BizTalk Group : BTSOrchHost service terminated unexpectedly.  It has done this 2 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.

Root Cause

There is a change in the .NET 4.5 CLR that results in the BizTalk process crashing during XLANG AppDomain shutdown.  XLANG AppDomain Shutdown is when the .NET AppDomain that contains the Orchestration Engine tears itself down during periods of inactivity or idleness. 

The Orchestration engine’s AppDomain shuts down by default after 20 minutes of idleness (all orchestration instances are dehydrateable) or 30 minutes of inactivity (no orchestration instances exist).

BTW, if you see this same issue in BizTalk 2010, it’s because you installed .NET 4.5 in your BizTalk 2010 environment – which is not tested or supported.  You need to be on .NET 4.0 if you’re running BTS2010.

Update:  BizTalk 2010 CU7 added support for .NET 4.5 so you should not see this issue once you are on CU7 - see Permanent Solution section for details.

Workarounds

The first and easiest workaround is to do nothing.  Note that this crash happens during periods of idleness/inactivity so your orchestrations aren’t doing anything anyway.  Also, Service Control Manager is configured to bring a BTS process up a minute after a crash so the process will come back up just fine and continue to process fine.  The crash won’t happen again until after there is orchestration activity and another period of idleness/inactivity occurs.

Note that if there are other things going on in the same host (like receive or send ports) then I wouldn’t be comfortable letting the host crash since non-orchestration work within the same host could be impacted.  In that case, you can separate out non-orchestration work into other hosts (that’s generally recommended anyway) or you can go with the below workaround. 

Another thing to consider is that, depending on the Windows Error Reporting (Dr. Watson) settings on the machine, these crashes can build up dump files on the drive.  The default location for these would be C:\ProgramData\Microsoft\Windows\WER\ReportQueue.  Check for any subfolders starting with "AppCrash_BTSNTSvc" - you can delete them if you don't need them.  So if you have limited disk space on the system drive, that might also be a reason to go with the below workaround. 

If allowing the host to crash during periods of orchestration idleness/inactivity is not an option, the workaround is to turn off XLANG AppDomain shutdown.  This is generally safe to do.  The only concern is if you have an orchestration that calls custom code that pins objects in memory to the appdomain (not good in general), then not tearing it down periodically could lead to excessive memory usage.  Still, any 24x7 environment will never have appdomain shutdown happening anyway and I’ve seen a number of environments that have very low latency requirements turn it off (since reloading XLANG engine after periods of inactivity is a perf hit to that first request). 

So, to prevent the crash from happening altogether, here’s what you do:

1. Go to your BTS folder (default is C:\Program Files (x86)\Microsoft BizTalk Server 2013)
2. First, save a copy of the BTSNTSvc64.exe.config file with a new name since we need to modify the original.  (BTSNTSvc.exe.config if it’s a 32 bit host that is crashing – you can check the error message to see if the crash is happening to BTSNTSvc.exe or BTSNTSvc64.exe)
3. Open the original file in notepad and directly below the <configuration> node, add the following:
    
       <configSections>
           <section name="xlangs" type="Microsoft.XLANGs.BizTalk.CrossProcess.XmlSerializationConfigurationSectionHandler, Microsoft.XLANGs.BizTalk.CrossProcess" />
       </configSections>
    
4. Then, directly below the </runtime> node, add the following:
    
       <xlangs>
           <Configuration>
               <AppDomains AssembliesPerDomain="50">
                                   <DefaultSpec SecondsIdleBeforeShutdown="-1" SecondsEmptyBeforeShutdown="-1"/>
               </AppDomains>
           </Configuration>
       </xlangs>
    
5. Recycle the host

Permanent Solution

Microsoft has created fixes for this issue in BizTalk 2013 and 2010.

BizTalk 2013:

• Fix will be included in BizTalk 2013 Cumulative Update 3
• The article at https://support.microsoft.com/kb/2555976 will be updated once 2013 CU3 is publicly released.
• If you need this fix before 2013 CU3 is publicly released, please create a support case with Microsoft and request standalone fix 2909928.
• 2013 CU2 is a prereq for this standalone fix.  Get CU2 from https://support.microsoft.com/en-us/kb/2555976.
• Please be sure to understand the information at https://support.microsoft.com/kb/2003907 before installing any BizTalk fix or CU.

BizTalk 2010:

• Fix is available in BizTalk 2010 Cumulative Update 7
• Get 2010 CU7 from https://support.microsoft.com/kb/3021712.
• This issue is officially documented at https://support.microsoft.com/en-us/kb/2909928.
• .NET 4.5 is not officially supported on 2010 without CU7.  It is OK to install CU7 before or after installing .NET 4.5.
• Please be sure to understand the information at https://support.microsoft.com/kb/2003907 before installing any BizTalk fix or CU.

It turns out that this customer also had several of their own SQL Agent jobs running on the BizTalk server. As part of configuring the destination environment, we attempt to recover all of the jobs running on the BizTalk server with one exception: we don’t support importing of SQL Agent jobs where the steps use more than one database. The script's logic iterates over each database to be recovered and logs the jobs associated with that database in bts_LogShippingJobs for later recovery. If a job has more than one database association, the script attempts to log it twice. But we never want to recover the same job more than once, so bts_LogShippingJobs doesn't allow duplicate jobs. When the script attempts to log the job the second time, it fails.

In general, we discourage running any jobs on the SQL Server that supports BizTalk Server other than the jobs that ship with the product.

For those customers who choose to run their own jobs and encounter this issue, the solution is to temporarily remove any job(s) associated with multiple databases while setting up the destination recovery environment. You will also need to develop your own recovery plan for any such job(s). Once the destination environment is configured, you can restore the jobs. The following T-SQL will identify jobs associated with multiple databases:

SELECT j.name, COUNT(DISTINCT js.database_name) AS dbcount
INTO #tmp FROM msdb.dbo.sysjobsteps js
JOIN msdb.dbo.sysjobs j WITH (NOLOCK)
ON j.job_id = js.job_id
GROUP BY j.name
SELECT * FROM #tmp WHERE dbcount > 1
DROP TABLE #tmp

You can also get some great information about this issue at https://blogs.msdn.com/b/biztalkcrt/archive/2012/02/22/edi-leap-year-hotfix-biztalk-2009-and-biztalk-2006-r2.aspx.

I've also compiled a list of questions that have been asked about this issue over the last week and created a detailed FAQ document about this.  Check it out below. 

Also, to download the pipeline sample mentioned in the FAQ, click here.

(FAQ last updated on 02/29/2012 at 12:22PM Central Time)

BTSLeapYearFAQ.docx

Carl Nolan of the UK Solution Development Team wrote an excellent blog post on the above topic.  He includes a script for creating partitioned views that can be used if modifications to the existing activity items are continuing to be made.

https://blogs.msdn.com/b/mcsuksoldev/archive/2011/11/15/creating-a-partitioned-view-in-the-bam-archiving-database.aspx

Say you have a large flatfile in an application that does messaging only.   You drop a single file and you notice elevated cpu for the receive host for several minutes.    What’s happening in the host is that a single messaging thread is using the ffdasm to disassemble the flat file.   This is a very cpu intensive activity

Now you drop 100 large flatfiles.  Since messaging engine threads per cpu is set to 20, if you have a two processors, then there will be 40 messaging engine threads created and operating on 40 inbound flatfiles.    cpu still goes to 100%, but it takes longer for the first flatfile to be processed because other threads are taking valuable cpu resources.  

In testing on a two processor system I was able to get the best end to end performance by specifying that only one messaging engine thread per cpu be used on the receive host when performance testing with 100 files.  Tuning messaging engine threads per cpu to a lower value results in faster processing of the first file through the system and additionally faster overall processing.

1 messaging engine thread per cpu on the receive host about 4 minutes to process the first file.  1 hour and 30 minutes for all files

10/21/2011  02:46 PM                 set messaging engine threads to 1 for full test.txt

10/21/2011  02:50 PM         2,752,368 {FC0AD608-1F76-43A2-B7D5-44DDB66E66DF}.xml

10/21/2011  04:16 PM         2,752,368 {F7241E83-C7F1-428D-A5DD-2582CF8ABE20}.xml

2 messaging engine threads per cpu on the receive host about 4 minutes to process the first file.  1 hour 42 minutes to finish all approx..

10/19/2011  04:29 PM               set messaging engine threads to 2 for full test.txt

10/19/2011  04:33 PM         2,752,368 {C29DB82A-98E8-4B41-B917-093DDC708D21}.xml

10/19/2011  05:02 PM         2,752,368 {E9AFFEC2-B60B-446C-9D63-1BBBB2B2FD12}.xml  33^rd file in 34 minutes.  Had to stop test early summary

extrapolated.

4 messaging engine threads per cpu on the receive host  10 minutes time to first message and 1 hour and 39 minutes total processing time.

10/19/2011  06:06 PM                 set messaging engine threads to 4 for full test.txt

10/19/2011  06:16 PM         2,752,368 {E96CD77C-8D24-4BB9-A111-871594B06F3F}.xml

10/19/2011  07:45 PM         2,752,368 {2F2EE1F9-C9F0-44A9-B2D1-1A02BC9F7EAD}.xml

20 messaging engine threads per cpu on the receive host (the default setting) 18 minutes.  1 hour and 42 minutes for all files to be processed.

10/20/2011  09:35 AM                 set messaging engine threads to 20 for full test.txt

10/20/2011  09:53 AM         2,752,368 {9BA6A42F-A634-4FEF-A29E-F16A774D823F}.xml

10/20/2011  11:17 AM         2,752,368 {262F3035-CACF-423A-BDFA-5EA817D2639B}.xml

50 messaging engine threads per cpu on the receive host  23 minutes before first file is processed.  1 hour 54 minutes for all files to be processed.

10/20/2011  02:56 PM                 set messaging engine threads to 50 for full test.txt

10/20/2011  03:19 PM         2,752,368 {40BF75AF-3935-4DC1-99C4-048CD61421D1}.xml

10/20/2011  04:50 PM         2,752,368 {5B72CC56-FB61-4A33-9186-3A3F954AC38A}.xml

I've been pinged a number of times on this so thought I should blog the workaround and an explanation. 

First, let's say MBV shows you something like the following in the Warning and Summary Report:

Or you just notice in MBV that there's a bunch of cache messages in one of the queue tables:

Well, according to my Using BizTalk Terminator to Resolve Issues article, you simply run the Terminate Caching Instances task: 

That should do it.  If it doesn't, make sure you have stopped all the BizTalk hosts (that includes the IIS app pool hosting the BizTalk isolated host if the caching items are there) and try again.  (Hey, you shouldn't be running Terminator without stopping all the BTS hosts anyway).

Now, that will definitely do it.

Well... like 99.9% of the time.

Let's say you do all of the above and then run one of those View tasks (or MBV)  and notice that Terminator left behind some of those caching instances (and their associated caching messages).  This happens even though Terminator claims to have terminated all of them successfully.  And rerunning the task doesn't help - Terminator will repeatedly say it successfully terminated those instances but either of the View tasks will show that they're still there.

Ok, so now you're most likely running into a very rare scenario that I've come across a few times. 

First, I should point out that the Terminate Caching Instances and Terminate Instances tasks use BizTalk's WMI API to interact with the messagebox - and that's key.  I had the opportunity to analyze some data from a customer environment that was running into this issue and it turns out that there are certain times when msgbox logic prevents the stored procs called by WMI from terminating “internal” instances – with caching being considered one of those "internal" types.  As far as when exactly the msgbox logic goes down this "rare" path, I've never had access to a repro environment where I could fully debug this so I don't have a good answer for that.

So I was going to write code in Terminator's WMI class to catch this scenario and warn the user that they need to use the workaround to clean up the remaining items but unfortunately the msgbox logic catches the failed call and doesn’t pass that info on to the WMI caller – so there’s no way for Terminator (or any WMI client) to know that some of the instances weren’t deleted.  As far as the WMI client is concerned, the stored proc call completed sucessfully so it assumes the instances it asked to be Terminated were actually terminated.

So what's the workaround?  Well, don't use WMI for this particular situation.  Instead, use Terminator's Hard Termination tasks (Terminate Multiple Instances (Hard Termination) or Terminate Single Instance (Hard Termination).

While I was working on Terminator's WMI class and having BizTalk engineers use Terminator on an internal-only basis within Microsoft, we noticed that on very rare occasions the termination tasks would not terminate something.  This would not be a limitation of Terminator and we could reproduce it with any WMI client (including the BTS Admin console).  I created the hard termination tasks specifically to handle those scenarios.  They bypass BizTalk's normal termination API and use SQL calls to directly interact with the BizTalk msgbox.  They can terminate anything (well, so far) - even internal instances.  Originally, we just had the Terminate Single Instance (Hard Termination) task to help terminate a one-off instance that just wouldn't terminate any other way.  That worked great for those one-off scenarios but I soon realized that sometimes there would be a larger number of instances that needed hard termination.  I left the single instance task to give users that functionality and wrote the Terminate Multiple Instances (Hard Termination) task.  That allows the user to choose the Host, Class, Status, and the Max number of instances to terminate and will do hard terminates on all items that fit the filter criteria. 

The only thing that is painful about the Terminate Multiple Instances (Hard Termination) task is that if you have instances across multiple hosts with various statuses, you will need to run the task for each permutation since it doesn't have the ability to handle, in one execution, multiple hosts and statuses (or classes) like the WMI-based Terminate Instances and Terminate Caching Instances tasks.  Since Terminator v2 supports Powershell, I'm hoping at some point to create a powershell-based hard terminate task that can provide this functionality - just haven't had the time to do that yet.

So, in short:

Problem:

The Terminate Caching Instances task is not cleaning up all cache items.

Solution:

Use the Terminate Multiple Instances (Hard Termination) task, choose Caching as the Class Parameter. 

You will need to re-run this task for each Host and Status that applies to the cache items you're trying to terminate - MBV or the Terminator View Count of Cache tasks should give you some info in this regard. 

BTW, I've seen active, dehydrated, and suspended cache items so you may need to cycle through all of those Statuses.

In general, if you ever find that any of the termination tasks are not terminating what you want, the two Hard Termination tasks (Single and Multiple) are the workaround. 

Remember that Hard Termination tasks bypass the normal BTS APIs so should only be used if the normal Terminate Instances or Terminate Caching Instances task is not working - and as always, be careful with Terminator - especially when doing any of the deletion tasks.