So first off, I should probably say that my posts might be a bit more far and few between (not that they are all that often as it stands :). A long time co-worker, Kartik, will probably be manning the ship on the blog as I have decided to expand my skills a bit and work on SQL Server Analysis Services. Figured I had been helping people build the processes which actually executed their business logic, now I wanted a bit more experience with the tools they use to analyze what exactly they have been doing at their business. This blog, though, will remain a BizTalk blog since it was never meant to be "my" blog ... just look at the name. Okay, enough of that. :)

 As a "parting gift", I managed to convince the team to let me add a small enhancement to persistence as long as I did not push changes to the UI. Ostensibly, this was really meant for PSS, but personally, I would rather have everyone else fix / figure out their problems without having to call PSS. So don't tell them I told you, but here you go :)

As part of default dehydration now, the orchestration engine also persists, in the clear, the name of the shape on which it was blocked. I have seen many customers which turn on shape level tracking for orchestration just so that if things get "hung up" and lots of orchestrations end up in the dehydrated state, they can open the orchestration in the debugger view and see what it has executed and what it is waiting on. Using orchestration debugger for this is just a tremendous overkill and adds overhead to the processing and now you have to manage the potential buildup of this information in the tracking db. All just so you can know what an orchestration is blocked on. With R2, you have the potential to figure this out yourself. :)

Unfortunately, we did not get this into the UI. Would have been really cool to view it in the MMC and dehydrated orchestrations would have the string of the blocking shape. But we gave you a start and now if we ever did add the UI tools, we wouldn't have to change to underlying db schema or any of the engine. So here are the details.

In the Instances table in the messagebox there is a new column called nvcLastAction. This column contains a guid for dehydrated instances. Sorry, all I had access to in the engine was a guid. However, in the DTA database you will find a table called dta_ServiceSymbols which contains some xml for each service which is deployed, regardless of whether you have any tracking on. This xml is used to translate the guids into string names for some of our tools. It is relatively easy to look at this and build a little tool which will map the guid to the actual shape name. Sorry I have not written the tool ... Remember about being very careful when looking directly in the messagebox. Be careful of locking tables and causing all sorts of blocking issues. Use NOLOCK hints and never, ever, ever update / delete anything. If you are at all unsure about rules around this, please see https://blogs.msdn.com/biztalk_core_engine/archive/2004/09/20/231974.aspx which is specific to 2004 and mostly obsolete for 2006 with the new enhancements to the MMC which makes these direct queries unnecessary, but the rules for quering are still valid. Be very, very, very careful. If you change things or cause problems by introducing blocking, your chances of support are pretty much nill.

 Couple of caveats. We did not do extensive testing on this for parallels where you might be blocking on more than one shape. It could technically pick either shape with no gaurantees. Also, we are simply depending on the orchestration engine to provide this information as we hook in to grab it in the same fashion as the tracking interceptors do.

So there you go. Have some fun with it. Perhaps if you ask nicely, you could convince someone to put this in for 2006 SP1. You have to ask nicely, though. :)

Hope everyone is having a great day.

Lee

 So first, let me say that the online documentation should be extremely good for our Disaster Recovery at this point. Michael McConnell, Shu Zhang and I spent a good bit of time making sure we covered every detail we needed and tried to order it so that it was more readable. If you have any complaints about the docs, please send feedback via this blog posting. I will get it to the appropriate person (Michael) to update the docs if necessary. You can find these docs at https://msdn2.microsoft.com/en-us/library/aa562140.aspx.

I will not repeat everything which is in the docs here, but most of the docs are spent explaining how to setup our backup jobs, how to configure log shipping, and how to restore the system in the event of a disaster. A number of companies have had their DBAs complain about having to use our backup process versus using their standard process which is to backup a database at perhaps 1am every day and then take periodic log backups. This link (https://msdn2.microsoft.com/en-us/library/aa577848.aspx), a sub topic of the above link, gives a good explanation of what we are doing ... transactional log marking. The problem with what the DBAs are doing is that they are treating each db as a completely separate entity. BizTalk, though, will at times use Distributed Transactions to ensure reliability of actions across databases. If you treat each db as a separated entity, then if DB1 and DB2 participate in a distributed transaction together and I do a log backup of DB1 before the transaction commits and a log backup of DB2 right after the transaction commits, then when I restore the two of them my environment will be in an inconsistent state. For that purpose we use transaction log marking with "restore to mark" for all the databases in our environment (as described in the previous link and in many online sql docs).

There is no magic here. This is all standard TSql. You can read through our code and figure out everything that we are doing. We never install custom components on database servers for doing external procedure calls and we never use xp_cmdshell so this is all code that any database developer could have written, it is just that most never seem to care about DTC (even if maybe they should).

The question has come up as to are there any alternatives. The quick answer is that if you want to use log backups, then no, there are no other alternatives. You would have to write all the same code that we do so you need to use the code that we wrote. However, if you don't really care so much about the data in your system and are okay with, say, 24 hour data loss windows (your data loss window with log backups is the interval between log backups), then you could optionally just have a period of time every day where you shut the entire system down and then just do full backups. Since there is no activity going on in the system when it is totally down, then you don't need to worry about distributed transactions. To shut the entire system down, you need to shutdown all BizTalk Host Instances, make sure no traffic is coming in via Isolated Hosts (for example http / soap receive adapters), shutdown the sql agent processes on the various databases, and make sure that no other administrative tasks are on going (like changing configuration, redeploying, adding a new sendport, ...). You should be able to verify that nothing is happening on the system via SQL profiler or some other device monitoring the databases. Once the whole system is down, you could take a backup of all of the databases in the system. To get a list of all of the databases which the BizTalk backup process was going to backup, you just need to "SELECT * FROM admv_BackupDatabases" from the BizTalkMgmtDb database. That is a view which all of our backup code uses during our DR process. If you add custom databases to our backup process (https://msdn2.microsoft.com/en-us/library/aa561198.aspx) then it will also show up in this list of databases. It is important to note that our log shipping story also copies sql agent jobs out of msdb when you configure the destination system and restores them when you finally restore to mark on the destination system. If you chose to not use our log shipping story and do the system shutdown full backup approach, you need to make sure you handle backing up and restoring msdb with the sql agent jobs.

At this point, if you are chosing to have a 24 hour data loss window and use scheduled, daily downtime to create a complete set of full backups for recovery, then you should also change the recovery mode on the databases to SIMPLE. This will automatically manage the size of the transaction log by essentially doing "truncate log on checkpoint". By default, our databases are all set to FULL recovery mode which will only truncate the log when the user performs a backup of log or when the user manually forces the log to truncate (ie BACKUP LOG WITH TRUNCATE ONLY type command). There are lots of docs out there on this, just search for SQL Server Recovery Models. The key is that if you leave it in FULL recovery mode and then don't do log backups (with our job), then your log will grow unbounded and you will eventually run out of disk space.

Hope that helps to clear up some confusion.

 Thx

Lee

 Oh yeah ... for those of you who always ask about database mirroring ... I am really sorry that we can't support it. If you check out https://msdn2.microsoft.com/en-us/library/ms366279.aspx you will find a write up by the SQL team on issues with mirroring and distributed transactions. Even if you put everything in one database, we still use DTC transactions occassionally (we did not optimize to detect everything in one database and it is only with the latest System.Transaction work is their automatic support for upgrading from local to distributed transactions). Thx.

What can't you do?

You may not make any changes which could effect how our queries are executed (ie their execution plan). I will list some specific things which can affect the query plans, but they are just examples. If you know of a setting which I don't list and you aren't sure, then don't change it. Here are a couple of things to understand around this:

1) Think of the sql code (stored procedures) which we ship as uncompiled C# (or whatever your favorite language is) code. Every customer who installs our product gets their own compiled version of this code courtesy of the SQL Server engine. You can read up on documentation on how the sql engine performs this compilation (or go to one of Paul Randall's talks at Tech Ed or some other conference) but it is important to realize that it is being done. SQL even has a performance counter which you can monitor called recompiles which will show you how often "code" is being recompiled potentially while your applicatin is running at full throttle. SQL Server uses various statistics about the table layout and cardinality of indexes and such to attempt to chose the best plan to reduce IO and provide the best performance. One thing that the optimizer cannot anticipate is the locking side-effects certain plan choices could have. It is therefore possible that in an environment in which lots of threads are accessing the database at very high rates, a plan which might be optimized for potential IO access could cause tremendous locking and hence be a much worse overall plan choice.

2) The next very, very important thing to understand is that while the database is installed on your server, it is not "your" database. It is our (or if you catch me on the wrong day, not thinking, "my") database. When you have performance problems with it, the buck stops over here, not at your DBAs desk. For that reason, it is unbelievably important that the system runs the same on every installation. If every customer who called in with a question was forced to gather a dump of the plan cache from within SQL Server and send it to us so that we could reconstruct exactly what sql server is doing, it would make support pretty much impossible.

3) Access to the messagebox is very controlled and very limited. Via our Admin MMC (in bts 2006) you can construct somewhat flexible queries against the data, but even that is somewhat limited. From the runtime, there are a set of precanned generic stored procedures and some per-host templated stored procedures which provide all the access that the runtime needs. Since we have locked down access to the database, we can understand all of the potential queries. With that in mind, we have done everything we possibly can to "hardcode" the query plans for all of our stored procedures. This means using index hints, force order hints, occassionally join hints, norecompile hints (why recompile if you are just going to generate the same plan) and every trick we can come up with the tie the hands of the optimizer (part of the sql engine which calculates the best query plan). The sql server team is well aware of what we are doing. Technically in SQL 2005 there is support for USE PLAN where we could really hardcode the plan completely, but (in order of importance): (a) we support SQL 2000, (b) it is only supported on SELECT statements which only ~30% of our code and (c) it is non-trivial for generated stored procedures like the ones we use for per host access. Along with attempting to force the optimizer to chose the plan we want, we also disable some of the features of SQL Server which "help" the optimizer chose the best plan. This includes statistics generation and updating. Statistics provide information about the layout of a table so that the optimizer might decide it is more efficient to scan the clustered index than to seek over the non-clustered index (as an example). Since we are hardcoding the plan, statistics are not so important. We also disable parallelism (setting at the server level MAX DOP to 1). Parallelism provides more options for the optimizer to chose amongst for plans. Typically it is very useful for queries over very large databases (warehouse type applications). We are a OLTP (online transaction processing) style app which executes the same sets of sprocs on multiple threads at very high rates. We already provide our own parallelism. I "apologize" that we chose to set this at the server level which could cause it to effect any other database you have installed on that server, but you may not change this setting or else we will simply push back on any support until you switch it back (no matter how large a customer you are). If you have only one server, install an alernate instance of sql server on the same machine and put us on the separate instance. In our testing, when SQL Server choses a plan with parallelism in it for one of our queries it can cause orders of magnitude performance degradataion.

4) We are not perfect (but we try :). Occassionally we do not include enough hints so that the optimizer can chose another plan which is worse. I have personally fixed two bugs in which we needed to add hints or slightly change the structure of our query because the optimizer chose a less than optimal plan. It can happen, but we do our darned'ist and I have only had I think 3 cases of it in about 3 years which is not bad. It also not fair to blame this on the optimizer. As I said above, since we are hardcoding the plans for all of our queries, we turn off a lot of the features which might help the optimizer (but add overhead to the overall system).

A quick list of things which can effect the query plans:

Statistics (don't enable these)

Parallelism (don't turn this on )

table structure (don't add indexes, columns, triggers, ... If you do you will hear silence when you call for help)

Stored procedures (don't change them. You can look all you want, but no touching)

So what can you do?

You can change the underlying storage model for our data on the physical disks. By this I mean stuff like file groups and files and such. By default all of our tables are setup on a single (PRIMARY) filegroup which has one file. This was admittedly perhaps not the best decision. You are more than welcome to create multiple files for a given file group and also to create multiple filegroups and move our tables around. This can give potentially significantly better IO performance and hence overall system performance. We are currently working on a paper (and perhaps tool to automate) which will give recommendations on how to layout out tables given as many phsical disks as you have access to. You can also change setings related to how sql pre-allocates space for the files and for some of their internal structures. None of this effects how our plans are executed, it simply allows for faster IO access and better overall performance. One interesting settings which we do not enable by default but have been playing with for a little while now is "TEXT IN ROW". This table level option is used to tell sql server whether to store the image column data in the same data page as the clustered index or in another data page. By default this is set to off. If you think about an application like Outlook / Exchange this makes a lot of sense, as the image data (you email message) is potentially only loaded when the user clicks on a specific email to read it (disregarding preview pane). Storing the image data in a separate page allows more rows to be stored in one page and hence less IO is needed to read a large set of data. However, in BizTalk, when a message is delivered to a subscribing service, that message is going to be "processed" and hence loaded every time (we do special handling for large messages to fragment them already). So instead of an optimization, we incur an extra IO to lookup the image data in the separate data page. I don't gaurantee it will always provide a performance benefit, but we have seen some decent gains when processing small messages. As always you should test this out in a QA environment first. You can read more about text in row simply by searching for "SQL SERVER text in row" and picking one of the links. Text in row is enabled at the table level, so you need to have some understanding of our table structures. Really, there are only two tables we need to worry about. The first table is the Spool. This table contains for every message, its MessageID, a timestamp, some other properites, and the blob form of the message context. The second table you need to worry about is called the parts table. This table contains both the part property bag (for things like content type and charset) as well as the first fragment for each part. Enabling text in row on these tables could provide a benefit for high throughput systems which are processing smaller messages. Since the size of a page can only be 8K, if your messages are greater than this, enabling text in row probably won't help since sql will be forced to push the data to a separate page anyways (although we do compress your data so it is smaller in storage than you might think). You can also play around with the max size for the image column before it is offloaded to a separate page (https://msdn2.microsoft.com/en-us/library/ms173530.aspx). As always, have someone with database experience read over this and understand it before testing anything out. A quick script to turn on the option would look like:

exec sp_tableoption N'Spool', 'text in row', 'ON'

exec sp_tableoption N'Parts', 'text in row', 'ON‘

So the conclusion is that you should never do anything which effects the way our queries are executed but you are more than welcome to try and optimize the IO access to improve performance. Hope this helps some of you out there.

Lee

Addendum: 

I was asked about Index Rebuilds. First, we do not support online index defragmentation. That process involves page locks which we explicitly disable as part of our installation on all of our tables. Page locks make lock acquisition ordering almost impossible and will cause us to deadlock all over the place. We are very carefull about how we acquire locks and expect that the granularity is at the row level so that our ordering is enforced. Having said that, offline rebuilding of indexes is perfectly okay and supported. In our labs, we have not seen significant performance gains from doing this but it is true that our indexes will fragment since many of them are on guids (most). SQL Server team is not a huge fan of guid based indexing, but we have done numerous tests which show that as long as you don't ever scan the table, they can perform better than identity based indexes (for our specific workloads). They will of course get fragmented, but usually our tables don't get too big as data flows in and out of them at a relatively steady pace. If you have large amounts of data which is expected to build up in the messagebox, you are more than welcome to periodically do index rebuilds during scheduled downtime. The same applies to the tracking database where you can rebuild indexes during downtime.

https://www.gotdotnet.com/codegallery/codegallery.aspx?id=67bbd6ea-850e-4d93-be87-df6788976cab

This can be a very usefull tool when used in association with the Archiving and Purging features of BizTalk 2006 which will also be available in BizTalk 2004 SP2. Now you can schedule regular archiving of your tracking database and then reconstruct one single database against which you can mine your data. Thanks to Vishal for putting together this tool for everyone.

For those of you heading to Boston for Tech-ed, drop by the Connected Systems area and say hi. I will be there all week and am doing a co-presentation on Wednesday (my bit is on performance) and have a chalk-talk on Tuesday on Ordered Delivery. Hope to see a lot of you there.

Lee 

My role on the team has changed a bit and now I have a team to help work on all the great features. Sumitra S has been working with me for a while now and I am going to work on getting her to post to this blog as appropriately. She rocks and along with a bunch of work on the messagebox core functionality she wrote the (internal) OM which exposes all of the data and operations in the Admin MMC Group Hub Page. Also joining us now is Adrian Hamza after working on the Share Point adapter most recently He has his own blog which you should all checkout at https://blogs.msdn.com/ahamza. He is great and we are really looking forward to his work on the core engine. Craig C has also joined the team and will be helping define some of the long term direction we are going towards. Should make an awesome team. :) :) :)

So now maybe I will give you some good technicall details so that you are not just reading fluff. :) One thing I have really wanting to post are some learnings I have had with using Yukon and 2 tips that could really help some people avoid the gotchas that we hit when porting code directly that must run on both Yukon and Shiloh (I'd post more but it is 1 am).

1) After a number of conversations with the optimizer team, we discovered a hidden gem in the use of WITH SCHEMABINDING on UDFs. Our UDF simply took in three datetime variables and did a somewhat complex comparison of them and returned 1 or 0. It did not touch any tables and so did not affect any data. In Yukon there is a new feature / attribute associated with a UDF called SystemDataAccess. This property can be seen via:

SELECT OBJECTPROPERTYEX(OBJECT_id(‘<MyFunction>’), 'SYSTEMDATAACCESS')

Without the WITH SCHEMABINDING option, this property is always set to 1 so the optimizer assumes that you might access the data and will add sort / spool to your plan to protect itself from data changes. Adding the schemabinding hint, causes the function to be parsed when it is created and then you can see this property set to 0 and the sort / spool disappears. It is really quite nice and can be a very large perf improvement which I believe will be KB’d at some point. In our case, this is a major gain in the performance of dequeue for large queue sizes. Thanks to the SysRepublic guys for discovering the perf issue with large queues and triggering this discovery. You will find that our peformance for large queues is better on Yukon in some scenarios because of this .

2) Always owner qualify tables and sprocs / udfs which you execute. SQL plan cache access actually uses this as part of its lookup in the cache and missing this can cause a certain amount of extra cache misses and plan (re)compiles resulting in higher CPU utilization and potentially lower performance. What I mean is instead of "SELECT * FROM Foo" you would write "SELECT * FROM dbo.Foo" assuming the owner of Foo is the dbo. Unfortunately this is something we found too late and it actually didn't make it into our RTM bits as it was not a functional bug and was such an extensive change that we could not risk our RTM date since we had reached our perf goals. We are pushing to QFE this, though, and I will let you know if / when this happens. All it means is that in certain scenarios, on Yukon, we would use more CPU than on Shiloh.

These two things are just so wierd that I figured I would post about them as case one could be a big gain for those who do this (which might not be a lot of you) and case two is probably all of us and these are fixes that can do which have no effect on Shiloh but can have large gains on Yukon. Hope this helps some of you and I expect the Yukon guys will KB this stuff at some point (as well as investigating why #2 is currently worse than Shiloh).

Next time, I will post something cool about BizTalk, but now I have a coulpe more things to catch up on before I go to sleep. :)

Thx

Lee

We have several open positions in BizTalk team. We just started planning our next release of BizTalk server and have a lot of exciting ideas and opportunities. Send email to kerryv@microsoft.com if you want to participate in building next generation of the BizTalk server, or know someone who can be interested in growing his/her career in Microsoft.

Thx

Lee

Lee

We’ve been working very hard over the past year to connect with folks just like you and want to include you in our community of Most Valuable Professionals (MVP), developers, information technology professionals, chief information officers, chief executive officers, or any other role within large, medium, and small companies that hang out in our online newsgroup communities. We want to have you join in this vibrant online community to ask those questions you always wanted to ask but did not know where to go. Well, now you know where to go, we want you to come on in and join us!

If you are new to BizTalk Server, try out the NewUser newsgroup, Microsoft.public.biztalk.newuser.

We’re offering two levels of interaction with Microsoft Corporation employees as follows:

1. Managed Newsgroup Support
2. Unmanaged Newsgroup Support

Managed Newsgroup Support

MSDN managed newsgroups are available in English to MSDN Universal, Enterprise, Professional and Operating Systems subscribers to receive free technical support on select Microsoft technologies as well as to share ideas with other subscribers. MSDN managed newsgroups provide:

• Unlimited on-line technical support - keep your PSS incidents
• A commitment to respond to your post within two business days
• Over 200 newsgroups to choose from
• Spam protection for your e-mail address when posting items

Go to the following URL to sign up: https://msdn.microsoft.com/newsgroups/managed .  These newsgroups are monitored by Microsoft support engineers and product group team member as described above.

Unmanaged Newsgroup Support

MSDN unmanaged newsgroups are available to all individuals.

Go to the following URL to participate: https://msdn.microsoft.com/newsgroups. These newsgroups are monitored by Microsoft product group members, other customers like you, most valuable professionals, and various other individuals.

Questions, suggestions, and direct feedback can be sent to me.

James Fort

BPI Community Lead

mailto:jfort@microsoft.com 

Thx
Lee