Now, the primary replica was running SQL Server 2012 Enterprise Edition. It was hosting several databases whose contents were accessed by a server product which supported SQL Server 2012 but, unfortunately, didn’t support SQL Server 2014 yet. That was an impediment which required them to have the instance hosting the primary replica running SQL Server 2012.

Since they didn’t ever plan to failover the availability group to the secondary, and in the event they accidentally did so they were fully aware of the consequences such action would imply, their question was whether or not such AOAG configuration would be one we support as a long term solution.

Every piece of documentation we had where mixed versions configurations were discussed, always referred to them in the context of a rolling upgrade. However, there was no topic in BOLs, KB article, post in the AlwaysON MSDN’s blog, or other authoritative source where we either stated that such setup was not supported as a permanent solution.

It happens that such configuration is not supported because it is possible to have a secondary replica running on a higher version than the primary, and it can even be configured for readable access, but in practice it cannot be accessed for read only until it has been “promoted” to primary. It’s only at that point when the metadata and physical structures in its files are upgraded to be at the level that version of the engine can work with.

Before having done so (ie having upgraded the version of the database) any attempt to startup the database via a USE statement or a reference to an object in that database from the context of a different database, would fail with error 946 (Cannot open database '%.*ls' version %d. Upgrade the database to the latest version), or a 18456 (Login failed for user '%.*ls'.%.*ls%.*ls) with 18337 (Reason: Failed to open the explicitly specified database '%.*ls'. ) for its reason when attempted via a login with explicit initial database specified in the login payload.

So, a doable alternative for what this customer required, could be setting up transactional replication to move data from the 2012 instance into the 2014 instance.

“We are redirecting read load of our SSIS Packages from Primary to Secondary, and our delta logic is dependent of @@dbts values.

On primary, the value of @@dbts always changes. But the same on secondary doesn’t change. Any experience around this.”

And this is what I answered based on my findings on how this works:

“On the secondary replica, it shouldn’t be static, it should increase but less often and not monotonically as in the primary replica, but in much bigger steps, typically equidistant to one another. This is by design. Let me explain...

@@DBTS is incremented by a builtin function which is incorporated in the query tree produced by the relational engine, while it algebrizes certain DML statements and binds a column of timestamp type.

The boot page of each database stores the high water mark value (dbt_maxDbTimestamp) that database uses for handing out timestamps in its context. When you create a database the initial value for that field is what the model database in that instance has in it (run DBCC DBINFO(model) WITH TABLERESULTS and look for the row where the column “Field” equals “dbi_maxDbTimestamp”.)

Now, during a database startup, initial/current value of DBTS is set to whatever the value of that high water mark is. And the first time a DML statement that includes a node to execute the builtin mentioned above is executed, it will increase the current value monotonically. But, before doing so, it will check whether the high water mark has been reached (which after a database startup is always the case). If the high mark was reached (ie current value equals high mark), it will bump up the value of the high mark and replace the previous value in the boot page with this new one. That modification of the row stored in the boot page gets logged and therefore will eventually be transferred to your secondary HADRON replicas. When the high water mark is bumped up, it is increased by a factor which is 1K times the number of visible schedulers in that instance, so it may well vary from one environment to another.

What happens on a secondary replica is that you begin restoring (with norecovery) a full backup which was taken on the primary replica. If at that point you check what’s the value of @@DBTS on the secondary, it will be whatever it has retrieved from the boot page and loaded into DBINFO structure in the dbt_maxDbTimestamp field. And, since no DMLs can be executed in a secondary  replica, that current value of @@DBTS will never be monotonically increased. However, there might come a point in the primary replica where current DBTS reaches the high mark, in which case it gets bumped up and logged, the log operation gets transferred to the secondary which, when redone, will modify the value of high water mark and also that of DBTS, since it becomes the new current value.

So it means the product is working as it was designed to do, and only we need to know you cannot count on the value returned by @@DBTS to say whether tables with timestamp columns have been affected by changes or not, if those changes come in the form of a transaction log operation which is simply redone. This is the case with non-recovered/standby in-restore databases, log shipped databases, database mirroring mirrors, and Always On Availability Groups secondary replicas.

You’ll have to resort to some a mechanism other than the value returned by @@dbts if you want to detect that changes have occurred on a secondary replica of a database.”

With SQL Server 2014, it is also possible and supported by the Setup code, to do an in-place upgrade of an existing instance of SQL Server 2005 without having to upgrade it first to any intermediate version (2008/2008R2/2012) which would put it in the n-2 range.

The supportability of those upgrade methods (backup/restore, detach/attach, or in-place upgrade) are documented in the “SQL Server 2014 Support for SQL Server 2005” section of the Supported Version and Edition Upgrades topic in he documentation.

And there is a note in it which reads “When a SQL Server 2005 database is upgraded to SQL Server 2014, the database compatibility level will be changed from 90 to 100. (In SQL Server 2014, valid values for the database compatibility level are 100, 110 and 120.)”

Because I had to investigate how this worked internally to provide an authoritative answer to the question asked by a customer who wanted to know what happened with databases he had in instances of SQL Server 2005 whose compatibility level set to 80, I’ll share my findings here.

When a database is upgraded from a previous version to SQL Server 2014, there’s an upgrade process which sequentially executes a chain of upgrade processes to get it from whatever version it was, up to what the storage engine in 2014 is coded to work with.

The more recent the version of the legacy database is the lesser processes will have to be executed during that upgrade phase.

Assuming your starting version is SQL Server 2005, it will execute (as far as the compatibility level upgrade goes):

1. one function (the one that would bring it to the SQL Server 2008/2008R2 level) which upgrades the compatibility level of the database to 80 if it was set to a value lower than that. Or to 100 if the database being upgraded is either TEMPDB or MODEL (these cases only happen during an in-place upgrade)
2. one function (the one that would bring it to the SQL Server 2012 level) which upgrades the compatibility level of the database to 90 if it was set to a value lower than that. Or to 110 if the database being upgraded is either TEMPDB or MODEL
3. one function (the one that would bring it to the SQL Server 2014 level) which upgrades the compatibility level of the database to 100 if it was set to a value lower than that. Or to 120 if the database being upgraded is either TEMPDB or MODEL

So, for example, if your database in SQL Server 2005 had its compatibility level set to 70 or 80, and you don’t bump it up before upgrading it to SQL Server 2014, the upgrade process will do it for you.

Next question I typically get is: How do I know whether any of the databases which I’m trying to protect participates or has participated in one of those two types?

The answer to that question is in the transaction log of the databases you are planning to protect. Whenever a cross-database or distributed transaction is committed, the commit happens in two phases. In the first phase, all involved resource managers (or databases, if it’s a cross-database transaction) participating in the transaction need to vote yes/no to the prepare request sent by the coordinator. For a SQL Server database, the fact that the database voted yes to commit is reflected in its transaction log through the write of a LOP_PREP_XACT operation.

So, if we find at least one record of that type in the transaction log of our database, we can tell for sure, that database has participated in a cross-database or distributed transaction. But, in the event we want to know in more detail, can we tell whether it was one or the other?

The answer is yes.

It happens that the PREP_XACT type of log operation has room for a couple of fields which are only used for cross-database transactions: “Master DBID” and “Master XDESID” (XDESID standing for transaction descriptor identifier.) They include the database id of the one database acting as the coordinator, and the descriptor of the transaction in that coordinating database. Since those two fields are useless in the context of a distributed transaction, in that case they are left empty.

So, if you want to find out whether there is evidence in the transaction log of a given database that it has participated in a cross database transaction, then you can set yourself in the context of that database and run the following batch:

if exists(select * from fn_dblog(NULL, NULL) where Operation = 'LOP_PREP_XACT' and [Master DBID] <> 0)

print 'Based on the active part of the transaction log read, there is evidence that this database has participated in cross-database transactions.'

else

print 'Based on the active part of the transaction log read, there is no evidence of this database having participated in cross-database transactions.'

You can use the following alternative predicate and it will work just the same:

Operation = 'LOP_PREP_XACT' and [Master XDESID] <> '0000:00000000'

As for finding out evidence of your database having transactions which were coordinated by a DTC (aka distributed transactions), run this one below:

if exists(select * from fn_dblog(NULL, NULL) where Operation = 'LOP_PREP_XACT' and [Master DBID] = 0)

print 'Based on the active part of the transaction log read, there is evidence that this database has participated in distributed transactions.'

else

print 'Based on the active part of the transaction log read, there is no evidence of this database having participated in distributed transactions.'

The following WHERE predicate would also do the trick:

Operation = 'LOP_PREP_XACT' and [Master XDESID] = '0000:00000000'

From SQL Server Management Studio, every single entry in the Processes section of the Activity Monitor showed as being classified as the “internal” in the “Workload Group” column, just like the screenshot below shows:

None of the reasons exposed by Bob in How It Works: Resource Governor ‘INTERNAL’ Group seemed appropriate for this particular case, since my colleague was observing it wasn’t a temporary condition at all.

What that post doesn’t explain is that there are two possible conditions under which each and EVERY session is classified as internal for their whole existence. Those two reasons are when the instance monitored corresponds to:

1. an Enterprise SKU with trace flag 8040 enabled via a –T startup parameter, or
2. an SKU for which RG is disabled (ie anything other than Evaluation, Developer, or Enterprise editions)

In either of those two cases, every session is classified in the same internal group, and never changes to any other (whether default or a user-defined one.)

However, the GetRow method of the internal table valued function sysrequests grabs that value and stores it into a CXVariant of type XVT_I4 which corresponds to a 32 signed integer covering the range from -2147483648 through 2147483647.

Therefore the following sequence of bits
01111111 11111111 11111111 11111111
both as a ULONG or a LONG represents the same decimal value: 2147483647.

But adding one to it results in
10000000 00000000 00000000 00000000
which as ULONG represents decimal 2147483648, but as a LONG represents -2147483648.

So, as a workaround, for columns with that problem you can refer to them like this:

case when total_elapsed_time < 0 then 2147483647+(2147483649+total_elapsed_time) else total_elapsed_time end as total_elapsed_time

and that way you always get the correct unsigned positive integer representation of whatever value, even when the DMV returns a negative signed representation.

Thanks,

Nacho

This is a flaw in the UA tool and there’s no current fix for it, neither plans to get it solved in the near future. So, proceed with the workaround which consist of creating a login for the account, and making it a member of sysadmin server role.

For further information about the “Maximum Relational Database Size” limit imposed by every SKU, refer to the list of Cross-Box Scale Limits section of the Features Supported by the Editions of SQL Server 2014 topic in the documentation of the product.