Failing to restore SQL Server database backup with internal consistency error.

ERROR
======
5 percent restored.
10 percent restored.
15 percent restored.
Server: Msg 3270, Level 16, State 1, Line 1
An internal consistency error occurred. Contact Technical Support for assistance.
Server: Msg 3013, Level 16, State 1, Line 1
RESTORE DATABASE is terminating abnormally.

Troubleshooting
===============
-- Customer had SQL 7 backup and trying to restore the database and failing to restore and giving some internal consistecny error.
-- It passes all the checks (Verify Header and Filelistonly) however when trying to restore.
-- CU do not have the DB anymore in production.
-- They have mdf and ldf but they were dettached after the database went into suspect mode.

-- Ran Restore headeronly from disk = backup file name, this came with below output.

BackupName      BackupType ExpirationDate     Compressed Position DeviceType UserName                  ServerName     DatabaseName      DatabaseVersion DatabaseCreationDate               BackupSize            
--------------- ---------- ------------------ ---------- -------- ---------- ------------------------- -------------- ----------------- --------------- ---------------------------------- ----------------------
NULL            1          NULL               0          1        2          <User Name>         <Servername>     <DBANme>         515             2009-01-02 13:54:13.000            2516844032            

-- Restore Verifyonly -- Gave Valid backup Set.
-- restore filelistonly -- Gave the result.
-- This was giving indication that the backup is good but still it was giving some inconsistency issue.
-- Looks like the Database backup was taken post the database encoutnered corruption.
-- Connected to SQL 2005 instance and try restoring the database from the backup here.
-- We saw that everytime we try to restore the DB using the back file, we saw the DB files getting created but it was failing to complete the restore.
-- Ran the restore using WITH CONTINUE_AFTER_ERROR paramter, this continued file with no errors.
-- This completed fine but it detected inconsistency error below:

5 percent processed.
10 percent processed.
15 percent processed.
20 percent processed.
25 percent processed.
30 percent processed.
35 percent processed.
40 percent processed.
45 percent processed.
50 percent processed.
55 percent processed.
60 percent processed.
65 percent processed.
70 percent processed.
75 percent processed.
80 percent processed.
85 percent processed.
90 percent processed.
95 percent processed.
Processed 307200 pages for database 'DBName', file 'DBName_Data' on file 1.
Processed 1 pages for database 'DBName', file 'DBName_Log' on file 1.
100 percent processed.
Damage to the backup set was detected.
RESTORE WITH CONTINUE_AFTER_ERROR was successful but some damage was encountered. Inconsistencies in the database are possible.
RESTORE DATABASE successfully processed 307201 pages in 142.521 seconds (17.657 MB/sec).

-- Checked SQL Server error from file.

2010-03-15 11:41:57.15 spid63      Starting up database 'DBName'.
2010-03-15 11:41:57.20 spid63      The database 'DBName' is marked RESTORING and is in a state that does not allow recovery to be run.
2010-03-15 11:41:58.79 spid63      SQL Server has encountered 1 occurrence(s) of cachestore flush for the 'Object Plans' cachestore (part of plan cache) due to some database maintenance or reconfigure operations.
2010-03-15 11:42:02.09 spid63      SQL Server has encountered 1 occurrence(s) of cachestore flush for the 'SQL Plans' cachestore (part of plan cache) due to some database maintenance or reconfigure operations.
2010-03-15 11:42:02.24 spid63      SQL Server has encountered 1 occurrence(s) of cachestore flush for the 'Bound Trees' cachestore (part of plan cache) due to some database maintenance or reconfigure operations.
2010-03-15 11:42:32.23 spid63      Error: 3183, Severity: 16, State: 1.
2010-03-15 11:42:32.23 spid63      RESTORE detected an error on page (0:0) in database "DBName" as read from the backup set.
2010-03-15 11:44:12.56 spid63      Starting up database 'DBName'.
2010-03-15 11:44:12.58 spid63      The database 'DBName' is marked RESTORING and is in a state that does not allow recovery to be run.
2010-03-15 11:44:13.22 Backup      Database was restored: Database: DBName, creation date(time): 2009/01/02(13:54:13), first LSN: 12375:571:29, last LSN: 12375:582:1, number of dump devices: 1, device information: (FILE=1, TYPE=DISK: {'D:\DIR\Backup\DBName_20100310_214921.bak'}). Informational message. No user action required.

-- In SSMS the status of the DB was restoring.
-- Ran --> Restore database with recovery, but failed with below error:

Server: Msg 4323, Level 16, State 2, Line 1
The database is marked suspect. Transaction logs cannot be restored. Use RESTORE DATABASE to recover the database.
Server: Msg 3013, Level 16, State 1, Line 1
RESTORE DATABASE is terminating abnormally.

-- Stopped SQL 2005 instabnce.
-- Copied the mdf and ldf file for this restoring state Database.
-- Dropped the DB in restoring state.
-- Tried to CREATE DATABASE DBNAme ON (Filename ='D:\File.mdf') FOR ATTACH_FORCE_REBUILD_LOG
-- This again failed with below error.

File activation failure. The physical file name "F:\Database\DBName\DBName_Log.LDF" may be incorrect.
New log file 'D:\MSSQL\SQL2005\Logs\DBNAme_log.LDF' was created.
Msg 1824, Level 16, State 1, Line 1
Cannot attach a database that was being restored.

-- Here I noticed that the log file was getting created on F Drive and we did not had Drive: F on the box.
-- Mapped another drive with letter F: and created the folder structure, but same error.

-- Tried EXEC sp_attach_single_file_db but failed with

File activation failure. The physical file name "F:\Database\DBName\DBName_Log.LDF" may be incorrect.
The log cannot be rebuilt because the database was not cleanly shut down.
Msg 1813, Level 16, State 2, Line 1
Could not open new database 'DBNameDBName'. CREATE DATABASE is aborted.

-- Now tried the sp_attach_db command but failed again.

EXEC sp_attach_db @dbname = N'AdventureWorks',
    @filename1 = N'C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\Data\AdventureWorks_Data.mdf',
    @filename2 = N'C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\Data\AdventureWorks_log.ldf';

Msg 1824, Level 16, State 1, Line 1
Database cannot be restored.

-- Copied the original Data and Log files.
-- And tried attaching the files here.

Msg 1813, Level 16, State 2, Line 1
Could not open new database 'DBName'. CREATE DATABASE is aborted.
Msg 9004, Level 21, State 1, Line 1
An error occurred while processing the log for database 'DBName'.  If possible, restore from backup. If a backup is not available, it might be necessary to rebuild the log.

RESOLUTION
==========
-- Finally resorted back to Restoring the database from SQL 7.0 backup taken with CONTINUE_AFTER_ERROR paramter.
-- Then ran sp_reset status follwoed by restore with recovery this got the DB in accessible mode and was able to read the table.
-- Ran CHECKDB and this showed inconsistency on user Table.
-- Ran repair and this fixed the corruption, however there was some data loss which could not be recovered.

Here the corruption was in system object 'sys.ifts_comp_fragment_1467152272_6792'. DBCC CHECKDB for the database was coming out with below error message.

ERROR

=======

Table error: Object ID 0, index ID -1, partition ID 0, alloc unit ID 72057596473638912 (type Unknown), page (1:2867044). Test (sorted[i].GetOffset () >= max) failed. Slot 0, offset 0x1 overlaps with the prior row.

CHECKDB found 0 allocation errors and 1 consistency errors not associated with any single object.

Object ID 10391752, index ID 1, partition ID 72057596477833216, alloc unit ID 72057596479799296 (type In-row data): Page (1:2867043) could not be processed.  See other errors for details.

Table error: Object ID 10391752, index ID 1, partition ID 72057596477833216, alloc unit ID 72057596479799296 (type In-row data), page (1:2867281). Test (sorted[i].GetOffset () >= max) failed. Slot 0, offset 0x1 overlaps with the prior row.

Table error: page (1:2867282) allocated to object ID 10391752, index ID 1, partition ID 72057596477833216, alloc unit ID 72057596479799296 (type In-row data) was not seen. The page may be invalid or may have an incorrect alloc unit ID in its header.

Table error: Object ID 10391752, index ID 1, partition ID 72057596477833216, alloc unit ID 72057596479799296 (type In-row data). Page (1:2867283) is missing a reference from previous page (1:2867282). Possible chain linkage problem.

CHECKDB found 0 allocation errors and 8 consistency errors in table 'sys.ifts_comp_fragment_1467152272_6792' (object ID 10391752).

CHECKDB found 0 allocation errors and 9 consistency errors in database 'DBName'.

repair_allow_data_loss is the minimum repair level for the errors found by DBCC CHECKDB (DBNAME, noindex).

ENVIRONMENT

=============

SQL = SQL 2008

Looking at the above error the corruption seems to be on system table, however this system object wasn’t visible in SQL user database, neither on master database.

Checked the error and could see corruption in system table 'sys.ifts_comp_fragment_1467152272_6792'

The problem database was approximately 23 GB in size.

There are 0 rows in 0 pages for object "TableName".

CHECKDB found 0 allocation errors and 9 consistency errors in database 'DBName'.

repair_allow_data_loss is the minimum repair level for the errors found by DBCC CHECKDB (DBName).

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

Checked the complete system and application log however no indication of Server crash or hardware issue.

DBCC CHECKDB on master was clean.

The problem object relates to Full text search and looks meta data for FTS fragments or document information.

Ran below query and this came out with below information. This query checks the fragmentation in objects and their size.

Select * from Sys.fulltext_index_fragments

table_id    fragment_id fragment_object_id timestamp          status      data_size            row_count

----------- ----------- ------------------ ------------------ ----------- -------------------- --------------------

1165247206  37755       1563165360         0x00000000001B9583 4           47043687             210215

1165247206  37813       183684448          0x00000000001B95EE 4           10254                    175

1165247206  37815       199684505          0x00000000001B95F1 4           5201                      89

1165247206  37835       567685816          0x00000000001B961C 4           53                           1

1165247206  37836       583685873          0x00000000001B961F 4           5308                      91

1467152272  6792        10391752           0x00000000001B7002 4           605258819            2133650

1467152272  7302        1547829272         0x00000000001B8632 4           2025872              23044

1467152272  7545        1701593846         0x00000000001B9253 4           1172022              15027

1467152272  7648        1243164220         0x00000000001B9561 4           577875               8024

1467152272  7676        535685702          0x00000000001B9619 4           167683 

Here i got interested information the table and the fragment ID. If we check the table witnessed in the error its suffix is table + Fragment ID

So ran the query, select * from sysobjects where id = 1467152272 and got the object name, this object was then confirmed to be used in FTS catalog

So confirmed that this table "Sys.fulltext_index_fragments" keeps information on the FTS object / index level fragment information and this seems to be corrupt for some reason.

Performed Re-indexing on this user object did not help.

Ran  DBCC CHECKTABLE ('Tablename') for this table and it came clean.

So the corruption is not on the user table but the system reference table.

Took the database backup and dropped the catalog using this table and re-created the catalog successfully.

Re-executed DBCC CHECKDB on the database and this time it came clean.

- When the domain authentication details are provided - in code it looks for the PDC.

- A function by name VerifyPDC is invoked to verify the PDC

- A parameter is passed to the VerifyPDC function that can hold a maximum of MAX_COMPUTERNAME_LENGTH+1, which is as below:

 max 15 characters of PDC name

and a Null terminator (\n)

- This function invokes another function by name DCGetDcName - this will return the name of the PDC.

- In the function DCGetDcName they were padding '\\' to the PDC name.

- So for example if the DC name is 13 characters in length, the computation goes as below:

      13  (length of Domain controller name)

        1  (for null terminator)

        2  (for \\)

     -------

      16 (total of 16 characters can be stored in the stack for that variable)

- If we exceed then the buffer overflow error is expected.

General understanding:

1. Issue occurs when we are upgrading to SP4 on the cluster. When DC's are installed on both the nodes of the cluster and Domain controllers having more than 14 or 15 characters.

2. The workaround specified in the bug - Will work till SP3:

Virtual server setup crashes if the computer name of PDC of the domain contains 14 or 15 characters

The workaround is to use debug version of sqlsui.dll and sqlsut.dll and replace them with the (sqlsui.dll & sqlsut.dll) files.

3. The way it works till SP3 is...

- The variables in debugger version was created with some extra space so it was capable of storing the extra characters.

4. However this workaround doesnt help in SP4.  In SP4 they have introduced a new switch.

5. The functionality of this switch is to throw an exception if there is any stack overflow and the exception is handled and it gives the following error message:

"Buffer Overflow detected"

6. This functionality has been added to avoid any malicious code being executed.

7. In this case, the computer name is 14 characters in length and in the code is padding '\\' and adding the \n terminator reaches to 17 characters which is out of the boundary of stack -

so it generates the above error.

8. The recommendation to fix the issue is - to rename the domain controller name with max of 13 characters in length.

2008-06-04 17:34:29.66 Server      Error: 17053, Severity: 16, State: 1.

2008-06-04 17:34:29.66 Server      UpdateUptimeRegKey: Operating system error 5(Access is denied.) encountered.

.....

2008-06-04 17:34:30.16 spid11s     Failed to check for new installation or a renamed server at startup. The logic for this check has failed unexpectedly. Run setup again, or fix the problematic registry key.

2008-06-04 17:34:30.16 spid11s     Server name is 'WHJPBS12\CRDNTPD01'. This is an informational message only. No user action is required.

2008-06-04 17:34:30.16 Server      Error: 17182, Severity: 16, State: 1.

2008-06-04 17:34:30.16 Server      TDSSNIClient initialization failed with error 0x5, status code 0x51.

2008-06-04 17:34:30.16 spid15s     Starting up database 'model'.

2008-06-04 17:34:30.16 Server      Error: 17182, Severity: 16, State: 1.

2008-06-04 17:34:30.16 Server      TDSSNIClient initialization failed with error 0x5, status code 0x1.

2008-06-04 17:34:30.16 Server      Error: 17826, Severity: 18, State: 3.

2008-06-04 17:34:30.16 Server      Could not start the network library because of an internal error in the network library. To determine the cause, review the errors immediately preceding this one in the error log.

2008-06-04 17:34:30.16 Server      Error: 17120, Severity: 16, State: 1.

2008-06-04 17:34:30.16 Server      SQL Server could not spawn FRunCM thread. Check the SQL Server error log and the Windows event logs for information about possible related problems.

SQL Service starts with Local Administrator account.

It also start with Domain Account with sufficient privileges

Having narrowed down to a permission issue, we granted the below privileges:

• Act as Part of the Operating System = SeTcbPrivilege

• Bypass Traverse Checking = SeChangeNotify

• Lock Pages In Memory = SeLockMemory

• Log on as a Batch Job = SeBatchLogonRight

• Log on as a Service = SeServiceLogonRight

• Replace a Process Level Token = SeAssignPrimaryTokenPrivilege

Still it failed. Took Processmon

It was failing with "Access Denied" on the below registry key

HKLM\SOFTWARE\Wow6432Node\Microsoft\Microsoft SQL Server\CRDNTPD01\MSSQLServer\SuperSocketNetLib\Np

Suggestion

=======

Grant permissions to the below registry keys

HKLM\Software\Microsoft SQL Server

HKLM\Software\MSSQLServer

HKLM\SYSTEM\CurrentConrolSet\Services\<service_name>

HKLM\SOFTWARE\Wow6432Node\Microsoft\Microsoft SQL Server

Grant permissions on the Folder path and the folder (or in a different location- where the data file and the log files are kept)

c:\Program files\Microsoft SQL Server

Grant the below priveleges

• Act as Part of the Operating System = SeTcbPrivilege

• Bypass Traverse Checking = SeChangeNotify

• Lock Pages In Memory = SeLockMemory

• Log on as a Batch Job = SeBatchLogonRight

• Log on as a Service = SeServiceLogonRight

• Replace a Process Level Token = SeAssignPrimaryTokenPrivilege

This should resolve this issue.

MSI (s) (48!C0) [15:00:19:089]: Note: 1: 2262 2: Error 3: -2147287038

MSI (s) (48!C0) [15:00:19:089]: Product: Microsoft SQL Server 2005 -- Error 29512. SQL Server Setup was unable add user sqlserver_service@domain.com to local group Domain\SqlServer_service.

While installing Service pack – if you get the above message –

1.     Check the service account for the SQL, SQL Agent, FT accounts – you can see the service account would be in a sqlserver_service@domain.com format

2.     We need to change that to domain\sqlserver_service format

This will help resolve the issue.

Snapmanager Log

===============

[15:30:19.480] Creating Snapshot of Virtual Disk Drive:

[15:30:19.480] Virtual Disk [F:], Virtual Disk [G:], Virtual Disk [E:]

[15:30:19.480] Snapshot name: sqlinfo__venkn__recent

[15:30:25.202] Snapshot SnapInfo drive(s) successfully completed.

[15:30:25.248] *** VERIFY DATABASE AFTER BACKUP

[15:30:25.248] Collecting verification information...

[15:30:25.248] Getting database backup information from SnapInfo file...

[15:30:25.248] Running verification from local server....

[15:30:25.264] Mounting Snapshot [sqlsnap__venkn__recent] for Virtual Disk E of Computer VENKSAMP

[15:30:41.949] This Snapshot is mounted as the drive [Z:].

[15:30:41.949] Mount Snapshot succeeded.

[15:30:41.980] Using existing connection to server [VENKSAMP]...

[15:30:42.027] Attaching database [VAS__Mount]...

[15:30:42.027] CREATE DATABASE [VAS__Mount] ON PRIMARY (NAME = 'VAS_dat', FILENAME = 'X:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\data\VAS.mdf') LOG ON (NAME = 'VAS_log', FILENAME = 'X:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\data\VAS.ldf') FOR ATTACH

[15:30:43.934] Database attached.

[15:30:43.934]

[15:30:43.934] DBCC CHECKDB (N'VAS__Mount') WITH NO_INFOMSGS, PHYSICAL_ONLY

[16:50:14.443]

[16:50:14.443] DBCC CHECKDB completed successfully.

[16:50:14.443] Detaching database [VAS__Mount]...

[16:50:34.487] Failed to detach the database.

[16:50:34.487] Dismounting Virtual Disk Z of SnapShot [sqlsnap__venkn__recent]...

[16:54:38.657] Dismount Snapshot succeeded.

As a part of backup process, Snapmanager software creates a database snapshot. It verifies the consistency of the backup by running CHECKDB. Once the CHECKDB completes successfully, it detaches the database snapshot.

In some cases it was failing to detach. Later with profiler traces we were able to identify that "AUTO SHRINK" was enabled. Apparently a system process started the shirink operation on the database. Disabling the "AUTO SHRINK" property helped resolve this issue.