User realm discovery failed. System: The remote server returned an error: (404) Not Found.

DataSource=asazure://westus.asazure.windows.net/tabularservername;Persist Security Info=True;User ID=serviceaccountname@contoso.com;Password=highlycomplexpassword

Log shipping which was running for an year, suddenly stopped working. Log shipping monitor says that there are no backups restored for last 14849 minutes... Why?

We started figuring this out, the copy and restore jobs were running fine but only the backup job was failing.

What was the error generated by the Log shipping backup job?

*** Error: Could not retrieve backup settings for primary ID 'abfb9a42-207e-451b-968c-3ac29edc16f9'.(Microsoft.SqlServer.Management.LogShipping) ***
*** Error: Failed to connect to server SQLSERVER2001.(Microsoft.SqlServer.ConnectionInfo) ***
*** Error: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)(.Net SqlClient Data Provider) ***
----- END OF TRANSACTION LOG BACKUP -----

But the name of the SQL Server name is SQLSERVER2012, so why is it looking for SQLSERVER2001? This is because the server was renamed from SQLSERVER2001 to SQLSERVER2012 recently but the sys.servers was not updated with this change.

As a result, SELECT @@SERVERNAME still points to old name so we ran commands given below to update sys.server entry and then RESTARTED SQL Server:

EXEC sp_dropserver 'Your_OLD_Computer_Name'

GO

EXEC sp_addserver 'Your_NEW_Computer_Name', 'local'

GO

Now we reconfigured log shipping to make it work. You can either do this or update the Log shipping backup with correct SQL Server instance name or create a SQL Server alias pointing back to new server name to make it work. Still not working? Leave the error message in the comments here....

Reference:

Rename a Computer that Hosts a Stand-Alone Instance of SQL Server - http://msdn.microsoft.com/en-us/library/ms143799.aspx

We are using SQL DMV's to find out the number of NUMA nodes and Powershell query to find out the number of processor cores and then use the recommendations given in http://support.microsoft.com/kb/2023536 to give a MAXDOP value to be set in SQL Server.

As stated in this KB:

"...The maximum value of 8 provided in the above guideline is applicable for typical SQL Server activity and the overhead for the exchange operators used in parallel query plans. You could vary this maximum value depending upon your specific application patterns and concurrent activity on the SQL Server.

For example:- 

If you have very small number of concurrently executing queries relative to the number of processors, then you can set maxdop to a higher value like 16.
If you have very large number of concurrently executing queries relative to the number of processors, then you can set maxdop to a lower value such as 4. 
Any value you propose to use should be thoroughly tested against the specific application activity or pattern of queries..."

You can then follow http://msdn.microsoft.com/en-us/library/ms189094.aspx and set Max. Degree of Parallelism in SQL Server.

Please note that we detect the number of cores available in the Server from OS perspective, so if you have limited the number of processors to SQL Server, this calculator will not be able to adjust the values accordingly so may not be useful to you.

This calculator assumes that there are equal number of processors per NUMA node.

This calculator works for SQL Server 2005 & above running on Windows Server 2008 and above. To make this to work on Windows 2003, please install the hotfix mentioned in http://blogs.technet.com/b/mapblog/archive/2012/02/03/why-doesn-t-the-map-toolkit-report-the-number-of-cores.aspx

Calculator: http://blogs.msdn.com/b/sqlsakthi/p/maxdop-calculator-sqlserver.aspx

Keep your feedbacks coming through comments section in this blog post...

Sakthivel Chidambaram, Microsoft

since I'm not able to invest my time

to improve it to make it "stay current".

Apologize for the inconvenience!

Hello SQL Community,

I have written a simplified “Max. Server Memory” calculator for calculating "Max. Server Memory" value for 64-bit SQL Server instances after seeing my blog referenced in MSDN forums, blogs etc., to automate the calculation of Max. Server Memory.

Few things to take care before you use this calculator:

IMPORTANT: This calculator is designed to help Novice SQL Database Administrators or someone who manages SQL Server but is really not an expert in SQL Server. So to put it simple, this calculator is meant for SQL Server professionals with Level 100 knowledge in SQL Server

Based on my support experience, lot of performance & memory related problems are because customer’s don’t set a Max. Server Memory and they don’t do that because they don’t know what to set. This is my personal interest to help those people who are in need but confused because there are so many recommendations around how to calculate & set this value.

Lot of other things like software based replication, HBA card, Fusion IO card, Antivirus Real-time scanning, Firewall monitoring the traffic flow through ports etc., need good amount memory which is not accounted here simply because we don't see these in all the Environments. If you use anything other than the features given in the calculator here, please make sure you adjust the Max. Server Memory accordingly!!!

So important thing is, take the Max. Server Memory value recommended here as a reference value and then manipulate the value according to your custom environment and then configure Max. Server Memory in SQL Server accordingly. This is not all, please refer to following instructions on how to monitor memory usage of SQL Server & OS and make sure you monitor the performance of the server during peak load to see whether the value is right or needs a change so that you don't hurt  the performance of other applications & Operating System!!!

This calculator will not help on server having more than 1 instance of SQL Server because the memory required for OS, SSIS, Antivirus etc., are shared so the calculation becomes little complex.

Max. Server Memory value shown here is not to be assumed as an official Microsoft recommendation

This calculator is applicable only for 64-bit SQL Server

Things have changed for SQL Server 2012 so this calculator is applicable only for SQL Server 2005, 2008, 2008 R2

If you think you know how to arrive at this value better than what is shown in the calculator, you can share the feedback in the comments section or skip the recommendations because you are not the right audience to use this calculator.

Max Worker thread calculation used to arrive at memory requirement for worker threads is based on Books On Line - http://bit.ly/KflAa8

 You can monitor the health of SQL Server Memory using these Performance monitor counters:

Page Life Expectancy Counter under SQL Server Buffer Manager which is the average number of seconds a page stays in cache. It should be atleast = 300 * (Total RAM in GB)/4. Though our recommendation earlier was PLE should be 300, if a page is staying just for 5 minutes on a server with 128 GB RAM then it is not a right value to say that there is no memory pressure.
Memory Grants Pending, in the Perfmon object SQL Server Memory Manager (Close to 0 or lower is better) (OR)
Memory grant queue waits in the Perfmon object SQL Server Wait Statistics Object (Close to 0 or lower is better)

To Monitor the health of memory available for other processes & OS:

Available MBytes under Perfmon object Memory - Refer http://blogs.msdn.com/b/slavao/archive/2006/11/13/q-a-does-sql-server-always-respond-to-memory-pressure.aspx for some reference values

More information on troubleshooting memory bottleneck is available at http://technet.microsoft.com/en-us/library/cc966401.aspx

Max server memory setting only controls size of the Buffer Pool and hence doesn't affect allocations going through Multi Page Allocator (MPA). You can monitor your  Multi Page Allocator usage by using sys.dm_os_memory_clerks DMV and adjust the Max. Server Memory accordingly.

So here we go

Calculator: http://blogs.msdn.com/b/sqlsakthi/p/max-server-memory-calculator.aspx

UPDATE: Per feedbacks from MVP's over twitter and blogs, modified the calculator to give more meaningful value for memory required by Operating System since we see more systems with high Physical memory loaded.

Formula used to calculate memory required for OS: 2GB + 12% of physical memory. (Thinking of how or why this value? It is 2 GB base and 7% for kernel pool (paged pool, NPP and PFN database) and 5% for cache.)

Technology used: Office Web Apps (Try it for free) & MSDN Static Pages.

You can download the Excel to view the formula used to arrive at the Max. Server Memory value.

Note:

Number of processors visible to SQL Server may not be equal to number of processors available in the server. Thinking why? these links will help:

http://blogs.msdn.com/b/psssql/archive/2012/01/23/assigning-sql-server-sql-agent-to-a-processor-group-oom-hang-performance-counters-always-zero-for-buffer-pool.aspx

http://blogs.msdn.com/b/psssql/archive/2009/06/16/sql-server-on-windows-7-windows-2008-r2-with-more-than-64-cpus.aspx

Post your feedback and comments in this blog post!

You may also try MAXDOP calculator available at http://blogs.msdn.com/b/sqlsakthi/archive/2012/05/24/wow-we-have-maxdop-calculator-for-sql-server-it-makes-my-job-easier.aspx

Sakthivel Chidambaram, Microsoft

I encourage you to upgrade to SQL Server 2012 using this guide (HOW TO UPGRADE), What’s New whitepaper (WHY UPGRADE) downloadable from http://download.microsoft.com/download/E/9/D/E9DBB1BB-89AE-4C70-AF02-AAFC29451A85/SQL_Server_2012_Whats_New_White_Paper.pdf and other resources in the web site http://www.microsoft.com/sqlserver/en/us/product-info/why-sql-server.aspx

So what happens when a transaction rollback fails? Database will shutdown automatically and restart itself to perform the Recovery.

Here is a sequence copied from SQL Server errorlog to explain what happens

Error: 802, Severity: 17, State: 20.
There is insufficient memory available in the buffer pool.

Error: 3314, Severity: 21, State: 4.
During undoing of a logged operation in database 'SAKTHIDB', an error occurred at log record ID (494194:2565:21). Typically, the specific failure is logged previously as an error in the Windows Event Log service. Restore the database or file from a backup, or repair the database.

Database SAKTHIDB was shut down due to error 802 in routine 'XdesRMReadWrite::RollbackToLsn'. Restart for non-snapshot databases will be attempted after all connections to the database are aborted.

Recovery of database 'SAKTHIDB' (5) is 0% complete (approximately 2968 seconds remain). Phase 1 of 3. This is an informational message only. No user action is required.

Recovery of database 'SAKTHIDB' (5) is 98% complete (approximately 19 seconds remain). Phase 3 of 3. This is an informational message only. No user action is required.

2 transactions rolled back in database 'SAKTHIDB' (5). This is an informational message only. No user action is required.

Recovery completed for database SAKTHIDB (database ID 5) in 937 second(s) (analysis 15261 ms, redo 889025 ms, undo 31435 ms.) This is an informational message only. No user action is required.

The “..Database SAKTHIDB was shut down due to error...” tells you what was the error that caused the database to shutdown in our case was due to error 802 (insufficient memory) and also tell the routine name which in our case is RollbacktoLSN. It also tells you that Restart of all non-snapshot databases will be attempted which happened in our case and eventually recovery completed.

There is also an error printed by the failed routine with error 3314 stating undoing of a logged operation failed. Refer http://msdn.microsoft.com/en-us/library/ff713991(v=sql.100).aspx for more information on 3314 errors.

Hope that Helps!

Sakthivel Chidambaram, SQL Server Support

An extended error has occurred.
Failed to save Local Policy Database.

This happens only when you try to add virtual accounts like

NT SERVICE\MSSQLSERVER

NT SERVICE\SQLSERVERAGENT

to grant certain privileges like Act as part of the operating system, Increase quotas, replace process level token, Log on as a batch job to the SQL Server Service SID and SQL Server Agent Service SID.

Granting privileges to Service SID was introduced in SQL Server 2008 since it was a pain in SQL Server 2005 when somebody accidentaly deleted the domain or local groups created by SQL Server to which all the permissions were granted. Read more about this here http://msdn.microsoft.com/en-us/library/cc280562.aspx

This error is a known issue w.r.t Group Policy Editor since it was not able to handle these virtual accounts. Fix for this issue is available through Windows hotfix http://support.microsoft.com/kb/2491998

Workaround is to use the ntrights utility available as part of Windows 2003 Resource Kit. You can use the Windows 2003 version of ntrights to do this in Windows 2008 or simply apply the hotfix to take care of granting privileges to the virtual accounts in the group policy editor.

Sample syntax to grant "Replace process level token" to SQL Server Service SID & SQL Server Agent Service SID:

ntrights +r SeAssignPrimaryTokenPrivilege -u "NT SERVICE\MSSQLSERVER"
ntrights +r SeAssignPrimaryTokenPrivilege -u "NT SERVICE\SQLSERVERAGENT"

Check the link http://msdn.microsoft.com/en-us/library/windows/desktop/bb530716(v=vs.85).aspx for the complete list of privileges to Privilege Name mapping

Here is how the SQL Server errorlog looked like

2012-04-19 03:43:26.50 Server Microsoft SQL Server 2005 - 9.00.4035.00 (Intel IA-64)
Nov 24 2008 12:59:53
Copyright (c) 1988-2005 Microsoft Corporation
Enterprise Edition (64-bit) on Windows NT 5.2 (Build 3790: Service Pack 2)
...
...
2012-04-19 03:43:26.52 Server Registry startup parameters:
2012-04-19 03:43:26.52 Server -d J:\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\master.mdf
2012-04-19 03:43:26.52 Server -e J:\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\ERRORLOG
2012-04-19 03:43:26.52 Server -l J:\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\mastlog.ldf
...
...
2012-04-19 03:43:26.52 Server -T 834
...
...
2012-04-19 03:43:26.53 Server Large Page Extensions enabled.
2012-04-19 03:43:26.53 Server Large Page Granularity: 16777216
2012-04-19 03:43:26.57 Server Large Page Allocated: 32MB
2012-04-19 03:43:26.59 Server Large Page Allocated: 32MB
2012-04-19 03:43:26.69 Server Using large pages for buffer pool.
2012-04-19 03:48:27.12 Server Service control: stop before startup

So I checked with another server on what should be the next step in the SQL Server error log after "Using larges pages..." since that's where SQL Server seems to have failed and it is supposed to be

"xxxxx MB of large page memory allocated"

So it seems like large page memory allocation failed. When we look at the startup parameters, we see trace flag 834 and this trace flag forces SQL Server to use large pages for Buffer pool memory allocations.

As per http://support.microsoft.com/kb/920093:

"... Trace flag 834 may prevent the server from starting if memory is fragmented and if large pages cannot be allocated. Therefore, trace flag 834 is best suited for servers that are dedicated to SQL Server...."

So in our case there was already another SQL instance running on the same node blocking this instance in allocating Large pages for buffer pool so we gave options to Cx to decide removing 834 trace flag or add more memory.

This could also happen if the large page allocation is taking longer time causing timeout. Note that server we troubleshooted is on a clustered environment.

Sakthivel Chidambaram, SQL Server Support

When you are restoring a database (in my case a database having 32 files), post 90% complete, all but one threads in sysprocesses went into SUSPENDED state waiting on BACKUPTHREAD.

One thread was in runnable state with a wait time of close to 6 hrs. and the wait type was PREEMPTIVE_OS_WRITEFILEGATHER

A quick search got me http://support.microsoft.com/kb/2091024 which explains the problem which in our case was a 380 GB LDF file and SQL Server was busy in zeroing out the file.

So all we have to do was to wait for it to complete

But how to prevent this situation or what can be done if you are in this situation?

1. Monitor the size of your LDF and %space used within the LDF using DBCC SQLPERF('logspace') to make sure you don't have a really big LDF file with just 2% space used. You should remember that when you take a backup of this database which has say 300 GB LDF file with 10% used space, still SQL Server has to create a 300 GB LDF file during Restore and zero out 270 GB of disk space so this is going to take lot of time.

2. If you are already in this situation and if you still have access to the source database which you took backup, try shrinking the LDF and take the backup again to perform the restore. This should be a quick process rather than waiting for zeroing to complete

3. Stop any activity (if possible) to free-up the target disk's header so that it can dedicatedly work on zeroing out the LDF file

FYI: This Instant File Initialization is not applicable to LDF which means SQL Server has to zero out the disk when LDF file is expanded to new offset. But you can prevent zeroing out in MDF file but granting "Perform Volume Maintenance Task" privilege to SQL Server service account. Reference: http://msdn.microsoft.com/en-us/library/ms175935.aspx

More info:

To see which files are being zeroed out, you can use the undocumented trace flag 3004 and the undocumented xp_readerrorlog. Undocumented means it is unsupported and may change or be removed with no notice at any time. Trace flag 3004 shows information about backups and file creations. Trace flag 3605 redirects the output to the SQL error log.

WARNING: These trace flags should be used under the guidance of Microsoft SQL Server support. They are used in this post for discussion purposes only and may not be supported in future versions.

DBCC TRACEON(3004,3605,-1)

GO

Somebody messed up the data at publication database (few rows updated with a value which they aren't supposed to) and publication database was restored from a older backup to rollback the change. Now when log reader agent was started it starting failing with an error:

2012-03-14 01:08:14.360 Publisher: {call sp_repldone ( 0x00159ebb000031250001, 0x00159ebb000031250001, 0, 0)}
2012-03-14 01:08:14.376 Status: 0, code: 20011, text: 'The process could not execute 'sp_repldone/sp_replcounters' on 'SQLPROD'.'.
2012-03-14 01:08:14.423 The process could not execute 'sp_repldone/sp_replcounters' on 'SQLPROD'.
2012-03-14 01:08:14.470 Repl Agent Status: 6
2012-03-14 01:08:14.517 Status: 0, code: 9003, text: 'The log scan number (1416891:12581:1) passed to log scan in database 'DBNAME' is not valid. This error may indicate data corruption or that the log file (.ldf) does not match the data file (.mdf). If this error occurred during replication, re-create the publication. Otherwise, restore from backup if the problem results in a failure during startup. '.
2012-03-14 01:08:14.579 Status: 0, code: 22017, text: 'The process could not set the last distributed transaction.'.
2012-03-14 01:08:14.610 Status: 0, code: 22037, text: 'The process could not execute 'sp_repldone/sp_replcounters' on 'SQLPROD'.'.

-- DBCC OPENTRAN in the publication database shows:

Replicated Transaction Information:
        Oldest distributed LSN     : (1416699:339145:57)
        Oldest non-distributed LSN : (1416699:341745:3)

-- So distribution database has more information about the publication database than what publication database has now because it was restored from a older backup. To be precise:
Publication database says it has sent this LSN                                -->         1416699:339145:57 (Oldest distributed LSN)
Publication database says it has to start sending from this LSN      -->         1416699:341745:3 (Oldest non-distributed LSN)

But Log reader agent says I have to start from here                        -->         1416891:12581:1 (High value comapred to what publication database says)

So how do you handle this situation? Well here is the guide:

Option 1:

The Log Reader Agent can be recovered by using the sp_replrestart stored procedure, which re-synchronizes the metadata between the Publisher and the Distributor. This can result in Subscribers potentially having more data than the Publisher. In the event that these changes are made to the publication database again and subsequently replicated, the Distribution Agent will fail with data consistency errors. To override these errors use the Distribution Agent profile “Continue on data consistency errors.”
Note:  Care should be taken when overriding these errors. For more information, see Skipping Errors in Transactional Replication in SQL Server Books Online.

How if works? sp_replrestart will fill NO-OP (No-Operation) transaction which will cause the LSN's to increase until the log LSN matches the LSN as per Distribution database. So from there-on, Logreader agent will be able to read the LSN, it is expecting.

Note: Depending on how old the backup of the Published database which was restored, it may take hours for this operation and may make the transaction log to grow big.
and grow by gigs, until they match.

Caution:  This option (Option 1) can lead to inconsistencies between data at the Publisher and data at the Subscribers!!!

Option 2:
--> Stop all applications accessing the publication database.

--> Run the Distribution Agent until all Subscribers are synchronized with the outstanding commands in the distribution database. Verify that all commands are delivered to Subscribers by using the Undistributed Commands tab in Replication Monitor or by querying the MSdistribution_status view in the distribution database.

--> Use the tablediff Utility or another tool to manually synchronize the Publisher with the Subscriber, which allows you to recover data from the subscription database that was not contained in the publication database backup.

--> Drop the subscription(s).

--> At the Publisher on the publication database, execute sp_addsubscription. Specify the name of the database at the Subscriber containing the published data for @destination_db, a value of push for @subscription_type, and a value of 'replication support only' for @sync_type.

--> At the Publisher on the publication database, execute sp_addpushsubscription_agent or sp_addpullsubscription_agent and start the distribution agent.

--> Now you can allow changes to the publication database.

 How it works? 'replication support only' in the @sync_type says that we already have data and schema at the subscriber and we are asking replication to just create stored procedures and other replication objects and start synchoronization.

Option 3: (Recommended)

Script the Publication -> Drop Subscriptions -> Drop Publication -> Re-create them

You can use "Sync with Backup" to avoid these scenarios but at the expense of high log reader latency depending on the frequency of the log backup at the publication database. Read more about "sync with backup" option from http://msdn.microsoft.com/en-us/library/ms152560.aspx

Please comment if you tried this and whether it helped you or you faced some issues.!

Sakthivel Chidambaram,

SQL Server Support, Microsoft