This also means that we do not support external customers using xp_servicecontrol for any work. The code of this XP can be changed anytime and customer should not be using this XP for anything.

The correct alternative in this case would be to either use xp_cmdshell 'SC query <servicename>' or use the WMI namespace. Using WMI provider will be the best way looking into future.

To resolve this, we removed the Microsoft.HostIntegration.DataAccessLibrary.dll file located in the C:\Windows\assembly directory
After this, we were able to resolve the connectivity issue.

But we ran into another issue with applying the initial snapshot in this transactional replication. When the snapshot was applied on the subscriber, the table names, column names and schema name used to have double quotes around them.

The root cause of this was DB2 does not understand the square brackets [] that table names and column names in the schema script produced by snapshot agent have (in the .sql file) and it interprets them to be quoted
After we added the -QuotedIdentifier to the distribution agent's command line, the issue with table name and column names went away.
But we were still left with the issue of double quotes in the schema name. To resolve this issue, we found that when the schema name (this is the db schema name) was specified in the connection string (while setting up subscriber) was replaced with default, the issue went away

> Wireless Network Adapter
> Display Adapter
> Modem
> Mass Storage Controller
> Multimedia Audio Controller

Out of these Wireless Network Adapter and Display Adapter were installed by windows update automatically (yeah, I had to reboot the box couple of times and run windows update multiple times). I think Windows Update has the drivers provided by Intel (which is what the Chipset is) here. For Modem, went into troubleshooter and it suggested to download and install the drivers from Motorola site (it actually gave me the link) and once I click the link to download it, that was installed fine as well (it shows as "Motorola SM65 Speakerphone Modem" in Device Manager)

Bigger problem was finding the right drivers for MultiMedia Audio Controller. I tried installing the drivers that came with the laptop (which was for XP) and I downloaded the latest from Realtek site, but all in vain. I tried uninstalling the one windows by default set there "multimedia audio controller" and then ran window update again and again with no help (the driver installation would always fail). Searched all over the internet and tried every possible forum entries,etc.

To resolve the problem with this, I did the following:

• Visited http://catalog.update.microsoft.com/v7/site/Home.aspx and searched for Contexant AC link audio
• There were 100+ matches, out of the ones that I downloaded, the succeeded one was "Conexant - Sound - Conexant AC-Link Audio"
• Downloaded it and unzipped the cab (you can use winRAR or 7Zip)
• set compatibility mode of HXFSetup.exe from the extracted files to XP SP2 and ran the EXE as administrator
• This made the audio controller fire up

I am left with Mass Storage Controller, but I am not worried about that since I am able to connect and work with my USB pen drives and external hard disks. Wrote the blog post for those who have old v2000 laptops and might have been struggling with similar issues. Please note, what worked for me may not work for you, so keep posting.

Description: 
> There is a high volume replication setup with thousands of subscriber and a very volatile(lot of transactions per second) publisher database
> Started observing huge blocking when multiple merge agents starts synchronizing
> Even if merge agents are stopped, the time taken for a single merge agent is huge
> There are millions of rows in merge metadata tables (msmerge_genhistory, msmerge_contents and msmerge_tombstone) and they are not getting cleaned up
> Running update stats every hour does not help

Cause:
 > The huge backlog in the merge metadata tables was causing the merge queries to take long times. These long times were resulting in blocking when multiple merge agents were sync'ing.
 > This is because the msmerge_genhistory metadata table is referred everywhere during merge sync. Any query that joins this table (and there are a lot of them) had to get blocked since some other queries had taken locks on the same tables rows.
 > The root cause of this problem was the millions of rows those metadata tables were not just getting cleaned up.

What I found out:
 > Once I found that the root cause of the issue is the backlog in the metadata tables, ran sp_mergemetadataretentioncleanup manually to force a cleanup and it returned immediately with 0 rows cleaned.
 > Ran it repeatedly and that did not help.
 > So, its obvious that the merge agents (Note: merge agents run retention based metadata cleanup as the first step during regular sync) were also not cleaning anything from those tables.
 > Further looking into the code of this proc (by running sp_helptext sp_mergemetadataretentioncleanup) found that it checks if any other agent is running sp_mergemetadataretentioncleanup (of it this SP is being manually on the server). If yes, it just skips execusting any code and returns.

<snip>
-- if somebody else is already cleaning up in this database, we simply return
    set @applockname= 'MS_sp_mergemetadataretentioncleanup' + convert(nvarchar(11), db_id())
    exec @retcode= sp_getapplock @Resource= @applockname, @LockMode= 'Exclusive', @LockOwner= 'Session', @LockTimeout= 0, @DbPrincipal = @DbPrincipal
    if @@error <> 0 or @retcode < 0 return (0)
</snip>

 > From DBCC OPENTRAN, found that there was one merge agent which was running sp_mergemetadataretentioncleanup and taking long time but the user might have hit cancel and it was hung while rolling back.
 > This was causing other executions of sp_mergemetadataretentioncleanup to not work and also causing the metadata to keep getting stale causing further performance (and blocking issues)
 > Killed that spid, and stopped all the merge syncs and ran sp_mergemetadataretentioncleanup manually.
 > This took long time as it always cleans the rows from metadata tables in a batch of 5000 rows. Also it checks expired subscriptions (using sp_MSmark_expired_subscriptions) and this can take some time when you have thousands of subscribers.
 > The complete execution took long time but it cleaned millions of rows from metadata tables. There were only a few hundred rows left in the metadata tables after this.
 > After this, the merge queries ran instantly and there was a almost no blocking.

If you come across a situation where you find sp_mergemetadataretentioncleanup is not cleaning up rows from metadata tables, it will be worth to: stop all merge agents, confirm that no spids that might be executing any merge metadata commands are lying around.
After this, run sp_mergemetadataretentioncleanup manually and wait till it finishes completely. Once it finishes completely, check the table count. After this, start the merge agents again.

-4 = Session ID of the blocking latch owner could not be determined due to internal latch state transitions.

What to do: 

* If you see this scenario, focus on the waitresource this spid -4 was working. In our case is was 2:1:103
* This was a page from tempdb (2:1:103  = 2 - database ID i.e. tempdb)
* If you see the -4 SPIDs being waiting for the same waitresource, you can be sure there is a page hotspot condition going on.
* In my case, found that a new code that uses table variables was implemented. Every client was firing this query almost every second.
* Since it was using a default setup, the table variable was getting allocated on the same page causing that db page (in our case 2:1:103) to become a hotspot causing tempdb contention

To resolve this issue, carefully follow east step from http://support.microsoft.com/kb/328551 and check for

Repeated create and drop of temporary tables (local or global).
Table variables that use tempdb for storage purposes.
Work tables associated with CURSORS.
Work tables associated with an ORDER BY clause.
Work tables associated with an GROUP BY clause.
Work files associated with HASH PLANS.

* You can implement TF 118 (as per the above KB article) to have the issue subsided (In my case, the blocking impact went down considerably).
* Final resolution is change the code to use temp tables instead of table variables

Once these messages were thrown, No applications could connect to SQL Server after this error.
When the issue happened, the only way to resolve it is by restarting the browser service.

Cause:
-----------
Unknown

Resolution: 
------------------

• We tried to collect dumps using SQLDumper and ADPLUS crash but could not do that. The reason for that was that the SSRP thread inside browser was not aborting with any exception. Had there been an exception, we should have got a dump.
• Direct action plan was to apply Service Pack 2 on OS since they were on Windows Server 2003 Service Pack 1.
• But we found that after the storage team rolled back the upgrade of the TSM client to the previous version (5.4.2.0), SQL Browser did not report any errors in the event log.
  It is suspected that upgrade of the TSM client to version 5.5.2.1 that caused the problem however we could not test it in development environment for confirmation.
      This is unverified from a proper reproduction test. Please test it in your development environment.

What you can do if you face similar issue:
 > Start SQLBrowser from command prompt with verbose mode "SQLBrowser.exe -c" and check if you get any more detailed messages when the issue occurs
 > Collect BIDTrace since browser everywhere has provisions to throwing more messages in the BIDTrace (follow http://msdn2.microsoft.com/en-us/library/aa964124.aspx)
 > Confirm that you are on latest OS and SQL Service Packs
 > Check if they have any Side by Side installation of SQL Server 2000 with SQL 2005/2008. If yes, follow http://support.microsoft.com/default.aspx?scid=kb;EN-US;922131
 > Confirm from "netstat -aon" that SQLBrowser was the one holding 0.0.0.0:1434 UDP port when the error messages are thrown.
 > Check if you can make SQL Server work on 1433 as a temporary workaround (since server running on 1433 will not require UDP broadcast)
 > Test uninstalling (not just disabling) third party softwares from the box for. ex. the one mentioned above.

How are these old (non-latest) snapshot folders (static and dynamic snapshots) purged:
  For snapshot and transactional replication: Distribution cleanup agent purges the non-latest folders
  For Merge replication: the snapshot agent itself will purge older folders.  Please click here to see comments of Raymond Mak (one of the developers on replication development team) on this.

Reason:
Whenever any one of "allow_anonymous" or "immediate_sync" properties of a publication are set to TRUE, these entries are added.
These entries are as such harmless and are added to signify that this publication can have ananymous subscriptions.
In one particular case, these entries were found to be linked with thousands of records into MSrepl_transaction and MSRepl_commands. Since these entries are not removed by distribution cleanup agent, they were causing some performance isssues.
But this is not true in every case and you should check if this is the case in your environment.

If you want to turn off (delete) these entries, you should recreate your publication with "allow_anonymous" and "immediate_sync" properties set to FALSE. If you don't want to recreate publication, you can try sp_changepublication to set these to false. But in my testing, sp_changepublication did not help and and recreating publication helped remove the entries.
If you create the publication using scripts, you should look for them on the sp_addpublication command line.

* We observed that the value of this @immediate_sync parameter makes a difference to the snapshot produced when a new article is added to an existing publication.
When the @immediate_sync parameter is set to 0 i.e. when the immediate_sync column of distribution..MSpublications table is set to "0" zero and you add a new table (article) to the existing publication, you will find that when you run the snapshot agent, snapshot of only that new article will be produced in the \REPLDATA folder. snapshot will be produced in concurrent mode. When you run distribution agent, it will apply the snapshot of only the newly article to the subscriber.

* When the @immediate_sync parameter is set to 1 i.e. when the immediate_sync column of distribution..MSpublications table is set to "1" one and you add a new table (article) to the existing publication, you will find that when you run the snapshot agent, concurrent snapshot of ALL the articles in the publication will be prepared. When you run distribution agent however, it will apply the snapshot of only the newly article to the subscriber.

* If @immediate_sync parameter is set to 0 and you add a new article to publication, distribution agent will not automatically produce the snapshot of the new article. You need to manually run the snapshot agent which will produce the snapshot of that article only. After this, when you run the distribution agent, it will now apply the snapshot of that article properly and proceed with synchronization.

* The behavior of producing the snapshot of only the newly added article may be very much desired in most cases because:
 > It means that we will not be require huge space for snapshot files
 > snapshot process will not lock (although the concurrent snapshot locks the tables for minimal time) the other existing tables
 > less processing by the distribution since the snapshot size is small.

* Please note that I have observed and repro'ed it on my sql server 2005 box but I have not tested all possible scenarios and additional testing may be required. Test it out in your environment before you take a decision to set this value for your purpose.

External Links:

http://www.extremeexperts.com/SQL/Articles/SQLCounters.aspx

http://www.sqlservercentral.com/articles/Administering/performancemonitoringbasiccounters/1348/

User Settable queries:

http://searchsqlserver.techtarget.com/generic/0,295582,sid87_gci1110440_tax301334,00.html?adg=301324&bucket=REF#