We were trying to install SP3 on SQL server 2008 R2 and got the error like below.

 <FuncName='Do_sqlFileSR'>

Failed to open file C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\devenv.exe.config.tmp

 Error Code: 1603

 MSI (s) (D0!88) [02:38:57:769]: Product: SQL Server 2008 R2 SP2 BI Development Studio -- Error 29528. The setup has encountered an unexpected error while Updating Installed Files

++ We searched for the file in this location and it was not there " C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies"

 ++ according to below KB article it was suggested to copy from the "Program files" location. 

https://support.microsoft.com/en-us/kb/2216468

++ But customer did not have "Microsoft Visual Studio 9.0" folder under "C:\Program Files"

 ++ So what we did is we went to different server and checked this file and compared between few servers to see if the content in that file is same or not 

++ Content is same , SO we copied from the server which had this file and placed in the server where we had problem, under C:\ProgramFiles (x86)\Microsoft Visual Studio9.0\Common7\IDE\PrivateAssemblies 

++ later SP3 completed successfully. 

So if you do not have "Microsoft Visual Studio 9.0" under C:\Program Files, Don't worry , just find another server where you have this file "devenv.exe.config.tmp" and place it in the location to fix this issue 

Disclaimer :- This is workaround , not a fix or root cause analysis.

Msg 1934, Level 16, State 1, Line 1

ALTER TABLE failed because the following SET options have incorrect settings: 'ANSI_WARNINGS'. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations.

We made sure that 'ANSI_WARNINGS' is set to ON from session and database perspective , We made sure that Triggers are disabled from table, database, server side.(for our troubleshooting perspective we deleted all indexes on that table as well). but still we were getting above error.

and that column we are trying to alter is not used in any another computed columns in the table.

Repro steps:-

SQL server 2012

CREATE TABLE [dbo].table1([column1] [int] IDENTITY(30000000,1) NOT NULL,column2 int null,column3 as(column2) persisted)
alter table table1 alter column [column1] bigint not null

Above repro we have done in SQL server 2012, later we thought lets test it in higher version, we tested in SQL server 2014

SQL server 2014

When executed the same steps in SQL server 2014 we got different error message , like below.

Msg 11424, Level 16, State 1, Line 1

Cannot alter the identity column 'column1' in the table 'table1' because this operation requires data modification and the table contains a persisted computed column. Remove the persisted computed column before modifying the identity column.

Actually SQL server 2014 gives the right error message, 2012 error is misleading us. So in this case ideally we need to drop all PERSISTED COMPUTED columns in the table and change the IDENTITY column type and then add all computed columns.

remember you will not have this problem if either table has no persisted columns or you are not changing the IDENTITY column type.

We have collected PSSDIAG, which is ideal in this situation because it has combination of tools in it  like perfmon, profiler trace, DMV's output.

first let me give you some information about what could be the capacity limits of normal single disk and SAN

Single disk :- Avg disk bytes/sec:-  50 to 70 Mb/sec (approximate values)

SAN := Avg disk bytes/sec 150-300 Mb/sec (approximate values) . Any thing more than above limits disk are expected to show some latency and are considered overwhelmed.

In our case we had normal single disk, but SQL server was doing more than 100 mb/sec (avg disk bytes/sec) including reads and writes which is high. so We wanted to know how to reduce this.

and drive which we wanted to know is W:\

If you see above screen shot, LogicalDisk(W:)\Disk Bytes/sec is fluctuating for every min or so. since this is almost happening for every one min first thing which came into my mind is CHECKPOINTs, So added CHECKPOINT counter to performance monitor.

and graph looks like below.

From the graph we can clearly see that as in checkpoint pages/sec increases  Disk bytes/sec is also increasing , So from the graph  IO is directly proportional to Checkpoint pages/sec.

Checkpoint pages/sec means:- Number of pages flushed by checkpoint or other operations that require all dirty pages to be flushed. 

So Checkpoint duty is to flush dirty pages from buffer pool to disk and pages in the buffer pool will become dirty as in they are modified , So more writes more dirty pages , more dirty pages more IO on disk.

This is all falling back to number of modification you are doing on databases So we need to focus on reducing the modification on the databases so that dirty pages count will reduce and IO on drive will reduce

So we started looking at top write intensive queries and found a purge process that is being executed for every minute from application , and that explains why you are seeing spikes in disk bytes/sec for every min. We reduced the amount of records that purge job deletes every min and that reduced the half of the IO request on Drive W:\

So ideal solution in this case going to be

Reduce the number of modification happening on the server (remove unnecessary writes)

Or

Use the Disk which can provide more throughput ( like SAN drives)

Or

Spilt the number of files on the drive to multiple drives so that load on that drive will reduce. ( we had 24 data files on this drive) 

General recommendations

1. Check the fragmentation on database and see you can defrag the databases , this might give some relief
2. apply any missing indexes and see if that help.

 high disk utilization can be because of multiple reason and above is one of such, hope this helps someone.

Customer was very eager to know how can this happen and even I was curious to repro. So troubleshooting went as following.

So first things first , on the second peer node We need to collect audit traces which will help us to identify where are the INSERTS,UPDATES and DELETES coming from , so We have collected SQL server audit traces and we have noticed an update on second peer node apart from replication store procedure, something like below.

select * from fn_get_audit_file('C:\repl_issue\*.sqlaudit',default,default) where statement like '%update%'

You need to check  additional_information column in the output, if only updates are coming from replication then you will see object_name='sp_MSupd_dboCustomers02030113239' but if you see anything apart from this (like below) then that means there is someone or something making changes.

<tsql_stack><framenest_level='2'database_name='repl_db'schema_name='dbo'object_name='update_customers'/></tsql_stack>

So there is definitely update coming from object "update_customers", so first thought came into my mind was whether it is a trigger, when we checked yes there was trigger with that name.

in the initial phase we have asked them to disable trigger , issue surfaced almost 90% but we still noticed one or two conflicts per day, why we are still getting conflicts even after disabling trigger i will explain in my repro (below) but final resolution for this issue is to modify trigger on all peer nodes and add

"NOT for Replication", something like below

 ALTER TRIGGER[dbo].[customer_updates]

ON [dbo].[customers]

AFTER UPDATE

NOT for Replication

AS 

UPDATE customers 

SET

customers.dateupdated=getutcdate() 

FROM 

inserted 

WHERE 

customers.ID=inserted.ID

GO

But Customer wanted to prove that trigger is the cause here and I was also interested so started doing repro. And this post is to give you some insight on how conflict detection and resolution happens.

Demo

use the below script to create database and table

create database p2p

use p2p 

create table customers(CumID nvarchar(10) CONSTRAINT[PK_Customers] PRIMARYKEY,Name nvarchar(50) notnull,datecreated datetimeoffset CONSTRAINT[CreationDate] DEFAULTG ETDATE(),datemodified datetimeoffset CONSTRAINT[modifiedDate] DEFAULTGETDATE())

then take backup and restore on second node and configure peer to peer replication. use the below script to create trigger on both the nodes. after configuring the peer to peer replication , change the publication property "Continue replication after conflict detection" to True, otherwise your distribution agent will be stopped considering this error is critical.

CREATE TRIGGER [dbo].[update_customers]
ON [dbo].[Customers]
AFTER UPDATE
AS
   UPDATE
 Customers
   SET
 Customers.dateModified = getutcdate()
   FROM
 inserted
   WHERE
 Customers.CumID = inserted.CumID

GO

Now lets insert a first record.

insert into customers values(111,'ravi',getutcdate(),getutcdate()) select

select * from customers  and make sure record present in both the nodes

now I want to pause my repro here and explain about the hidden column that gets added to table when you configure peer to peer replication.

$sys_p2p_cd_id is the column name that gets added , and you need to use DAC connection to see that column and there are functions that can be used to convert column values, something like below.

select cumID,name,datecreated,datemodified,sys.fn_replvarbintoint( $sys_p2p_cd_id) as originator_id , sys.fn_replp2pversiontotranid($sys_p2p_cd_id) as tran_id from customers

output will be like this.

$sys_p2p_cd_id include originator ID from where that record got originated and transaction ID. and this is the value used to detect the conflicts, according to this MSDB article  https://msdn.microsoft.com/en-us/library/bb934199.aspx "In peer-to-peer replication, conflicts are detected by the stored procedures that apply changes to each node, based on a hidden column in each published table. This hidden column stores an ID that combines an originator IDthat you specify for each node and the version of the row. During synchronization, the Distribution Agent executes procedures for each table. These procedures apply insert, update, and delete operations from other peers. If one of the procedures detects a conflict when it reads the hidden column value, it raises error 22815 that has a severity level of 16:

A conflict of type '%s' was detected at peer %d between peer %d (incoming), transaction id %s  and peer %d (on disk), transaction id %s"

Hope now you know how conflicts are detected, The conflict is resolved based on the originator ID value that you specified during configuration: the row that originated at the node with the highest ID wins the conflict.

Now going back to demo. first disable triggers on both the nodes. and update the record and check the originator ID.

in my case , Node1 originator ID=1 and Node 2 originator ID=202. now I updated a row on Node 2 (ID 202)

update customers set name='ravi_1' where CumID=111
select cumID,name,datecreated,datemodified,sys.fn_replvarbintoint( $sys_p2p_cd_id) as originator_id , sys.fn_replp2pversiontotranid($sys_p2p_cd_id) as tran_id from customers

So  now I see that record got updated and we have correct originator ID on both sides. Lets enable the trigger on Node 1 (originator id =1)

after enabling trigger on Node1, update the same row again in Node 2(originator id=202).

update customers set name='ravi_1_tigger_enabled_Node1' where CumID=111
select cumID,name,datecreated,datemodified,sys.fn_replvarbintoint( $sys_p2p_cd_id) as originator_id , sys.fn_replp2pversiontotranid($sys_p2p_cd_id) as tran_id from customers

execute the above select statement on both the nodes and I am sure you will see the difference in originator ID column in both the nodes, something like below.

That confirms that trigger is causing the issue and we have difference in originator ID in both the server, we need to do another update to set this correctly even after disabling trigger on node1, so that is the reason you will see one more conflict even after disabling the triggers.

If you open the conflicts you should see conflict as well. there is lot of backend work done by replication store procedure, I do not want to put that in this article, it will be overhead, I will try to post another article on how replication will send records to another node. Hope this helps someone at least.

1. Please bring SQL server resource offline and remove the network name from SQL server resource dependencies list  in WFC and please delete the IP address resource in failover cluster manager