I thought I would expand on my initial question with more background details.
The client machines in our company run a windows app that can either connect directly to the PROD SQL Server OR connect to a local install of SQL Server 2012 Express. There is an option in the app to synchronize the local database to the PROD DB via a merge pull subscription.
The sync process uses web synchronization via repliaspi.dll on an IIS server.
This setup allows for flaky/not permanent network connections.
This has worked for for a while now, but are seeing more sync issues so a migration is in order.
We are migrating our Prod and Dev SQL Servers from SQL Server 2012 to 2017 as well as upgrading the client app and SQL Server Express install to the 2017 version.
When the app was recompiled to use the SQL Server 2017 dlls (Microsoft.SqlServer.ConnectionInfo, Microsoft.SqlServer.Replication, Microsoft.SqlServer.Rmo) and the remote option was used (this option will manage the local subscription to the PROD DB) the error I mentioned in my initial question appeared.
After the synchronization is started it runs for approx 60secs then gives the "{The processing of the response message failed.} System.Exception {Microsoft.SqlServer.Replication.ComErrorException}} error.
The repliaspi log file shows nothing out of the ordinary (ie. no error messages)
CReplicationListenerWorker , 2021/05/18 07:27:00.380, 13012, 174, S2, INFO: =============== START PROCESSING REQUEST ==============
CReplicationListenerWorker , 2021/05/18 07:27:00.459, 13012, 298, S2, INFO: Processed request type: MESSAGE_TYPE_UploadEmpty.
CReplicationListenerWorker , 2021/05/18 07:27:00.459, 13012, 396, S2, INFO: =============== DONE PROCESSING REQUEST ===============
CReplicationListenerWorker , 2021/05/18 07:27:00.755, 13012, 174, S2, INFO: =============== START PROCESSING REQUEST ==============
CReplicationListenerWorker , 2021/05/18 07:27:00.755, 13012, 298, S2, INFO: Processed request type: MESSAGE_TYPE_UploadEmpty.
CReplicationListenerWorker , 2021/05/18 07:27:00.755, 13012, 396, S2, INFO: =============== DONE PROCESSING REQUEST ===============
CReplicationListenerWorker , 2021/05/18 07:27:00.755, 13012, 174, S2, INFO: =============== START PROCESSING REQUEST ==============
CHttpListener , 2021/05/18 07:27:00.755, 13012, 258, S2, INFO: Exchange ID = 407D627A-8E92-475D-ADE0-413CC831A099.
CReplicationListenerWorker , 2021/05/18 07:27:00.755, 13012, 298, S2, INFO: Processed request type: MESSAGE_TYPE_SyncContentsUpload.
DatabaseReconciler , 2021/05/18 07:27:01.287, 13012, 13289, S2, INFO: Reading profile: AgentID:25, AgentType:4, ProfileName:
DatabaseReconciler , 2021/05/18 07:27:02.787, 13012, 25669, S2, INFO: [WEBSYNC_PROTOCOL] Received client ReconcilerPhase WebSyncReconcilerPhase_ReinitSchemaAndFiles
Replprov.dll , 2021/05/18 07:28:17.422, 13012, 3019, S2, :T:,80,83,0,pub,65,,,,,
Replprov.dll , 2021/05/18 07:28:17.422, 13012, 2493, S2, :T:,70,83,0,pub,0,,,,,
DatabaseReconciler , 2021/05/18 07:28:17.422, 13012, 4778, S2, :T:,10,83,0,65,0,0,65,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7768, S2, :T:,100,83,1,65,3e097e77-8cb5-4a07-a603-18e53e7b750b,,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,66,66,ccde725c-326c-4d90-934b-0d762fc808fd,3,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,67,67,52d84fad-41d9-47e3-9c45-fc8c5c1e8e5c,3,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,68,68,8e70dcca-c384-4acb-99ba-0e6f141ec445,1,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,69,69,1810f392-10d1-465a-b055-7cc745ad9f07,1,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,70,70,c339f093-e00b-467c-8ca5-23d551137bdc,9,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,71,71,8981cd10-bb83-4af7-9baf-b132fde9b2f5,9,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,72,72,921d5260-8774-41ef-83ab-db3476fa4e3c,2,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,73,73,0969390a-b6c8-4568-86e8-e7e31e2f40b2,2,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,74,74,29bf6743-fdd3-42b4-9fd9-2672ce5b2d52,2,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,75,75,0b35a2f7-bb87-4543-8385-501ad4d764e7,2,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,76,76,5a7f0ab8-0aec-4ae7-bebc-07e072b9b947,3,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,77,77,ae8267d2-65c7-4248-b6b7-f970e5223a7c,3,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,78,78,a13e6084-a41b-4c3f-90ab-8af2ee10a195,4,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,79,79,6f0c7c61-831f-449b-9971-bc9c947f2934,4,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,80,80,db1129a0-ee58-4e98-94a7-7e3d3ee02b9f,5,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,81,81,ed4d1072-4db9-4ef6-af15-3a28e1e355e7,5,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,82,82,d3e6aecf-f906-4f88-ac93-5b1c75340285,3,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,83,83,0550cb93-f121-4b71-b520-e56600a60732,1,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,84,84,bd984769-6a8c-4f64-ac75-ccd7cccc72b2,2,,,,
DatabaseReconciler , 2021/05/18 07:28:18.228, 13012, 7758, S2, :T:,30,83,1,85,85,e1720052-dc81-4460-b7d6-2376e8813dfd,7,,,,
DatabaseReconciler , 2021/05/18 07:28:18.393, 13012, 25753, S2, INFO: Reading profile: AgentID:25, AgentType:4, ProfileName:
DatabaseReconciler , 2021/05/18 07:28:18.393, 13012, 25632, S2, INFO: [WEBSYNC_PROTOCOL] Sending server ReconcilerPhase WebSyncReconcilerPhase_ReinitSchemaAndFiles
DatabaseReconciler , 2021/05/18 07:28:18.393, 13012, 3939, S2, :T:,110,83,32776,,,,,,,
DatabaseReconciler , 2021/05/18 07:28:18.393, 13012, 3946, S1, INFO: ID = 83, Session Highlights: REINIT, WEBSYNC_SERVER,
CReplicationListenerWorker , 2021/05/18 07:28:18.424, 13012, 344, S2, INFO: Statistics: RequestQueueWaitTime 0 ms, SyncDuration 77672 ms
CReplicationListenerWorker , 2021/05/18 07:28:18.424, 13012, 396, S2, INFO: =============== DONE PROCESSING REQUEST ===============
CReplicationListenerWorker , 2021/05/18 07:28:18.534, 13012, 174, S2, INFO: =============== START PROCESSING REQUEST ==============
CReplicationListenerWorker , 2021/05/18 07:28:18.534, 13012, 298, S2, INFO: Processed request type: MESSAGE_TYPE_UploadEmpty.
CReplicationListenerWorker , 2021/05/18 07:28:18.534, 13012, 396, S2, INFO: =============== DONE PROCESSING REQUEST ===============
CReplicationListenerWorker , 2021/05/18 07:28:18.534, 13012, 174, S2, INFO: =============== START PROCESSING REQUEST ==============
CHttpListener , 2021/05/18 07:28:18.534, 13012, 258, S2, INFO: Exchange ID = 407D627A-8E92-475D-ADE0-413CC831A099.
CReplicationListenerWorker , 2021/05/18 07:28:18.534, 13012, 298, S2, INFO: Processed request type: MESSAGE_TYPE_FileRequest.
CReplicationListenerWorker , 2021/05/18 07:28:18.534, 13012, 396, S2, INFO: =============== DONE PROCESSING REQUEST ===============
I have also set the sync log file on as well
2021-05-18 07:26:59.563 Connecting to OLE DB Subscriber at datasource: 'Local\SQLEXPRESS', location: '', catalog: 'LocalReplicatedDB', providerstring: '' using provider 'SQLNCLI11'
2021-05-18 07:26:59.753 OLE DB Subscriber: Local\SQLEXPRESS
DBMS: Microsoft SQL Server
Version: 14.00.1000
catalog name: LocalReplicatedDB
user name: dbo
API conformance: 0
SQL conformance: 0
transaction capable: 1
read only: F
identifier quote char: "
non_nullable_columns: 0
owner usage: 15
max table name len: 128
max column name len: 128
need long data len:
max columns in table: 1000
max columns in index: 16
max char literal len: 131072
max statement len: 131072
max row size: 131072
2021-05-18 07:26:59.755 OLE DB Subscriber 'Local\SQLEXPRESS': select SERVERPROPERTY ('ProductVersion')
2021-05-18 07:26:59.755 OLE DB Subscriber 'Local\SQLEXPRESS': {call sp_MSreplcheck_subscribe}
2021-05-18 07:26:59.756 OLE DB Subscriber 'L:oacl\SQLEXPRESS': set nocount on declare @dbname sysname select @dbname = db_name() declare @collation nvarchar(255) select @collation = convert(nvarchar(255), databasepropertyex(@dbname, N'COLLATION')) select collationproperty(@collation, N'CODEPAGE') as 'CodePage', collationproperty(@collation, N'LCID') as 'LCID', collationproperty(@collation, N'COMPARISONSTYLE') as 'ComparisonStyle',cast(case when convert (int,databasepropertyex (@dbname,'comparisonstyle')) & 0x1 = 0x1 then 0 else 1 end as bit) as DB_CaseSensitive,cast(case when convert (int,serverproperty ('comparisonstyle')) & 0x1 = 0x1 then 0 else 1 end as bit) as Server_CaseSensitive set nocount off
2021-05-18 07:26:59.793 OLE DB Subscriber 'Local\SQLEXPRESS': {?=call sp_helpsubscription_properties (N'Remote', N'RemoteDB', N'RemoteReplicationName')}
2021-05-18 07:26:59.810 Distributor security mode: 1, login name: , password: ********.
2021-05-18 07:26:59.828 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MShelp_profilecache(N'DEFAULT')}
2021-05-18 07:26:59.830 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSadd_merge_history90 (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}
2021-05-18 07:26:59.911 The upload message to be sent to Publisher 'Remote' is being generated
2021-05-18 07:26:59.911 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSadd_merge_history90 (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}
2021-05-18 07:26:59.913 The merge process is using Exchange ID '407D627A-8E92-475D-ADE0-413CC831A099' for this web synchronization session.
2021-05-18 07:26:59.913 Connecting to OLE DB at datasource: 'Local\SQLEXPRESS', location: '', catalog: 'LocalReplicatedDB', providerstring: '' using provider 'SQLNCLI11'
2021-05-18 07:26:59.917 OLE DB : Local\SQLEXPRESS
DBMS: Microsoft SQL Server
Version: 14.00.1000
catalog name: LocalReplicatedDB
user name: dbo
API conformance: 0
SQL conformance: 0
transaction capable: 1
read only: F
identifier quote char: "
non_nullable_columns: 0
owner usage: 15
max table name len: 128
max column name len: 128
need long data len:
max columns in table: 1000
max columns in index: 16
max char literal len: 131072
max statement len: 131072
max row size: 131072
2021-05-18 07:26:59.918 OLE DB 'Local\SQLEXPRESS': set nocount on declare @dbname sysname select @dbname = db_name() declare @collation nvarchar(255) select @collation = convert(nvarchar(255), databasepropertyex(@dbname, N'COLLATION')) select collationproperty(@collation, N'CODEPAGE') as 'CodePage', collationproperty(@collation, N'LCID') as 'LCID', collationproperty(@collation, N'COMPARISONSTYLE') as 'ComparisonStyle',cast(case when convert (int,databasepropertyex (@dbname,'comparisonstyle')) & 0x1 = 0x1 then 0 else 1 end as bit) as DB_CaseSensitive,cast(case when convert (int,serverproperty ('comparisonstyle')) & 0x1 = 0x1 then 0 else 1 end as bit) as Server_CaseSensitive set nocount off
2021-05-18 07:26:59.941 OLE DB 'Local\SQLEXPRESS': exec sp_helpmergepublication @publication = N'RemoteReplicationName', @reserved = 'internal', @publisher = N'Remote', @publisher_db = N'RemoteDB'
2021-05-18 07:26:59.972 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSgetreplicainfo(?,?,?,?,?,?,?,90)}
2021-05-18 07:26:59.989 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSgetreplicainfo(?,?,?,?,?,?,?,90)}
2021-05-18 07:27:00.468 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSsetreplicainfo (?,?,?,?,?,?,?,?,?,?,90,?,?,0)}
2021-05-18 07:27:00.486 OLE DB Subscriber 'Local\SQLEXPRESS': {call sp_MSmergeupdatelastsyncinfo (?,?,?)}
2021-05-18 07:27:00.488 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSenumreplicas90 }
2021-05-18 07:27:00.492 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSenum_metadataaction_requests(?,?,?,?)}
2021-05-18 07:27:00.494 OLE DB Subscriber 'Local\SQLEXPRESS': {call sp_MSgetlastsentrecgens (?)}
2021-05-18 07:27:00.495 OLE DB Subscriber 'Local\SQLEXPRESS': {call sp_MSgetmakegenerationapplock_90(?,?)}
2021-05-18 07:27:00.603 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSmakegeneration (?, NULL,NULL,NULL,100)}
2021-05-18 07:27:00.726 OLE DB Subscriber 'Local\SQLEXPRESS': {call sp_MSreleasemakegenerationapplock}
2021-05-18 07:27:00.732 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSadd_merge_history90 (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}
2021-05-18 07:27:00.733 No data needed to be merged.
2021-05-18 07:27:00.733 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSadd_merge_history90 (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}
2021-05-18 07:27:00.735 Request message generated, now making it ready for upload.
2021-05-18 07:27:00.735 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSadd_merge_history90 (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}
2021-05-18 07:27:00.737 Upload request size is 1349 bytes.
2021-05-18 07:28:18.429 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSadd_merge_history90 (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}
2021-05-18 07:28:18.490 Uploaded a total of 1 chunks.
2021-05-18 07:28:18.491 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSadd_merge_history90 (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}
2021-05-18 07:28:18.492 The request message was sent to 'https://mergeweb:444/SQLMerge/replisapi.dll'
2021-05-18 07:28:18.524 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSadd_merge_history90 (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}
2021-05-18 07:28:18.525 Downloaded a total of 45 chunks.
2021-05-18 07:28:18.538 OLE DB Subscriber 'Local\SQLEXPRESS': {call sys.sp_MSadd_merge_history90 (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}
2021-05-18 07:28:18.541 The processing of the response message failed.
2021-05-18 07:28:18.541 OLE DB Subscriber 'Local\SQLEXPRESS': {call sp_MSmergeupdatelastsyncinfo (?,?,?)}
2021-05-18 07:28:18.554 OLE DB Subscriber 'Local\SQLEXPRESS': {call sp_MSmergeupdatelastsyncinfo (?,?,?)}
2021-05-18 07:28:18.554 Disconnecting from OLE DB Subscriber 'Local\SQLEXPRESS'
2021-05-18 07:28:18.555 Disconnecting from OLE DB Subscriber 'Local\SQLEXPRESS'
2021-05-18 07:28:18.555 Disconnecting from OLE DB 'Local\SQLEXPRESS'
2021-05-18 07:28:18.556 Disconnecting from OLE DB 'Local\SQLEXPRESS'
Also the IIS log file for the mergeweb service
Version: 1.0
Date: 2021-05-18 07:27:00
Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2021-05-18 07:27:00 W3SVC2 MERGEWEB 10.1.3.50 HEAD /SQLMerge/replisapi.dll - 444 - 10.1.3.151 HTTP/1.1 SQL+Replication+WebSync - - MERGEWEB:444 401 2 5 248 184 187
2021-05-18 07:27:00 W3SVC2 MERGEWEB 10.1.3.50 HEAD /SQLMerge/replisapi.dll - 444 USER 10.1.3.151 HTTP/1.1 SQL+Replication+WebSync - - MERGEWEB:444 200 0 0 217 271 250
2021-05-18 07:27:00 W3SVC2 MERGEWEB 10.1.3.50 HEAD /SQLMerge/replisapi.dll - 444 USER 10.1.3.151 HTTP/1.1 SQL+Replication+WebSync - - MERGEWEB:444 200 0 0 217 271 0
2021-05-18 07:28:18 W3SVC2 MERGEWEB 10.1.3.50 POST /SQLMerge/replisapi.dll - 444 USER 10.1.3.151 HTTP/1.1 SQL+Replication+WebSync - - MERGEWEB:444 200 0 0 367711 1642 77746
2021-05-18 07:28:18 W3SVC2 MERGEWEB 10.1.3.50 HEAD /SQLMerge/replisapi.dll - 444 USER 10.1.3.151 HTTP/1.1 SQL+Replication+WebSync - - MERGEWEB:444 200 0 0 217 271 0
2021-05-18 07:28:18 W3SVC2 MERGEWEB 10.1.3.50 POST /SQLMerge/replisapi.dll - 444 USER 10.1.3.151 HTTP/1.1 SQL+Replication+WebSync - - MERGEWEB:444 200 0 0 793 1208 0
Any thoughts or comments will be much appreciated