VSS problems backing up windows 2012 R2 server VM in hyper-v server

Premier IT 1 Reputation point
2021-12-17T10:06:36.63+00:00

Hi everyone,

I have a single Hyper-V 2012 R2 server host with 2 VMs running windows 2012 R2 server with the latest updates.

The VMs are backed up using Altaro VMBACKUP which runs on the host.

The first VM is a DC which is backed up successfully.

The second VM OS is Windows 2012 R2 with Exchange Server 2013 onpremise.

This VM has been backing up for years without any problem and since 2 months ago the backup has been failing.

The Altaro VMBACKUP shows the following error:

There was a problem while rolling back the latest failed backup for this VM to the last consistent state. (Error code 'ROLLBACK_306')
The rollback will be attempted again automatically before any subsequent backups or restores on this VM until it is successful, so the health of the backup data should not be affected.
Additional information is available:
There was a problem while rolling back the latest failed backup for this VM to the last consistent state. (Error code 'ROLLBACK_016')
The rollback will be attempted again automatically before any subsequent backups or restores on this VM until it is successful, so the health of the backup data should not be affected.

I contacted Altaro who told me that this is a VSS error on the VM.

I ran the following command:

vssadmin list writers and I get:

C:\Windows\System32>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error

Writer name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error

Writer name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No error

Writer name: 'SqlServerWriter'
Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
Writer Instance Id: {e3fe5bef-4d5d-4bfb-ba2d-70b26bc79c1f}
State: [5] Waiting for completion
Last error: No error

Writer name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {c728413d-e449-43a3-90d7-3a2c808d6f5a}
State: [1] Stable
Last error: No error

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {3fa2fe09-b897-467d-81b0-27e087733089}
State: [5] Waiting for completion
Last error: No error

Writer name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {c872e448-32e3-4a64-b87f-d327029fceed}
State: [1] Stable
Last error: No error

Writer name: 'MSMQ Writer (MSMQ)'
Writer Id: {7e47b561-971a-46e6-96b9-696eeaa53b2a}
Writer Instance Id: {b78ccc41-d903-43f6-b66b-f0b63efc5de7}
State: [5] Waiting for completion
Last error: No error

Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {5563ea37-874b-4454-8c7b-f9d600dbae6f}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {b9b56105-58d7-4c57-9a89-ab1deb5c855f}
State: [5] Waiting for completion
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {b3133987-1fcb-4ba6-9bbc-7061fa422092}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {425abb99-ed35-41cf-9bfc-91d88d40a748}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {ef8f79ff-0be3-4d5d-a116-5dd26f4ae389}
State: [1] Stable
Last error: No error

Writer name: 'IIS Config Writer'
Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
Writer Instance Id: {cc66af8d-b858-490b-a67c-7a8473411603}
State: [5] Waiting for completion
Last error: No error

Writer name: 'IIS Metabase Writer'
Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
Writer Instance Id: {3139badb-8634-4e7d-a358-c329e1fcc174}
State: [5] Waiting for completion
Last error: No error

The state of the following writers shows waiting for completion

SqlServerWriter Waiting
Microsoft Exchange Writer
MSMQ Writer (MSMQ)
WMI Writer
IIS Metabase Writer
IIS Config Writer

The other writers are stable.

I restarted these services:

Microsoft Software Shadow Copy Provider
Volume Shadow Copy
Cryptographic services

Nothing has changed.

The I ran the following script to fix VSS:

-------------------------------------------------------------------------------

@Echo off
cd c:\windows\system32
Net Stop VSS
Net Stop SWPRV
regsvr32 /s ole32.dll
regsvr32 /s oleaut32.dll
regsvr32 /s vss_ps.dll
vssvc /register
regsvr32 /s /i swprv.dll
regsvr32 /s /i eventcls.dll
regsvr32 /s es.dll
regsvr32 /s stdprov.dll
regsvr32 /s vssui.dll
regsvr32 /s msxml.dll
regsvr32 /s msxml3.dll
regsvr32 /s msxml4.dll
vssvc /register
Net Start SWPRV
Net Start VSS

-------------------------------------------------------------------------------

Rebooted the server, still same backup issue.

Updated Windows 2012 R2 with the latest available updates.

Now Altaro tells me that if I upgrade to the new version this may resolved.
But I feel that there is something wrong with the VSS.

Does anyone have any ideas on how to fix this because I have just come across a brick wall?

Thank you

Best regards

Alfred

Windows Server
Windows Server
A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.
12,207 questions
Hyper-V
Hyper-V
A Windows technology providing a hypervisor-based virtualization solution enabling customers to consolidate workloads onto a single server.
2,560 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Limitless Technology 39,396 Reputation points
    2021-12-17T16:00:35.417+00:00

    Hello @Premier IT

    Windows Server backup may fail because of the SQL Server VSS Writer.

    When Windows Server backup attempts to back up a disk volume, a Volume Shadow Copy Snapshot is created for the volume. When the snapshot is created, any Volume Shadow Copy Service (VSS) writer associated with the volume is called. If any of the VSS writers encounter an error, the entire backup job will fail.

    The error is typically caused by a problem with one of the SQL Server instances. Here is an article to help you out with the resolution https://learn.microsoft.com/en-us/troubleshoot/windows-server/backup-and-storage/backup-fails-vss-writer

    Hope this resolves your Query!!

    --If the reply is helpful, please Upvote and Accept it as an answer--

    0 comments
  2. Premier IT 1 Reputation point
    2021-12-18T04:35:49.203+00:00

    Hi LimitlessTechnology,

    Thank you for your reply.

    I installed Windows Server Backup on the Exchange 2013 Virtual machine.

    I did a one off backup to a NAS remote shared folder and it backed up without an issue.

    158658-windows-server-backup.jpg

    When I tried the Altaro VMBACKUP running on the host it fails by stating:

    Waiting to retry failed shadowcopy
    Then it bombs out.

    These are the failed writers on the Exchange 2013 Virtual Machine:

    Writer name: 'SqlServerWriter'
    Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
    Writer Instance Id: {2d7dd263-f302-49b6-a3b1-0304d65ab7cf}
    State: [10] Failed
    Last error: Timed out

    Writer name: 'Microsoft Exchange Writer'
    Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
    Writer Instance Id: {943a9e15-a80b-404d-8d62-bca3e52d9cb9}
    State: [10] Failed
    Last error: Timed out

    Writer name: 'COM+ REGDB Writer'
    Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
    Writer Instance Id: {c7a407fc-2f39-4364-9d9f-5eedc475e045}
    State: [10] Failed
    Last error: Timed out

    Writer name: 'IIS Metabase Writer'
    Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
    Writer Instance Id: {dcc18db5-ad32-4964-8cd6-0f5abffa534a}
    State: [10] Failed
    Last error: Timed out

    Writer name: 'Shadow Copy Optimization Writer'
    Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
    Writer Instance Id: {5df4fe4b-89bc-418a-8f8e-64f5b8daaf9c}
    State: [10] Failed
    Last error: Timed out

    Writer name: 'MSMQ Writer (MSMQ)'
    Writer Id: {7e47b561-971a-46e6-96b9-696eeaa53b2a}
    Writer Instance Id: {10694806-f790-4dcc-9c27-346791d57497}
    State: [10] Failed
    Last error: Timed out

    Writer name: 'IIS Config Writer'
    Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
    Writer Instance Id: {b77fb86c-70a2-4047-b27f-e0b88bb2e7d0}
    State: [10] Failed
    Last error: Timed out

    Writer name: 'WMI Writer'
    Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
    Writer Instance Id: {175ad6fd-292f-4dfb-bcf8-674e30d9b755}
    State: [10] Failed
    Last error: Timed out

    I also stopped systematically every SQL service, but the result is the same.

    On the virtual machine event viewer application:

    Event 24583, Sqlwriter
    Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013
    Error state: 1, Severity: 16
    Source: Microsoft SQL Server Native Client 10.0
    Error message: BACKUP DATABASE is terminating abnormally.
    SQLSTATE: 42000, Native Error: 3271
    Error state: 1, Severity: 16
    Source: Microsoft SQL Server Native Client 10.0
    Error message: A nonrecoverable I/O error occurred on file "{00AB327F-BA10-4520-A119-362942581D2D}1:" 995(The I/O operation has been aborted because of either a thread exit or an application request.).
    SQLSTATE: 01000, Native Error: 4035
    Error state: 1, Severity: 0
    Source: Microsoft SQL Server Native Client 10.0
    Error message: Processed 0 pages for database 'model', file 'modeldev' on file 1.

    Event 3041
    BACKUP failed to complete the command BACKUP DATABASE model. Check the backup application log for detailed messages.

    ---------------------------------------------------------------------------------------------------------------------------

    Event 1
    SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=3844. Thread=2472. Client. Instance=SOPHOS. VD=Global{A19916EB-5A86-4801-BB90-D883A049BB8B}1_SQLVDIMemoryName_0.

    ---------------------------------------------------------------------------------------------------------------------------

    Event 12293
    Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details PreFinalCommitSnapshots({5ed8691f-d2c6-4ed4-9671-4d2756625205}, 3) [hr = 0x800705b4, This operation returned because the timeout period expired.
    ].

    Operation:
    Executing Asynchronous Operation

    Context:
    Current State: DoSnapshotSet

    ----------------------------------------------------------

    On the host event viewer I see this message:

    EVENT 8229
    A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,
    the error is likely to reoccur.
    . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

    This is beyond me.

    0 comments