Troubleshooting the Volume Shadow Copy Service
Microsoft Exchange-aware Volume Shadow Copy Service (VSS) backups are supported for both active storage groups and databases and for passive storage groups and databases. The passive copy backup solution that Microsoft provides is a VSS-only solution. It is implemented by the Exchange Replica VSS Writer, which is part of the Replication service. Streaming backups are supported only from the active storage groups. You cannot use streaming backup APIs to back up a replica database. A VSS backup program must be used together with a VSS Requestor for the Exchange writer.
In Microsoft Exchange Server 2003, you can run two backups at the same time against two different storage groups by using streaming backups. However, you cannot use VSS for this. For example, by using VSS in Exchange 2003, you cannot back up Storage Group Two until the backup on Storage Group One is finished.
By using Microsoft Exchange Server 2007, you can run two separate VSS backup jobs against the same Exchange server. The Exchange 2007 Writer also lets you restore Exchange data to an alternative location. The Exchange 2003 Writer does not let you target VSS restore operations to a location other than the original path.
VSS backups cannot be restored to the replica location by using the Exchange Writer. However, they can be restored to an alternative location, such as the replica copy directory.
In Exchange 2007, the Store.exe VSS writer enables restore operations to any storage group. This includes the recovery storage group (RSG). The Store.exe VSS writer also enables the restoration of databases to a folder location that is not associated with a storage group. If you do this, you can use the Jet database engine to replay the restored transaction logs to the database to bring the database to a consistent and mountable state. Then, you can move the database to another location.
Troubleshooting the Volume Shadow Copy Service
By default, VSS is installed on a Windows Server 2003 server. It is set to start manually, and it runs if a requester (a backup program) can use the writers in the system. Windows Server 2003 Service Pack 2 (SP2) addresses some issues that affect the performance of VSS.
For a list of the issues that are fixed in Windows Server 2003 SP2, see Microsoft Knowledge Base article 914962, List of updates in Windows Server 2003 Service Pack 2.
Although you can install only the updates for VSS, we recommend that you install the full service pack. Additionally, the Microsoft Knowledge Base article for the relevant update rollup package can help resolve some VSS snapshot issues in Windows Server 2003.
For more information, see Knowledge Base article 940349, Availability of a Volume Shadow Copy Service (VSS) update rollup package for Windows Server 2003 to resolve some VSS snapshot issues
The following hotfixes are available to resolve VSS issues that you may experience.
| KBID Number | Title |
|---|---|
A Volume Shadow Copy Service (VSS) update package is available for Windows Server 2003 |
|
The backup process may fail and a time-out error may occur in Volume Shadow Copy Service writers |
|
Event ID 9840 or 9607 is logged when a VSS backup operation fails in Exchange 2007 or in Exchange 2003 |
|
A snapshot backup of an Exchange Server 2003 database fails, and event ID 9607 is logged |
|
The backup application stops responding when you perform a backup that uses a volume snapshot in Windows Server 2003 with SP1 |
|
A snapshot copy backup of a storage group fails and Event IDs are logged in the Application log in Exchange Server 2003 |
Before a backup operation runs, the Exchange writer must be in a stable state, and must experience no errors. You can determine the state of the Exchange writer by running the following command:
VSSadmin list writers
The output from this command lists all writers that are registered by using VSS. In the command results, examine the State field of the writers. Notice whether the State field displays a state other than Stable. The following example output shows the Exchange writer in a failed state.
Writer name: 'Microsoft Exchange Writer' Writer Id: {GUID} Writer Instance Id: {GUD} State: [14] Failed Last error: Retryable error |
If the Exchange writer is in a failed state, restart the Microsoft Exchange Information Store service, and then run the VSSadmin list writers command to determine whether the writer state returns to a stable state. The following example output shows the Exchange writer in a stable state.
Writer name: 'Microsoft Exchange Writer' Writer Id: {GUID} Writer Instance Id: {GUD} State: [1] Stable Last error: No error |
If the output from the VSSadmin list writers command is blank, there may be a registry error that prevents the operating system from detecting and displaying the appropriate writer information. For more information about how to fix this issue and about how to re-create the list of VSS writers, see Microsoft Knowledge Base article 940184, Error message when you run the "vssadmin list writers" command on a Windows Server 2003-based computer: "Error: 0x8000FFFF".
The VSSadmin list providers command lists the registered Volume Shadow Copy providers on the computer. Typically, when you run this command, "Microsoft Software Shadow Copy provider 1.0" is listed as one of the providers or as the only provider. Other third-party backup programs may use their own providers. The following example output shows the results from the VSSadmin list providers command.
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool (C) Copyright 2001 Microsoft Corp. Provider name: 'Microsoft Software Shadow Copy provider 1.0' Provider type: System Provider Id: {GUID} Version: 1.0.0.7 |
For a successful backup, make sure that the provider or providers are listed and that the Exchange writer is in a stable state. For more information about other tests that you can run, type VSSadmin /? at a command prompt.
Tracing the Volume Shadow Copy Service
If you still experience backup issues, you should enable VSS tracing. For more information about how to enable tracing, see Microsoft Knowledge Base article 887013 How to enable the Volume Shadow Copy service's debug tracing features in Microsoft Windows Server 2003.
You can create a batch file to set the registry entries that are described in article 887013. To do this, type or copy the following text into a text file. Then, save the file as "EnableVSSTracing.bat." This batch file saves tracing information in a file that is named C:\trace.txt.
net stop swprv
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceFile /t REG_SZ /d C:\trace.txt /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceLevel /t REG_DWORD /d 0xffffffff /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceEnterExit /t REG_DWORD /d 1 /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceToFile /t REG_DWORD /d 1 /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceToDebugger /t REG_DWORD /d 1 /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceFileLineInfo /t REG_DWORD /d 1 /f
reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /v TraceForceFlush /t REG_DWORD /d 0 /f
To trace VSS, follow these steps:
Modify the Windows registry to enable tracing. To do this, follow the steps in article 887013, or run the EnableVSSTracing.bat file that you created previously.
Reproduce the VSS issue that you experience.
After you receive the VSS error that corresponds to the issue that you want to troubleshoot, stop the VSS services. To do this, run the following commands:
net stop VSS net stop SWPRVDisable VSS tracing. To do this, run the following command:
reg delete HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Debug\Tracing /fReview the trace file that was generated.
Tracing the Exchange Writer
The Exchange Writer is the built-in VSS writer that is provided together with the Microsoft Exchange information store. The writer runs in the Store.exe process. The Exchange Writer is used when the VSS requester (VSS backup program) requests a VSS backup from the active database. Like all the other components in Exchange Server 2007, the tracing is generated in an event trace log (ETL) file.
The information store provides the following tag for the Exchange writer.
| Tag | Description |
|---|---|
TagVSS |
In Exchange Server 2007, the only tag that is available for the Exchange writer is TagVSS. This tag combines all the VSS tags that are provided in Exchange 2003, including the following:
|
Note
The default freeze time-out for the Exchange writer is 20 seconds.
Tracing the Replica Writer
The Exchange Cluster Replica Writer is the built-in VSS writer that is provided together with the replication service. This writer runs in the M.E.Cluster.Replay.exe process. The Exchange Cluster Replica Writer is used when the VSS requester (VSS backup program) requests a VSS backup from the copy database. Like other components in Exchange Server 2007, the tracing is generated in ETL format.
The Replica Service provides the following tag for the Exchange Cluster Replica Writer.
| Tag | Description |
|---|---|
ReplicaVssWriterInterop |
|
Use Extra.exe to trace the Replica Writer
You can also use the Microsoft Exchange Troubleshooting Assistant (Extra.exe) to trace the Exchange VSS writer. To do this, follow these steps:
Run the following command from the Exchange Management Shell:
Set-EventLogLevel "MSExchange Repl\Exchange VSS Writer" -level expertClick Start, click Run, type extra.exe, and then click OK.
Click Go to the Welcome Screen, and then click Select a Task.
Click Trace Control, and then click OK if you receive a message that states that Exchange does not have a module to interpret traces.
Specify a location to save the trace file, and then set the trace file size to 200 megabytes (MB).
Click Set manual trace tags, and then click to select the following trace type check boxes:
PFD
Fatal
Error
Warning
Info
Debug
In the Components to trace list, click to select the Cluster.Replay check box.
Note
This check box is only applicable for a clustered Exchange environment.
In the Trace Tags list, click to select the following check boxes:
ReplicaInstance
ReplicaVssWriterInterop
In the Components to trace list, click to select the Store check box.
In the Trace Tags list, click to select the following check boxes:
tagVSS
tagMostError
Click Start Tracing.
Start an Exchange VSS backup. When the backup fails, click Stop tracing now in the Exchange Troubleshooting Assistant.
Running the BETest tool
BETest is a VSS requester that tests advanced backup and restore operations. This tool can be used to test an application's use of complex VSS features, such as the following:
Incremental and differential backup
Complex restore options, such as authoritative restore
Rollforward options
You can use the BETest tool to take a VSS backup of an Exchange storage group and then test the Exchange VSS writer. BETest can perform most of the operations that a VSS requestor can perform. You can also use BETest to take a VSS Snapshot of the active and replica databases on an Exchange 2007 server.
BETest is available in the VSS SDK 7.2. To obtain this SDK, see Microsoft Download Center article Volume Shadow Copy Service SDK 7.2.
The version of BETest that is available in the SDK uses a text-based version of the Components file (Components.txt). A version of BETest that uses an XML-based version of the Components.txt file is available in the Windows SDK. For more information about how to obtain and use the Windows SDK version of BETest, see BETest Tool.
Note
The steps in this topic use the text-based version of the Components file in BETest.
To use the BETest tool, follow these steps:
Obtain and install the BETest tool. By default, the i386 version of the tool is installed in the following folder:
C:\Program Files(x86)\Microsoft\VSSSDK72\TestApps\betest\obj\i386
Note
An AMD64-based version of BETest is also available. Always change to the directory of the appropriate operating system version before you run BETest.
Run the following command to obtain a list of available writers:
betest.exe > AvailableWriters.txtCreate a Components.txt file to specify the VSS components for BETest. The Components file uses the following format to specify options:
"<WriterId>": "<component-logical-path>" {"target" # "new target", ...}, ..."<component-logical-path>" : '"<subcomponent-logical-path>,...";Obtain the value for the WriterId entry for use with the Components.txt file. To do this, follow these steps:
Open the AvailableWriters.txt file by using a text editor, such as Notepad.
Search for the following text:
Microsoft Exchange Writer
Note the value for the WriterId field. For example, 76fe1ac4-15f7-4bcd-987e-8e1acb462fb7.
Determine the path for the <component-logical-path> placeholder in the Components.txt file. This path may be specified in one of the following formats:
<logical-path>
<logical-path>\<component-name>
<component-name> (if no logical path exists)
For example, Microsoft Exchange Server\Microsoft Information Store\<ServerName>\<StorageGroupGUID>.
Add the values to the Components.txt file, and then save the changes to the file. The following is example content of a Components.txt file.
|
In this example, the first GUID represents the Exchange writer. The second GUID represents a specific storage group. You can choose to run the BETest tool against only one storage group if you have multiple storage groups. To obtain the GUID for a particular storage group, you can use the LDP.exe tool or the Get-StorageGroup cmdlet. If you use the LDP tool, locate the following object, and then note the value for objectGUID in the details pane.
CN=First Storage Group,CN=InformationStore,CN=<ServerName>,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=First Organization,CN=Organizations,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=fourthcoffee,DC=com
Note
This path uses the default names for the organization and administrative group. Modify this path as appropriate for your organization.
To obtain the storage group GUID from the command-line, run the following command at the Exchange Management Shell:
Get-StorageGroup |fl Guid
Streaming backups are only supported from active storage groups. If you want to back up a passive database copy, you must use a VSS backup. If the database is a Continuous Cluster Replication database, or if you use Local Continuous Replication, the Components.txt file will resemble one of the following.
For a CCR replica copy
|
For an LCR replica copy
|
Note
The success or failure of a streaming backup does not indicate the health of the Exchange VSS writer.
After the Components.txt file has been saved, run the following command to start BETest:
BETEST.exe /B /E /T 1 /S output.XML /C components.txt /D c:\betest > Output.txtThis command creates a backup in the C:\BETest directory.
Note
You can also run this command without the /E option.
If the backup is unsuccessful, examine the contents of the Output.txt file to review any error messages that have been logged.
For More Information
Exchange Server 2007 Service Pack 1 (SP1) and several Exchange Update Rollup packages include fixes for VSS issues. We recommend that you install the latest Exchange 2007 service pack together with the latest Update Rollup package for Exchange. For more information, see How to Obtain the Latest Service Pack or Update Rollup for Exchange 2007.