What is DoubleSpace and How Does It Work?
| Archived content. No warranty is made as to technical accuracy. Content may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist. |
This chapter provides technical information and troubleshooting tips for DoubleSpace.
DoubleSpace is a technology that compresses data stored by the FAT file system in real time. Real time means that data is compressed and decompressed as it is written and read.
Real-time compression provides a transparent way to store data more compactly. Unlike file-compression utilities such as PKZIP, real-time compression compresses files when you store them and decompresses them when you need them. With a file-compression utility, you must run the utility once to compress a file, and again to decompress the file when you need to gain access to the file's contents.
On This Page
How Does DoubleSpace Compress Data?
How Does DoubleSpace Store Compressed Data on a Drive?
What is DBLSPACE.BIN?
What Happens During DoubleSpace Setup?
What Happens When MS-DOS 6 Starts With DoubleSpace?
MS-DOS 6 DoubleSpace Files and Their Functions
Removing DoubleSpace from Your Computer
Troubleshooting DoubleSpace
DoubleSpace System API Specification
Microsoft Real-Time Compression Interface (MRCI) Specification
How Does DoubleSpace Compress Data?
DoubleSpace compresses data by analyzing a block of data for repeated sequences, and then encoding these repeated sequences as a match in a very compact form. Consider the following sentence:
The rain in Spain falls mainly on the plain.
To compress this sentence, DoubleSpace first identifies the repeated sequences, and then writes them as <offset, length>. Offset is the number of bytes to the left of where the match starts, and length is the number of bytes matched. For example, this sentence would be compressed as follows:
The rain <3,3>Sp<9,4>falls m<11,3>ly on <34,37>pl<15,3>.
Once it identifies the matches, DoubleSpace encodes this information compactly. The key is to analyze the frequency of occurrence of the offset and length values, and then to choose very short encoding for the most common values.
How Does DoubleSpace Store Compressed Data on a Drive?
Compressed files are stored in a compressed volume file, or CVF. The CVF resides on an uncompressed drive, or host drive, and typically takes up almost all of the space on it. After you compress the existing files on a drive, the CVF contains the files from that drive. Usually, the host drive contains very little free space after compression, since the compressed volume file uses so much space. Because it is compressed, a CVF can store more data than the space it uses; for example, a typical CVF might use 10 MB of space on the host drive but contain 20 MB of compressed data.
A CVF is just a Hidden, Read-Only file until you mount it. Mounting a CVF establishes a connection between it and a drive letter, so that you can use that CVF as a drive.
DoubleSpace manages the files and free space on the CVF. A small amount of space is left over on the host drive for data that cannot or should not be compressed.
What is DBLSPACE.BIN?
DBLSPACE.BIN is a subsystem of the MS-DOS 6 operating system. It provides access to the data on compressed drives. DBLSPACE.BIN has two components:
A disk space manager
A MRCI server (a compress/decompress engine)
The Disk Space Manager
DBLSPACE.BIN is a compression subsystem that appears to the FAT file system like a block device driver; to the FAT file system, the CVF appears like any other drive. FAT stores information about files and directories, while DoubleSpace stores information only about the sector space it manages within the CVF. DoubleSpace can be thought of as a disk space manager.
The smallest FAT allocation unit is the cluster, which on a typical drive is 8K. On such a drive, a 1-byte file takes up 8K on the disk, and a 40K+1-byte file takes up 48K on the disk. This end-of-file waste, or cluster overhang, can be significant if the drive contains many small files.
The smallest DoubleSpace allocation unit is the sector, which is 512 bytes on most systems. This finer granularity is possible because DoubleSpace maintains its own disk allocation data structures within the CVF that map FAT clusters to DoubleSpace sectors. As a consequence, DoubleSpace minimizes cluster-overhang waste.
The Compress/Decompress Engine
The other component of DBLSPACE.BIN compresses and decompresses files as they are written to and read from the disk. There are tradeoffs in choosing a compression algorithm: speed of compression, speed of decompression, achieved compression ratio, and memory consumption. No algorithm provides the best of all. Speed is clearly important for real-time compression, and it is understood throughout the software industry which algorithms are optimal for use with real-time compression. Therefore, the compress/decompress engine in DBLSPACE.BIN, like that of add-in compression products, is based on Lempel-Ziv algorithms. Since Lempel-Ziv algorithms are extremely well-understood, it is not surprising that all real-time compression products, including DoubleSpace, achieve roughly the same compression ratios.
The efficiency of the compress/decompress code affects the speed of compression. DBLSPACE.BIN contains code that has been hand-tuned for optimum efficiency with the 8086/8088, 80286, and 80386 processors.
What Happens During DoubleSpace Setup?
When you run MS-DOS Setup, it installs MS-DOS 6 on your computer but does not compress your files. To actually compress your files, you must run DoubleSpace Setup by typing dblspace at the command prompt.
When DoubleSpace Setup starts, you can choose either Express or Custom Setup. Express Setup, which is the default, compresses the existing files on drive C. In Custom Setup, you can either compress the existing files on a selected drive (including drives other than C) or create a new, empty compressed drive and leave your existing files uncompressed.
When compressing the existing files on a drive, DoubleSpace first creates an empty CVF. DoubleSpace then repeats the following steps until all the files on that drive are compressed:
DoubleSpace reads an uncompressed file from the host drive.
It compresses the file.
It writes the file to the CVF.
It deletes the uncompressed file from the host drive.
DoubleSpace repeats steps 1 through 4 until there is no more space on the CVF. When there is no more space on the CVF, DoubleSpace increases the size of the CVF and resumes the process at step 1.
Compressing existing files (also called compressing in place) takes roughly one minute per megabyte of data on the drive.
The compression process is both safe and restartable. DoubleSpace ensures the safety of the data by flushing disk buffers regularly. Because of this, no data ever exists only in memory; any data currently in memory is also on the disk. If DoubleSpace is interrupted while compressing data (for example, if the computer is accidentally restarted), DoubleSpace recovers automatically. DoubleSpace keeps track of its progress by adding information to the AUTOEXEC.BAT file; if interrupted, DoubleSpace uses this information to resume compressing exactly where it left off.
When compressing existing files, DoubleSpace compresses and moves most files from that drive to the CVF. When DoubleSpace finishes compressing the drive, nothing is left on the uncompressed (host) drive except a few system files, which are either Hidden/System or Hidden/System/Read-Only. These system files are:
Filename |
Function |
|---|---|
IO.SYS |
An MS-DOS system file. The computer will start only if this file is stored on the uncompressed drive. |
MSDOS.SYS |
An MS-DOS system file. The computer will start only if this file is stored on the uncompressed drive. |
DBLSPACE.BIN |
The MS-DOS system file that provides access to compressed drives. The computer will start only if this file is stored on the uncompressed drive. |
DBLSPACE.000 |
The CVF that holds the compressed data. When mounted, the CVF appears to be a normal drive. |
DBLSPACE.INI |
File that contains configuration information for DoubleSpace. |
386SPART.PAR |
The Windows permanent swap file, which must be stored on the uncompressed drive. |
What Happens When MS-DOS 6 Starts With DoubleSpace?
When you start a computer on which DoubleSpace has been installed, the following occurs:
IO.SYS loads, and then looks on the startup drive for the DBLSPACE.BIN file. If DBLSPACE.BIN is there, IO.SYS loads it at the top of conventional memory. If DBLSPACE.BIN is not there, IO.SYS detects that the computer is not running DoubleSpace, and continues the startup process as it does without compression.
DBLSPACE.BIN reads settings from the DBLSPACE.INI file. These settings specify which CVFs DoubleSpace should mount as compressed drives and which drive letter to assign to each drive.
If DBLSPACE.BIN cannot find the DBLSPACE.INI file on the startup drive or on drive C, DLBSPACE.BIN unloads itself from memory, and MS-DOS continues as it would on an uncompressed system.
DBLSPACE.BIN mounts the CVFs specified by the DBLSPACE.INI file and assigns them the specified drive letters. (A CVF named DBLSPACE.000 indicates that the existing files on that drive were compressed. When DoubleSpace mounts a CVF with the filename DBLSPACE.000, it swaps the CVF's drive letter with the letter of its host drive.
MS-DOS processes the CONFIG.SYS file exactly as it would on an uncompressed system.
If MS-DOS encounters the command devicehigh=c:\dos\dblspace.sys /move in the CONFIG.SYS file and upper memory is available, then MS-DOS moves DBLSPACE.BIN from the top of conventional memory to upper memory. (DBLSPACE.BIN cannot initially load in upper memory because IO.SYS loads it before processing the CONFIG.SYS file. At this point in the startup process, no upper-memory manager is present, so upper memory blocks do not yet exist. Therefore IO.SYS must load DBLSPACE.BIN into conventional memory, which is less than optimal on 386 and 486 systems. But DBLSPACE.BIN loads at the top of conventional memory, not the bottom, so it can later relocate to upper memory without leaving a hole in lower conventional memory and fragmenting it.)
If the CONFIG.SYS file does not contain a command for DBLSPACE.SYS, MS-DOS automatically relocates DBLSPACE.BIN from the top of conventional memory to the bottom of conventional memory when it has finished processing the CONFIG.SYS file (loading DBLSPACE.SYS by using the device command has the same effect).
Note: The sys and format /s commands now copy DBLSPACE.BIN in addition to IO.SYS and MSDOS.SYS, so an MS-DOS 6 system floppy disk can be used to start a compressed system.
MS-DOS 6 DoubleSpace Files and Their Functions
The following table lists the DoubleSpace files and their functions.
Filename |
Function |
|---|---|
DBLSPACE.BIN |
The portion of MS-DOS that provides access to the data on compressed drives. DBLSPACE.BIN is loaded at system startup, before MS-DOS processes the CONFIG.SYS file. |
DBLSPACE.EXE |
Program you run to create or modify DoubleSpace drives. The first time you run this program, it runs as DoubleSpace Setup. Thereafter, it runs as the DoubleSpace maintenance program. |
DBLSPACE.HLP |
Help file for the DBLSPACE program.. |
DBLSPACE.INF |
Information file used by DoubleSpace Setup. |
DBLSPACE.INI |
DoubleSpace information file. |
DBLSPACE.WIN |
Used to keep track of Windows during DoubleSpace installation. |
DBLSPACE.SYS |
Device driver that, when loaded, determines the final location of DBLSPACE.BIN in memory. Can be used to move DBLSPACE.BIN to the top of conventional memory or to upper memory. |
DBLSPACE.00x |
Compressed volume file, which contains an entire compressed drive. |
DBLWIN.HLP |
Help file for the Windows DoubleSpace Info dialog. |
For definitions of terms used in this section, run DoubleSpace, and then choose Index from the Help menu.
DBLSPACE.BIN
DBLSPACE.BIN is the component of the MS-DOS operating system that mounts the compressed volume file so you can access the files on your compressed drive. MS-DOS loads DBLSPACE.BIN before processing the commands in your CONFIG.SYS file. (The file must be located in the root directory of your startup drive); no device command is needed to load DBLSPACE.BIN. (For information about loading DBLSPACE.BIN into upper memory, see the section below on DBLSPACE.SYS.)
DBLSPACE.EXE
This program allows you to create, mount, or modify DoubleSpace drives. If you have not yet installed DoubleSpace, typing dblspace at the command prompt starts the DoubleSpace Setup program. If you have already installed DoubleSpace, then typing dblspace runs the DoubleSpace maintenance program, which provides menu commands you can use to modify or create DoubleSpace drives. You can also perform many of these functions from the command prompt, by adding switches and other parameters to the dblspace command. For more information, type help dblspace at the command prompt.
DBLSPACE.HLP
This file contains the help for the DoubleSpace Setup and maintenance programs. To use DoubleSpace help, press F1 from any DoubleSpace screen or dialog box, or choose Contents or Index from the Help menu in the DoubleSpace maintenance program.
DBLSPACE.INF
DoubleSpace Setup uses the DBLSPACE.INF file when you are creating a new DoubleSpace drive.
Commands under the [SpecialFiles] section indicate actions to be taken for specific files; the possible actions are copy=, ignore=, and move=. Without a path, any file with that name will match; wild cards (* and ?) are acceptable.
Programs under the [CopyFiles] section are optional utilities that will be left on the original startup drive if there is enough space.
Programs under the [dangerous] section will be disabled during DoubleSpace Setup.
Programs under the [prior] section will have the command devicehigh=c:\dos\dblspace.sys /move placed above them in the CONFIG.SYS file.
DBLSPACE.INI
The DBLSPACE.INI file is a text file with the System, Read-Only, and Hidden attributes. DoubleSpace stores this file in the root directory of your startup drive. The DBLSPACE.INI file contains variables that DoubleSpace uses when your computer starts. Although it is possible to change these variables yourself, you should do so only if you understand what they do and what the results might be. Before changing the DBLSPACE.INI file, you should make a backup copy of the file.
The DBLSPACE.INI file can contain one or more of the following variables:
MaxRemovableDrives=n
Specifies how many additional drives DoubleSpace should allocate memory for when your computer starts. DoubleSpace allocates 96 bytes of memory for each additional drive. The MaxRemovableDrives variable determines how many additional compressed drives you can create or mount without restarting your computer.
FirstDrive=X
Set by DoubleSpace each time it modifies the DBLSPACE.INI file. Do not change the FirstDrive variable yourself.
LastDrive=Y
Specifies the highest drive letter available for use by DoubleSpace. (If another program uses one of the drive letters specified for DoubleSpace, the highest drive letter available to DoubleSpace will be higher than the LastDrive variable.)
MaxFileFragments=n
Set by DoubleSpace to specify the degree of fragmentation to allow in all mounted compressed volume files. Do not change the MaxFileFragments variable yourself.
ActivateDrive=X,Yn
Specifies a compressed volume file that DoubleSpace should mount automatically when your computer starts. The DBLSPACE.INI file can contain as many ActivateDrive variables as there are CVFs.
The ActivateDrive variable requires the X, Y, and n parameters. The way DoubleSpace uses these parameters depends on whether the specified CVF was created by compressing existing files or by using free space.
If the specified CVF was created by compressing existing files
If the filename of the CVF is DBLSPACE.000, then that CVF was created by compressing existing files. In that case, DoubleSpace uses the parameters for the ActivateDrive variable as follows:
X |
Specifies the drive letter to be assigned to the uncompressed (host) drive. |
Y |
Specifies the drive letter to be assigned to the compressed drive. |
n |
Specifies the filename extension of the compressed volume file. If the CVF was created by compressing existing files, n is set to 0. |
For example, the following is a sample ActivateDrive variable for a CVF that was created by compressing existing files:
ActivateDrive=K,C0
This sample ActivateDrive variable specifies that the CVF's filename is DBLSPACE.000. When mounted, the compressed drive will be assigned drive letter C; the uncompressed drive (which contains the CVF) will be assigned drive letter K.
If the specified CVF was created by using free space on a drive
If the filename of the CVF is not DBLSPACE.000 (for example, DBLSPACE.002), then that CVF was created by using free space on an existing drive. In that case, DoubleSpace uses the parameters for the ActivateDrive variable as follows:
X |
Specifies the drive letter to be assigned to the compressed drive. |
Y |
Specifies the drive letter to be assigned to the uncompressed (host) drive. |
n |
Specifies the filename extension of the compressed volume file. If the CVF filename is DBLSPACE.001, set n to 1; if the CVF filename is DBLSPACE.002, set n to 2; and so on |
For example, the following is a sample ActivateDrive variable for a CVF that was created by using free space on an existing drive:
ActivateDrive=J,D1
This sample ActivateDrive variable specifies that the CVF's filename is DBLSPACE.001. When mounted, the compressed drive will be assigned drive letter J; the uncompressed drive (which contains the CVF) will be assigned drive letter D.
DBLSPACE.WIN
The DBLSPACE.WIN file is a temporary file that DBLSPACE.BIN generates to keep track of Windows information when compressing a drive. DoubleSpace deletes the file automatically after it finishes compressing the drive.
DBLSPACE.SYS
The DBLSPACE.SYS device driver does not provide access to compressed drives; it simply moves DBLSPACE.BIN to its final location in memory. For more information, type help dblspace.sys at the command prompt.
DBLSPACE.00x
A file with a name in the form DBLSPACE.00x is a compressed volume file (CVF) and contains all the files on a compressed drive. A compressed volume file is stored in the root directory of the uncompressed (host) drive.
When you compress the files on an existing drive, DoubleSpace names the associated compressed volume file DBLSPACE.000. When you create a new compressed drive, DoubleSpace names the associated CVF by using a number such as 001 (for example, DBLSPACE.001).
Most CVFs can store more data than the space they use; for example, a typical CVF might use 10 MB of space on its host drive but contain 20 MB of compressed data. When a CVF is mounted, or made active, it is assigned a drive letter and appears as a disk drive. The CVF's contents are then accessible and appear as normal files.
DBLWIN.HLP
This file contains help for the Windows DoubleSpace Info dialog. (To display this dialog, choose the DoubleSpace Info command from the Tools menu in File Manager.)
Removing DoubleSpace from Your Computer
There is no method for automatically removing DoubleSpace. There are two ways to remove it manually:
Back up the files on all your compressed drives, remove DoubleSpace, and then restore the backed-up files onto your uncompressed drive. (Note that all the files currently on your compressed drive(s) might not fit on your hard disk after you remove DoubleSpace.)
Move as many files as possible from your compressed drive to your uncompressed drive, reduce the size of your compressed drive to free space on the uncompressed drive, and keep moving files and shrinking the compressed drive until no more files remain on the compressed drive. Then, remove DoubleSpace.
Important: The drive letter of your uncompressed drive may change after you remove DoubleSpace. If it does, any files or programs configured for use on the uncompressed drive (for example, your Windows permanent swap file) will need to be reconfigured.
The procedures in this section explain each method of removing DoubleSpace.
To remove DoubleSpace by using the back-up-and-restore method
Delete any unnecessary files from your compressed drives.
Back up the files on all compressed drives. If your backup program is located on a compressed drive, make sure you copy the backup program files to an uncompressed drive or to a floppy disk.
If you are using Microsoft Backup for MS-DOS, you need to copy the following program files:
MSBACKUP.EXE
MSBACKUP.OVL
MSBACKUP.INI
MSBACKDB.OVL
MSBACKDR.OVL
MSBACKFB.OVL
MSBACKFR.OVL
MSBCONFG.OVL
DEFAULT.SET
MSBACKUP.LOG
MSBACKUP.RSTTo determine which drive is your uncompressed drive, type dblspace /list at the command prompt. The uncompressed drive is listed under the CVF Filename column. For example, if H:\DBLSPACE.000 is the CVF Filename associated with drive C, drive H is the uncompressed drive.
If you are removing DoubleSpace from your startup drive, copy the COMMAND.COM file from your compressed drive to the root directory of your uncompressed drive.
Make your uncompressed drive the current drive. For example, if drive H is your uncompressed drive, type H: at the command prompt. To change to the root directory, type cd\ at the command prompt. If you want to delete all of your DoubleSpace drives, type the following at the command prompt:
deltree dblspace.*
To delete just one of your DoubleSpace drives, use the deltree command to delete the CVF for the drive. (The dblspace /list command also shows the CVF names for your drives.)
For example, if the CVF is DBLSPACE.000, type the following at the command prompt:
deltree dblspace.000
Restart your computer.
Restore your backed-up files. If your Backup program files are on a floppy disk, copy them to the hard disk first. Then run the Backup program from your hard disk.
Note: You might need to retrieve your catalog file from your backup floppy disks. To do so, choose the Catalog button in the Restore dialog box.
To remove DoubleSpace by using the move-and-resize method
Delete any unnecessary files from your compressed drives.
To determine which drive is your uncompressed (host) drive, type dblspace /list at the command prompt. The uncompressed drive is listed under the CVF Filename column. For example, if H:\DBLSPACE.000 is the CVF Filename associated with drive C, drive H is the uncompressed drive.
Delete any unnecessary files from the uncompressed drive, including your Windows permanent swap file (if any).
Change to your compressed drive, and then type dblspace /size at the command prompt. DoubleSpace will reduce the drive's size as much as possible, which will free some space on the uncompressed drive. (If you have more than one compressed drive, carry out this step for each one.)
If DoubleSpace cannot reduce a compressed drive's size because the drive is too fragmented, run Microsoft Defragmenter by typing defrag at the command prompt. When Defragmenter completes, type dblspace /size at the command prompt.
Use the move command to move files from the compressed drive to the uncompressed drive until only .5 MB of free space remains on the uncompressed drive.
Repeat steps 3 and 4 until your compressed drives do not contain any files you want to keep.
If you are removing DoubleSpace from your startup drive, copy the COMMAND.COM file from your compressed drive to the root directory of your uncompressed drive.
Make your uncompressed drive the current drive. For example, if drive H is your uncompressed drive, type H: at the command prompt. To change to the root directory, type cd\ at the command prompt. If you want to delete all of your DoubleSpace drives, type the following at the command prompt:
deltree dblspace.*
If you want to delete just one of your DoubleSpace drives, use the deltree command to delete the CVF for the drive. (The dblspace /list command also shows the CVF names for your drives.) For example, if the CVF is DBLSPACE.000, type the following at the command prompt:
deltree dblspace.000
Remove all references to dblspace from your CONFIG.SYS and AUTOEXEC.BAT files.
Restart your computer.
Troubleshooting DoubleSpace
The following section provides tips on troubleshooting DoubleSpace. For additional tips, see the DoubleSpace section of the README.TXT file.
If You Receive a Corrupt Swap File Warning After Removing a Compressed Drive
If you remove a compressed drive, you might receive the message "The swap file is corrupt" when you start Windows.
To solve this problem
Open Control Panel, and then double-click the 386 Enhanced icon.
Choose the Virtual Memory button. Windows displays a dialog box stating that a corrupt swap file was found and asks if you want to set the file's length to zero.
Choose the Yes button. Windows displays another Virtual Memory dialog box.
Choose the Change button. Windows displays swap-file settings.
In the Drive list box, select a drive that is not compressed. In the Type list box, select "Permanent."
If your uncompressed drive does not have enough free space to create a permanent swap file, create a temporary swap file on either your compressed or uncompressed drives. (For information about freeing space on your uncompressed drive, see the DoubleSpace section of the README.TXT file.)
When you have finished specifying swap-file settings, choose OK twice, and follow the instructions on your screen.
If Your Computer Restarts When You Are Specifying DoubleSpace Commands
If your computer restarts unexpectedly after you type dblspace /ratio or other DoubleSpace commands, edit your CONFIG.SYS file and move the command line for DBLSPACE.SYS before any commands that start network drivers.
DoubleSpace Setup places the command line for DBLSPACE.SYS before any commands for network drivers, so this problem should not occur unless you have moved the command line yourself.
If You See the Message "DoubleSpace cannot defragment because of an unknown error"
If you are running a screen-saver program when you run DoubleSpace Setup, you may receive the following error message:
DoubleSpace cannot defragment drive C because an unknown error occurred.
To work around this problem
Edit your CONFIG.SYS or AUTOEXEC.BAT file and disable or remove the command that starts the screen-saver program.
Restart your computer.
Run DoubleSpace Setup by typing dblspace at the command prompt.
After DoubleSpace Setup completes, edit your CONFIG.SYS or AUTOEXEC.BAT file and reenable the command that starts the screen-saver program.
If You See the Message "There are no more drive letters..."
If you have not installed DoubleSpace on your hard disk and you try to mount a compressed floppy disk, you receive the following error message:
There are no more drive letters reserved for DoubleSpace to use. To add more drive letters, choose the Options command from the Tools menu.
You must load DBLSPACE.BIN and create a DBLSPACE.INI file to mount compressed floppy disks if you have not installed DoubleSpace on your hard disk.
To mount a compressed floppy disk without running DoubleSpace Setup
Using a text editor such as MS-DOS Editor, create a file that contains only the following commands:
MaxRemovableDrives=2
LastDrive=F
The lastdrive command must specify one letter higher than your last logical drive letter. For example, if your last drive is E, the command should appear as lastdrive=f.
Save the file in the root directory of your startup disk; give it the filename DBLSPACE.INI.
Copy the DBLSPACE.BIN file from your MS-DOS directory to the root directory of your startup drive.
Restart your computer.
Mount the compressed floppy disk by using the dblspace /mount command. For example, if the compressed floppy disk is in drive A, type the following at the command prompt:
dblspace /mount a:
DoubleSpace System API Specification
MS-DOS 6 DoubleSpace integrated disk compression includes the DoubleSpace System Application Programming Interfaces (API), which are supported by DBLSPACE.BIN. You might want to make use of the DoubleSpace System API if you are a developer of disk-utility software. With the DoubleSpace System API, software such as defragmentation programs, disk-repair programs, disk-caching programs, and backup programs will be able to take advantage of DoubleSpace integrated disk compression.
To receive more information about the DoubleSpace System API, contact Microsoft Developer Relations at 1-800-227-4679.
Microsoft Real-Time Compression Interface (MRCI) Specification
MRCI is the Microsoft Real-Time Compression Interface, a software interface definition that allows a MRCI client to request compression services from a MRCI server that supports the Microsoft Real-Time Compression Format (MRCF). A MRCI server may implement support for MRCF compression and decompression either fully in software or with partial or complete hardware assistance. Today, MRCI servers exist only as software (for example, the Microsoft DoubleSpace compressed file system includes a software MRCI server), but in the future hardware-based MRCI servers will be common.
MRCI has two purposes:
To define standard, system-level, compress/decompress services that can be used by ISVs, independent of the implementation of those services.
To define a standard way for IHVs and PC manufacturers to implement dedicated compression hardware.
Note that MRCI defines a standard for lossless compression, which is different from the compression used for pictorial and video images and defined in standards such as JPEG and MPEG. Lossless compression is useful for many operating system and application functions, especially where the type of data is unknown and so cannot benefit from data type-specific algorithms. The benefits of lossless compression include:
Increasing the effective storage capacity of media
Maximizing the effective data bandwidth between two computers
Maximizing the effective data bandwidth between a computer and a storage device
MRCI is currently used by the Microsoft DoubleSpace compressed file system, the Microsoft Flash File System, and the Microsoft Backup program in MS-DOS 6.
Since MRCI defines an interface standard that allows for hardware implementations, it is important to note the advantages that compression/decompression hardware has over compression/decompression software:
Performance: Tightly integrated compression/decompression hardware (on the local bus, for example) will improve performance, but until there are actual implementations it is difficult to pinpoint the magnitude of improvement.
Multitasking performance: When used by a multithreaded operating system such as Windows NT, compression/decompression hardware acts as a highly specialized second CPU; this frees up the main CPU to execute other threads not requesting compression services.
Better compression: By using more intensive matching algorithms, compression/decompression hardware can improve compression ratios by 10 percent to 15 percent over software without paying any performance penalty.
Standards Defined by MRCI
MRCI defines the following standards:
Query API: MRCI defines a rendezvous API that lets an application check for the presence of MRCI-compliant compression services (possibly implemented in hardware).
Compress and decompress APIs: If a MRCI server is present, an application can disable its own internal software compression/decompression routines. Using the address returned from the query, the application can then transfer data to and from the MRCI server that it needs compressed or decompressed.
Compression format: A standard compression format means that given a stream of uncompressed bytes as input, all MRCI-compliant hardware will output exactly the same stream of compressed bytes, and vice versa for decompression. MRCI enables the exchange of compressed data between systems by defining a format which is a variant of Lempel-Ziv encoding.
How to Implement MRCI
ISVs that want take advantage of MRCI will need the following:
MRCI specification: This section of the Resource Kit, available to the general public.
MRCI software libraries for MS-DOS and Windows: The libraries (MRCFDOS.LIB and MRCFWIN.LIB) provide very fast real-time compression/decompression services in software and adhere to the MRCI compression format. These are the same libraries used in the MS-DOS 6 DoubleSpace and Backup programs. The libraries check for the presence of a MRCI server and use it if present. If not present, the libraries use their own internal software routines. So, by simply linking in a library, you get fast, real-time, MRCI-compliant compression/decompression services regardless of the hardware environment. There is no need to learn about the details of MRCF, since these details are hidden by the libraries.
To use the MRCI software libraries, you must license them from Microsoft. There is no licensing fee. For licensing information, contact Microsoft Developer Relations at 1-800-227-4679.
IHVs that want to develop MRCI-compliant hardware will need the following:
MRCI specification: This section of the Resource Kit, available to the general public.
MRCI software libraries for MS-DOS and Windows: See above. These libraries are necessary to test hardware for MRCI compatibility.
MRCI algorithms: C language reference code that implements the MRCI compression and decompression algorithms.
MRCI compression format: A description of the MRCI compression format.
To use the MRCI software libraries, algorithms, and compression format, you must license them from Microsoft. There is no licensing fee. For licensing information, contact Microsoft Developer Relations at 1-800-227-4679.
If you have a CompuServe account, you can obtain additional MRCI source examples and any updates to the MRCI information included in this guide.
Operating System Support
MRCI currently supports the MS-DOS and Microsoft Windows operating systems. MRCI for Windows NT will be defined at a later date, as an extension to the Hardware Abstraction Layer (HAL).
Overview of MRCI Services
The MRCI services are briefly described in the following table.
Service |
Description |
MRCQuery |
Return MRCI server information |
MRCCompress |
Compress a data buffer using StandardCompression |
MRCDecompress |
Decompress a data buffer |
MRCMaxCompress |
Compress a data buffer using MaxCompression |
MRCIncrementalDecompress |
Decompress a portion of a data buffer (optimization for DoubleSpace) |
MRCQuery enables a client or server to establish communication with an existing MRCI server. A software interrupt is used to make this call, the interrupt number being chosen to permit the server to be supplied in any manner (as code in ROM, or as a DOS device driver or TSR, or as a Windows VxD). A new MRCI server can install itself in the system, partially or fully replacing a previously installed MRCI server. Only the last MRCI server installed is accessible via MRCQuery.
The compression and decompression services, MRCCompress, MRCMaxCompress, MRCDecompress, and MRCIncrementalDecompress, permit a client to call a server to compress and decompress data in the MRC Format. These services are invoked via a direct call to the server (the address having been obtained by MRCQuery), for optimum performance.
Note: A client must enter the Windows Disk Critical Section before calling the direct call entry point of a MRCI server. This protects the MRCI server from being reentered. Failure to do so will cause data corruption and data loss in a multitasking environment. For more information, see the sample code in the "MRCI Definitions" section later in this chapter.
MRCI Interrupts
MRCI defines a new interrupt 2Fh call (AX=4A12h) to allow a MRCI client to detect a MRCI server, and a new interrupt 1Ah call (AX=B001h) to allow the first MRCI server or client to detect a ROM BIOS (presumably hardware-based) MRCI server. MRCI clients and servers first check for the presence of an existing MRCI server by issuing the INT 2Fh call. If that fails, they issue the INT 1Ah call. If that fails, then there is no MRCI server present.
These two rendezvous interrupts are provided to ensure compatibility with existing MS-DOS software. If a software MRCI server intends to install itself over an existing INT 1Ah MRCI server, or a MRCI client is going to remain resident (either as an MS-DOS device driver or terminate-and-stay-resident (TSR) program), then it must hook INT 2Fh and create a RAM copy of the MRCINFO structure (see below). The procedure allows subsequent programs to supersede the MRCI server without cutting the initial resident program out of the loop.
MRCI Definitions
The following ASM headers document MRCI. These are specified in ASM because that is the highest-performance interface. These may be called from C with suitable C "wrapper" functions.
;*** MRCI.INC - Microsoft Real-Time Compression Interface definitions
;
; MRCI version 1.00.07 12-Mar-1993
intMRCI equ 2Fh ; MRCI interrupt number
mrciDETECT equ 04A12h ; intMRCI AX for detecting MRCI server
intMRCIROM equ 1Ah ; ROM MRCI interrupt number
mrciDETECTROM equ 0B001h ; intMRCIROM AX for detecting MRCI server
;*** mcXXXX - flag values passed to MRCI operations
;
; MRCCompress and MRCDecompress take a flag to indicate whether the
; client is a *system* component (and hence may call with InDOS
; set), or an *application*.
;
; mcSYSTEM clients must ensure the following is true before calling
; MRCI:
; 1) The Windows Disk Critical Section is owned
; 2) The InDOS flag is *set*
;
; mcAPPLICATION clients must ensure the following is true before calling
; MRCI:
; 1) The Windows Disk Critical Section is owned
; 2) The InDOS flag is *clear*
; FAILING TO FOLLOW THE ABOVE RULES WILL LIKELY RESULT IN A SYSTEM
; HANG AND LOSS OF USER DATA.
;
mcAPPLICATION equ 0 ; Client is an application
mcSYSTEM equ 1 ; Client is a file system driver
;*** micapXXXXX - bit flags for MRCINFO.mi_flCapabilities
;
; These define both the capabilities of the Server, and also double
; as *operation* codes passed to the mi_pfnOperate entry point in
; the server.
; 111111
; 5432109876543210
; ----------------
micapNONE equ 0000000000000000b ; No capabilities
micapSTANDARD equ 0000000000000001b ; Standard compress
micapDECOMPRESS equ 0000000000000010b ; Standard compress
micapRESERVED_1 equ 0000000000000100b ; RESERVED for future use
micapMAX equ 0000000000001000b ; MaxCompress
micapRESERVED_2 equ 0000000000010000b ; RESERVED for future use
micapINCDECOMP equ 0000000000100000b ; Incremental Decompress
;
; Remaining bits (6..14) are RESERVED and must be 0
;
micapREADONLY equ 1000000000000000b ; MRCINFO structure is read-only
micapDEINSTALL equ 1111111111111111b ; Server deinstall service
;*** MRCINFO - MRC Information data structure
;
; A pointer to a MRCINFO structure is returned from MRCQuery, and sent
; on MRCNotifyLoad. This structure contains information on the MRCI
; server and its capabilities.
MRCINFO struc
mi_lVendor dd ? ; A 4-byte vendor ID.
; Microsoft's vendor ID is "MSFT".
mi_wVendorVersion dw ? ; Version number of the MRC server.
; High byte is major number, low byte is minor.
; EXAMPLES: v3.20 = 0314h, v10.01 = 0A01h
mi_wMRCIVersion dw ? ; Version number of the MRCI supported by
; this server.
mi_pfnOperate dd ? ; Far pointer of the server compression entry
; point.
; NOTE: Caller must ensure that the Windows
; critical section is held *before*
; calling this entry point!
mi_flCapability dw ? ; Bit field of server capabilities
; See micapXXX for bit definitions
mi_flHWAssist dw ? ; Bit field of hardware assisted
; server capabilities. One-to-one
; correspondance with mi_flCapability
; bits. A bit set in this field
; indicates the corresponding
; capability is hardware assisted.
mi_cbMax dw ? ; Maximum number of bytes that the compression
; services provider can compress or decompress.
; Requests to compress or decompress buffers in
; excess of this length will fail.
; All MRCI servers are to support at least
; 8192 byte (8Kb) blocks.
MRCINFO ends
;*** MRCREQUEST - MRC compress/decompress Request packet
;
; This structure is used to pass parameters to the server for
; compress/decompress operations.
;
; General Notes
; -------------
; (1) <mr_pbSrc,mr_cbSrc> and <mr_pbDst,mr_cbDst> MUST NOT
; OVERLAP!
;
; (2) The safest practice is for mr_cbSrc and mr_cbDst to be
; identical.
; Details on Structure Members
; ----------------------------
; mr_pbSrc
; This points to the *source* buffer.
;
; On a *compress* operation, the contents of this buffer are
; *uncompressed* data.
;
; On a *decompress* operation, the contents of this buffer are
; *compressed* data.
;
; On a *incremental decompress* operation, this field points to
; the next section of compressed data to be uncompressed. The
; server updates the offset portion of this address after each
; incremental decompress call and the application should not modify
; this address between incremental decompress calls on the same block
; of compressed data.
;
; mr_cbSrc
; This is the size of the *source* buffer.
;
; On a *compress* operation, this is the amount of data to
; be compressed.
;
; For a *decompress* operation, this value is ignored. The amount
; of data to be decompressed is specified by the mr_cbDst parameter,
; described below.
;
; mr_RESERVED
; RESERVED for future use. Should be 0.
;
; mr_pbDst
; This points to the *destination* buffer.
;
; On a *compress* operation, this buffer receives the *compressed*
; result of the operation.
;
; On a *decompress* operation, this buffer receives the
; *uncompressed* result of the operation.
;
; On a *incremental decompress* operation, this field points to
; the next location in the destination buffer where uncompressed
; data is to be stored. The server updates the offset portion of
; this address after each incremental decompress call, and the
; application should not modify this address between incremental
; decompress calls on the same block of compressed data.
;
; mr_cbDst
; On INPUT, for a *compress* operation, this is the size of the
; *destination* buffer. If the compressed data would overflow
; this buffer length, then the operation fails and the server
; returns the error MRCI_ERROR_BUFFER_OVERFLOW.
;
; On INPUT, for a *decompress* operation, this must be EXACTLY the
; number of bytes that will be decompressed, as the MRCI server
; uses this information to determine when to stop decompressing.
;
; On INPUT, for an *incremental decompress* operation, this is
; the number of bytes that should uncompressed at this time. This
; will typically be less than the original uncompressed size of the
; compressed block. A single compressed block can be uncompressed in
; steps by making multiple incremental decompress calls for smaller
; sized blocks. However, to incrementally decompress an entire
; compressed block requires that the sum of the individual mr_cbDst
; counts be EXACTLY the number of bytes that were originally
; compressed so the MRCI server can determine when to stop
; decompressing.
;
;
; On OUTPUT, the Server updates this field with the actual size
; of the resulting compressed/uncompressed data.
;
; mr_cbChunk
; This is information that the Server compress routines can use to
; "early out" of the compression as early as possible.
;
; Valid values are 1 (client is interested in savings as small as
; 1 byte) to 32767. DblSpace passes 512, and Flash File System
; passes 1.
;
; This field is most easily explained by giving an example:
;
; Example:
; DblSpace does space allocation in chunks of 512 bytes (the
; common sector size on a disk).
;
; The compression server can use this information for two
; optimizations:
;
; (1) If the Server cannot compress the uncompressed data
; enough to save at least 512 bytes, then the data is
; *incompressible* as far as DblSpace is concerned, even
; if it could be compressed to save fewer than 512 bytes.
;
; (2) While compressing, if the Server gets to a point where
; the remaining uncompressed data is of such a length that
; it can be encoded simply (without table lookups, etc.)
; and not cross a 512 byte boundary, then the Server can
; do the simple encoding.
;
; It is likely that these optimizations will be hard to perform
; quickly in software, but it is possible that hardware can do
; these optimizations without any performance loss.
;
; mr_dwIncDecomp
; NOTE: This is used for Incremental Decompression only.
;
; For the first *incremental decompression* call on a compressed
; block, this value must be set to zero. Upon return, the field
; will contain state information for use on the next incremental
; decompress call. This value must not be modified between
; subsequent incremental decompress calls on the same compressed
; block.
MRCREQUEST struc
mr_pbSrc dd ? ; Pointer to source buffer
mr_cbSrc dw ? ; Size of source buffer, in bytes
mr_RESERVED dw ? ; RESERVED for future use.
mr_pbDst dd ? ; Pointer to destination buffer
mr_cbDst dw ? ; Size of destination buffer, in bytes
mr_cbChunk dw ? ; Client compressed data storage chunk size (see above!)
mr_dwIncDecomp dd ? ; Incremental Decompression state
MRCREQUEST ends
;*** MRCI_ERROR_XXX definitions
;
; Error codes returned from MRCIOperate
;
MRCI_ERROR_NONE equ 0 ; No error
MRCI_ERROR_NOT_SUPPORTED equ 1 ; Unsupported operation requested
MRCI_ERROR_BUSY equ 2 ; Server is busy
MRCI_ERROR_BUFFER_OVERFLOW equ 3 ; Destination buffer too small
MRCI_ERROR_NOT_COMPRESSIBLE equ 4 ; Data could not be compressed
MRCI_ERROR_BAD_MRC_FORMAT equ 5 ; Compressed data format is bad
;*** DefineMRCQuery - Macro to generate MRCQuery function
;
; Put this macro somewhere in your code segment. It will define the
; MRCQuery routine, which you can then call far. See the MRCQuery
; header below for documentation on its behavior.
DefineMRCQuery macro
;*** sigOLD_CX, sigOLD_DX, sigNEW_CX, sigNEW_DX - MRCI Server detection
;
; These values are used to verify that the response from issuing
; intMRCI is coming from a MRCI server, and not some other piece
; of code.
;
; The *old* values are passed on the mrciQUERY call, and the server
; must change CX/DX to the *new* values, so that the caller can
; trust that the MRCI server was responding, and not some other
; interrupt hook.
;
; The Server uses this code sequence to transform CX/DX:
;
; ;------------- entry: cx='ab' dx='cd'
;
; xchg ch,cl ; cx='ba' dx='cd'
; xchg dh,dl ; cx='ba' dx='dc'
; xchg dx,cx ; cx='dc' dx='ba'
;
sigOLD_CX equ 'MR'
sigOLD_DX equ 'CI'
sigNEW_CX equ 'IC'
sigNEW_DX equ 'RM'
;*** MRCQuery - Detect presence of MRCI server, return MRCINFO
;
; Detect presence of MRCI server safely, and if present return
; pointer to the server's MRCINFO structure. NOTE that we check
; first for a RAM-based server, and then for a ROM-based server.
;
; Entry
; none
;
; Exit-Success
; ax = 0, MRCI server is present.
; es:di -> MRCINFO structure
;
;
; Exit-Failure
; ax = 1, NO MRCI server is present.
;
; Uses
; ax,di,es,flags
MRCQuery proc near
;* Save caller's registers
SaveReg <ax,bx,cx,dx,si,bp,ds>
;* Check intMRCI vector before we issue the interrupt.
xor ax,ax ; Segment of interrupt vector table
mov ds,ax
lds si,ds:[intMRCI*4] ; ds:si -> MRCI server
;* Test if vector is plausible
mov ax,ds
or ax,ax ; Vector hooked?
jz mdr ; NO, go make another check
;* Call the server
mov ax,mrciDETECT ; Function
mov cx,sigOLD_CX ; Signatures for validation
mov dx,sigOLD_DX
int intMRCI ; Call server
cmp cx,sigNEW_CX ; Signature match?
jne mdr ; NO, go make another check
cmp dx,sigNEW_DX ; Signature match?
je mdp ; YES, have server
;* Server not present, check for ROM based server
mdr:
xor ax,ax
mov ds,ax
lds si,ds:[intMRCIROM*4] ; ds:si -> ROM MRCI server
;* Test if vector is plausible
mov ax,ds
or ax,ax ; Vector hooked?
jz mde ; NO, fail
;* Call the server
mov ax,mrciDETECTROM ; Function
mov cx,sigOLD_CX ; Signatures for validation
mov dx,sigOLD_DX
int intMRCIROM ; Call ROM server
cmp cx,sigNEW_CX ; Signature match?
jne mde ; NO, go make another check
cmp dx,sigNEW_DX ; Signature match?
jne mde ; No, fail
;* Server is present
mdp:
xor ax,ax ; Indicate success
jmp short mdx ; Go exit
;* Set error
mde: mov ax,1 ; Indicate failure
;* Restore caller's registers and exit
mdx: RestoreReg <ds,bp,si,dx,cx,bx,ax>
ret
MRCQuery endp
endm ;; DefineMRCQuery
;*** DefineMRCCompress - Macro to generate MRCCompress function
;
; Put this macro somewhere in your code segment. It will define the
; MRCCompress routine, which you can then call far. See the MRCCompress
; header below for documentation on its behavior.
DefineMRCCompress macro
;*** MRCCompress - Compress an uncompressed data buffer
;
; Entry
; ax = operation to perform:
; micapSTANDARD
; micapMAX
; cx = type of client:
; mcAPPLICATION - application
; mcSYSTEM - file system client
;
; ds:si -> MRCREQUEST structure
; mr_pbSrc - Pointer to uncompressed data buffer
; mr_cbSrc - Length of uncompressed data
; mr_pbDst - Pointer to compressed data buffer
; mr_cbDst - Length of compressed data buffer
; mr_cbChunk - Granularity of compressed data storage
;
; es:bx -> MRCINFO structure returned by MRCQuery
;
; Exit-Success
; ax = 0, compress operation completed
; ds:[si].mr_cbDst has length of compressed data in mr_pbDst.
;
; Exit-Failure
; Contents of mr_pbDst buffer and value of mr_cbDst are
; undefined.
; ax = non-zero error code:
; MRCI_ERROR_NOT_SUPPORTED
; Server does not support this operation.
;
; MRCI_ERROR_BUSY
; Server is busy with another operation. Try again later.
; NOTE: The most common case where this could occur is if an
; application calls the server while a disk cache
; (like SmartDrive) is writing its lazy-write queue at
; interrupt time to a compressed drive. The application
; should try the operation again
;
; MRCI_ERROR_BUFFER_OVERFLOW
; The destination buffer size (mr_cbDst) was not large enough
; to hold the compressed data.
;
; MRCI_ERROR_NOT_COMPRESSIBLE
; The data was not compressible, i.e., the size of the compressed
; data would have been greater than (mr_cbDst - mr_cbChunk).
;
; Uses
; ax,flags
MRCCompress proc near
;* Save caller's registers
SaveReg <bx,cx,dx,bp,si,di,ds>
;* Enter a Windows disk critical section. This must be done to prevent
; the MCRI server from being reentered if multiple VMs are making MRCI
; calls under 386 Enhanced mode Windows. The MRCI server sets the
; InDOS flag, but that is not enough, as InDOS is a per-VM variable.
push ax ; NOTE: Use this exact sequence of
mov ax,8001h ; instructions because Windows
int 2ah ; expects it and will patch
pop ax ; other code here
call dword ptr es:[bx].mi_pfnOperate ; Call Server
;* Release the disk critical section. The MRCI server is now available.
push ax ; NOTE: Use this exact sequence of
mov ax,8101h ; instructions because Windows
int 2ah ; expects it and will patch
pop ax ; other code here
;* Restore caller's registers and exit
RestoreReg <ds,di,si,bp,dx,cx,bx>
ret
MRCCompress endp
endm ;; DefineMRCCompress
;*** DefineMRCDecompress - Macro to generate MRCDeCompress function
;
; Put this macro somewhere in your code segment. It will define the
; MRCDecompress routine, which you can then call far.
; See the MRCDecompress header below for documentation on its behavior.
DefineMRCDecompress macro
;*** MRCDecompress - Decompress an compressed data buffer
;
; Entry
; cx = type of client:
; mcAPPLICATION - application
; mcSYSTEM - file system client
;
; ds:si -> MRCREQUEST structure
; mr_pbSrc - Pointer to compressed data buffer
; mr_cbSrc - Length of compressed data
; mr_pbDst - Pointer to uncompressed data buffer
; mr_cbDst - Length of uncompressed data buffer
;
; es:bx -> MRCINFO structure returned by MRCQuery
;
; Exit-Success
; ax = 0, decompress operation completed
; ds:[si].mr_cbDst has length of uncompressed data in mr_pbDst.
;
; Exit-Failure
; Contents of mr_pbDst buffer and value of mr_cbDst are
; undefined.
; ax = non-zero error code:
; MRCI_ERROR_NOT_SUPPORTED
; Server does not support this operation.
;
; MRCI_ERROR_BUSY
; Server is busy with another operation. Try again later.
; NOTE: The most common case where this could occur is if an
; application calls the server while a disk cache
; (like SmartDrive) is reading ahead at interrupt time to
; a compressed drive. The application should try the
; operation again.
;
; MRCI_ERROR_BUFFER_OVERFLOW
; The destination buffer size (mr_cbDst) was not large enough
; to hold the uncompressed data.
;
; MRCI_ERROR_BAD_MRC_FORMAT
; The compressed data format was invalid (generally only
; detectable as an overrun of the source buffer length, because
; the MRC Format has no redundancy).
; NOTE: Most software implementations will not generate this
; error, since it is to expensive (in time) to check for
; buffer overrun. Hardware implementations may be able
; to check for this without performance penalty, however.
;
; Uses
; ax,flags
MRCDecompress proc near
;* Save caller's registers
SaveReg <bx,cx,dx,bp,si,di,ds>
;* Enter a Windows disk critical section. This must be done to prevent
; the MCRI server from being reentered if multiple VMs are making MRCI
; calls under 386 Enhanced mode Windows. The MRCI server sets the
; InDOS flag, but that is not enough, as InDOS is a per-VM variable.
push ax ; NOTE: Use this exact sequence of
mov ax,8001h ; instructions because Windows
int 2ah ; expects it and will patch
pop ax ; other code here
mov ax,micapDECOMPRESS
call dword ptr es:[bx].mi_pfnOperate ; Call Server
;* Release the disk critical section. The MRCI server is now available.
push ax ; NOTE: Use this exact sequence of
mov ax,8101h ; instructions because Windows
int 2ah ; expects it and will patch
pop ax ; other code here
;* Restore caller's registers and exit
RestoreReg <ds,di,si,bp,dx,cx,bx>
ret
MRCDecompress endp
endm ;; DefineMRCDecompress
MRCI System Client Example
;*** MRCSYSCL.ASM - MRCI System Client EXAMPLE
;
; MRCI version 1.00.07 12-Mar-1993
;
; NOTE: This example is implemented assuming the code segment is
; writeable, to simplify the exposition.
;
; This is an example "system client" of the Microsoft Real-time
; Compression API. A "system client" is a device driver or TSR
; that loads and stays resident for the lifetime of the system.
; DoubleSpace, Flash File System 2, and network drivers are examples
; of system clients.
;
; By contrast, an "application client" is a program like MSBackup,
; which loads, uses MRCI, and then terminates.
;
; This example demonstrates the proper technique for:
;
; 1) Calling the MRCI server to get its capabilities
; 2) Calling the MRCI server to perform Standard Compress
; 3) Calling the MRCI server to perform Decompress
include utility.inc
include mrci.inc
;***********************
;* Private Definitions *
;***********************
cbBUFFER equ 8192 ; Our compression block size
;****************
;* CODE segment *
;****************
code segment
assume cs:code
;*************
;* DATA AREA *
;*************
pmi dd 0 ; Pointer to MRCINFO structure
buffer1 db cbBUFFER dup (?) ; One data buffer
buffer2 db cbBUFFER dup (?) ; Another data buffer
mr MRCREQUEST <> ; MRCI request block
;*********************
;* USE MRCI Services *
;*********************
main proc
call InitMRC ; Get MRCI server information
or ax,ax ; MRCI server present
jnz max ; NO, exit
;* Compress some data
;
; Assume buffer1 has uncompressed data
;
mov ax,cs ; Get CS in AX, for buffer pointers
mov ds,ax
mov si,OFFSET mr ; ds:si -> our MRCREQUEST packet
;* Set source pointer, length
mov ds:[si].mr_pbSrc.offst,OFFSET buffer1
mov ds:[si].mr_pbSrc.segmt,ax
mov ds:[si].mr_cbSrc,cbBUFFER
;* Set destination pointer, length
mov ds:[si].mr_pbDst.offst,OFFSET buffer2
mov ds:[si].mr_pbDst.segmt,ax
mov ds:[si].mr_cbDst,cbBUFFER
;* Set storage chunk granularity
mov ds:[si].mr_cbChunk,1 ; We store at byte granularity
;* Get MRCINFO pointer for call
les bx,cs:pmi
;* Set operation and client type
mov ax,micapSTANDARD
mov cx,mcSYSTEM
call MRCCompress ; Compress!
or ax,ax ; Did compress work?
jnz max ; NO, go exit
;* buffer2 now has compressed data, and
; ds:[si].mr_cbDst has the size of the compressed data.
;* Decompress some data
;
; Assume buffer1 has compressed data
;
mov ax,cs ; Get CS in AX, for buffer pointers
mov ds,ax
mov si,OFFSET mr ; ds:si -> our MRCREQUEST packet
;* Set source pointer, length
mov ds:[si].mr_pbSrc.offst,OFFSET buffer1
mov ds:[si].mr_pbSrc.segmt,ax
mov ds:[si].mr_cbSrc,cbBUFFER
;* Set destination pointer, length
mov ds:[si].mr_pbDst.offst,OFFSET buffer2
mov ds:[si].mr_pbDst.segmt,ax
mov ds:[si].mr_cbDst,cbBUFFER
;* Get MRCINFO pointer
les bx,cs:pmi
;* Set client type
mov cx,mcSYSTEM
call MRCDecompress ; Compress!
or ax,ax ; Did compress work?
jnz max ; NO, go exit
;* buffer2 now has decompressed data, and
; ds:[si].mr_cbDst has the size of the decompressed data.
;* Do something else interesting
;* Nothing else to do, this is just an example.
max: ret
main endp
;** BEGIN: Include code for MRCCompress/MRCDecompress here!
;
DefineMRCCompress
DefineMRCDecompress
;
;** END: Include code for MRCCompress/MRCDecompress here!
;******************
;* INIT-TIME CODE *
;******************
;** BEGIN: Include code for MRCQuery here!
;
DefineMRCQuery
;
;** END: Include code for MRCQuery here!
;** InitMRC - Get address of MRCI Server
;
;
; Entry
; none.
;
; Exit-Success
; ax = 0
; pmi = pointer to MRCI server MRCINFO structure
;
; Exit-Failure
; ax != 0, no MRCI server present
;
; Uses
; ax,bx,es,flags
InitMRC proc near
assume ds:nothing,es:nothing,ss:nothing
call MRCQuery ; es:bx -> MRCINFO, if server present
or ax,ax ; Is server present?
jnz imx ; NO, go fail (ax != 0)
;* Server present, save MRCINFO pointer
mov cs:pmi.offst,bx ; Store offset
mov ax,es
mov cs:pmi.segmt,ax ; Store segment
xor ax,ax ; Indicate success
imx: ret
InitMRC endp
code ends
end
MRCI Server Example
;*** MRCSRV.ASM - MRCI Server EXAMPLE
;
; MRCI version 1.00.07 12-Mar-1993
;
; NOTE: This example is implemented assuming the code segment is
; writeable, for simplicity.
;
; EXPORTED FUNCTIONS:
; Server - intMRCI interrupt handler
; Operate - Compress/Decompress entry point
; Strategy - DOS device driver strategy entry point
; Interrupt - DOS device driver interrupt entry point
;
;
; INTERNAL FUNCTIONS:
; InitMRCServer - Install Server
; ShouldWeInstall - Test if we should replace existing server
; GetInDOSFlagPointer - Get address of InDOS flag
; DoCompress - Unimplemented compress routine
; DoDecompress - Unimplemented decompress routine
include utility.inc
include mrci.inc
;***********************
;* Private Definitions *
;***********************
MYVERSION equ 0100h ; Version of this server
MRCIVERSION equ 0100h ; Version of MRCI supported by this server
MYVENDOR equ 5446534Dh ; "MSFT" (reversed for byte ordering)
MYVENDORhi equ 5446h ; High word of MYVENDOR
MYVENDORlo equ 534Dh ; Low word of MYVENDOR
cbMAX equ 8192 ; Maximum compression block size
;* This server's capabilities
micapMine equ micapSTANDARD or micapDECOMPRESS
;******************
;* PRIVATE MACROS *
;******************
;*** EnterCriticalSection - Grab server critical section
;
; Entry
; cx - Flag indicating whether the caller may already be inside DOS.
; mcAPPLICATION (0) - InDOS must be 0.
; mcSYSTEM (1) - InDOS may have any value.
;
; Exit-Success
; Zero Flag Set, critical section entered
; InDos is guaranteed to be non-zero
; fBusy set to non-zero
;
; Exit-Failure
; Zero Flag clear, critical section NOT entered
;
; Uses
; dx,si,ds,flags
EnterCriticalSection macro reg
cli ; Grab our lock atomically
cmp cs:fBusy,0 ; Are we already working on something?
jnz ecsx ; YES, return failure
;* Make InDOS non-zero
;
; NOTE: We check for failure here. We should not have to, since if we
; are busy, fBusy would be set, and we would have detected it above.
; The other case is a non-file system client calling, but InDOS is
; set. This case should not happen, because if InDOS is set, no
; non-file system client should gain control of the CPU and try to
; call us. So, if such a non-file system client tries this, we
; fail the call.
;
lds si,cs:fpInDOS ; ds:si -> InDOS flag
mov dl,ds:[si] ; dl = InDOS flag
or dl,dl ; In DOS?
jz ecs10 ; NO, okay to take critical section
;* InDOS is non-zero, see if caller allows this
; Zero Flag is CLEAR (=> not zero) from falling through the "jz" above
;
; NOTE: We allow a system client to call even if InDOS is not set, since
; it is possible that a system client may need to operate at
; interrupt time for "background" operations.
;
; *** THIS IS DISCOURAGED BEHAVIOR ***
;
jcxz ecsx ; NO, caller is not file system
.errnz mcAPPLICATION
;* InDOS was zero, or caller was a file system
ecs10: mov cs:oldInDOS,dl ; Save old InDOS flag, for later
mov byte ptr ds:[si],1 ; Set InDOS flag
mov cs:fBusy,1 ; We are now busy!
xor dl,dl ; Set zero flag to indicate success
ecsx: ; done with macro, zero flag indicates success/failure
sti ; Done being atomic
endm ;; EnterCriticalSection
;*** LeaveCriticalSection - Leave server critical section
;
; Entry
; cx - Flag indicating whether the caller may already be inside DOS.
; mcAPPLICATION (0) - Application client
; mcSYSTEM (1) - File System client
;
; Exit
; InDos restored to value saved by EnterCriticalSection
; fBusy set to zero
;
; Uses
; ax,si,ds,flags
LeaveCriticalSection macro reg
cli ; Release our lock atomically
;; Should assert that fBusy is set
;; cmp cs:fBusy,0 ; Are we already working on something?
;; jnz lcs20 ; YES, but not any longer
;;lcs10:
;;
;; Handle case of server attempting to leave critical section
;; without being inside critical section!
;;lcs20:
;* Restore InDOS
lds si,cs:fpInDOS ; ds:si -> InDOS flag
;; Should assert that InDOS is non-zero
;; cmp ds:[si],0 ; InDOS set?
;; jz lcs10 ; NO, but should have been!
mov al,cs:oldInDOS ; al = old InDOS flag
mov ds:[si],al ; Restore old InDOS flag
;* Clear our busy flag
mov cs:fBusy,0 ; No longer busy
sti ; Done being atomic
endm ;; LeaveCriticalSection
;****************
;* CODE segment *
;****************
code segment public byte
assume cs:code,ds:nothing,es:nothing,ss:nothing
;; org 0 ; Driver starts at 0
;*************
;* DATA area *
;*************
;** Character Device Header
;
DHlink dd 0FFFFFFFFh ; Next device => none
DHattr dw 8000h ; Simple character device
DHstrat dw Strategy ; Device strategy entry point
DHinter dw Interrupt ; Device interrupt entry point
DHname db 'MRCISRV$' ; Device name
;** mrcinfo - MRCINFO to return to clients
;
; NOTE: This structure must be writeable, so that a new Server
; can install and hook this structure with its values.
;
; vendor,server version,MRCI version,entry point,capabilities,cbMAX
;
mi MRCINFO <MYVENDOR,MYVERSION,MRCIVERSION,Operate,micapMine,cbMAX>
;** fpOldintMRCI - previous contents of intMRCI interrupt vector
;
fpOldintMRCI dd ?
;** pmi - pointer to active MRCINFO structure
;
; If we replace an old server, then that server's MRCINFO structure
; is the one that we use. So, we need a level of indirection to it.
; The more common case is that we are the only server, so this
; variable will just point to our mi.
;
pmi dd ?
;** rpDOS - pointer to DOS request packet
;
rpDos dd ?
;** fpInDOS, oldInDOS - pointer to DOS "InDOS" flag, old InDOS flag value
;
; This is REQUIRED for a software-only MRCI server, to signal that
; the server is busy, and so prevent reentrancy. Whenever the server
; is about to become non-reentrant (starting a compress/decompress
; operation, for example), it must ensure that InDOS is set.
;
; ATTENTION--BEGIN
;
; Furthermore, ALL callers of MRCI *must* ensure that the Windows
; Disk Critical Section is held *prior* to calling MRCI. This prevents
; programs in different Windows virtual machines from reentering MRCI
; and encountering the MRCI_ERROR_BUSY error.
;
; Block device drivers, and system extensions like DBLSPACE.BIN, do not
; need to grab this critical section because the MS-DOS kernel has
; already done so before giving them control.
;
; But TSRs and application programs *must* grab this critical section
; prior to calling MRCI.
;
; ATTENTION--END
;
; The following worst-case scenario demonstrates why this is needed,
; and why returning *Busy* (see fBusy below) is of no help:
;
; 1) MSBackup calls MRCI server to do a decompress operation
; 2) While MRCI server is busy, user brings up a TSR (e.g., SideKick)
; 3) SideKick does a DOS file open and read on a compressed drive
; 4) DOS FAT file system calls DoubleSpace
; 5) DoubleSpace calls MRCI server, which returns busy
;
; At this point, the system is deadlocked! DoubleSpace has no choice
; but to try the request again, or return some sort of disk error!
;
; By setting the DOS InDOS flag, however, the MRCI server signals to
; TSRs (and Windows) that it is entering a critical section, and so
; a well-written TSR will not try to make DOS calls or MRCI calls.
fpInDOS dd ? ; Pointer to DOS InDOS flag
oldInDOS db ? ; Old InDOS flag value
;** fBusy - TRUE if we are compressing/decompressing
;
; Setting the DOS InDOS flag will protect us in most cases, but
; having this flag lets us prevent being reentered by an interrupt-
; time caller.
;
fBusy db ? ; 0 => not busy, !0 => busy
;********
;* CODE *
;********
;*** Strategy - Device driver strategy entry point
;
; Entry
; es:bx - pointer to DOS request packet
;
; Exit
; es:bx saved in data:rpDos
;
; Uses
; None
;
Strategy proc far
assume cs:code,ds:nothing,es:nothing
mov cs:rpDos.offst,bx ; Store request packet pointer
mov cs:rpDos.segmt,es
ret
Strategy endp
;*** Interrupt - Device driver interrupt entry point
;
; Entry
; rpDOS - pointer to DOS request packet saved by Strategy function
;
; Exit
; ???
;
; Uses
; None
;
Interrupt proc far
SaveReg <ax,bx,cx,dx,si,di,ds,es>
push cs ; Make access to driver segment faster
pop ds
assume ds:code
les bx,rpDos ; es:bx -> DOS request packet
mov al,es:[bx].rpFunction ; al = function
cmp al,DEVINIT ; Init call?
mov ax,STERR+03h ; Assume ERROR+"Unknown command"
jne itrx ; NO, fail call
;* Init device
call GetInDOSFlagPointer ; Needed for critical section calls
SaveReg <bx,es>
call InitMRCServer
RestoreReg <es,bx>
mov cx,OFFSET cs:EndOfResidentCode ; Assume we are installed
or ax,ax ; Are we installed?
jz itr10 ; YES, set break address, note that
; ax=status=fine!
;* Did not install MRCI server
xor cx,cx ; NO, make break address small
mov ax,STERR ; What error code should we use?
;* Set break address
;
; cx = break address
; ax = status code
itr10: mov es:[bx+14].offst,cx
mov es:[bx+14].segmt,cs
;* es:bx -> DOS request packet
; ax = status code
itrx: or ax,STDONE ; Set DONE bit in status
mov es:[bx].rpStatus,ax ; Store status
RestoreReg <es,ds,di,si,dx,cx,bx,ax>
ret
Interrupt endp
;*** Server - MRCI server interrupt entry point
;
; Entry
; mov ax,MRCI function ; Set function
; int intMRCI ; Call us here
;
; Exit
; See MRCI spec. IRET to return.
;
; Uses
; AX, DI, ES, flags
Server proc
assume ds:nothing,es:nothing,ss:nothing
;* Verify that caller is asking for a MRCI server
cmp cx,sigOLD_CX ; Signature match?
je srv10 ; YES, keep testing
jmp dword ptr cs:[fpOldintMRCI] ; Chain to previous hooker
srv10: cmp dx,sigOLD_DX ; Signature match?
je srv20 ; YES, caller wants us
jmp dword ptr cs:[fpOldintMRCI] ; Chain to previous hooker
;* Dispatch MRCI function
srv20: cmp ax,mrciDETECT ; Detect call?
je dms10 ; NO, continue
;* mrciQUERY call
push cs
pop es
mov di,OFFSET mi ; es:di -> our MRCINFO structure
;* Update CX/DX signatures so caller knows we are a MRCI server
; entry: cx='ab' dx='cd'
xchg ch,cl ; cx='ba' dx='cd'
xchg dh,dl ; cx='ba' dx='dc'
xchg dx,cx ; cx='dc' dx='ba'
xor ax,ax ; Indicate success
iret ; Return to caller
dms10: mov ax,1 ; Indicate failure
;* EmptyIRET - address for fpOldintMRCI if intMRCI was unhooked.
;
; To improve performance of Server when it chains a non-intMRCI request,
; if we are the first hooker of intMRCI, we store the pointer to this
; IRET. This removes the need for Server to check that the fpOldintMRCI
; value is valid, to prevent from INTing into hyperspace.
;
EmptyIRET:
iret
Server endp
;*** Operate - MRCI server operation entry point
;
; Handle Compress/Decompress requests.
;
; Entry
; ax = Requested operation (micapXXXX)
; cx = File System Caller flag
; mcAPPLICATION (0) - Application client
; mcSYSTEM (1) - File System client
; ds:si -> MRCREQUEST packet
;
; Exit-Success
; ax = 0
; See MRCI spec for details on returned values.
;
; Exit-Failure
; See MRCI spec for details on returned values.
;
; Uses
; All (but CS:IP, SS:SP)
Operate proc far
assume ds:nothing,es:nothing,ss:nothing
and ax,micapMine ; Is this an operation we support?
jz opre ; NO, fail request
;* Make sure we are not being reentered
EnterCriticalSection cx ; Grab critical section
jc opre1 ; Already grabbed, go fail
SaveReg <cx> ; Save fFileSystemCaller flag
;* Dispatch operation
;
; NOTE: We assume Standard Compress is the most common operation,
; so we optimize the flow to reduce jumps for that operation.
;
cmp ax,micapSTANDARD ; Standard compress?
jnz opr10 ; YES, go do it
cmp ax,micapDECOMPRESS ; Decompress?
jz opre ; NO, invalid operation (user had
; more than one bit set!)
;* Do Decompress
opr20: call DoDecompress
jmp short oprLCS
;* Do Standard Compress
opr10: call DoCompress
;* Release critical section
oprLCS: RestoreReg <cx> ; Get fFileSystemCaller flag
LeaveCriticalSection cx ; Leave critical section
ret ; Return status
opre: mov ax,MRCI_ERROR_NOT_SUPPORTED
ret
opre1: mov ax,MRCI_ERROR_BUSY
ret
Operate endp
;*** DoCompress - Unimplemented compress routine
;
;
DoCompress proc near
DoCompress endp
;*** DoDecompress - Unimplemented decompress routine
;
;
DoDecompress proc near
DoDecompress endp
;******************
;* INIT-TIME CODE *
;******************
EndOfResidentCode label byte ; Truncate device here after init is complete
;** BEGIN: Include code for MRCQuery here!
;
DefineMRCQuery
;
;** END: Include code for MRCQuery here!
;** InitMRCServer - Initialize our MRCI Server
;
; If a MRCI server IS NOT present, then we install our server.
;
; If a MRCI server IS present, then we get the pointer to its
; MRCINFO structure, and check to see if we are "better" than the
; present server. If so, then we install, and edit the old
; server's MRCINFO structure so that its clients can call us.
;
; Entry
; none.
;
; Exit-Success
; ax = 0
; Server installed
; pmi = pointer to MRCI server MRCINFO structure
;
; Exit-Failure
; ax = 1, another, better server already present
;
; Uses
; ax,bx,si,ds,es,flags
InitMRCServer proc near
assume ds:nothing,es:nothing,ss:nothing
call MRCQuery ; Check for MRCI server presence
or ax,ax ; Is server present?
jz ims20 ; NO, install ourselves
;* Existing Server present
;
; es:di -> old server's MCRINFO structure
;
; We need to copy our MRCINFO structure over the old server's MCRINFO
; structure, so that clients of the previous server that call the
; server indirectly using the old MRCINFO structure will call us,
; instead.
;
; NOTE: We are guaranteed that no one is using the old server right now,
; because we have the CPU.
;
; So, we disable interrupts while we update the old MRCINFO structure.
;; dbgPrint 'Existing server present'
call ShouldWeInstall
or ax,ax ; Do install?
jz ims10 ; YES, go install
ret ; NO, return "did not install"
;* Copy our MRCINFO to old server's MRCINFO
ims10: cli ; Protect old MRCINFO from being
; in an inconsistent state.
push cs
pop ds
assume ds:code
mov si,OFFSET mi ; ds:si -> our MRCINFO structure
mov cx,SIZE mi
cld
rep movsb ; Copy our MRCINFO to old one
;* Fall through to hook interrupt chain.
;
; NOTE: Interrupts remain off, until we have hooked the interrupt chain.
; We have to leave the old server in the chain, so that it can
; chain on to previous non-MRCI hookers.
;* Hook into intMRCI chain
ims20: xor ax,ax ; Segment of interrupt vector table
mov ds,ax
assume ds:nothing
mov si,intMRCI*4 ; ds:si -> intMRCI vector
cli ; Editing the interrupt vector table
mov ax,OFFSET Server
xchg ds:[si].offst,ax ; Set offset, ax = old offset
mov bx,cs
xchg ds:[si].segmt,bx ; Store segment, bx = old segment
or ax,ax ; Old pointer valid?
jnz ims30 ; YES, go save
mov ax,OFFSET EmptyIRET ; NO, point at on of our IRETs
mov bx,cx
ims30: mov cs:fpOldintMRCI.offst,ax ; Save old offset, for chaining
mov cs:fpOldintMRCI.segmt,bx ; Save old segment, for chaining
sti ; Done editing the intvec table
ret
InitMRCServer endp
;*** ShouldWeInstall - check if our server should supercede another server
;
; We can install only if all of the following are true:
; 1) Our capabilities are at least as good as present server
; 2) Our MRCI version is at least as high
; 3) If the vendor is the same, our vendor server version must
; be greater than the present server. If the vendor does not
; match, then we cannot compare vendor versions, so we assume
; we should install.
;
; Entry
; es:di -> MRCINFO structure of current installed server
;
; Exit-Success
; ax = 0, we should install
;
; Exit-Failure
; ax = 1, we should not install
; Existing driver is more capable than we are.
;
; Uses
; ax,bx,flags
ShouldWeInstall proc near
;* Our capabilities must be at least as good as present server
mov ax,es:[di].mi_flCapability ; server capabilities
and ax,not micapMine ; ax = server caps that I do not have
or ax,ax ; Does the server have caps I do not?
jnz swie ; YES, do NOT install
;* We are at least as good on capabilities, so...
;* Compare MRCI versions
mov ax,es:[di].mi_wMRCIVersion ; ax = present server MRCI version
cmp ax,MRCIVERSION ; Are we at least as good as server?
ja swie ; NO, do not install
jb swi10 ; We are better than server, install
;* We support the same MRCI version, so...
;* See if we supersede the old <vendor,vendor version>
mov ax,es:[di].mi_lVendor.loword
cmp ax,MYVENDORlo ; Match low word?
jne swi10 ; NO, DO install
mov ax,es:[di].mi_lVendor.hiword
cmp ax,MYVENDORhi ; Match high word?
jne swi10 ; NO, DO install
mov ax,es:[di].mi_wVendorVersion
cmp ax,MYVERSION ; Am I a newer driver version?
jae swie ; NO, do NOT install
;* Tests pass -- we should install
swi10: xor ax,ax ; Indicate DO install
ret
swie: mov ax,1 ; Indicate do NOT install
ret
ShouldWeInstall endp
;*** GetInDOSFlagPointer - Get address of InDOS flag
;
; Entry
; none
;
; Exit
; fpInDOS = pointer to DOS InDOS flag
;
; Uses
; ax,flags
GetInDOSFlagPointer proc near
SaveReg <bx,es>
mov ah,34h
int 21h ; Get In DOS flag
mov cs:fpInDOS.offst,bx
mov cs:fpInDOS.segmt,es
RestoreReg <es,bx>
ret
GetInDOSFlagPointer endp
code ends
end