File System Performance Tests for Flash (Compact 7)
3/12/2014
The File System Performance Test measures the performance of a file system driver (FSD) that uses the Win32 application programming interface. You can use this test to obtain throughput metrics for various file system usage scenarios on a storage device block driver that runs under the FAT, ExFAT, TexFAT, or a custom FSD. The test can also measure the performance of the Object Store file system.
To judge the performance data produced by this test, you need a basis for comparison. For instance, you might use a known theoretical throughput maximum for a specific storage device, or you might compare two data sets obtained by running this test on two different devices, FSDs, or block drivers. The following are examples of useful applications of the File System Performance Test:
* Comparing file system I/O to block driver I/O (the latter can be obtained by running the Storage Device Block Driver Performance Test).
* Comparing file system performance on storage devices from different manufacturers.
* Running a performance test every time there is a known change in any of the underlying components in the file system stack.
This test will generate a number of performance metrics. After running the test, some of these tests can be viewed visually via the GraphTool in the Windows Embedded Compact Test Kit (CTK) application. Other tests' results can be viewed in the fsperflog.xml log file that is generated. These tests were not graphed because they generate one dimensional or a single data point.
| Test Name | In GraphTool | Description |
|---|---|---|
Throughput Writefile |
Yes |
Measures file write throughput using varying buffer sizes |
Throughput Readfile |
Yes |
Measures file read throughput using varying buffer sizes |
Throughput Writefile Random |
Yes |
Measures the throughput while writing varying buffer sizes of data at random offsets |
Throughput Readfile Random |
Yes |
Measures the throughput while reading varying buffer sizes of data from random offsets |
Throughput Reverse Readfile |
Yes |
Measures the throughput while reading a file in reverse using varying buffer sizes |
Throughput Overwrite |
Yes |
Measures the throughput to overwrite an existing file with varying buffer sizes |
Create Directory Tree |
Yes |
Measures the directory depth that can be created per second |
SetFilePointer and SetEndOfFile |
No |
Measures the time it takes to create a large file with SetEndOfFile |
Seek Speed(read) |
No |
Measures how long it takes to perform a specific number of seeks followed by file reads |
Seek Speed(write) |
No |
Measures how long it takes to perform a specific number of seeks followed by file writes |
Read/Append to Fragmented File |
No |
Time to read and write to a large fragmented file |
Copy Files(from storage to device root) |
No |
Time to copy a set of files from storage to device root |
Copy Files(from device root to storage) |
No |
Time to copy a set of files from device root to storage |
FindFirstFile FindNextFile |
No |
Time to perform FindFileFirst and FindFileNext on 50/100/200 files of the same type |
FindFirstFile FindNextFile Random |
No |
Time to perform random FindFileFirst and FindFileNext on 50/100/200 files of the same type |
Create File CREATE_NEW |
No |
Time to create files with different depths across different directories |
Create File OPEN_EXISTING |
No |
First create files with different depths across different directories and measure the time to CreateFile with the OPEN_EXISTING attribute |
Test Prerequisites
Your device must meet the following requirements before you run this test.
The following table shows the hardware requirement for the File System Performance Test for Flash.
| Requirement | Description |
|---|---|
Onboard flash memory |
The device must have onboard available that can be used for storage. |
A storage device is not required if the test is intended to measure the performance of the Object Store.
The following table shows the software requirements for the File System Performance Test for Flash.
| Requirement | Description |
|---|---|
Tux.exe |
Tux Test Harness, required for executing the test. |
Kato.dll |
Kato logging engine, required for logging test data. |
Fsperflog.dll |
Test library. |
ceperf.dll |
Module containing functions that monitor and log performance. |
perfscenario.dll |
Module containing functions that monitor and log performance. |
Note:When you run the File System Performance Test, the CTK temporarily copies files to the root directory of the target device. Before running the test, verify that there is at least 0.5 megabytes (MB) of free space in the root directory of the target device. If there is not sufficient space in the root directory of the target device, the test cannot run.
Subtests
The table below lists the subtests included in this test.
| SubTest ID | Description |
|---|---|
2001 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 128 bytes using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 128 byte read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2002 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 256 bytes using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 256 byte read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2003 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 512 bytes using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 512 byte read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2004 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 1KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 1KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2005 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 2KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 2KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2006 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 4KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 4KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2007 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 8KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 8KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2008 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 16KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 16KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2009 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 32KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 32KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2010 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 64KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 64KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2011 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 128KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 128KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2012 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 256KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 256KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2013 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 512KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 512KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
2014 |
Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 1MB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 1MB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail. |
3001 |
Measures the speed of creating a deep nested directory tree 10 levels deep using the CreateDirectory function. The test fails if any of the directories cannot be created. |
3002 |
Measures the speed of creating a deep nested directory tree 15 levels deep using the CreateDirectory function. The test fails if any of the directories cannot be created. |
3003 |
Measures the speed of creating a deep nested directory tree 20 levels deep using the CreateDirectory function. The test fails if any of the directories cannot be created. |
3004 |
Measures the speed of creating a 500MB file by using SetFilePointer and SetEndOfFile functions. The test fails if the file cannot be created, or if either of the operations fail. |
3005 |
Measures the seek speed by performing 1000 random seeks followed by 1 byte reads using the SetFilePointer and ReadFile functions. The test fails if the file cannot be created, or if any of the seeks or reads fail. |
3006 |
Measures the seek speed by performing 1000 random seeks followed by 1 byte writes using the SetFilePointer and WriteFile functions. The test fails if the file cannot be created, or if any of the seeks or writes fail. |
4001 |
Measures the I/O throughput by continuously reading and appending data to a 100MB file using the SetFilePointer, ReadFile, and WriteFile functions. This test is intended to simulate the behavior of a Digital Video Recorder (DVR) device, which often records and plays back data from the same file. The test fails if the file cannot be created, or if any of the seeks, reads, or writes fail. If the storage device has insufficient space (~300MB), the test will skip. |
4002 |
Measures the speed of copying a folder with 100 files from the storage device to the root directory of the device using the FindFirstFile, FindNextFile, and CopyFile functions. The test fails if the directory or files cannot be created, or if the copy process fails. If this test is run on the Object Store rather than a storage device, then the files will be both created and copied on the Object Store. |
4003 |
Measures the speed of copying a folder with 100 files from the root directory to a storage device using the FindFirstFile, FindNextFile, and CopyFile functions. The test fails if the directory or files cannot be created, or if the copy process fails. If this test is run on the Object Store rather than a storage device, then the files will be both created and copied on the Object Store. |
5001 |
Measures the time to perform FindFileFirst and FindFileNext on same type of 50 files. |
5002 |
Measures the time to perform FindFileFirst and FindFileNext on same type of 100 files. |
5003 |
Measures the time to perform FindFileFirst and FindFileNext on same type of 200 files. |
6001 |
Measures the time to perform FindFileFirst and FindFileNext randomly on same type of 50 files |
6002 |
Measures the time to perform FindFileFirst and FindFileNext randomly on same type of 100 files. |
6003 |
Measures the time to perform FindFileFirst and FindFileNext randomly on same type of 200 files. |
7001 |
Measures the time to CreateFile with the CREATE_NEW attribute up to 10 files across different directories and nested directories. |
7002 |
Creates files with different depths with 10 files across different directories and nested directories. Then measures the time to CreateFile with the OPEN_EXISTING attribute. |
Setting Up the Test
* The onboard flash memory must be mounted and able to read/write accessible.
* You must modify the default command line of the test to specify the profile name of the flash storage if it is not "FlashDisk." Or use other parameters to specify the existence of the flash memory volume.
* The onboard flash memory can be partitioned and formatted using the Storage Manager Control Panel Applet. This applet can be added to the image using the SYSGEN_STOREMGR_CPL. After the image has been built, the Storage manager applet can be accessed by My Device->Control Panel->Storage Manager.
Running the Test
The command line for the File System Performance Test for Flash must be modified to point to the onboard flash memory. For example, if the flash memory uses the profile name of "FlashDisk" then use the option, "-p FlashDisk".
The test will generate raw performance data in a log file named, fsperflog.xml, in the release directory.
You can modify the test by editing the command line. The following table shows the command line parameters for the File System Performance Test for Flash.
tux -o -d fsperflog -c "[-p <profile>|-r <path>|-root] [-nb -wt -flush] [-cpu] [-desc <description>]"
| Command line parameter | Description |
|---|---|
-p <profile> |
Specify the storage profile to which to restrict testing, found in the registry under the HKEY_LOCAL_MACHINE\System\StorageManager\Profiles key. |
-r <\path> |
Specify the directory in which to run the test. If you provide this option, the test only runs in the specified directory and ignores the -p option. This flag is useful for a file system driver that does not use storage profiles such as a network file system driver. The path does not require quotation marks; for example, -r \Storage Card |
-root |
Specify that the test should run on the root file system. If you provide this option, the test runs in the root directory and ignores the -p and -r options. |
-nb |
Specify that all the test files should be opened with FILE_FLAG_NO_BUFFERING. This option can be combined with -wt. |
-wt |
Specify that all the test files should be opened with FILE_FLAG_WRITE_THROUGH. This option can be combined with -nb. |
-flush |
Include the calls to the CloseHandle and FlushFileBuffers functions in performance timing calculations. |
-cpu |
Include the CPU usage information in the performance log. |
-desc <string> |
Include a user-specified description string in the performance log. |
-iosize # |
The I/O buffer size for test cases (default = 512) |
-filesize |
The file size for test cases 2001-2014 (default = 2MB) |
-rand |
For tests that perform random I/O, uses TUX-provided random seed. |
Verifying the Test
When the test completes running, verify that "PASS" appears in the test log for all sub-tests. A "PASS" test result indicates that the test ran successfully, but the test engineer must manually evaluate the results for each device. Investigate any warning messages that are present in the test log.
After running the test, you can view the results in two ways:
* CEPerf output log - fsperflog.xml
* GraphTool output view
In GraphTool output view, you will be able to see the graphs for tests which generate a series of data. You will have to determine if the performance numbers reported by the test meet the performance required by the device. If not, additional optimizations may be needed. The tests which are graphed are:
* Throughput Writefile
* Throughput Readfile
* Throughput Writefile Random
* Throughput Readfile Random
* Throughput Reverse Readfile
* Throughput Overwrite
* Create Directory Tree
Troubleshooting the Test
Ensure that the flash memory is available and is formatted with the specific file system that are supported by the device.
For additional platform specific issues, consult the CTK articles on the TechNet wiki.