NTFS Sparse files range boundaries

Question

NTFS Sparse files range boundaries

Pierre Chatelier 81

NTFS Sparse file https://learn.microsoft.com/en-us/windows/win32/fileio/sparse-files are not really well documented.

For instance you can read The default data value of a sparse file is zero; however, it can be set to other values., but I found no further information.

More important : what are the requirements on a FSCTL_SET_ZERO_DATA to be efficient ?

I expected that it has an effect only on boundaries and size of clusters (default 4KB on my drives), but a few tests revealed that I had to write at least 64KB of 0 to see the size of the file reflect a "sparse".

Where does this value come from ? Is there something to know here ?

David Lowndes 2,745 Reputation points MVP

2023-12-09T21:02:41.17+00:00

The same question was made here some time ago and the opinion there seems to be that there's no easy way to change from the zero default.
Xiaopo Yang - MSFT 12,736 Reputation points Microsoft External Staff

2023-12-11T03:13:39.5733333+00:00

According to FSCTL_SET_ZERO_DATA,

Fills a specified range of a file with zeros (0). If the file is sparse or compressed, the NTFS file system may deallocate disk space in the file.
Pierre Chatelier 81 Reputation points

2023-12-11T06:50:19.8933333+00:00

I am sorry, but you are not answering any of my 2 questions, you are just pointing to the same documentation that is rather... sparse !

Example of things to explain.

Let's start from an empty file, and write packets of 64 KB. "Virtual" and "Compressed" file sizes refer to GetFileSize() and GetCompressedFIleSize().

Scenario 1),

-append packet of data (new file size 64KB)

-append packet of sparse zero : (new file size : 128 KB "virtual", but 64 KB "compressed" : OK)

-append packet of data (new file size 192 KB : no more compression)

-punch a hole by rewriting sparse 0 in the range [64KB;+64KB] : the "compression" is effective again

This scenario tends to show that

-when the filesystem extends a file, it fills the gap of unallocated holes instead of keeping sparse info. Why ?

Scenario 2)

fill a file with data on a 4KB-clusters drive (the default)

-try to punch holes of 32KB, even contiguous : no "compressed size" appears. 32KB is not enough for sparsing. Why ?
Xiaopo Yang - MSFT 12,736 Reputation points Microsoft External Staff

2023-12-11T09:45:46.92+00:00

For instance you can read The default data value of a sparse file is zero; however, it can be set to other values., but I found no further information.

It seems **FSCTL_SET_ZERO_DATA **can be used with nonzero data.

Zeros being written over nonzero data (only with FSCTL_SET_ZERO_DATA) may result in a deallocation of disk space.

Scenario 2)

It's hard to say the 32KB is contiguous but definitely there must be a criterion which could be 64KB as you tested and the system uses to deallocate the physical disk storage when the entire contiguous disk storage is zeroed.

A sparseness-aware application may also set an existing file to be sparse. If an application sets an existing file to be sparse, it should then scan the file for regions which contain zeros, and use FSCTL_SET_ZERO_DATA to reset those regions, thereby possibly deallocating some physical disk storage.

Scenario 1),

How about appending packet of data lastly?
Pierre CHATELIER 11 Reputation points

2023-12-11T10:51:34.2166667+00:00
You are quoting two unrelated things.

It seems **FSCTL_SET_ZERO_DATA **can be used with nonzero data.

Using FSCTL_SET_ZERO_DATA with non zero data is not documented. How would you write an API call for that? I would rather expect "you don't write anything anyway, you just ask the filesystem to return some dummy value for non-allocated ranges", so the "non zero" would be some kind of READ parameter. It is currently unclear.

Zeros being written over nonzero data (only with FSCTL_SET_ZERO_DATA) may result in a deallocation of disk space.

Yes, it just means that calling FSCTL_SET_ZERO_DATA will not necessarilly punch a real hole, depending on the range you hit. But it does not give any hint about what is expected to be effective. I perfectly understand some cluster-aligned boundaries, but the doc tells nothing.

Scenario 1), How about appending packet of data lastly?

I can't change the scenario, it describes a real-life incoming data scenario. It is just hard to understand why an already sparsed file will lose the last hole when extending its size.

It's hard to say the 32KB is contiguous

What do you mean ?
Gary Nebbett 6,411 Reputation points

2023-12-11T19:15:54.71+00:00

Hello Pierre,

I have seen hints that a sparse range must be a multiple of 0x10000 bytes (64 kb) - this might explain why you can't punch holes less than that size.

I tried part of your scenario 1:

Scenario 1), -append packet of data (new file size 64KB) -append packet of sparse zero : (new file size : 128 KB "virtual", but 64 KB "compressed" : OK) -append packet of data (new file size 192 KB : no more compression)

After those 3 steps, the sparse region was still present. Did you write your own test program for this scenario? If so, can you share it so that one can check how/why it differs from one's own experience?

Gary

Pierre CHATELIER 11

I did write the code, here it is for scenario 1

The console output is :

Writing 65536 1 at offset 0
Flush
New file size : 65536 (virtual) 65536 (compressed)
New file size : 65536 (virtual) 65536 (compressed)
Writing 65536 sparse 0 at offset 65536
Flush
New file size : 131072 (virtual)        65536 (compressed)
New file size : 131072 (virtual)        65536 (compressed)
Writing 65536 1 at offset 131072
Flush
New file size : 196608 (virtual)        196608 (compressed)
New file size : 196608 (virtual)        196608 (compressed)

typedef enum {ONE, ZERO, ZERO_SPARSE} data_t;

const wchar_t* filePath = L"E:\\temp\\sparse.tmp";

LARGE_INTEGER EXGetFileSize(HANDLE f)
{
  LARGE_INTEGER result = {0};
  if (f != nullptr)
    GetFileSizeEx(f, &result);
  return result;
}
//end EXGetFileSize()

void logFileSize(HANDLE f)
{
  LARGE_INTEGER vsize = EXGetFileSize(f);
  LARGE_INTEGER fsize = {0};
  DWORD highPart = 0;
  fsize.LowPart = GetCompressedFileSize(filePath, &highPart);
  fsize.HighPart = highPart;
  std::cout << "New file size : " << vsize.QuadPart << (!f ? "?" : "") << " (virtual)" << "\t" << fsize.QuadPart << " (compressed)" << std::endl;
}
//end logFileSize()

void flushFile(HANDLE f)
{
  FlushFileBuffers(f);
  std::cout << "Flush" << std::endl;
  logFileSize(f);
}
//end flushFile()

void writePacket(HANDLE f, size_t offset, data_t dataType, size_t size, bool byteByByte, bool flush)
{
  std::cout << "Writing " << size << " " << ((dataType == ZERO_SPARSE) ? "sparse 0" : (dataType == ZERO) ? "real 0" : "1") << " at offset " << offset << std::endl;
  const bool isSparse = (dataType == ZERO_SPARSE);

  std::vector<unsigned char> v(size);
  std::fill(v.begin(), v.end(), (dataType == ONE) ? 1 : 0);

  LARGE_INTEGER _offset = {0};
  _offset.QuadPart = offset;
  BOOL b1 = SetFilePointerEx(f, _offset, nullptr, FILE_BEGIN);
  BOOL b2 = FALSE;
  DWORD numberOfBytesWritten = 0;
  if (!byteByByte)
  {
    if (isSparse)
    {
      FILE_ZERO_DATA_INFORMATION fzdi = {0};
      fzdi.FileOffset.QuadPart = offset;
      fzdi.BeyondFinalZero.QuadPart = offset + size;
      if (fzdi.BeyondFinalZero.QuadPart > EXGetFileSize(f).QuadPart)
      {
        BOOL b2 = SetFilePointerEx(f, fzdi.BeyondFinalZero, nullptr, FILE_BEGIN);
        BOOL b4 = SetEndOfFile(f);
      }
      b2 = DeviceIoControl(f, FSCTL_SET_ZERO_DATA, &fzdi, sizeof(fzdi), nullptr, 0, &numberOfBytesWritten, nullptr);
    }//end if (isSparse)
    else
      b2 = WriteFile(f, v.data(), v.size(), &numberOfBytesWritten, nullptr);
  }
  else//if (byteByByte)
  {
    for(size_t i = 0 ; i<v.size() ; ++i)
    {
      if (isSparse)
      {
        FILE_ZERO_DATA_INFORMATION fzdi = {0};
        fzdi.FileOffset.QuadPart = offset+i;
        fzdi.BeyondFinalZero.QuadPart = offset+i+1;
        b2 = DeviceIoControl(f, FSCTL_SET_ZERO_DATA, &fzdi, sizeof(fzdi), nullptr, 0, &numberOfBytesWritten, nullptr);
      }
      else
        b2 = WriteFile(f, v.data()+i, 1, &numberOfBytesWritten, nullptr);
      if (flush)
        FlushFileBuffers(f);
    }
  }//end if (byteByByte)

  if (flush)
    flushFile(f);

  logFileSize(f);
}
//end writePacket()

int main()
{
  HANDLE f = CreateFile(filePath, GENERIC_READ|GENERIC_WRITE, 0, nullptr, CREATE_ALWAYS, 0*FILE_FLAG_DELETE_ON_CLOSE, nullptr);
  DWORD bytesReturned = 0;
  BOOL b = DeviceIoControl(f, FSCTL_SET_SPARSE, nullptr, 0, nullptr, 0, &bytesReturned, nullptr);

  const size_t ps = 64*1024;
  size_t pi = 0;
  writePacket(f, (pi++)*ps, data_t::ONE, ps, false, true);
  writePacket(f, (pi++)*ps, data_t::ZERO_SPARSE, ps, false, true);
  writePacket(f, (pi++)*ps, data_t::ONE, ps, false, true);

  CloseHandle(f);
  logFileSize(nullptr);
  return 0;
}

Gary Nebbett 6,411 Reputation points

2023-12-11T20:32:11.53+00:00

Hello Pierre,

That code looks as though it performs the same steps as my test program.

Let's say that your test program is named sparse.exe. Could you try executing the following command sequence and either seeing if the output (why.etl) helps you or sharing the output here (via a link to some file sharing site) so that we can take a look?

logman start gary -ets -p Ntfs_NtfsLog -bs 64 -nb 640 -o why.etl & sparse & logman stop gary -ets

Gary
Pierre CHATELIER 11 Reputation points

2023-12-11T20:46:53.3833333+00:00

Here it is : https://chachatelier.fr/tmp/why.zip
Gary Nebbett 6,411 Reputation points

2023-12-11T21:11:54.1533333+00:00

Hello Pierre,

Something went wrong with that trace - why.etl just contains a single entry (the trace header).

Analysing a trace from my test program, I saw that the FSCTL_SET_ZERO_DATA component of step 2 was superfluous - the SetFilePointerEx and SetEndOfFile calls extended the file with a sparse region.

When the trace works, this is the type of information included in the trace:

The SetInformation entry is setting the end of the file and the FSCTL entry is performing the FSCTL_SET_ZERO_DATA.

Gary
Pierre CHATELIER 11 Reputation points

2023-12-11T21:46:06.6133333+00:00

I run the code under Windows 7, maybe the GUID or NTFS_NTFSLOG is unknown ?
Xiaopo Yang - MSFT 12,736 Reputation points Microsoft External Staff

2023-12-12T01:47:23.06+00:00

@Pierre CHATELIER, it works fine.

Answer accepted by question author

0 additional answers

Your answer

David Lowndes 2,745 Reputation points MVP

2023-12-09T21:02:41.17+00:00

The same question was made here some time ago and the opinion there seems to be that there's no easy way to change from the zero default.
Xiaopo Yang - MSFT 12,736 Reputation points Microsoft External Staff

2023-12-11T03:13:39.5733333+00:00

According to FSCTL_SET_ZERO_DATA,

Fills a specified range of a file with zeros (0). If the file is sparse or compressed, the NTFS file system may deallocate disk space in the file.
Pierre Chatelier 81 Reputation points

2023-12-11T06:50:19.8933333+00:00

I am sorry, but you are not answering any of my 2 questions, you are just pointing to the same documentation that is rather... sparse !

Example of things to explain.

Let's start from an empty file, and write packets of 64 KB. "Virtual" and "Compressed" file sizes refer to GetFileSize() and GetCompressedFIleSize().

Scenario 1),

-append packet of data (new file size 64KB)

-append packet of sparse zero : (new file size : 128 KB "virtual", but 64 KB "compressed" : OK)

-append packet of data (new file size 192 KB : no more compression)

-punch a hole by rewriting sparse 0 in the range [64KB;+64KB] : the "compression" is effective again

This scenario tends to show that

-when the filesystem extends a file, it fills the gap of unallocated holes instead of keeping sparse info. Why ?

Scenario 2)

fill a file with data on a 4KB-clusters drive (the default)

-try to punch holes of 32KB, even contiguous : no "compressed size" appears. 32KB is not enough for sparsing. Why ?
Xiaopo Yang - MSFT 12,736 Reputation points Microsoft External Staff

2023-12-11T09:45:46.92+00:00

For instance you can read The default data value of a sparse file is zero; however, it can be set to other values., but I found no further information.

It seems **FSCTL_SET_ZERO_DATA **can be used with nonzero data.

Zeros being written over nonzero data (only with FSCTL_SET_ZERO_DATA) may result in a deallocation of disk space.

Scenario 2)

It's hard to say the 32KB is contiguous but definitely there must be a criterion which could be 64KB as you tested and the system uses to deallocate the physical disk storage when the entire contiguous disk storage is zeroed.

A sparseness-aware application may also set an existing file to be sparse. If an application sets an existing file to be sparse, it should then scan the file for regions which contain zeros, and use FSCTL_SET_ZERO_DATA to reset those regions, thereby possibly deallocating some physical disk storage.

Scenario 1),

How about appending packet of data lastly?
Pierre CHATELIER 11 Reputation points

2023-12-11T10:51:34.2166667+00:00

You are quoting two unrelated things.

It seems **FSCTL_SET_ZERO_DATA **can be used with nonzero data.

Using FSCTL_SET_ZERO_DATA with non zero data is not documented. How would you write an API call for that? I would rather expect "you don't write anything anyway, you just ask the filesystem to return some dummy value for non-allocated ranges", so the "non zero" would be some kind of READ parameter. It is currently unclear.

Zeros being written over nonzero data (only with FSCTL_SET_ZERO_DATA) may result in a deallocation of disk space.

Yes, it just means that calling FSCTL_SET_ZERO_DATA will not necessarilly punch a real hole, depending on the range you hit. But it does not give any hint about what is expected to be effective. I perfectly understand some cluster-aligned boundaries, but the doc tells nothing.

Scenario 1), How about appending packet of data lastly?

I can't change the scenario, it describes a real-life incoming data scenario. It is just hard to understand why an already sparsed file will lose the last hole when extending its size.

It's hard to say the 32KB is contiguous

What do you mean ?
Gary Nebbett 6,411 Reputation points

2023-12-11T19:15:54.71+00:00

Hello Pierre,

I have seen hints that a sparse range must be a multiple of 0x10000 bytes (64 kb) - this might explain why you can't punch holes less than that size.

I tried part of your scenario 1:

Scenario 1), -append packet of data (new file size 64KB) -append packet of sparse zero : (new file size : 128 KB "virtual", but 64 KB "compressed" : OK) -append packet of data (new file size 192 KB : no more compression)

After those 3 steps, the sparse region was still present. Did you write your own test program for this scenario? If so, can you share it so that one can check how/why it differs from one's own experience?

Gary
Gary Nebbett 6,411 Reputation points

2023-12-11T20:32:11.53+00:00

Hello Pierre,

That code looks as though it performs the same steps as my test program.

Let's say that your test program is named sparse.exe. Could you try executing the following command sequence and either seeing if the output (why.etl) helps you or sharing the output here (via a link to some file sharing site) so that we can take a look?

logman start gary -ets -p Ntfs_NtfsLog -bs 64 -nb 640 -o why.etl & sparse & logman stop gary -ets

Gary
Pierre CHATELIER 11 Reputation points

2023-12-11T20:46:53.3833333+00:00

Here it is : https://chachatelier.fr/tmp/why.zip
Gary Nebbett 6,411 Reputation points

2023-12-11T21:11:54.1533333+00:00

Hello Pierre,

Something went wrong with that trace - why.etl just contains a single entry (the trace header).

Analysing a trace from my test program, I saw that the FSCTL_SET_ZERO_DATA component of step 2 was superfluous - the SetFilePointerEx and SetEndOfFile calls extended the file with a sparse region.

When the trace works, this is the type of information included in the trace:

The SetInformation entry is setting the end of the file and the FSCTL entry is performing the FSCTL_SET_ZERO_DATA.

Gary
Pierre CHATELIER 11 Reputation points

2023-12-11T21:46:06.6133333+00:00

I run the code under Windows 7, maybe the GUID or NTFS_NTFSLOG is unknown ?
Xiaopo Yang - MSFT 12,736 Reputation points Microsoft External Staff

2023-12-12T01:47:23.06+00:00

@Pierre CHATELIER, it works fine.

Answer 1

Gary Nebbett 6,411

Hello Pierre,

Using Windows 7 almost certainly explains the trace behaviour and probably explains the unexpected filling of sparse regions that you seem to be encountering.

The posting at https://bugs.launchpad.net/ubuntu/+source/ntfs-3g/+bug/1958180 might help inform one's understanding of NTFS sparse files. A non-Windows implementation of NTFS sparse files can create sparse regions at the smallest possible size (a cluster); a Windows autochk of the volume will report this but can still work with the file.

Using sparse files can add to the fragmentation of a volume (a newly created large file might initially occupy a single range of clusters; making some parts of the file sparse might lead to these freed clusters being used by other files; when the large file is deleted, the initially contiguous free space is now fragmented).

One idea would be to use a maximally sized FSCTL_SET_ZERO_DATA whenever possible (ignoring alignment and size of the region); if adjacent regions are known to be zero (because the space in the file is being managed by software - perhaps a database application) then include them in the region to be zeroed. This would allow the NTFS implementation to make the best choice about how to perform FSCTL_SET_ZERO_DATA on the region (make sparse or write zeros).

Gary

Pierre CHATELIER 11 Reputation points

2023-12-12T10:12:00.6533333+00:00

Using Windows 7 almost certainly explains the trace behaviour and probably explains the unexpected filling of sparse regions that you seem to be encountering.

I was not expecting a major change in behaviour regarding sparse files between Windows 7 and Windows 10/11. I can't remember any NTFS relase notes.

The problem is really the lack of documentation.

One idea would be to use a maximally sized FSCTL_SET_ZERO_DATA

Yes, but knowning the limits helps post-processing a file to detect zeros areas that could be sparsed. It helps by limiting the ranges to be checked for being zeroes. For instance : "test every 64KB block against 0", instead of "find sequences of zeros anywhere in the file".
Gary Nebbett 6,411 Reputation points

2023-12-12T10:50:42.4533333+00:00

Hello Pierre,

The filling of the sparse region might just be a bug that has been fixed rather than a change in designed/documented behaviour - there was perhaps no release note or similar.

My advice for post-processing the file would still be similar - choose for yourself a minimum size (e.g. the year of your birth - 1961, in my case; it need not be a "meaningful" computer science number like 0x1000) and apply FSCTL_SET_ZERO_DATA to all regions that are greater or equal to this size. If that does something useful then great; if it is effectively a "nop" (no operation) for some/many regions then at least you know that you have done your best.

The documentation is possibly deliberately vague, perhaps because providing guarantees and specific values would be too constraining on possible implementations.

Gary
Pierre CHATELIER 11 Reputation points

2023-12-12T11:07:43.72+00:00

Hi,

I am not sure we mean the same thing for "post-processing". My idea is to compact a file (to save storage resources) if by chance it already contains large ranges of 0s. (Actually, my use case is for files generated by a numerical processing involving sparse matrices, so it is very likely to occur, it is not just a recreative filesystem experience :-) ).

Thus, I have to read the file to detect possible 0 ranges. Knowing that sparse boundaries are 64KB will help doing quick tests and limit useless file reads.

Your proposal of using a minimum size is for 0-ranges known in advance, where I can indeed let the filesystem decide if there's something to deallocate.

That said, I don't know if my minimum size of 1980 would be more efficient than yours :-)
Gary Nebbett 6,411 Reputation points

2023-12-12T11:30:27.74+00:00

Hello Pierre,

I think that we do "understand" the same thing by "post-processing".

Before writing my own code to "efficiently" detect zero regions, I would at least consider/check whether "regular expressions" might be a useful approach (\x00{1961,})

Gary
Pierre CHATELIER 11 Reputation points

2023-12-12T11:47:08.4266667+00:00

;Before writing my own code to "efficiently" detect zero regions, I would at least consider/check whether "regular expressions" might be a useful approach (\x00{1961,})

My approach to detect if a range R of N bytes is all 0s :

const bool isAllZeros = !R[0] && !R[N-1] && !R[N/2] && ! cv::hasNonZero(R);

The big advantage is that the hasNonZero() is vectorized, and it works at constant memory size.

Using a regex does not offer such guarantees for non-fixed range sizes.
Gary Nebbett 6,411 Reputation points

2023-12-12T12:03:47.7833333+00:00

Hello Pierre,

Initially, we don't know where (in the file) R starts, nor how big N is.

Finding all reasonably large zero ranges in a single evaluation of a regular expression and applying FSCTL_SET_ZERO_DATA to the matches could be coded in just a few lines and clearly expresses the intent and logic of the program (easy to maintain many years into the future).

The are obviously many circumstances under which this would not be a suitable approach but I do like the KISS (Keep It Simple, Stupid) approach :-)

Gary
Pierre Chatelier 81 Reputation points

2023-12-12T18:34:08.6566667+00:00

At this point, you gave me almost all the info I need.
By chance, do you know anything about the ghost feature

The default data value of a sparse file is zero; however, it can be set to other values
Gary Nebbett 6,411 Reputation points

2023-12-12T19:00:05.6766667+00:00

Hello Pierre,

I think that this is just an error in the documentation.

The on-disk format of NTFS is more-or-less completely known (people can and have developed NTFS drivers for Linux based on the available information) and I don't think that there is anywhere to store on-disk a "default" value for sparse bytes.

There would probably also be a large number of unanswerable questions about the behaviour of such a feature (especially if setting the sparse attribute and the sparse default value was not an atomic operation).

Finally (and least convincingly), a quick look at the disassembly of "sparse" related routines in ntfs.sys shows no signs to me of support for such a feature.

Gary
Pierre Chatelier 81 Reputation points

2023-12-12T19:50:13.0966667+00:00
Thank you.

As for my initial request, I think we can consider this thread closed, you brought very valuable answers.

To summarize for future readers :

non zero sparse data seems to be a doc error and does not exist

there seem to be a 0x10000 length size affinity for sparse ranges (probably identical for offset alignment)

Windows 7 may not behave as well as Windows 10+ regarding sparsed data conservation during file extents
Gary Nebbett 6,411 Reputation points

2023-12-12T19:52:46.9866667+00:00

Hello Pierre,

One minor correction, for accuracy's sake - the magic length value appears to be 0x10000 (one more trailing zero).

Gary
David Lowndes 2,745 Reputation points MVP

2023-12-12T20:21:39.26+00:00

Has anyone added feedback to the documentation questioning the validity of the non-zero default values? It'd be good to get that removed from the documentation if it's incorrect.
Pierre CHATELIER 11 Reputation points

2023-12-12T20:35:26.6333333+00:00

Has anyone added feedback to the documentation questioning the validity of the non-zero default values? It'd be good to get that removed from the documentation if it's incorrect.

I just did.

Xiaopo Yang - MSFT 12,736 Microsoft External Staff

I also found the FILE_ZERO_DATA_INFORMATION_EX structure, in which there is a flag.

flag	description
FILE_ZERO_DATA_INFORMATION_FLAG_PRESERVE_CACHED_DATA	Indicates not to purge the contents of the cache corresponding to this range of the file. Only drivers can set this flag.

Anyway, there is a lack of documentation, or the documentation is incorrect in Win32. @Pierre Chatelier, could you please mark answer for completion of the question?

Share via

NTFS Sparse files range boundaries

0 additional answers

Your answer