extents reported in USN_RECORD_V4 struct are a lot bigger than actual changes

Question

extents reported in USN_RECORD_V4 struct are a lot bigger than actual changes

Kanak Agrawal 6

We use USN change journal to track changes on SQL data files for performing incremental backups.

For big files (of size 5 TB), we have seen USN_RECORD_V4 have extents reporting that a big portion of file is modified but actual change on the file is very less.

I want to understand how and when USN_RECORD_EXTENT are added to a USN_RECORD_V4.

In what scenarios is it possible that data is not overwritten but there is an extent added in USN_RECORD_V4.

On one day, I observed, 943 records for a file of size 5 TB. Of these 943 records, there are three consecutive records for this file which had extents which corresponds to changed data of around 330 GB, 220 GB and 4300 GB. All other 940 records combined reported 70 GB data is changed.

But actual change of that file is only of 600 GB (We did a diff between file backups of the two versions of file at 16KB granularity). So, the three big records corresponds to some actual changes in the file but are reporting some very big extents which are not modified.

Due to this, our backup software is doing way more work than what it should.

Xiaopo Yang - MSFT 12,731 Reputation points Microsoft External Staff

2023-11-30T02:54:21.26+00:00

You need checking the Reason of records. According to Change Journal Records,

In addition, multiple changes to the same file may result in only one reason flag being added to the current record. If the same kind of change occurs more than once, the NTFS file system does not write a new record for the changes after the first. For example, several write operations with no intervening close and reopen operations result in only one change record with the reason flag USN_REASON_DATA_OVERWRITE set.

And can you reproduce it using fsutil usn?
Kanak Agrawal 6 Reputation points

2023-12-01T21:59:39.58+00:00

@Xiaopo Yang - MSFT I'm able to reproduce the bigger extents by checking fsutil usn. Reason in records is 0x80000001: Data overwrite | Close

One thing I observed is all the big extents are of size multiple of 64MB and in a 30 min interval. I suspect it might be something related to how MSSQL writes. But actual data is not written on the disk. I'm not certain right now why these extents are reported in USN v4 record

2 answers

Your answer

Xiaopo Yang - MSFT 12,731 Reputation points Microsoft External Staff

2023-11-30T02:54:21.26+00:00

You need checking the Reason of records. According to Change Journal Records,

In addition, multiple changes to the same file may result in only one reason flag being added to the current record. If the same kind of change occurs more than once, the NTFS file system does not write a new record for the changes after the first. For example, several write operations with no intervening close and reopen operations result in only one change record with the reason flag USN_REASON_DATA_OVERWRITE set.

And can you reproduce it using fsutil usn?
Kanak Agrawal 6 Reputation points

2023-12-01T21:59:39.58+00:00

@Xiaopo Yang - MSFT I'm able to reproduce the bigger extents by checking fsutil usn. Reason in records is 0x80000001: Data overwrite | Close

One thing I observed is all the big extents are of size multiple of 64MB and in a 30 min interval. I suspect it might be something related to how MSSQL writes. But actual data is not written on the disk. I'm not certain right now why these extents are reported in USN v4 record

Answer 1

The USN_RECORD structure, used in NTFS file systems for tracking changes, might sometimes report larger extents of file modifications than what actually occurred. Here are simpler reasons:

Sparse Files: If your file has empty or zeroed regions, the system may mistakenly report changes there.

Fragmentation: If your file is scattered in different parts of the disk, changes from those parts might get combined into one report.

Non-contiguous Changes: Changes happening in separate places within the file may be grouped together, causing larger reported extents.

Metadata Changes: Changes to file details (like attributes or security info) can contribute to larger reported extents.

Overhead from Applications: Some programs may interact with the file system in ways that make the reported extents bigger.

To investigate, check for file fragmentation, sparse file regions, and how the file is being modified

Answer 2

Husnain Ali 0

The USN_RECORD structure, used in NTFS file systems for tracking changes, might sometimes report larger extents of file modifications than what actually occurred. Here are simpler reasons:

Sparse Files: If your file has empty or zeroed regions, the system may mistakenly report changes there.

Fragmentation: If your file is scattered in different parts of the disk, changes from those parts might get combined into one report.

Non-contiguous Changes: Changes happening in separate places within the file may be grouped together, causing larger reported extents.

Metadata Changes: Changes to file details (like attributes or security info) can contribute to larger reported extents.

Overhead from Applications: Some programs may interact with the file system in ways that make the reported extents bigger.

To investigate, check for file fragmentation, sparse file regions, and how the file is being modified

Kanak Agrawal 6 Reputation points

2023-12-03T17:20:24.8033333+00:00

Non-contiguous Changes: Changes happening in separate places within the file may be grouped together, causing larger reported extents. Overhead from Applications: Some programs may interact with the file system in ways that make the reported extents bigger.

@Husnain Ali I'm following changes reported for a SQL data file. I've seen that at times changes are reported of size of 64 MB or multiple of that but actual change in data is less than that. Can you share some more info/documentation of how changes can be grouped together and cause a larger reported change. I thought change journal will always have exact extents which were written by the application.

In our case, the file is not a sparse file. Also, I didn't understand how fragmentation could result in larger extents. Which bytes are modified in a file should be independent of fragmentation.
Xiaopo Yang - MSFT 12,731 Reputation points Microsoft External Staff

2023-12-04T01:56:30.2933333+00:00

Are your problematic records all Close records?

As far as Close operation is concerned, there is a little description in Change Journal Records,

The final record, created when the file is closed, adds the USN_REASON_CLOSE flag. This record represents a summary of changes to the file, but unlike the prior records, gives no indication of the order of the changes.

So, I suppose, based on the following document, all write operations except the first write operation are accumulated in the last Close record.

In addition, multiple changes to the same file may result in only one reason flag being added to the current record. If the same kind of change occurs more than once, the NTFS file system does not write a new record for the changes after the first. For example, several write operations with no intervening close and reopen operations result in only one change record with the reason flag USN_REASON_DATA_OVERWRITE set.

Second write operation The NTFS file system does not write a new USN record. Because USN_REASON_DATA_OVERWRITE is already set for the existing record, no changes are made to the record.

Share via

extents reported in USN_RECORD_V4 struct are a lot bigger than actual changes

2 answers

Your answer