Windows Searching Platform

Using Find My Files in Your App

adamdwilson — Wed, 24 Oct 2018 21:14:43 GMT

The new indexing mode of called Find My Files Enhanced expands the number of locations where apps can use the indexer to search. Any app can issue a query against the search indexer for Find My Files locations and build a great search experience.

What is Find My Files

To understand why Find My Files is interesting you must understand the basics of how the indexer has worked since the beginning of time - defined here not as 1970 but the much more recent XP SP2 release

When the indexer sees a new file on the disk it does two things:

Get the metadata about the file from the file system and make it searchable (such as file name and ACLs)
Extract data from the file and make it searchable (the full text of documents or emails, EXIF properties on images)

Historically, these two operations were linked together. First the metadata would be extracted, then the full content. However, getting content from the file is up to 100x slower than just getting the metadata from the file system. The filesystem (especially on NTFS) can almost instantly give you the metadata, whereas reading a file stream from disk, parsing it, pulling out all the text or metadata, and storing that in the index is a CPU and disk intensive process.

This means that although the indexer could have indexed an entire hard drive, the extra volume of indexing work would cause the machine to grind to a halt. Thus, users were stuck without fast search outside of their user profile.

For Find My Files steps 1 and 2 are separated. Files included under Find My Files locations, only their file system metadata will be indexed, and the file content will not be opened. This lets the indexer index more files and folders using fewer resources.

The trade-off though is clear. In locations where only the file system metadata is gathered, the indexer doesn’t have any data about the content of files. This means that many types of queries cannot be served such as sorting pictures by date taken or location.

There are now 3 types of locations on a computer with Find My Files enabled:

Type of Location	Examples	What is indexed?
Type of Location	C:\Windows C:ProgramData Removable drives Temporary file locations	Nothing – all searches are handled by opening files at query time and walking through them
Find My Files Locations	C:\MyFolder D:\PicturesBackup C:\Users\Adam\MySpecialFolder	Only file system metadata: File Path System.ItemNameDisplay System.FileExtension System.Kind System.DateCreated System.DateModified System.Shell.SFGAOFlagsStrings System.Link.TargetParsingPath
Indexed Locations	C:\Users\Adam\Pictures C:\Users\Adam\Videos	All metadata and content for the file

Using Find My Files In Your App

Adding search over a Find My Files location is almost identical to using Windows Search normally or like using any other OLE DB connector. The full documentation for Windows Search is here, or if you’re looking for a quick start Josh Twist has a great write up.

The only gotcha to be aware of is that in Find My File locations, only the file system metadata will be available.

Below is a simple app that just gets the first 5 files off the D: drive that start with img*.

static void Main(string[] args)
{
    string query = @"SELECT TOP 3 System.ItemNameDisplay, System.DateCreated
                    FROM SYSTEMINDEX
                    WHERE scope='file:D:/'
                    AND (CONTAINS(System.ItemNameDisplay, 'img*'))";
    using (OleDbConnection objConnection =
           new OleDbConnection
           ("Provider=Search.CollatorDSO;Extended Properties='Application=Windows';"))
    {
        objConnection.Open();
        OleDbCommand cmd = new OleDbCommand(query, objConnection);
        using (OleDbDataReader rdr = cmd.ExecuteReader())
        {
            for (int i = 0; i < rdr.FieldCount; i++)
            {                       
                Console.Write(rdr.GetName(i));
                Console.Write('\t');
            }
            while (rdr.Read())
            {
                Console.WriteLine();
                for (int i = 0; i < rdr.FieldCount; i++)
                {
                    Console.Write(rdr[i]);
                    Console.Write('\t');
                }
            }
            Console.ReadKey();
        }
    }
}

The output is:

 SYSTEM.ITEMNAMEDISPLAY SYSTEM.DATECREATED
 IMG_0002.JPG 5/4/2013 7:14:19 PM
 IMG_0005.JPG 5/4/2013 7:18:06 PM
 IMG_0038.JPG 5/5/2013 8:22:53 PM

And that is it. Seriously – with only 25 lines of code we can pull the same results from the indexer that are used in the system search experience for your own app. With Find My Files, this code is even more valuable as users can search more locations on their machine using the search indexer.

There are countless ways to get fancy here. RANK BY COERCION gives you pinpoint control over how results are ordered, or a simple ORDER BY clause can do simple ordering easily. There are also countless variations you can make to the WHERE clause to filter by file types or kind of file. This is all documented in the Window Search Developer Guide, along with more samples to inspire your coding.

Fast File Enumeration with Partially Initialized StorageFiles

adamdwilson — Wed, 20 Dec 2017 17:35:35 GMT

Summary:

Enumerating files in a directory is now 10x faster for certain scenarios in RS3
Requires only a small change to existing code
Backwards compatible with your existing codebase

Intro

Frequently when UWP developers talk to me about quickly accessing files, it comes down to a common pattern: Getting the filename and properties for a large number of files, and then interacting with a few. This pattern spans from media apps listing all the songs available to play and only playing one, to LOB apps showing a list of datafiles or templates before the user picks one to use. Since you’ve made it clear this is a common pattern for so many apps, we’ve made a few changes in the Fall Creators Updated to help you to write faster, more memory efficient file enumeration code. You can leverage these changes with only a minor modification to your existing app How much faster? For large data sets it is almost 9x faster. The data below shows how stark the difference is - the top row is in the Creators Update, the bottom row is the Fall Creators Update.

API Used	1017 .jpgs, including ImageProperties	10170 .jpgs, including ImageProperties
Existing (Full StorageFiles)	3 000ms	32 000ms
New(Partial StorageFiles)	450ms	3 700ms

That’s an almost 9x improvement in performance from just a simple change to the code. I was able to make the change in under 10 minutes to my app, and the improvement was instantly noticeable. We’ll cover the code in detail below, but it enumerates all the pictures on a machine. Before we can make the change, a little bit of background on the API surface is needed. If you’re looking for a quick code sample, check out the MSDN quick start. This post is going to go into a lot of gritty detail to help you understand what is happening under the covers. The quick start covers very similar code but with a focus on getting it in your app right now.

Technical Background:

StorageFiles are the core part of the existing UWP surface and are the common currency for access to a file. Once your app has a StorageFile, it has many different options including:

Sharing permissions to the file with another app
Reading/Modifying/Deleting/Copying/Moving the file
Get a unique identifier for the file
Accessing properties of the file
Finding out if the file is from a cloud storage provider

These are rich objects used throughout the platform, but this richness comes at a cost. Fetching properties, cloud provider details, and permissions all take CPU cycles and memory. This richness is great for apps providing complex interactions with the file but slows down apps which only need some of these features. In the Fall Creators Update there is a new flag for creating StorageFile queries that will inform the system to not fully initialize the StorageFile objects it returns to your app. Instead, it will follow a lazy initialization pattern and only create a partial StorageFile object. These partial StorageFiles are much faster to create and only contain basic file properties. However they must be initialized into full StorageFiles to before any of the rich features are used.

What are Partially Initialized StorageFiles

Partially Initialized StorageFiles (or partial StorageFiles for short) are for when apps need the filename and properties from a lot of a files but will only interact with a few of them. Apps first notify the system that they want partial StorageFiles when creating a file query. The system will create partial StorageFiles containing only the data requested in the query. These partial StorageFiles can then have the properties read or be initialized to full StorageFiles as needed. The result is the initial enumeration and property retrieval is extremely fast since partial StorageFiles are much faster to create. Partial StorageFile objects only contain the some properties and are not fully initialized with all the backing data that you expect to have in a fully initialized StorageFile - such as a connection to the system thumbnail DB or the ability to share the file using an app contract. These connections are created later when requested by the app. This means partial StorageFiles are not going to be useful in all scenarios.

When should I use Partial StorageFiles

Partial StorageFiles are designed for cases where you need the file properties from a lot of different files but don’t need interact deeply with all the files. Here are a few common cases where this might happen:

A music app wants to show the titles and artist for all the tracks in an album, but only plays 1 song at a time
CNC mills which have many preset templates, but only use one at a time
Movie or image gallery apps which show a bunch of media, but only play one at a time

These scenarios, along with many others can be improved by using partial StorageFiles. However, there are some cases where using this new option doesn’t make sense. Any apps interacting in depth with most or all the files returned from a query should not use partial StorageFiles. I’ll dive into the details why later, but here are a few examples of apps where this technique doesn’t provide a performance boost:

Apps that bulk update the properties on collections of files
Apps open many files to display a base experience. For example, opening all the files in a directory to display a 3D model

Make the choice that is right for your app and your scenario. Partial StorageFiles are not a replacement for all your existing code using StorageFiles but are there to help in a few common scenarios.

Interacting with Partial Storage Files

Interacting with partial StorageFiles in your code is very similar to using full StorageFiles. The same methods and properties are all there and you can call them normally. There are just a couple differences that you should be aware of. Accessing the file properties from a partial StorageFile is synchronous as opposed to regular StorageFiles which have asynchronous access. This provides a performance boost versus full StorageFiles. When an app invokes any operation using a partial StorageFile that requires a full StorageFile, the system will convert the partial StorageFile into a full StorageFile on the app’s behalf. The process of fully initializing a partial StorageFile requires some computation which results in a slight time delay (~1ms). Fast access to file properties requires the Windows indexer, which places a restriction on partial StorageFiles - they are only available in indexed locations. Fortunately, all user libraries are indexed by default and your app can prompt the user to add new locations to the library at any time. To help clarify, here is a chart of what you can and can’t do with partial and full StorageFiles.

Full StorageFile (from all existing APIs, and most future ones)	Partial StorageFile (from fast indexed properties)
Can access basic properties of a file (file path)	Same as full StorageFile
Can access extended properties of the file (album artist)	Can access only properties requested at creation
Can copy/move/delete the underlying file	Initializes a full StorageFile automatically
Can read or modify the underlying file	Initializes a full StorageFile automatically
Can be used to pass permissions to another app	Initializes a full StorageFile automatically
Contain a unique ID to use for the file (different than the path or filename in cases where the user is using files with the same file name)	Initializes a full StorageFile automatically
Can work outside of indexed locations	Cannot be created outside indexed locations

Code Sample and Walkthrough

Let’s build up a simple app to demonstrate how to create partial StorageFiles. We will print out the GPS coordinates of all the pictures in the picture’s library, so the user can see where they’ve been. You can adapt this code into your app or hopefully just use it as a guideline for updating your existing code. I will skip over the details on getting permission and the basics of StorageFiles. There are tutorials on permissions and libraries, basic enumeration, and other StorageFile topics if you are interested in learning more about some of the choices I’m making in the sample.

Setting Up

StorageFolder folderToEnumerate = KnownFolders.PicturesLibrary;
//check if the folder is indexed before doing anything
IndexedState folderIndexedState = await folderToEnumerate.GetIndexedStateAsync();
if (folderIndexedState == IndexedState.NotIndexed || folderIndexedState == IndexedState.Unknown)
{
   //Only possible in indexed directories. 
   return;
}

The first step is to grab a StorageFolder for the target location and verify partial StorageFiles are available in that location. Partial StorageFile only work in indexed locations so it is important to check if the folder is indexed. In this sample, we will accept partially indexed locations, because speed is more important than completeness. Your app might have to make a different choice based on your users’ expectations. Sidebar: IndexedState.PartiallyIndexed only refers to the indexer’s scopes, not to the status of the indexing process. Partially indexed means there could be child folders which are not indexed under a root folder that is indexed. This commonly happens if the user has a reparse point under an indexed folder which points to an unindexed network share. When using partial StorageFiles, partially indexed folders will return results from the indexed locations and ignore the unindexed subfolders. There may be files in fully indexed locations which aren’t in the index yet because the indexer is behind the file system. This is not reflected in the IndexedState for the folder. The indexing status cannot be provided because the indexer is unable predict how many files it doesn’t know about.

Query Basics

QueryOptions picturesQuery = new QueryOptions()
{
   FolderDepth = FolderDepth.Deep,
   //Filter out all files that have WIP enabled on them
   ApplicationSearchFilter = "System.Security.EncryptionOwners:[] ",
   IndexerOption = IndexerOption.OnlyUseIndexerAndOptimizeForIndexedProperties
};
picturesQuery.FileTypeFilter.Add(".jpg");

Next, we will setup a query against the pictures library. The most important part of a query for partial StorageFiles is setting IndexerOption.OnlyUseIndexerAndOptimizeForIndexedProperties. This tells the system to return partial StorageFiles to your app. Any other IndexerOption value will return full StorageFiles and not see any performance gains. Note that we’re filtering out any files with System.Security.EncryptionOwners set. This is a quick way to make sure that your app doesn’t get any WIP protected files back in the results. It isn’t required for using partial StorageFiles but not handling encrypted files makes the sample code simpler.

Property Prefetch

string[] otherProperties = new string[] { SystemProperties.GPS.LatitudeDecimal,
                                          SystemProperties.GPS.LongitudeDecimal };
picturesQuery.SetPropertyPrefetch(PropertyPrefetchOptions.BasicProperties | 
                                  PropertyPrefetchOptions.ImageProperties,
                                  otherProperties);

In this step, we are declaring the set of properties the partial StorageFiles will contain. The app will need the image properties and two GPS properties. Accessing any of these properties from the partial StorageFiles returned in this query will be fast, but other properties will require initializing a full StorageFile. In general, you’ll always include PropertyPrefetchOptions.BasicProperties, which has properties such as the file path and file name. Add as many properties as you need at query time. Fetching more properties is much faster during the initial query than fetching extra properties by initializing in a full StorageFile. There is still a slight cost for each additional property at query time, so be sure not to include surplus properties. Most common file properties are represented in the SystemProperties class, but there are other properties you might need. In advanced cases, the property system reference includes all the details that you’ll need. Partial StorageFiles only work with properties where isColumn = true. For more information about how to read the property system doc please see my video and post on it.

Sort Order

SortEntry sortOrder = new SortEntry()
{
   AscendingOrder = true,
   PropertyName = "System.FileName"
};
picturesQuery.SortOrder.Add(sortOrder);
if (!folderToEnumerate.AreQueryOptionsSupported(picturesQuery))
{
   log("Querying for a sort order is not supported in this location");
   picturesQuery.SortOrder.Clear();
}

The app will also sort the pictures by the file name, so we create a SortEntry for the query. Sorting is a tricky beast because it requires there be an ordering available for the file system language and for the location to be fully indexed. Since it isn’t always possible to sort at query time, it is good practice to check if the sort order is supported before issuing a query. If it isn’t supported, then clear out the SortOrder from the query. Your app can always sort the results later using a LINQ query.

Getting the Results

//Create the query and get the results
uint index = 0;
const uint stepSize = 100;
StorageFileQueryResult queryResult = folderToEnumerate.CreateFileQueryWithOptions(picturesQuery);
IReadOnlyList images = await queryResult.GetFilesAsync(index, stepSize);
while (images.Count != 0)
{
   foreach (StorageFile file in images)
   {
      //With the OnlyUseIndexerAndOptimizeForIndexedProperties set, this won't
      //be async. It will be run synchronously
      var imageProps = await file.Properties.GetImagePropertiesAsync();

      //Build the UI
      log(String.Format("New: {0} at {1}, {2}",
      file.Path,
      imageProps.Latitude,
      imageProps.Longitude));
   }
   index += stepSize;
   images = await queryResult.GetFilesAsync(index, stepSize);
}

Finally, the app gets the results from the query and outputs them to the user. If you’ve read any of my other articles on fast file enumeration, this code will be familiar. The best practice is to page in manageable sets of files to limit your commit usage in cases where the user has millions of files. You can process each set of files while waiting for the next one to be returned. The significant difference for partial StorageFiles is GetImagePropertiesAsync. With full StorageFiles this method causes an out of process call as the system fetches the properties from either the indexer or the file. Partial StorageFiles however will short circuit this call and return the results from in your processes memory assuming the properties were specified in the prefetch. This makes the retrieval much faster and reduces the commit used by the system on behalf of your app. Finally, the app will log the image properties to an output pane. There are no limits to the experiences you can build using users’ files, and I look forward to seeing what you do with partial StorageFiles.

Details on Intializing Full StorageFiles

In the above examples we never initialize full StorageFiles, and this is a common case. Many apps will never need a full StorageFiles - everything they use is in partial StorageFiles. There are cases however where initializing a full StorageFile is required, such as if you are going to access the thumbnail with GetThumbnailAsync. In these cases, you are going to see the following behaviour:

StorageFile myFile = … //Get a partial StorageFile from a query

//Partial StorageFile
var name = myFile.Name;

//Partial StorageFile
var imageProperites = await myFile.Properties.GetImagePropertiesAsync();

//Initializes a full StorageFile (~1ms penalty)
var relativeId = myFile.FolderRelativeId;

//Uses existing full full StorageFile (no additional penalty)
var thumbnail = await myFile.GetThumbnailAsync(ThumbnailMode.PicturesView);

There is no difference in the code that you need to write, the system takes care of initializing the full StorageFile for you when it is needed. A full StorageFile will not be initialized in until it is absolutely needed, and the time delay to create it will only happen once per instance. It is important to know what properties are going to cause initialization: AttributesYes

Property/Method	Is it available from a partial StorageFile
Content Type	Yes
Date Created	Yes
Display Name	Yes
Display Type	Yes
File Type	Yes
FolderRelativeId	No, cannot be requested with a query and only available from full StorageFiles
IsAvailable	No. Only available from full StorageFiles
Name	Yes
Path	Yes
Properties	If requested Properties requested in the query will be included in the partial StorageFile, otherwise they will require a full StorageFile
All methods	No. Any operation on a StorageFile requires a full StorageFile

Note that all operations where you pass a StorageFile into another system function will require a full StorageFile.

Wrap – Up

There are a few critical points to remember when using partial StorageFiles to ensure everything goes smoothly:

Partial StorageFiles are designed for reading properties from many files and only interacting with a few files
Use IndexerOption.OnlyUseIndexerAndOptimizeForIndexedProperties to fetch partial StorageFiles
Only use partial StorageFiles in fully or partially indexed locations
Request all the properties your app will need in the prefetch options.
Initialize as few full StorageFiles as possible to save memory and CPU

If you follow these guidelines, you can create responsive and engaging experiences for your users using their files.

Sample Function

In case you’re looking for a complete function to try out in your app, here is the full snippet from the article.

private async void EnumeratePictureLibrary()
{
   StorageFolder folderToEnumerate = KnownFolders.PicturesLibrary;
   //check if the folder is indexed before doing anything
   IndexedState folderIndexedState = await folderToEnumerate.GetIndexedStateAsync();
   if (folderIndexedState == IndexedState.NotIndexed || folderIndexedState == IndexedState.Unknown)
   {
      //Only possible in indexed directories.
      return;
   }

   QueryOptions picturesQuery = new QueryOptions()
   {
      FolderDepth = FolderDepth.Deep,
      //Filter out all files that have WIP enabled on them
      ApplicationSearchFilter = "System.Security.EncryptionOwners:[] ",
      IndexerOption = IndexerOption.OnlyUseIndexerAndOptimizeForIndexedProperties
   };
   picturesQuery.FileTypeFilter.Add(".jpg");
   string[] otherProperties = new string[] { SystemProperties.GPS.LatitudeDecimal,
                                             SystemProperties.GPS.LongitudeDecimal };

   picturesQuery.SetPropertyPrefetch(PropertyPrefetchOptions.BasicProperties | 
                                    PropertyPrefetchOptions.ImageProperties,
                                    otherProperties);
   SortEntry sortOrder = new SortEntry()
   {
      AscendingOrder = true,
      PropertyName = "System.FileName"
   };
   picturesQuery.SortOrder.Add(sortOrder);

   if (!folderToEnumerate.AreQueryOptionsSupported(picturesQuery))
   {
      log("Querying for a sort order is not supported in this location");
      picturesQuery.SortOrder.Clear();
   }
   //Create the query and get the results
   uint index = 0;
   const uint stepSize = 100;
   StorageFileQueryResult queryResult = folderToEnumerate.CreateFileQueryWithOptions(picturesQuery);
   IReadOnlyList images = await queryResult.GetFilesAsync(index, stepSize);
   while (images.Count != 0)
   {
      foreach (StorageFile file in images)
      {
         //With the OnlyUseIndexerAndOptimizeForIndexedProperties set, this won't
         //be async. It will be run synchronously
         var imageProps = await file.Properties.GetImagePropertiesAsync();

         //Build the UI
         log(String.Format("New: {0} at {1}, {2}",
         file.Path,
         imageProps.Latitude,
         imageProps.Longitude));
      }
      index += stepSize;
      images = await queryResult.GetFilesAsync(index, stepSize);
   }
}

Build 2017 Code Samples

adamdwilson — Thu, 01 Jun 2017 21:09:19 GMT

UPDATE DECEMBER 2017 Hi everyone, thanks for watching our build talk and following the link to the code sample. At this time I have even more bad news. As many of you suspected, the code for using CreateFile2 to access brokered locations was pulled from the Fall Creators Update at the last minute. There were issues with the security model that resulted in us being unable to ship it in the current form. We have not stopped development though and are working to try and bring CreateFile2 to the universal platform without compromising the security of the operating system. We will make another announcement once there is a working prototype in the system for you to try out again. Sorry for the delay, and thanks in advance for your patience

Change Tracking: More Betterness

adamdwilson — Tue, 04 Oct 2016 18:40:50 GMT

In a couple previous posts on this blog we have covered how a UWP can track changes that are happening in a user's library. Although there were notifications and some apps could enable the scenarios they are looking for, the code required was still complicated. We heard a lot of feedback from developers that it could be a made simpler and took it to heart. So for the Windows 10 Anniversary Update I'm happy to announce the StorageLibraryChangeTracker. The StorageLibraryChangeTracker is a new feature that is going to allow apps to easily track files and folders as users move them around the system. Using the StorageLibraryChangeTracker, an app can now easily track:

File operations including add, delete, modify
Folder operations such as renames and deletes
Items moving through the system to differentiate between a move within a library, a move outside a library and a delete.

Since the change tracker tracks folder level operations and file deletions, it greatly reduces the effort for an app to track changes in a library. In this blog post we are going to briefly walk through the programing model for working with the change tracker, a brief outline of some sample code, and the different types of file operations that are tracked.

Using the Change Tracker

The change tracker is implemented on system as a circular buffer storing the last N file system operations. Apps are able to read the changes off of the buffer and then process them into their own experiences. Once the app is finished with the changes it marks the changes as processed and will never see them again

Enabling the change tracker

The first thing that the app needs to do is to tell the system that it is interested in change tracking a given library. It does this by calling Enable() on the change tracker for the library of interest

StorageLibrary videosLib = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Videos);
StorageLibraryChangeTracker videoTracker = videosLib.ChangeTracker;
videoTracker.Enable();

A few important notes:

Make sure your app has permission to the correct library in the manifest before creating the StorageLibrary object
Enable is thread safe, will not reset your pointer, and can be called as many times as you'd like (more on this later)

Waiting for changes

Once the change tracker is initialized it will begin to record all of the operations that occur within a library, even while the app isn’t running. If you'd like your app's background task to be activated when changes happen, please see the StorageLibraryChangedTrigger or my blog post.

Reading the Changes

The app can then poll for changes from the change tracker, and receive a list of the changes since the last time the app checked. The code to get a list of changes is below, but we will cover how to understand those changes in another section.


StorageLibrary videosLibrary = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Videos);
videosLibrary.ChangeTracker.Enable();
StorageLibraryChangeReader videoChangeReader = videosLibrary.ChangeTracker.GetChangeReader();
IReadOnlyList changeSet = await changeReader.ReadBatchAsync();

The app is now responsible for processing the changes into its own experience or database as needed. Pay special attention to the second call to enable here, this is to defend against a race condition if the user adds another folder to the library. Without the extra call to Enable(), the code will work in most scenarios, but if the user is changing their library folders it will fail with the exception ecSearchFolderScopeViolation (0x80070490).

Accepting the changes

Once your app is done processing the changes that have been delivered to it, it should tell the system to never show it those same changes again. This is done by calling:


await changeReader.AcceptChangesAsync();

Note:

If changes have happened between calling ReadBatchAsync and AcceptChangesAsync, the pointer will be only be advanced to the most recent change the app has seen. Those other changes will still be available the next time it calls ReadBatchAsync
Not accepting the changes will cause the system to return the same set of changes the next time the app calls ReadBatchAsync.

Important Things to Remember

When using the change tracker, there are a few things that you should keep in mind to make sure that everything is working correctly.

Buffer Overruns

Although we try to reserve enough space in the change tracker to hold all the operations happening on the system until your app can read them, it is very easy to imagine a scenario where the app doesn't read the changes before the circular buffer overwrites itself. Especially if the user is restoring data from a backup or syncing a large collection of pictures from their camera phone. In this case a special error will be returned from ReadBatchAsync, StorageLibraryChangeType.ChangeTrackingLost. If your app receives this error code, it means a couple things:

The buffer has overwritten itself since the last time you looked at it. The best course of action is to recrawl the library, since any information from the tracker will be incomplete
The change tracker will not return any more changes until you call Reset(). Once the app calls reset, the pointer will be moved to the most recent change and tracking will resume normally.

It should be rare to get these cases, but in scenarios where the user is moving a large number of files around on their disk we don't want the change tracker to balloon and take up too much storage. This should allow apps to react to massive file system operations while not damaging the customer experience in Windows.

Changes to a StorageLibrary

StorageLibraries are an interesting concept in UWPs. They exist as a virtual group of root folders that may contain other folders. To reconcile this with a file system change tracker, we made the following choices:

Any changes to descendent of the root library folders will be represented in the change tracker
- The root library folders can be found using StorageLibrary.Folders
Adding or removing root folders from a StorageLibrary (through RequestAddFolderAsync and RequestRemoveFolderAsync) will not create an entry in the change tracker
- These changes can be tracked through the DefinitionChanged event on StorageLibrary or by enumerating the root folders in the library using StorageLibrary.Folders.
- This will also mean that if a folder with content already in it is added to the library, there will not be a change notification or change tracker entries generated. Any subsequent changes to the descendants of that folder will generate notifications and change tracker entries.

Calling Enable()

It’s mentioned above, but just to make sure that it is clear: Apps should call ChangeTracker.Enable() as soon as they start tracking the file system and before every enumeration of the changes. This will ensure that the change tracker is not going to miss changes to the folders included in the library.

Putting It all Together

Here is all the code that is used to register for the changes from the video library and start pulling the changes from the change tracker.

private async void EnableChangeTracker()
{
    StorageLibrary videosLib = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Videos);
    StorageLibraryChangeTracker videoTracker = videosLib.ChangeTracker;
    videoTracker.Enable();
}

private async void GetChanges()
{
    StorageLibrary videosLibrary = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Videos);
    videosLibrary.ChangeTracker.Enable();
    StorageLibraryChangeReader videoChangeReader = videosLibrary.ChangeTracker.GetChangeReader();
    IReadOnlyList changeSet = await changeReader.ReadBatchAsync();


    //Below this line is for the blog post. Above the line is for the magazine
    foreach (StorageLibraryChange change in changeSet)
    {
                    
        if (change.ChangeType == StorageLibraryChangeType.ChangeTrackingLost)
        {
            //We are in trouble. Nothing else is going to be valid. 
            log("Reseting the change tracker");
            videosLibrary.ChangeTracker.Reset();
            return;
        }                    
        if (change.IsOfType(StorageItemTypes.Folder))
        {
            await HandleFileChange(change);
        }
        else if (change.IsOfType(StorageItemTypes.File))
        {
            await HandleFolderChange(change);
        }
        else if (change.IsOfType(StorageItemTypes.None))
        {
            if (change.ChangeType == StorageLibraryChangeType.Deleted)
            {
                RemoveItemFromDB(change.Path);
            }
        }
    }
    await changeReader.AcceptChangesAsync();
}

Sorting the changes

The file system can be a complicated beast, and there are many different ways that users or programs can change how it will appear to your app. To help make it easier to understand what is going on, the change tracker provides two pieces of information to your app - the type of item that was changed, and the change that occurred. The following code shows an example of how the changes might sorted in an app that is going to be maintaining a cache of the state of the file system. (Yes, the indexer team does write indexers for their indexer code samples - you really don't want us writing anything else)

private async Task HandleFileChange(StorageLibraryChange change)
{
    //Temp variable used for instantiating StorageFiles for sorting if needed later
    //We are only going to create the StorageFile if needed to save the overhead
    StorageFile newFile = null;
    switch (change.ChangeType)
    {
        case StorageLibraryChangeType.Created:
        case StorageLibraryChangeType.MovedIntoLibrary:
            if (IsPathInAppDB(change.Path))
            {
                //Should never hit this, but check to make sure our sync isn't busted
                RemoveItemFromDB(change.Path);
            }

            //Need to filter the item to make sure that it is going to be something we are interested in
            newFile = (StorageFile)(await change.GetStorageItemAsync());
            if (newFile.ContentType.Contains("audio") || newFile.FileType == ".mp4")
            {
                AddItemToDB(newFile);
            }
            break;

        case StorageLibraryChangeType.Deleted:
        case StorageLibraryChangeType.MovedOutOfLibrary:
            if (IsPathInAppDB(change.Path))
            {
                //If we are tracking the file then lets remove it from the database
                RemoveItemFromDB(change.Path);
            }
        break;

        case StorageLibraryChangeType.MovedOrRenamed:
            if (IsPathInAppDB(change.PreviousPath))
            {
                MoveExistingItemInDB(change.PreviousPath, change.Path);
            }
            else
            {
                //Need to filter the item to make sure that it is going to be something we are interested in
                newFile = (StorageFile)(await change.GetStorageItemAsync());
                if (newFile.ContentType.Contains("audio") || newFile.FileType == ".mp4")
                {
                    AddItemToDB(newFile);
                }
            }
        break;

    case StorageLibraryChangeType.EncryptionChanged:
    case StorageLibraryChangeType.IndexingStatusChanged:
    case StorageLibraryChangeType.ContentsReplaced:
        if (IsPathInAppDB(change.Path))
        {
            UpdateExistingItemInDB(change.Path);
        }
    break;

    default:
    return;
    }
}

Now let's walk through what is going on in the code and each of the possible states of a StorageLibraryChange.

Types of Items in a StorageLibraryChange

The types of items that can be changed are by calling StorageLibraryChangeType.IsOfType(StorageItemTypes). There are 3 options that are possible here:

StorageItemType	Description
StorageItemType.File	The item changed was a file
StorageItemType.Folder	The item changed was a folder
StorageItemType.None	The type of the item changed couldn't be discovered by the change tracker. This is most likely to happen on operations for FAT based file systems, since they don't record the type of the item before deleting it.

Your app should be able to handle all 3 types of items being returned, or at least fail gracefully if a type is returned that it isn’t expecting.

Types of Changes in a StorageLibraryChange

There are 10 different values of StorageLibraryChangeType that can be returned to your app. Each change type gives enough information about a change to the files system for your app to know what item is being changed and to find the new location of it on disk.

StorageLibraryChangeType Value	StorageItemType.File	StorageItemType.Folder	StorageItemType.None
Created	A new file was created in the library. StorageLibraryChange.Path will contain the path of the new file	A new folder was created in the library. StorageLibraryChange.Path will contain the path of the new file	This should not be expected
Deleted	A file was deleted from the library. StorageLibraryChange.Path will contain the former path of the file that was deleted	A folder was deleted from the library. StorageLibraryChange.Path will contain the former path of the folder that was deleted. All subfolders and files of this folder will also generate delete notifications	An item was deleted from a FAT drive. In this case the change tracker cannot determine the item type, but will include the path
MovedOrRenamed	The file was moved or renamed. Path will contain the new path of the item and previous path will include the path of the item before the operation	The folder was moved or renamed. Path will contain the new path of the item and previous path will include the path of the item before the operation	This should not be expected
ContentsChanged	The file was modified in some way. This can include changing the file contents or just changing the metadata about the file (such as the file attributes)	The folder itself, not a descendent of the folder, was modified. In most cases this is going to mean a folder attribute was changed	This should not be expected
MovedOutOfLibrary (see below)	The file was moved from within a library to outside of the tracked location. This is returned on a best effort basis, otherwise MovedOrRenamed will be used	The folder was moved from within a library to outside of the tracked location. This is returned on a best effort basis, otherwise MovedOrRenamed will be used	This should not be expected
MovedIntoLibrary (see below)	The file was moved from somewhere else on the system into a library. This is returned on a best effort basis, otherwise MovedOrRenamed will be used	The file was moved from somewhere else on the system into a library. This is returned on a best effort basis, otherwise MovedOrRenamed will be used	This should not be expected
ContentsReplaced	The contents of the file were replaced. This normally results from a hardlink being changed.	The contents of the folder were replaced. This normally results from a hardlink being changed.	This should not be expected
IndexingStatusChanged	The file was either added or removed from the indexing scope. Check FILE_ATTRIBUTE_NOT_CONTENT_INDEXED (FANCI) to see what happened. If the FANCI bit is set, the file will not appear in indexed queries	The folder was either added or removed from the indexing scope. Check FILE_ATTRIBUTE_NOT_CONTENT_INDEXED (FANCI) to see what happened. If the FANCI bit is set, the folder and all its descendants will not appear in indexed queries	This should not be expected
EncryptionStatusChanged	The file has been encrypted or decrypted, generally a result of the enterprise changing some policy	The folder's contents have been encrypted or decrypted, generally a result of the enterprise changing a policy	This should not be expected
ChangeTrackingLost	All further result from the change tracker are invalid, see above for more details	All further result from the change tracker are invalid, see above for more details	All further result from the change tracker are invalid, see above for more details

MovedOutOfLibrary/MovedOutOfLibrary

These are going to be returned as best as possible by the platform. In cases where we cannot determine if the destination or origin is in a library, it will return MovedOrRenamed. The theory is that it would be great to have all the information about if the file was moved out of the library right away, but sometimes it isn't possible to calculate that. In those cases, your app will still be able to see the change and react accordingly.

Coalescing the change notifications

It is possible that between two subsequent reads of the change tracker by an application, there may be a number of operations to a single file. If possible the change tracker will attempt to reduce the load on the application by simplifying the operations. For example, an add then move operation will be combined to a single add operation in the file’s final location. In cases where it isn't possible to simplify the operations they will be presented separately, for example if there was a move then a modify operation.

Finishing Up

And that's it. Your app can now track all the changes that are happening in a given library and find out exactly where the files are going. I'm looking forward to seeing all the new apps that are built using the change tracker and the exciting experiences they bring to users. Happy tracking

Searching Private App Data in Windows 10

adamdwilson — Wed, 30 Mar 2016 17:40:04 GMT

Up until this point I’ve walked through how you can use the indexer to search through public data on a device, but what if your data cannot be stored in a public location or isn’t resident on the device but you’d like local search. In this case the indexer provides two different methods to help you add fast, internationalized search to your app. The methods come with a couple of names, but we will refer to them as the ContentIndexer and the indexed folder.

What is the ContentIndexer?

The ContentIndexer class enables apps to provide a property bag full of metadata that the system will then index and make searchable. The property bag can contain any properties that are present in the shell property system (see note below) and will be searchable as soon as the API call returns. The ContentIndexer was initially designed to be used by Edge for searching its history. When a user visits a webpage, the metadata is pushed into the index. Then when the user starts typing in the address box if there is a match with anything in the history it will be included in the drop down.

What is the Indexed Folder?

The indexed folder is a special folder that can be created in the local storage of a UWA. When the folder is created the indexer will add that location to its scope and indexed it as if the location was in a library. The contents will still say private to your app but you will have the same search, sorting, and filtering capabilities as the public libraries An example of this being used in the system is the settings app in Windows 10. All the of the settings entries in the app have a special file in the apps indexed folder. When a user types a search query, the app passes it to the indexer to match to the files in the indexed folder. It then maps the results back to the settings entries that they represent.

Which option to use?

The general rule of thumb is that if your data is going to be frequently updated, the ContentIndexer will probably make more sense. But as always there are nuances that need to be considered. The following chart will try to outline the differences between the two options

Feature	ContentIndexer	Indexed Folder
How data gets into the index	App pushes in the data, must listen to make sure indexing was successful	App creates files on the disk. Indexer manages the indexing process automatically
Type of data provided by app	Property bags	Any file type, metadata only appcontent-ms files can be used
Indexing Priority Control	High priority and app controls indexing order	No control over order, can force indexing priority bump by holding a query open over the entire folder
Behaviour on reset	App required to re-push data	Indexer automatically reindexes all the files

A Note on the Shell Property System and Search

The shell properties system is the list of metadata fields that can be used to describe an item on Windows. The entire list of properties includes hundreds of properties that can be used to describe everything from faxes to video streams. Any of the properties can be used to describe any IShellItem, but in practice only a subset of the most relevant properties are ever used for a given item.

Reading the Documentation for Search

In our case we only need to know how to read a small part of the system in order to use the search APIs. We need to know if the property is going to be indexed, if it will be stored intact, and what data type the property is expecting. Let’s take a look at the documentation for System.Music.AlbumArtist and make sure that we can use it for searching. I’ve copied the relevant section below:

propertyDescription
   name = System.Music.AlbumArtist
   shellPKey = PKEY_Music_AlbumArtist
   formatID = 56A3372E-CE9C-11D2-9F0E-006097C686F6
   propID = 13
   searchInfo
      inInvertedIndex = true
      isColumn = true
      isColumnSparse = true
      columnIndexType = OnDisk
      maxSize = 128
      mnemonics = album artist
   typeInfo
      type = String
      groupingRange = Discrete
      isInnate = false
      canBePurged
      multipleValues = false
      isGroup = false
      aggregationType = Default
      isTreeProperty = false
      isViewable = true
      isQueryable = true
      includeInFullTextQuery = false
      conditionType = String
      defaultOperation = Equal

Walking through the properties we can see what each of them means:

Property	Meaning	Values (For illustration purposes only, not complete)	Notes
inInvertedIndex	Is the property contained in the inverted index and is it searchable	True = Property is searchable False = Property cannot be searched with the indexer	If the property isn’t in the inverted index, then searching against it isn’t going to work. Try to find another property instead.
isColumn	Is the property stored in the index for fast retrieval. Data stored in the inverted index cannot be recovered and returned in search results instead it must be written in a separate location in the database.	True = Property can be retrieved from the indexer as a part of the query results False = The indexer is going to have to try to recover this data from the original item if it is requested as a part of search results	Requesting query results to contain data not held in a column results in a >100x slowdown in query performance
maxSize	The number of bytes reserved in the index for a given property to be stored in		Trying to store anything larger than max size isn’t supported.
isColumnSparse columnIndexType	These are used to describe how the indexer is going to set up the columns on disk		These can be safely ignored assuming that you have isColumn = true.
Type	The datatype of the property. Datatype mismatches are going to result in the indexer ignoring that field	String, FILETIME (DateTimeOffset in C#), UINT32, Custom Enumeration	Make sure you are using the international ready values for shell enumerations using the # sign
isInnate	Indicates if the datatype can be set by the metadata in the file or comes from the system	True = the property value will be derived from file system metadata False = the property can be set by information in the file	Important to note that setting innate properties with the ContentIndexer sometimes will fail silently. Makes for exciting to find bugs, so try to avoid them.

The other properties describe parts of the property system that we aren’t going to need for searching. And in the interest of not turning this into a rant about IShellItems they are ignored in the above chart. This means if we are going to use a property in a query, we’ll need to make sure it is a column and in the inverted index. As well, if you are planning on pushing data you’ll need to ensure that the data being pushed in matches the type the index is expecting.

Indexed Folder

The indexed folder was designed to be as simple to use as possible. The app only has to create a folder named “indexed” in its app data directory StorageFolder localFolder = ApplicationData.Current.LocalFolder; //Don't internationalize the folder name StorageFolder indexedFolder = await localFolder.CreateFolderAsync("indexed"); And that is it. Any file that you create in the indexed folder are going to be automatically indexed. All the fancy AQS queries that work in public libraries are now available to your app’s private data.

Appcontent-ms Files

There are a few rare cases where your app may have data that it wants to search, but doesn’t want to deal with the complexity of the ContentIndexer. For example if you have data that isn’t going to change over time, such as the settings on a PC. In this case the appcontent-ms files are a great way to provide rich local search with very little work in your app The documentation gives a very complete example of how to use these files, so I won’t rehash any of that information here. Instead, I’ll highlight a few important things that might be helpful when using the indexed folder. All the information from my previous post about reading the shell property system documentation holds. There are no restrictions about which properties that you can populate with an appcontent-ms file other than the data type restrictions from the shell schema. The indexer will very happily let you put email addresses in System.Music.AlbumTile, but if you try and store a string in System.Music.IsCompilation it will ignore the value and index the rest of the file. Indexer isn’t instant, there will be some time lag between the when you create the appcontent-ms files and when indexing completes. On an otherwise idle machine the delay should be only a few milliseconds, but that is going to be a lot slower than the very next line of code in your app. Give it some time before you start querying, and if you need to query right away use the IndexerOptions.UseIndexerWhenAvailable option. That way the system will make the smart choice about using the indexer vs. the slower option of just scanning the disk. All subfolders of the indexed folder will be indexed as well. The settings app has a neat setup where they will have a subfolder for each language installed on the machine. Makes their code for adding and removing languages really clean and easy to maintain.

Content Indexer

The ContentIndexer class and its related APIs were added to the system in Windows 8.1 as a new way for apps to push data into the indexer. They were created as a direct replacement for the protocol handler model that was previously used and have been used by a number of apps internally. The biggest customers by far are Edge, pushing in the user’s history, and Groove Music, pushing in the user’s cloud music collection. Both of these cases are highlight what these APIs are great for, data that can be recreated from another source but needs to be easy to search. The indexer is especially powerful in the case of Edge, since URLs often contain bunched together words in a number of languages combined with unusual punctuation. The indexer is able to separate out the words from the other text tokens and make the URLs easier to search.

Using the APIs

I’m not going to rehash the basic samples here, the basic code works great on Windows 10. Instead I’d like to walk through a few important things to note about using ContentIndexer APIs.

Read the shell property system documentation

Make sure you understand the previous section about reading the documentation for which properties can be used. The indexer doesn’t have any way of knowing if you want a property to be searchable (isInvertedIndex), retrievable from a query result (isColumn), or both.

Check your data types

Before storing data make sure that you have the right datatypes and that you aren’t going to be overflowing text fields. In C# it is really easy to end up with a massive string that isn’t going to fit into the particular field you’re using.

Implementing IIndexableContent is a Good Thing

Every successful app I’ve seen use the ContentIndexer has had a domain specific class that implements IIndexableContent. Not only does it make your data model easier but it isolates the nasty pattern of retrieve property  check for null  cast to correct data type  make sure that doesn’t fail  Sanity check resulting value to one location in your code.

Check for indexer resets

The indexer can be reset which will cause all the data in it to be lost. On a Windows upgrade (not just a patch, it has to be a significant diff) the indexer will detect that the shell schema has changed and will rebuild to match the new schema. An app simply needs to check the ContentIndexer.Revision property to see if there has been a rebuild. If there is a rebuild, all the previous data will be gone and the app has to re-push any data they’d like to be searchable.

Unit test storage and retrieval until you’re sick of it

This is for my own wellbeing as well as yours. A couple unit tests that hammer storing, searching, and retrieving a value go a long way to making sure your code works. It’s embarrassing the number of hours I’ve wasted debugging apps only to realize the property being used doesn’t support retrieval. It takes a minute to write the test and will save both of us a lot of hassle. And that’s it. Let me know if you have any more questions about the ContentIndexer, indexed folder or reading the shell properties in the comments below. I look forward to seeing how you use it in your apps.

Accelerating File Operations with the Windows Indexer

adamdwilson — Wed, 11 Nov 2015 10:26:00 GMT

The indexer is in MSDN magazine this November! I cover ways that you can improve the performance of the StorageFile/StorageFolder APIs using the indexer in UWP app as well as cover a few best practices that should be used with every app depending on the indexer.

If you have any questions or topics you'd like a deeper dive on, let me know. This blog doesn't have a word count so I can wax poetically for as long as needed

Change tracking Libraries in Universal Windows Apps

adamdwilson — Fri, 31 Jul 2015 14:26:00 GMT

December 2016: This content is out of date for Windows Anniversary Update and later

I've been working to make change tracking easier, and it shipped in the Windows Anniversary Update. There is an MSDN Magazine article and new blog post with all the details. I'm going to leave this post as a historical reference how how hard it used to be.

Many developers have been looking for a way to bring the change tracking power of Win32 apps to universal apps. Whether it be for uploading user's files to a cloud backup or for preparing the in app experience ahead of time, apps need to know about changes to the users' files.

In Windows 10 it is easier than ever to track changes to a user's files and keep up-to-date even when your app isn't running.

Why Change Tracking?

An app could want to track changes to a user's files for a number of reasons, but the most common ones are:

Being notified when a picture is taken to do some post processing
Being notified when a file is modified to sync the changes to the cloud
Being notified when media is downloaded so it is ready to play in your app right away

All of these scenarios and more can be accomplished in Windows 10 with ease. And we are so confidant of this that we built the OneDrive, Photos, and Xbox Music applications using the very same APIs and techniques that I will outline below.

How we make it easier in Windows 10

There are two new things that will make change tracking easier than ever for you in Windows 10. First of all, the indexer and Windows.Storage.Search are enabled on all device families. This means developers are going to have access to the same powerful searching engine that powers Cortana on everywhere.

Secondly the new StorageLibraryContentChangedTrigger will allow apps to be activated in the background whenever something changes in a library. By leveraging this your app will be notified whenever there is a change in a library, even if it isn't running.

Sample Walkthrough

We are going to walk through a quick example of how an application can change track. In this case we are only going to be looking for files that are new or modified. To do this, there are 3 steps:

Snapshot the current state of the file system
Register the background task to be run when there is a change
Track changes in the file system since the last snapshot

The sample will only be adding the files into an in app database, but your app can do a number of different things with the files that is finds.

This is the same method that is used by OneDrive and the Photos on Windows 10, although I've simplified the code somewhat to factor out their business logic.

Manifest Changes

The sample app is also going to need the picturesLibrary capability added to the manifest so let's do that first. You can either use the designer in VS, or if you love editing XML, add the line:

    <uap:CapabilityName="picturesLibrary"/>

You'll note that the namespace has changed for UWAs, but the capability has the same name as in Windows and Windows Phone 8.1.

First Run Grovel

The very first time your app runs, it is going to have to find the current state of the file system. In a perfect world, the file system would be completely empty the first time your app runs, but the real world is rarely that nice.

What we're going to do is first find all the files in the library that we are interested in. We will also have to track the time of the snapshot since step 3 will rely on this information.

First let's grab the current time, so that we know when the snapshot was taken

lastSearchTime = DateTimeOffset.UtcNow;
//Store it in the application settings so the background task can read it
//We are not checking if there is an existing value here because it is 
//going to trample the DB entirely
var appSettings = ApplicationData.Current.LocalSettings;
appSettings.Values["LastSearchTime"] = lastSearchTime;

This is done first so that if there is a change to the file system while the rest of the code is running, we will pick it up in the change tracking later.

Next, I create a query for all files in the pictures library that my app is able to handle.

StorageFolder photos = KnownFolders.PicturesLibrary;
//supportedExtentions is a list of file extensions my app supports
QueryOptions option = newQueryOptions(CommonFileQuery.OrderByDate, supportedExtentions);
//We are going to use indexer only properties for the query later            
option.IndexerOption = IndexerOption.OnlyUseIndexer;
option.FolderDepth = FolderDepth.Shallow;
StorageFileQueryResult resultSet = photos.CreateFileQueryWithOptions(option);

Now that all the files are included in the query there is an optional step. You could attach an event handler to the ContentChanged event on the resultSet object. This isn't strictly necessary in this case because we've already saved the time, so the future change tracking will include changes that happen while this query is open.

Next I'm going to enumerate the files and add them to an in app database for tracking.

uint currentIndex = 0;
constuint stepSize = 10;
IReadOnlyList<StorageFile> files = await resultSet.GetFilesAsync(currentIndex, stepSize);
currentIndex += stepSize;
for (; files.Count != 0; 
  files = await resultSet.GetFilesAsync(currentIndex,stepSize), currentIndex += stepSize)
{
  foreach (StorageFile file in files)
  {
      addOrUpdateDatabase(file);
  }
}

There is a very important note here: NEVER CALL GetFilesAsync() ON A DIRECTORY YOU DON'T CONTROL!! At this point the app doesn't know if there are 10 or 10 million files in the pictures library. If the user has a lot of files in the directory, creating all those StorageFile objects is going to cause the system to run out of memory.

The full function to copy and paste is here:

lastSearchTime = DateTimeOffset.UtcNow;
//Store it in the application settings so the background task can read it
//We are not checking if there is an existing value here because it is 
//going to trample the DB entirely
var appSettings = ApplicationData.Current.LocalSettings;
appSettings.Values["LastSearchTime"] = lastSearchTime;
StorageFolder photos = KnownFolders.PicturesLibrary;
//supportedExtentions is a list of file extensions my app supports
QueryOptions option = newQueryOptions(CommonFileQuery.OrderByDate, supportedExtentions);
//We are going to use indexer only properties for the query later            
option.IndexerOption = IndexerOption.OnlyUseIndexer;
option.FolderDepth = FolderDepth.Shallow;
StorageFileQueryResult resultSet = photos.CreateFileQueryWithOptions(option);

uint currentIndex = 0;
constuint stepSize = 10;
IReadOnlyList<StorageFile> files = await resultSet.GetFilesAsync(currentIndex, stepSize);
currentIndex += stepSize;
for (; files.Count != 0; 
  files = await resultSet.GetFilesAsync(currentIndex,stepSize), currentIndex += stepSize)
{
  foreach (StorageFile file in files)
  {
      addOrUpdateDatabase(file);
  }
}

Now we have all the files currently on the disk tracked in the app database, let's set up the app to be trigged when something changes.

Registering for StorageLibraryChangedTrigger

By registering for the StorageLibraryChangedTrigger, your app will be activated in the background whenever something changes in the library you are interested in. The registration is the same as for other background tasks, so we are going to walk through it really quickly.

If you want more details on how background tasks are registered, you can see overview or the quickstart article.

Manifest Registration

Your app will need to register for the general background task type using either the designer or entering the raw XML.

<Extensions>              
  <ExtensionCategory="windows.backgroundTasks"
  EntryPoint="StorageLibraryChangedBackgroundTask.StartupTasks">
      <BackgroundTasks>
        <TaskType="general"/>
      BackgroundTasks>
    Extension>
Extensions>

The entry point is going to change depending on the name of your background task.

Registering the Task in Code

Again the code registration is very similar to the registrations for the other task types, so we are going to focus on the unique parts.

First you are going to want to make sure that your task hasn't been already registered

foreach (var cur inBackgroundTaskRegistration.AllTasks)
{
  if (cur.Value.Name == backgroundTaskName)
  {
    //already registered the task
    return;
  }                
}

Next your application is going to have to get access to the libraries that it wants to change track and create the appropriate trigger.

var library = awaitStorageLibrary.GetLibraryAsync(KnownLibraryId.Pictures);
var trigger = StorageLibraryContentChangedTrigger.Create(library);

Remember to make sure that you have added the correct library capabilities in the manifest before requesting access.

Note, change tracking is only available in the libraries, but your application can track as many libraries as it would like. In this case I'm only going to track the pictures library because my application is only able to handle images. This includes pictures that are coming from the camera on devices that have one.

Finally I'll request access for the background task run and register it.

var access = awaitBackgroundExecutionManager.RequestAccessAsync();
if (access == BackgroundAccessStatus.Denied)
{
  //If you hit this, make sure the manifest registration is correct
 return;
}
string taskEntryPoint = typeof(StorageLibraryChangedBackgroundTask.StartupTasks).ToString();
var builder = newBackgroundTaskBuilder();
builder.Name = backgroundTaskName;
builder.TaskEntryPoint = taskEntryPoint;
builder.SetTrigger(trigger);
BackgroundTaskRegistration task = builder.Register();

And that is it. The background task will now be activated anytime that something changes in the pictures library.

Tracking Incremental Changes

Now that your app is going to be notified when something is changed we need to be able to find out what file changed. In this case we are only going to track additions and modifications, while ignoring deletions.

The tracking is done from a background task so that it can be run when the app isn't in the foreground. The background tasks can be triggered when the app is in the foreground though, so be aware of potential concurrency issues when you are writing your own app.

The first thing that we are going to do when the app is triggered is to hold the deferral (since there will be async operations run in the task) and retrieve the last search time from the application settings.

var appSettings = ApplicationData.Current.LocalSettings;
if (!appSettings.Values.ContainsKey("LastSearchTime"))
{
 //We haven't done a first run crawl, so lets just bail out
  return;
}
DateTimeOffset lastSearchTime = (DateTimeOffset)appSettings.Values["LastSearchTime"];

In this case, I'm electing not to do a first crawl in the background since it is going to be too resource intensive for a background task.

Next we are going to create the query that will find the items which have been changed since the last time the app was able to crawl. To do the filtering we will leverage the System.Search.GatherTime property from the Windows Property system. This property is an indexer only property that tracks the last time that the file was modified, created, or moved.

To use this property there are two important preconditions. First of all, the query must be set to use the indexer with the IndexerOption.OnlyUseIndexer property. Without the indexer this query will not work because System.Search.GatherTime isn't a property of the file, it is a calculated property stored by the indexer.

The other important note is that dates in the indexer are stored using Zulu time format. Thus, the ApplicationSearchFilter will have to be created with the lastSearchTimeConverted to Zulu time.

StorageFolder photos = KnownFolders.PicturesLibrary; 
QueryOptions option = newQueryOptions(CommonFileQuery.OrderByDate, supportedExtentions);
//This is important because we are going to use indexer only properties for the query
option.IndexerOption = IndexerOption.OnlyUseIndexer;
option.FolderDepth = FolderDepth.Shallow;
//Files that have been modified or created since the last search time. The date time 
//format being used is Zulu
string timeFilter = "System.Search.GatherTime:>=" + 
    lastSearchTime.ToString("yyyy\\-MM\\-dd\\THH\\:mm\\:ss\\Z");           

option.ApplicationSearchFilter += timeFilter;
StorageFileQueryResult resultSet = photos.CreateFileQueryWithOptions(option);
lastSearchTime = DateTimeOffset.UtcNow;

Once we have the query in hand, we can reuse the same code from the first run grovel to check for new or modified files.

We are ignoring the case where files may be deleted and not show up in the query, since the app is only interested in new files. If your app is interested in files that have been deleted, it will have to compare a full query of the location with the results in its database.

uint currentIndex = 0;
constuint stepSize = 10;
IReadOnlyList<StorageFile> files = await resultSet.GetFilesAsync(currentIndex, stepSize);
currentIndex += stepSize;


for (; files.Count != 0; files = await resultSet.GetFilesAsync(currentIndex, stepSize),
                            currentIndex += stepSize)
{
  foreach (StorageFile file in files)
  {
    //Note we are assuming that the database will handle any concurrency issues
      addOrUpdateDatabase(file);
  }
}

And on the way out, release the deferral and store the last time that we completed a successful search for the next time that the background task is triggered.

appSettings.Values["LastSearchTime"] = DateTimeOffset.UtcNow;
deferal.Complete();

The full function is below for those who are looking for something to copy and paste:

public async void Run(IBackgroundTaskInstance taskInstance)
{
  BackgroundTaskDeferral deferal = taskInstance.GetDeferral();

  var appSettings = ApplicationData.Current.LocalSettings;
  if (!appSettings.Values.ContainsKey("LastSearchTime"))
  {
    //We haven't done a first run crawl, so lets just bail out
    return;
  }
  DateTimeOffset lastSearchTime = (DateTimeOffset)appSettings.Values["LastSearchTime"];


  StorageFolder photos = KnownFolders.PicturesLibrary;
  QueryOptions option = newQueryOptions(CommonFileQuery.OrderByDate, 
                                          supportedExtentions);
  //This is important because we are going to use indexer only properties for the query
  //later
  option.IndexerOption = IndexerOption.OnlyUseIndexer;
  option.FolderDepth = FolderDepth.Shallow;
  //Set the filter to things that have changed since the lastSearchTime. Note that the 
  //time must be formatted as Zulu time as shown here.
  string timeFilter = "System.Search.GatherTime:>=" + 
                    lastSearchTime.ToString("yyyy\\-MM\\-dd\\THH\\:mm\\:ss\\Z");
  option.ApplicationSearchFilter += timeFilter;


  StorageFileQueryResult resultSet = photos.CreateFileQueryWithOptions(option);
  lastSearchTime = DateTimeOffset.UtcNow;
            

  uint currentIndex = 0;
  const uint stepSize = 10;
  IReadOnlyList<StorageFile> files = await resultSet.GetFilesAsync(currentIndex,
      stepSize),
  currentIndex += stepSize;
  
  for (; files.Count != 0; files = 
      await resultSet.GetFilesAsync(currentIndex, stepSize),
      currentIndex += stepSize)
  {
    foreach (StorageFile file in files)
    {
    //Note we are assuming that the database will handle any concurrency issues
      addOrUpdateDatabase(file);
    }
  }
  //Update the last time that we searched for files
  //Edit 2/9/16 fixed a race condition, see comments for details
  appSettings.Values["LastSearchTime"] = lastSearchTime;
  deferal.Complete();
}

Conclusions

Using the methods above an app is able to leverage the same change tracking technology that is used by first party experiences. It allows your app to be up to date with the latest changes on the device so your users can jump right into being delighted as soon as your app is opened.

Edit: (10/26)

We have discovered an issue where sometimes app registrations for the notification are getting lost if the app or machine were not shut down properly. As a workaround, please check the registration for the background task every time your app starts up and re-register as needed.

Using Neighbouring File Queries to Power your App

adamdwilson — Tue, 28 Jul 2015 16:54:00 GMT

Neighbouring files queries (NFQ) were originally introduced in Windows 8 and we have extended them for the Universal Windows Platform. NFQs allow for users to maintain context on their files as they move between apps by allow apps to pass a list of related files along with a normal file activation.

What is an Neighbouring file query?

It is a group of files that are included with a file activation to allow the target app to keep the user's context. The most common example of this in Windows is when you open a picture from a folder in file explorer. File explorer creates an NFQ which it includes with the file activation. The photos app is then able to provide an experience for users to move through the rest of the files in the same folder, without being explicitly activated for those files.

Who is expected to set NFQs?

Any apps using a file activation in their experiences and wish to allow their users to potentially see other, related files in the target app as well. For example if the user has received a number of images as a message attachment, they expect to be able to scroll through their images in a photo viewer without having to go back to the messaging app each time.

An NFQ should not be used for activating apps when the user is expecting the target app to use all the files in the query immediately, such as a multi-select scenario in files explorer. It should only be used as an optional hint to the target app.

Who should respond to NFQs?

Any app that registers for file activations and can do something interesting if there are files related to the one they are opening. Video and picture viewing apps are a good example of an app that can do something interesting with a series of files. A document editing app might not respond to an NFQ, as they don't have a scenario where a user is scrolling through multiple documents.

Example: Setting an NFQ in a Source Apps

We are going to walk through a quick example of how to create an NFQ for a source app that has a picture album stored in its local folder. In this case the source app wants to launch the photo viewer to view a single image, but enable the user to be able to scroll through all the images in the same folder.

Creating an NFQ starts with first finding the file that you are going to launch. In this case we are going to use a file that is stored in our app's local folder, but it is possible to do this with any file your app has access to.

StorageFolder sourceFolder = ApplicationData.Current.LocalFolder;
StorageFile fileToLaunch = await sourceFolder.GetFileAsync("launchingFile.jpg");

Next we need to find any files that are related to the file we are launching. To do this we build up a query that includes all the pictures in the same folder.


QueryOptions queryOptions = newQueryOptions();
//Note that you can have a more restrictive condition here (ext:.jpg OR ext:.jpeg) 
//but I've chosen to error on the side of giving the target app more files to work 
//with and if it can only handle jpegs, it needs to do its own filtering
queryOptions.ApplicationSearchFilter = "kind:picture";  queryOptions.FolderDepth = FolderDepth.Shallow;
//Important since we are in the app data container where there is no indexing
IndexerOption.DoNotUseIndexer;
StorageFileQueryResult nfq = SourceFolder.CreateFileQueryWithOptions(queryOptions);

You can get as creative here as you'd like with the ApplicationSearchFilter and other options, but I'm keeping the result set very broad. This will enable the target app to provide a richer experience or do further filtering themselves.

The final step is to attach the NFQ to the launcher options and launch the target app.

LauncherOptions launcherOptions = new LauncherOptions();
launcherOptions.NeighboringFilesQuery = nfq;
Launcher.LaunchFileAsync(fileToLaunch, launcherOptions);

And that is it. The default handler will be launched along to handle the file you've chosen along with another other files in the NFQ. For example you can see in the following image how I've been able to launch the photos app and have it displaying the arrow hints on the side which indicate the user can scroll to view more images in the collection

Example: Consuming an NFQ in a Target App

Consuming an NFQ is a very simple process, and can allow you to build more useful experiences for your users. There are two parts to consuming an NFQ, first the manifest must include the correct registration information and then a bit of code in the target app the handle the activation.

We will briefly touch on file activations, but for more detail see the article How to handle file activation.

Manifest Registration

Your app is going to have to register for the file types that it can be activated for and the more general kind of files it would be able to handle in an NFQ.

Files Type Registration

This is where your app can register for the types that is going to be activated to handle. There is an edit experience in Visual Studio and more details available but for now we are going to manually edit the package.appxmanifest to add a registration for .jpeg and .jpg.

        <uap:ExtensionCategory="windows.fileTypeAssociation">  
          <uap:FileTypeAssociationName="image"DesiredView="useMore">  
            <uap:EditFlagsOpenIsSafe="true"AlwaysUnsafe="false"/>  
            <uap:SupportedFileTypes>  
              <uap:FileTypeContentType="image/jpeg">.jpeguap:FileType>  
              <uap:FileTypeContentType="image/jpeg">.jpguap:FileType>  
            uap:SupportedFileTypes>  
          uap:FileTypeAssociation>  
        uap:Extension>

Note that the uap namespace is going to be new for Windows 10, but otherwise the registration is unchanged from Windows and Windows Phone 8.1 universal apps.

File Kind Registration

Your app also has to register for the kind of files that it is interested in being able to view in an NFQ. It does this by registering for access to the library of the kind of files it would like to see.

For example, my app is going to be able to view picture files, so it will register for the pictures library capability.

    <uap:CapabilityName="picturesLibrary"/>

Apps that can handle video or music types would register for the appropriate libraries. The mapping between capabilities and files extensions is outlined in more detail here.

The NFQ that is received by your app will be filtered down to only the kind of files it is registered for. This it to make sure that apps are not going to be stuck dealing with massive queries filled with files they can't use.

Note that if you don't have any library capabilities and try to access a NFQ you will get an AccessDenied exception.

Opening the NFQ in Code

To get access to the NFQ in your app you are first going to have to implement the OnFileActivated method in your App class.

  protectedoverridevoid OnFileActivated(FileActivatedEventArgs e)

Then to get the NFQ you simply have to read the NeighboringFilesQuery from the FileActivatedEventArgs as shown below.

  StorageFileQueryResult query = e.NeighboringFilesQuery;

And that is it. Your app now has access to all the files in the query and can use them in the experience to help your customers complete whatever action they are doing easily.