Got conflicts or missing files? Read this (and don't uninstall for now)

A few users have reported either files missing when they view Live Mesh folders on their computer (but appearing as expected on their Live Desktop) or their computer showing all files but reporting many more conflicts than expected (and their Live Desktop showing duplicate copies of any file with a conflict.) We’ve spent the weekend investigating this, and we have a few details below. But the key news is this:

• Live Mesh is not deleting or losing files. Even if they don’t all appear in Live Mesh folders on your computer, they are all present in a hidden state, as well as on your Live Desktop if the folder is synchronized there.
• Uninstalling the Live Mesh software from a PC will delete the files that are in a hidden state on that PC. If the folder is not synchronized on any other computer or your Live Desktop, then uninstalling will result in the hidden files no longer being available.
• If you stopped synchronizing a folder on your PC, then re-synchronized it, that folder is likely to generate some number of false conflicts.

We're testing a fix for this now and will deploy it as soon as we're confident it’s ready. In the meantime, if you have a Live Mesh folder that appears to be missing files (that is, synchronization appears to be complete but expected files are still not present) or one that has an unexpectedly large number of conflicts, in order to prevent data loss, do not uninstall the Live Mesh software. In addition, do not re-synchronize any folders you had previously stopped synchronizing.

 

Why is this happening? A technical explanation

 

In the interest of being open with you, our beloved and valued technology preview testers, we’re happy to provide more detail on what’s causing this behavior. The key to a good synchronization algorithm is understanding the history of all the files in a folder. You want to make sure that any client, even one that hasn’t synchronized in a good long while, can get an accurate list of what changed and what needs to be brought up to date. Part of a complete folder history is keeping track of files or folders that have been deleted, so that if a client comes online and tries to sync a file that doesn’t exist in the up-to-date folder list, we can tell whether it’s a new file or a file someone deleted a while back but is still on this particular client because the file deletion was never synchronized to it. We use the industry jargon tombstone for these deleted files.

 

What we discovered recently is an interesting edge case around tombstones. Live Mesh performs its synchronization in batches of roughly 50 files at a time. The client asks for the first 50 changes, processes those, then asks for the next 50, and so on. If, through sheer chance, the next batch of 50 items is made up entirely of tombstones—that is, it’s a list of 50 files that have all been deleted since this particular client last synchronized—the storage server gets a little confused. It returns that list of tombstones and then concludes that synchronization has ended. If there are additional file changes, they never get sent to the client. So this can lead to two patterns:

• If you’re synchronizing a folder for the first time, and that folder has this pattern of 50 tombstones in a row, then you get only a partial sync. Any files the server returned before this block of 50 tombstones show up just fine, but any files after that won’t be synchronized to the client.
• If you have first stopped and then restarted synchronization for a particular folder, it gets a little more complicated. In this scenario, when synchronization resumes, the client ought to just quickly see which of the files have changed and get everything up to date. But if a client hits this 50-tombstone-in-a-row bug, the Live Mesh service ends up confused. The client might have tens or even hundreds of files that you and I know are up to date and part of the folder, but the server failed to tell the client about them. So the client has no choice but to conclude that you have a bunch of new files, and that’s what it tells the server. The server, of course, recognizes that files with those names already exist, and so it mistakenly thinks the client has just created a bunch of new files that unfortunately have the exact same names as files already in the folder. Result: sync conflicts, and potentially lots of them.

Now, conflicts are an inevitable part of any synchronization system, and so both the server and client are built to understand what a conflict is and to store any conflicting files in a separateholding area, where they remain until the user decides how to resolve the conflicts. While they are in the holding area, the client software might not display the files as being part of the folder, while the Live Desktop instead represents the conflict by showing both files (yes, this is something we are already working on improving and making consistent ;). So the files are all in your mesh; it’s just not obvious where they are or how to access them.

 

The one catch is that the client holding area is stored in the Live Mesh folder for temporary application files. This directory is something we remove when the Live Mesh software is uninstalled (also something we are already working on improving—either warning you that uninstalling will remove any files in a conflicted state or just copying all conflicts somewhere else where they won’t get removed on uninstall). So if you somehow end up in a state where the folder is present only on one client, uninstalling the Live Mesh software from that client may remove all files in a conflicted state. That’s bad, we know it’s bad, and that’s why we suggest you don’t uninstall until we fix this issue.

 

Thanks as always for your continued participation and feedback. Ferreting out issues like this is exactly what our technical preview is designed to do, and so while it’s sometimes a bit painful, we appreciate your help and your patience as we work to get the issue fixed.

 

Technorati Tags: Live Mesh

Comments

• Anonymous
  September 08, 2008
  Nice to hear about progress, gl with the work!

• Anonymous
  September 09, 2008
  Thank you for being so forthright in describing and addressing this issue.  I'm glad these issues are getting sorted out now, rather than after launch (cough, Apple).

• Anonymous
  September 09, 2008
  If you decide to "just copying all conflicts somewhere else where they won’t get removed on uninstall" i think we need to be aware of this so we can remove them if we please. The last thing we need is more files taking up space on our HDD that we do not know about. At least give us a choice to delete or keep them! Thanks! and great product by the way!

• Anonymous
  September 09, 2008
  If you read my blog much, you'll know I can't run Live Mesh at the office, so what's the

• Anonymous
  September 09, 2008
  I have the latest version of the Live Mesh client.  For some reason, since the last update, Live Mesh has been repeatedly crashing.  It crashes, I get an error message, it restarts, and crashes again.  Plus, documents and files are disappearing.  I've checked the folders for hidden files, but nothing is there.  Is there any way for you to help me?

• Anonymous
  September 09, 2008
  @ Master Devwi, There a couple of ways to get help.  You can open a bug through our Microsoft Connect site (https://connect.microsoft.com/LiveMesh), or you can email your Live Mesh logs to lmprev@microsoft.com and describe the problem you're having. Also, while they are currently offline (for just a couple more hours), our forums are a great place to get support for Live Mesh: http://forums.community.microsoft.com/en/LiveMesh/threads/

• Anonymous
  September 09, 2008
  @jsbenjamin Thanks.  I forgot about Microsoft Connect.  I'll go ask the question over there.  Thanks for the help!

• Anonymous
  September 10, 2008
  I'm on 0.9.3103.14 and just noticed that moe.exe is consuming 19,500 handles!  That seems a bit excessive.

• Anonymous
  September 11, 2008
  I would love to see some options to throttle the cpu and bandwidth LiveMesh uses.  Moe frequently dominates my processes, and syncing eats up all my upload bandwidth.

• Anonymous
  September 14, 2008
  This tombstone business is a bummer. I've added a synced folder for the first time to btw my father and I's computer (1.65gb). It stopped 15% of the way and thats it. A really good feature would be to see what files are even missing. I have 3.65gb worth of school data on your servers, not sure if its all of the 3.65gb. That might have stopped half way. As a backup utility or everyday reliance there needs to be a way to be sure of the integrity of the data.  I'm so excited for this project.

• Anonymous
  September 16, 2008
  This bug has been doing its damage on my friends computer now for a week. He has officially 11GB of missing data because this bug just keeps on trying to sync and dropping everything from the prior sync. He had all 59GB of data sharing with foldershare flawlessly prior to this.. boy do i feel dumb for trying to help him out. Now for a week all i can say is "Uh yeah they're gonna fix it soon, and you might get your 11GB back" This is a joke

• Anonymous
  September 17, 2008
  The comment has been removed

• Anonymous
  September 17, 2008
  Last week, we posted about an issue that might cause your Live Mesh Folders to either appear to have

• Anonymous
  September 17, 2008
  Were you wondering, just like us, why it got so quiet after the last warning about unmap/remap/uninstalling