Comment by compsciphd | Hacker News Reader

Comment by compsciphd | Hacker News Reader

even with file system snapshots, your backup can be corrupt.

the example of the browser profile is a classic case.

imagine multiple writes occur and all of them have to occur for the profile to be correct (either multiple files are being written or firefox will write multiple blocks to the file). if the snapshot operation occurs in the middle, then the snapshot will be "corrupt".

I believe this is actually the entire point of Windows' volume shadow service (which is sort of poo-pooed in the article), to enable applications to tell the snapshot mechanism "wait, I'm in the middle of a file system transaction" and then to pause writes until the snapshot operation occurs after they finish the in process transaction.

without such a mechanism, you are always going to be at risk with snapshots.

In https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf we avoided this problem by combining 2 mechanisms without having modifying applications with such a service

1) we used a log structured file system (that was inherently a snapshot, every log entry was individually mountable) and 2) we used a checkpoint/restart mechanism that saved process state and enabled us to restart the processes combined with the file system state as it was at checkpoint time. (Checkpoint would also sync all dirty pages to disk and that fs state after the sync is what we tied to the checkpoint state).

So when a process would be resumed, the file system would look exactly as the process expected it, even if the process was in the middle of what can be referred to a transaction. But that only worked because the processes were restored along with file system, if we only restored the file system, it could be inconsistent.

> I believe this is actually the entire point of Windows' volume shadow service (which is sort of poo-pooed in the article), to enable applications to tell the snapshot mechanism "wait, I'm in the middle of a file system transaction" and then to pause writes until the snapshot operation occurs after they finish the in process transaction.

Former maintainer of VSS here. Yes, that's exactly right. In fact, filesystem snapshots are usually not enough for true application level consistency. As others have noted elsewhere, a filesystem snapshot is the equivalent to yanking the power cord out of the back of your computer. It's good, assuming your filesystem does atomic writes / copy-on-write / write-to-new. But we can do even better.

Imagine I have two databases - one traditional relational DB storing my app content, and a second log database. You want to keep these in sync. Well, good luck doing this with the filesystem alone. Usually your DBMS will need to be involved as well, and this is where the VSS "writer" concept comes in. When a snapshot is being taken, applications such as SQL will be invited to participate. Typically, this means they'll start to hold up writes so that things will be quieter for the snapshot. But they will also have a chance, after the fact, and to actually clean up the snapshot itself. In this case, the DBMS could roll back the log database to then match the content database.

It's correct that NTFS doesn't support snapshots natively, but Windows has the volsnap.sys driver that takes care of it. For all intents and purposes, NTFS does support copy-on-write snapshots.

Complicated? Sure. But it was quite capable, and actually a pretty cool (but sadly under appreciated) piece of technology.

Thanks for writing this! Am I correct in understanding that VSS requires writer support and that support is not widespread in desktop applications, eg web browsers?

This article also fails to consider that on modern systems writes are often batched and not immediate. The only way to get an actually consistent and safe backup of any file / file system, is to actually shut down the machine so that all applications, and the OS, have shut down gracefully.

As an intermediate step, it may be practical to consider shutting down your application, flush all writes (e.g. via sync command), and taking the file system snapshot then, restarting the application, and then taking the backup from that file system snapshot. At least then your critical data should have a consistent and safe backup, even if the rest of the OS _may_ be in a suspect state.

The worst part is that the article itself says this. This is exactly like "pulling the power on the running system" and then imaging the hard disk. Even if it happens to work, that is not really a good idea.

Reply on news.ycombinator.com