Strange behavior after "rm -rf //"

Strange behavior after "rm -rf //"

[Ivan Sizov]

I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after five
seconds, but some files had been deleted. I'd tried to shutdown the
system, but couldn't (a lot of files in /bin had been deleted and
systemd didn't work). After hard reboot (by reset button) and booting
to a live USB a strange thing was discovered.

Deleted files are present when I "mount -r" the disk, but
btrfs-restore tells they are deleted ("We have looped trying to
restore files too many times to be making progress").

What does it mean? Will those files be deleted after RW mount?

-- 
Ivan Sizov (SIvan)

Re: Strange behavior after "rm -rf //"

[Chris Murphy]

On Mon, Aug 8, 2016 at 10:30 AM, Ivan Sizov <sivan606@gmail.com> wrote:
> I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after five
> seconds, but some files had been deleted. I'd tried to shutdown the
> system, but couldn't (a lot of files in /bin had been deleted and
> systemd didn't work). After hard reboot (by reset button) and booting
> to a live USB a strange thing was discovered.
>
> Deleted files are present when I "mount -r" the disk, but
> btrfs-restore tells they are deleted ("We have looped trying to
> restore files too many times to be making progress").
>
> What does it mean? Will those files be deleted after RW mount?

Just a wild guess, the deletions may be in the tree log and haven't
been applied to the other trees (fs tree, extent tree, etc). So yes
I'd expect they get deleted on a rw mount.

This is what kernel? Because kernel 4.6 offers mount option
"nologreplay" which suggests even if you do mount -r that log replay
happens, so you shouldn't see these deleted files unless you mount ro
*and* use nologreplay mount option.

Anyway, even 5 seconds of rm -rf damages too much. If you don't have
recent snapshots then it's not sanely salvageable, just reinstall.


-- 
Chris Murphy

Re: Strange behavior after "rm -rf //"

[Ivan Sizov]

2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>:
> Just a wild guess, the deletions may be in the tree log and haven't
> been applied to the other trees (fs tree, extent tree, etc). So yes
> I'd expect they get deleted on a rw mount.
>
> This is what kernel? Because kernel 4.6 offers mount option
> "nologreplay" which suggests even if you do mount -r that log replay
> happens, so you shouldn't see these deleted files unless you mount ro
> *and* use nologreplay mount option.

Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue
zero-log" and then mount RW? Will the files safe in that case?

> Anyway, even 5 seconds of rm -rf damages too much. If you don't have
> recent snapshots then it's not sanely salvageable, just reinstall.

As I could see, almost all the "deleted" files are present. Certainly,
I'll make an rsync diff between two-week-ago snapshot and the current
FS state. But it will better if in-place recover without backup is
possible.

P.S. IMHO, log replay by default is a quite dangerous thing. I didn't
know about that change and I could lose all files if the live USB had
4.6 kernel))

-- 
Ivan Sizov (SIvan)

Re: Strange behavior after "rm -rf //"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1951 bytes --]

On Mon, Aug 08, 2016 at 09:38:28PM +0300, Ivan Sizov wrote:
> 2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>:
> > Just a wild guess, the deletions may be in the tree log and haven't
> > been applied to the other trees (fs tree, extent tree, etc). So yes
> > I'd expect they get deleted on a rw mount.
> >
> > This is what kernel? Because kernel 4.6 offers mount option
> > "nologreplay" which suggests even if you do mount -r that log replay
> > happens, so you shouldn't see these deleted files unless you mount ro
> > *and* use nologreplay mount option.
> 
> Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue
> zero-log" and then mount RW? Will the files safe in that case?
> 
> > Anyway, even 5 seconds of rm -rf damages too much. If you don't have
> > recent snapshots then it's not sanely salvageable, just reinstall.
> 
> As I could see, almost all the "deleted" files are present. Certainly,
> I'll make an rsync diff between two-week-ago snapshot and the current
> FS state. But it will better if in-place recover without backup is
> possible.
> 
> P.S. IMHO, log replay by default is a quite dangerous thing. I didn't
> know about that change and I could lose all files if the live USB had
> 4.6 kernel))

   Log reply on mount has _always_ been the default, and should remain
so. It gives you the expected semantics after a power loss: all th
efiles that you'd written up to the point of the power loss actually
appear afterwards. (If this didn't happen, you could lose up to 30s of
writes from before the crash).

   It's only very recently that there's been an option to prevent it,
which is useful in a limited number of cases (such as trying to
undelete a file, which is not really a supported operation in any
case).

   Hugo.

-- 
Hugo Mills             | "I lost my leg in 1942. Some bastard stole it in a
hugo@... carfax.org.uk | pub in Pimlico."
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Strange behavior after "rm -rf //"

[Ivan Sizov]

2016-08-08 21:52 GMT+03:00 Hugo Mills <hugo@carfax.org.uk>:
> On Mon, Aug 08, 2016 at 09:38:28PM +0300, Ivan Sizov wrote:
>> P.S. IMHO, log replay by default is a quite dangerous thing. I didn't
>> know about that change and I could lose all files if the live USB had
>> 4.6 kernel))
>
>    Log reply on mount has _always_ been the default, and should remain
> so. It gives you the expected semantics after a power loss: all th
> efiles that you'd written up to the point of the power loss actually
> appear afterwards. (If this didn't happen, you could lose up to 30s of
> writes from before the crash).
>
>    It's only very recently that there's been an option to prevent it,
> which is useful in a limited number of cases (such as trying to
> undelete a file, which is not really a supported operation in any
> case).

I mean only RO mount, of course.

So, will zero-log prevent my files during RW mount?


-- 
Ivan Sizov (SIvan)

Re: Strange behavior after "rm -rf //"

[Duncan]

Ivan Sizov posted on Mon, 08 Aug 2016 19:30:16 +0300 as excerpted:

> I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after five
> seconds, but some files had been deleted. I'd tried to shutdown the
> system, but couldn't (a lot of files in /bin had been deleted and
> systemd didn't work). After hard reboot (by reset button) and booting to
> a live USB a strange thing was discovered.
> 
> Deleted files are present when I "mount -r" the disk, but btrfs-restore
> tells they are deleted ("We have looped trying to restore files too many
> times to be making progress").
> 
> What does it mean? Will those files be deleted after RW mount?

Chris is likely correct in your case, but I'd like to point out three 
things.

1)  The looping ... warning in btrfs restore is obviously there for a 
reason, because under some circumstances the filesystem will be damaged 
in such a way that restore /can/ loop without making progress, but that's 
not always the case, and in fact, in my own experience, has /never/ been 
the case.

Far more common, at least from my own experience, is seeing that warning 
simply due to directories containing a large number of files, even when 
restore /is/ working properly and restoring the files.  I don't know 
where the cutover is, but there's a reason it's a warning that allows you 
to say continue, and in every single case from my own experience, 
continuing /enough/ times eventually resulted in a successful restore 
with no missing files that I could tell (tho I didn't do a before/after 
comparison, just never missed anything but symlinks, etc, before the 
option to restore them too was added).

So if you haven't tried it yet, tell restore to continue despite the 
warning and see if it eventually does make progress.

Some people even automate the process using yes | btrfs restore ... or 
similar, tho I've never needed that here, possibly because I use multiple 
relatively small partitions (all under 50 GiB each except for my media 
partition and its backup).  I guess if they do decide btrfs restore is in 
an infinite loop, say after hours with no increase in the total size of 
the files restored, they'd have to break out of the loop manually, tho 
I've seen several posts where people were asking for restore to have a 
built-in continue option, or where they used automation, and none where 
they had to break the loop manually, so I'd guess it's actually pretty 
rare that a real infinite loop actually happens.

And because btrfs is copy-on-write and the old roots stay around for 
awhile, provided you take pains not to mount the filesystem writable or 
if you do not to write too much to it, since the more you write the less 
likely you are to be able to fully recover older transactions, you can 
likely use restore manually with the -t <transid> option and btrfs-find-
root to find an appropriate transid, to get the files back even if they 
do otherwise appear to be deleted.

See the wiki for instructions on that.  If you have a new enough btrfs-
progs, the page should be referenced in the btrfs-restore manpage.  But 
here it is anyway, since I have the manpage open ATM:

https://btrfs.wiki.kernel.org/index.php/Restore

2) Primarily because you didn't mention it and it can be handy in other 
circumstance, if you're unaware of it, read up on magic sysrequest, aka 
sysrq aka srq.

$KERNDIR/Documentation/sysrq.txt ... and various googlable articles on 
the subject.

Basically, any time you'd otherwise resort to a hard reboot, try a magic-
srq sequence first.  Longer version: reisub.  Shorter version, just the 
sub.  That's emergency Sync, remoUnt-read-only, reBoot (thus s-u-b).

It won't always work, particularly for kernel crashes, but even if it 
doesn't you can get a feel for how bad the crash was by the response or 
lack thereof (if the s and u light up the storage device activity LED, 
the kernel was alive and considered it safe to still write to storage, if 
they don't show activity but the b still reboots, the kernel was alive 
but either nothing dirty to write or the kernel considered itself damaged 
and thus wasn't going to risk writing to storage, if none work, the 
kernel itself was dead).

Because your problem this time was userspace, simply no binaries to run, 
that should have worked, safely shutting down the filesystem.

Altho arguably in this case a hard reboot was the better choice, since 
that final commit might have been lower risk for the filesystem, but 
would have likely finalized those deletions that you can now recover.  
(Tho with btrfs being copy-on-write, there's a fair chance you'd have 
been able to restore the files anyway, if done right away, using restore 
and manually pointing it at an earlier root.)

So you arguably did the right thing with a hard reboot here anyway, but 
in other cases, magic-srq is incredibly useful to know and may just save 
your butt, as I believe it has mine a few times by now.

3) I did something similar a couple years ago.  In my case, I was 
(unwisely) testing a script as root, with a typo in a variable name so it 
was an empty variable and thus started from / instead of the intended 
path.

Fortunately, I have backups, tho I don't keep them as current as I might, 
and it took out /bin and /boot and then warned me about /dev, which it 
couldn't delete due to that being the devfs mountpoint.  It proceeded 
into /etc, but that's where I stopped it after the warning about /dev, so 
I still had /usr/bin and the libs as well as /home, and could rebuild 
/bin and /etc from backups.

But the point it drove home to me is one I had heard before and 
fortunately was living by, that an admin has as much to fear from fat-
fingering something as he does from device, filesystem or software update 
failure.  And of course I shouldn't have been testing that script as 
root, and anything that scripts rm -r /$variable/* deletions like that 
needs at minimum an empty-var test that only proceeds with the rm if the 
variable isn't empty/null.

But the primary point is that if it's not backed up, by the inaction of 
failing to do that backup, you are in a very real and non-negotiable 
after-the-fact way, defining that data as worth less than the time and 
resources required to do the backup.

Fortunately I did have a (tested, if it's not tested it's not yet a 
backup!) backup, tho I don't always keep my backups current.  But at 
least I know the risk is limited to the updates between that backup and 
the current time, and I recognize that by not doing more regular backups, 
I am in a very real way defining that data in the gap as of only trivial 
value, to the point that I recognize the risk and when I start getting 
uncomfortable with the size of the data in that difference gap, I know 
it's time to do another backup.


And by that definition, it's impossible to lose data more valuable than 
the cost of an additional level of backup that would have kept it safe, 
whether that's no backup for data of trivial value, only a single on-site 
backup for data worth a bit more, or a hundred (or a thousand) levels of 
backup at 50 sites in 20 countries on 5 continents, because the data 
really is /that/ valuable.

So if you /think/ you value the data, have the backups demonstrating that 
value, because if you don't, you have a very real possibility of 
demonstrating that you did /not/ value the data as much as you claimed 
to, because it wasn't backed up and that lack of backup demonstrated the 
lie in any claim to the contrary.  IOW, backups speak louder than words!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Strange behavior after "rm -rf //"

[Chris Murphy]

On Mon, Aug 8, 2016 at 12:38 PM, Ivan Sizov <sivan606@gmail.com> wrote:
> 2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>:
>> Just a wild guess, the deletions may be in the tree log and haven't
>> been applied to the other trees (fs tree, extent tree, etc). So yes
>> I'd expect they get deleted on a rw mount.
>>
>> This is what kernel? Because kernel 4.6 offers mount option
>> "nologreplay" which suggests even if you do mount -r that log replay
>> happens, so you shouldn't see these deleted files unless you mount ro
>> *and* use nologreplay mount option.
>
> Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue
> zero-log" and then mount RW? Will the files safe in that case?

Depends on what's in the log that you're zeroing out. It's entirely
possible other things are lost, not just the incomplete deletion. And
also I have no idea if the deletion is entirely contained in only the
tree log.

>
>> Anyway, even 5 seconds of rm -rf damages too much. If you don't have
>> recent snapshots then it's not sanely salvageable, just reinstall.
>
> As I could see, almost all the "deleted" files are present. Certainly,
> I'll make an rsync diff between two-week-ago snapshot and the current
> FS state. But it will better if in-place recover without backup is
> possible.
>
> P.S. IMHO, log replay by default is a quite dangerous thing. I didn't
> know about that change and I could lose all files if the live USB had
> 4.6 kernel))

The change is only a mount time option to avoid log replay and that
option only applies to ro mounts. All file systems with logs always
replay the log on mount by default, that's the entire point of the
log.



-- 
Chris Murphy

Re: Strange behavior after "rm -rf //"

[Duncan]

Chris Murphy posted on Tue, 09 Aug 2016 11:10:08 -0600 as excerpted:

> On Mon, Aug 8, 2016 at 12:38 PM, Ivan Sizov <sivan606@gmail.com> wrote:
>> 2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>:
>>> Just a wild guess, the deletions may be in the tree log and haven't
>>> been applied to the other trees (fs tree, extent tree, etc). So yes
>>> I'd expect they get deleted on a rw mount.
>>>
>>> This is what kernel? Because kernel 4.6 offers mount option
>>> "nologreplay" which suggests even if you do mount -r that log replay
>>> happens, so you shouldn't see these deleted files unless you mount ro
>>> *and* use nologreplay mount option.
>>
>> Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue
>> zero-log" and then mount RW? Will the files safe in that case?
> 
> Depends on what's in the log that you're zeroing out. It's entirely
> possible other things are lost, not just the incomplete deletion. And
> also I have no idea if the deletion is entirely contained in only the
> tree log.

It's worth noting a critical difference between btrfs replay logs and 
conventional filesystem replay logs, however, with the result being that 
there's a fair chance the log replay has absolutely nothing to do with 
this case at all, and that it's simply commit vs. crash timing.

Btrfs is copy-on-write, with commits designed to be atomic -- changes 
work their way up the tree until a root commit finalizes them, and if a 
crash occurs, all changes since the last successful commit (with a commit 
every 30 seconds by default, and a mount option to change that) are 
normally lost.  Because the filesystem is copy-on-write, that means the 
filesystem should be consistent at that commit, and changes made after 
that will be in different locations that haven't made it into the tree 
yet, since the next commit wasn't able to happen due to the crash.  Thus, 
the stuff that conventional filesystems log simply doesn't apply to btrfs 
at all.

By contrast, conventional filesystems rewrite a lot of data and metadata 
in-place, and logging lets them write out to a temporary area the changes 
they intend to make before they actually write them to the permanent 
location, so that in the event of a crash, any data partially written to 
the permanent location will be replayed from the log, while if the crash 
happened when writing the log so it's corrupt, that record won't be 
replayed, and the old content will remain in place.

Tho of course writing all data twice tends to hit performance rather 
hard, so what most event logging filesystems do is only log the metadata, 
not the actual data.  This lets them be much faster than if they were 
logging the data, and normally protects the filesystem structure, but 
there's some chance that files rewritten in-place will be corrupt if a 
crash happens at the wrong moment.  But it limits the damage to only the 
file being written at the time, and does away with the requirement to fsck 
the entire filesystem after every crash.

So what /does/ the btrfs log do, then?  Good question! =:^)  Rather 
simply, keeping in mind that commits only normally trigger every 30 
seconds, the btrfs log tracks fsyncs (individual file syncs, as opposed 
to whole filesystem syncs), recording them in a replay log, so the 
filesystem can return success on the fsync, that the file was actually 
synced to permanent storage (often ssds these days, so not always "disk" 
as it used to be), without having to either wait upto 30 seconds for the 
next root tree commit, or forcing a full filesystem sync and commit, 
possibly including many other files, when only the one was requested.

So with btrfs, it's *only* fsyncs that are logged to the replay log, and 
that only to be able to truthfully return that the file was written to 
permanent storage, not normal filesystem operations, which are already 
atomic due to the copy-on-write semantics, and thus don't need logged.

So then, the question becomes one of whether rm -rf, or whatever other 
actual command was used to do the deletes, called fsync, or not.  If the 
command didn't call fsync, then it would have been the normal btrfs 
commit mechanism, again, every 30 seconds by default, that would have 
been in play here, and the btrfs log replay wouldn't have anything to do 
with it.

Which I actually strongly suspect to be the case.  It's likely that the 
last commit wasn't completed, so the btrfs reverted to the last atomic 
commit.  That would also explain why a read-only mount /without/ the 
nologreplay option still showed the files, since read-only does normally 
still replay that fsync log, so if the files were caught in it, they 
shouldn't show up at all. 


Meanwhile, back to the original scenario, just another demonstration of 
what every good sysadmin knows, often from hard experience, admin fat-
fingering -- the human factor --  PEBKAC -- is as much of a danger to the 
data and the system, if not more, than device or software failure.  If 
would-be backups can't protect from that, they're not backups.  Which is 
why simple RAID fails as a backup method, even if it can protect against 
device failure.  And of course, there's only two cases for the value of 
the data, it's either worth the hassle and resources to backup, or it's 
not, and if it's not backed up, by definition of not having that backup, 
you're defining it as the latter, no matter any claims to the contrary.  
In this case, as too many unfortunate people eventually find out, 
actions, or the lack of them, speak louder than words, and if the data is 
lost due to not having a backup, well, the only thing to do is to be 
happy that the thing your actions defined as worth more than that data, 
the time/hassle/resources necessary to do it, was saved.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Strange behavior after "rm -rf //"

[Christian Kujau]

On Mon, 8 Aug 2016, Ivan Sizov wrote:
> I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after five

Out of curiosity, what version of coreutils is this? The --preserve-root 
option is the default for quite some time now:

> Don't include dirname.h, since system.h does it now.
> (usage, main): --preserve-root is now the default.
> 2006-09-03 02:53:58 +0000
http://git.savannah.gnu.org/cgit/coreutils.git/commit/src/rm.c?id=89ffaa19909d31dffbcf12fb4498afb72666f6c9

Even coreutils-6.10 from Debian/5 refuses to remove "/":

$ sudo rm -rf /
rm: cannot remove root directory `/'

Christian.
-- 
BOFH excuse #243:

The computer fleetly, mouse and all.

Re: Strange behavior after "rm -rf //"

[Russell Coker]

http://selinux.coker.com.au/play.html

There are a variety of ways of giving the same result that rm doesn't reject. "/*" Wasn't caught last time I checked. See the above URL if you want to test out various rm operations as root. ;)

On 10 August 2016 9:24:23 AM AEST, Christian Kujau <lists@nerdbynature.de> wrote:
>On Mon, 8 Aug 2016, Ivan Sizov wrote:
>> I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after
>five
>
>Out of curiosity, what version of coreutils is this? The
>--preserve-root 
>option is the default for quite some time now:
>
>> Don't include dirname.h, since system.h does it now.
>> (usage, main): --preserve-root is now the default.
>> 2006-09-03 02:53:58 +0000
>http://git.savannah.gnu.org/cgit/coreutils.git/commit/src/rm.c?id=89ffaa19909d31dffbcf12fb4498afb72666f6c9
>
>Even coreutils-6.10 from Debian/5 refuses to remove "/":
>
>$ sudo rm -rf /
>rm: cannot remove root directory `/'
>
>Christian.

-- 
Sent from my Nexus 6P with K-9 Mail.

Re: Strange behavior after "rm -rf //"

[Christian Kujau]

On Fri, 12 Aug 2016, Russell Coker wrote:
> There are a variety of ways of giving the same result that rm
> doesn't reject. "/*" Wasn't caught last time I checked. See the above 
> URL if you want to test out various rm operations as root. ;)

Oh, yes - "rm -r /*" would work, even with a current coreutils version. 
But since the OP stated "rm -rf //" I was curious about the userspace
part on that.

Thanks for the link to the SELinux playground, nice setup!

Christian.
-- 
BOFH excuse #346:

Your/our computer(s) had suffered a memory leak, and we are waiting for them to be topped up.

Re: Strange behavior after "rm -rf //"

[Ivan Sizov]

Duncan, you was right. The commit didn't happen and nothing was
deleted except ext4 /boot. the System booted normally after GRUB2 and
kernel recovery.

Thank you much.

P.S. I'm sorry for the late answer.

2016-08-09 23:30 GMT+03:00 Duncan <1i5t5.duncan@cox.net>:
> Chris Murphy posted on Tue, 09 Aug 2016 11:10:08 -0600 as excerpted:
>
>> On Mon, Aug 8, 2016 at 12:38 PM, Ivan Sizov <sivan606@gmail.com> wrote:
>>> 2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>:
>>>> Just a wild guess, the deletions may be in the tree log and haven't
>>>> been applied to the other trees (fs tree, extent tree, etc). So yes
>>>> I'd expect they get deleted on a rw mount.
>>>>
>>>> This is what kernel? Because kernel 4.6 offers mount option
>>>> "nologreplay" which suggests even if you do mount -r that log replay
>>>> happens, so you shouldn't see these deleted files unless you mount ro
>>>> *and* use nologreplay mount option.
>>>
>>> Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue
>>> zero-log" and then mount RW? Will the files safe in that case?
>>
>> Depends on what's in the log that you're zeroing out. It's entirely
>> possible other things are lost, not just the incomplete deletion. And
>> also I have no idea if the deletion is entirely contained in only the
>> tree log.
>
> It's worth noting a critical difference between btrfs replay logs and
> conventional filesystem replay logs, however, with the result being that
> there's a fair chance the log replay has absolutely nothing to do with
> this case at all, and that it's simply commit vs. crash timing.
>
> Btrfs is copy-on-write, with commits designed to be atomic -- changes
> work their way up the tree until a root commit finalizes them, and if a
> crash occurs, all changes since the last successful commit (with a commit
> every 30 seconds by default, and a mount option to change that) are
> normally lost.  Because the filesystem is copy-on-write, that means the
> filesystem should be consistent at that commit, and changes made after
> that will be in different locations that haven't made it into the tree
> yet, since the next commit wasn't able to happen due to the crash.  Thus,
> the stuff that conventional filesystems log simply doesn't apply to btrfs
> at all.
>
> By contrast, conventional filesystems rewrite a lot of data and metadata
> in-place, and logging lets them write out to a temporary area the changes
> they intend to make before they actually write them to the permanent
> location, so that in the event of a crash, any data partially written to
> the permanent location will be replayed from the log, while if the crash
> happened when writing the log so it's corrupt, that record won't be
> replayed, and the old content will remain in place.
>
> Tho of course writing all data twice tends to hit performance rather
> hard, so what most event logging filesystems do is only log the metadata,
> not the actual data.  This lets them be much faster than if they were
> logging the data, and normally protects the filesystem structure, but
> there's some chance that files rewritten in-place will be corrupt if a
> crash happens at the wrong moment.  But it limits the damage to only the
> file being written at the time, and does away with the requirement to fsck
> the entire filesystem after every crash.
>
> So what /does/ the btrfs log do, then?  Good question! =:^)  Rather
> simply, keeping in mind that commits only normally trigger every 30
> seconds, the btrfs log tracks fsyncs (individual file syncs, as opposed
> to whole filesystem syncs), recording them in a replay log, so the
> filesystem can return success on the fsync, that the file was actually
> synced to permanent storage (often ssds these days, so not always "disk"
> as it used to be), without having to either wait upto 30 seconds for the
> next root tree commit, or forcing a full filesystem sync and commit,
> possibly including many other files, when only the one was requested.
>
> So with btrfs, it's *only* fsyncs that are logged to the replay log, and
> that only to be able to truthfully return that the file was written to
> permanent storage, not normal filesystem operations, which are already
> atomic due to the copy-on-write semantics, and thus don't need logged.
>
> So then, the question becomes one of whether rm -rf, or whatever other
> actual command was used to do the deletes, called fsync, or not.  If the
> command didn't call fsync, then it would have been the normal btrfs
> commit mechanism, again, every 30 seconds by default, that would have
> been in play here, and the btrfs log replay wouldn't have anything to do
> with it.
>
> Which I actually strongly suspect to be the case.  It's likely that the
> last commit wasn't completed, so the btrfs reverted to the last atomic
> commit.  That would also explain why a read-only mount /without/ the
> nologreplay option still showed the files, since read-only does normally
> still replay that fsync log, so if the files were caught in it, they
> shouldn't show up at all.
>
>
> Meanwhile, back to the original scenario, just another demonstration of
> what every good sysadmin knows, often from hard experience, admin fat-
> fingering -- the human factor --  PEBKAC -- is as much of a danger to the
> data and the system, if not more, than device or software failure.  If
> would-be backups can't protect from that, they're not backups.  Which is
> why simple RAID fails as a backup method, even if it can protect against
> device failure.  And of course, there's only two cases for the value of
> the data, it's either worth the hassle and resources to backup, or it's
> not, and if it's not backed up, by definition of not having that backup,
> you're defining it as the latter, no matter any claims to the contrary.
> In this case, as too many unfortunate people eventually find out,
> actions, or the lack of them, speak louder than words, and if the data is
> lost due to not having a backup, well, the only thing to do is to be
> happy that the thing your actions defined as worth more than that data,
> the time/hassle/resources necessary to do it, was saved.
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Ivan Sizov