* Strange behavior after "rm -rf //" @ 2016-08-08 16:30 Ivan Sizov 2016-08-08 17:13 ` Chris Murphy ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Ivan Sizov @ 2016-08-08 16:30 UTC (permalink / raw) To: Btrfs BTRFS I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after five seconds, but some files had been deleted. I'd tried to shutdown the system, but couldn't (a lot of files in /bin had been deleted and systemd didn't work). After hard reboot (by reset button) and booting to a live USB a strange thing was discovered. Deleted files are present when I "mount -r" the disk, but btrfs-restore tells they are deleted ("We have looped trying to restore files too many times to be making progress"). What does it mean? Will those files be deleted after RW mount? -- Ivan Sizov (SIvan) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-08 16:30 Strange behavior after "rm -rf //" Ivan Sizov @ 2016-08-08 17:13 ` Chris Murphy 2016-08-08 18:38 ` Ivan Sizov 2016-08-08 19:02 ` Duncan 2016-08-09 23:24 ` Christian Kujau 2 siblings, 1 reply; 12+ messages in thread From: Chris Murphy @ 2016-08-08 17:13 UTC (permalink / raw) To: Ivan Sizov; +Cc: Btrfs BTRFS On Mon, Aug 8, 2016 at 10:30 AM, Ivan Sizov <sivan606@gmail.com> wrote: > I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after five > seconds, but some files had been deleted. I'd tried to shutdown the > system, but couldn't (a lot of files in /bin had been deleted and > systemd didn't work). After hard reboot (by reset button) and booting > to a live USB a strange thing was discovered. > > Deleted files are present when I "mount -r" the disk, but > btrfs-restore tells they are deleted ("We have looped trying to > restore files too many times to be making progress"). > > What does it mean? Will those files be deleted after RW mount? Just a wild guess, the deletions may be in the tree log and haven't been applied to the other trees (fs tree, extent tree, etc). So yes I'd expect they get deleted on a rw mount. This is what kernel? Because kernel 4.6 offers mount option "nologreplay" which suggests even if you do mount -r that log replay happens, so you shouldn't see these deleted files unless you mount ro *and* use nologreplay mount option. Anyway, even 5 seconds of rm -rf damages too much. If you don't have recent snapshots then it's not sanely salvageable, just reinstall. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-08 17:13 ` Chris Murphy @ 2016-08-08 18:38 ` Ivan Sizov 2016-08-08 18:52 ` Hugo Mills 2016-08-09 17:10 ` Chris Murphy 0 siblings, 2 replies; 12+ messages in thread From: Ivan Sizov @ 2016-08-08 18:38 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS 2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>: > Just a wild guess, the deletions may be in the tree log and haven't > been applied to the other trees (fs tree, extent tree, etc). So yes > I'd expect they get deleted on a rw mount. > > This is what kernel? Because kernel 4.6 offers mount option > "nologreplay" which suggests even if you do mount -r that log replay > happens, so you shouldn't see these deleted files unless you mount ro > *and* use nologreplay mount option. Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue zero-log" and then mount RW? Will the files safe in that case? > Anyway, even 5 seconds of rm -rf damages too much. If you don't have > recent snapshots then it's not sanely salvageable, just reinstall. As I could see, almost all the "deleted" files are present. Certainly, I'll make an rsync diff between two-week-ago snapshot and the current FS state. But it will better if in-place recover without backup is possible. P.S. IMHO, log replay by default is a quite dangerous thing. I didn't know about that change and I could lose all files if the live USB had 4.6 kernel)) -- Ivan Sizov (SIvan) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-08 18:38 ` Ivan Sizov @ 2016-08-08 18:52 ` Hugo Mills 2016-08-08 19:00 ` Ivan Sizov 2016-08-09 17:10 ` Chris Murphy 1 sibling, 1 reply; 12+ messages in thread From: Hugo Mills @ 2016-08-08 18:52 UTC (permalink / raw) To: Ivan Sizov; +Cc: Chris Murphy, Btrfs BTRFS [-- Attachment #1: Type: text/plain, Size: 1951 bytes --] On Mon, Aug 08, 2016 at 09:38:28PM +0300, Ivan Sizov wrote: > 2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>: > > Just a wild guess, the deletions may be in the tree log and haven't > > been applied to the other trees (fs tree, extent tree, etc). So yes > > I'd expect they get deleted on a rw mount. > > > > This is what kernel? Because kernel 4.6 offers mount option > > "nologreplay" which suggests even if you do mount -r that log replay > > happens, so you shouldn't see these deleted files unless you mount ro > > *and* use nologreplay mount option. > > Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue > zero-log" and then mount RW? Will the files safe in that case? > > > Anyway, even 5 seconds of rm -rf damages too much. If you don't have > > recent snapshots then it's not sanely salvageable, just reinstall. > > As I could see, almost all the "deleted" files are present. Certainly, > I'll make an rsync diff between two-week-ago snapshot and the current > FS state. But it will better if in-place recover without backup is > possible. > > P.S. IMHO, log replay by default is a quite dangerous thing. I didn't > know about that change and I could lose all files if the live USB had > 4.6 kernel)) Log reply on mount has _always_ been the default, and should remain so. It gives you the expected semantics after a power loss: all th efiles that you'd written up to the point of the power loss actually appear afterwards. (If this didn't happen, you could lose up to 30s of writes from before the crash). It's only very recently that there's been an option to prevent it, which is useful in a limited number of cases (such as trying to undelete a file, which is not really a supported operation in any case). Hugo. -- Hugo Mills | "I lost my leg in 1942. Some bastard stole it in a hugo@... carfax.org.uk | pub in Pimlico." http://carfax.org.uk/ | PGP: E2AB1DE4 | [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-08 18:52 ` Hugo Mills @ 2016-08-08 19:00 ` Ivan Sizov 0 siblings, 0 replies; 12+ messages in thread From: Ivan Sizov @ 2016-08-08 19:00 UTC (permalink / raw) To: Hugo Mills, Ivan Sizov, Chris Murphy, Btrfs BTRFS 2016-08-08 21:52 GMT+03:00 Hugo Mills <hugo@carfax.org.uk>: > On Mon, Aug 08, 2016 at 09:38:28PM +0300, Ivan Sizov wrote: >> P.S. IMHO, log replay by default is a quite dangerous thing. I didn't >> know about that change and I could lose all files if the live USB had >> 4.6 kernel)) > > Log reply on mount has _always_ been the default, and should remain > so. It gives you the expected semantics after a power loss: all th > efiles that you'd written up to the point of the power loss actually > appear afterwards. (If this didn't happen, you could lose up to 30s of > writes from before the crash). > > It's only very recently that there's been an option to prevent it, > which is useful in a limited number of cases (such as trying to > undelete a file, which is not really a supported operation in any > case). I mean only RO mount, of course. So, will zero-log prevent my files during RW mount? -- Ivan Sizov (SIvan) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-08 18:38 ` Ivan Sizov 2016-08-08 18:52 ` Hugo Mills @ 2016-08-09 17:10 ` Chris Murphy 2016-08-09 20:30 ` Duncan 1 sibling, 1 reply; 12+ messages in thread From: Chris Murphy @ 2016-08-09 17:10 UTC (permalink / raw) To: Ivan Sizov; +Cc: Chris Murphy, Btrfs BTRFS On Mon, Aug 8, 2016 at 12:38 PM, Ivan Sizov <sivan606@gmail.com> wrote: > 2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>: >> Just a wild guess, the deletions may be in the tree log and haven't >> been applied to the other trees (fs tree, extent tree, etc). So yes >> I'd expect they get deleted on a rw mount. >> >> This is what kernel? Because kernel 4.6 offers mount option >> "nologreplay" which suggests even if you do mount -r that log replay >> happens, so you shouldn't see these deleted files unless you mount ro >> *and* use nologreplay mount option. > > Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue > zero-log" and then mount RW? Will the files safe in that case? Depends on what's in the log that you're zeroing out. It's entirely possible other things are lost, not just the incomplete deletion. And also I have no idea if the deletion is entirely contained in only the tree log. > >> Anyway, even 5 seconds of rm -rf damages too much. If you don't have >> recent snapshots then it's not sanely salvageable, just reinstall. > > As I could see, almost all the "deleted" files are present. Certainly, > I'll make an rsync diff between two-week-ago snapshot and the current > FS state. But it will better if in-place recover without backup is > possible. > > P.S. IMHO, log replay by default is a quite dangerous thing. I didn't > know about that change and I could lose all files if the live USB had > 4.6 kernel)) The change is only a mount time option to avoid log replay and that option only applies to ro mounts. All file systems with logs always replay the log on mount by default, that's the entire point of the log. -- Chris Murphy ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-09 17:10 ` Chris Murphy @ 2016-08-09 20:30 ` Duncan 2016-08-21 17:54 ` Ivan Sizov 0 siblings, 1 reply; 12+ messages in thread From: Duncan @ 2016-08-09 20:30 UTC (permalink / raw) To: linux-btrfs Chris Murphy posted on Tue, 09 Aug 2016 11:10:08 -0600 as excerpted: > On Mon, Aug 8, 2016 at 12:38 PM, Ivan Sizov <sivan606@gmail.com> wrote: >> 2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>: >>> Just a wild guess, the deletions may be in the tree log and haven't >>> been applied to the other trees (fs tree, extent tree, etc). So yes >>> I'd expect they get deleted on a rw mount. >>> >>> This is what kernel? Because kernel 4.6 offers mount option >>> "nologreplay" which suggests even if you do mount -r that log replay >>> happens, so you shouldn't see these deleted files unless you mount ro >>> *and* use nologreplay mount option. >> >> Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue >> zero-log" and then mount RW? Will the files safe in that case? > > Depends on what's in the log that you're zeroing out. It's entirely > possible other things are lost, not just the incomplete deletion. And > also I have no idea if the deletion is entirely contained in only the > tree log. It's worth noting a critical difference between btrfs replay logs and conventional filesystem replay logs, however, with the result being that there's a fair chance the log replay has absolutely nothing to do with this case at all, and that it's simply commit vs. crash timing. Btrfs is copy-on-write, with commits designed to be atomic -- changes work their way up the tree until a root commit finalizes them, and if a crash occurs, all changes since the last successful commit (with a commit every 30 seconds by default, and a mount option to change that) are normally lost. Because the filesystem is copy-on-write, that means the filesystem should be consistent at that commit, and changes made after that will be in different locations that haven't made it into the tree yet, since the next commit wasn't able to happen due to the crash. Thus, the stuff that conventional filesystems log simply doesn't apply to btrfs at all. By contrast, conventional filesystems rewrite a lot of data and metadata in-place, and logging lets them write out to a temporary area the changes they intend to make before they actually write them to the permanent location, so that in the event of a crash, any data partially written to the permanent location will be replayed from the log, while if the crash happened when writing the log so it's corrupt, that record won't be replayed, and the old content will remain in place. Tho of course writing all data twice tends to hit performance rather hard, so what most event logging filesystems do is only log the metadata, not the actual data. This lets them be much faster than if they were logging the data, and normally protects the filesystem structure, but there's some chance that files rewritten in-place will be corrupt if a crash happens at the wrong moment. But it limits the damage to only the file being written at the time, and does away with the requirement to fsck the entire filesystem after every crash. So what /does/ the btrfs log do, then? Good question! =:^) Rather simply, keeping in mind that commits only normally trigger every 30 seconds, the btrfs log tracks fsyncs (individual file syncs, as opposed to whole filesystem syncs), recording them in a replay log, so the filesystem can return success on the fsync, that the file was actually synced to permanent storage (often ssds these days, so not always "disk" as it used to be), without having to either wait upto 30 seconds for the next root tree commit, or forcing a full filesystem sync and commit, possibly including many other files, when only the one was requested. So with btrfs, it's *only* fsyncs that are logged to the replay log, and that only to be able to truthfully return that the file was written to permanent storage, not normal filesystem operations, which are already atomic due to the copy-on-write semantics, and thus don't need logged. So then, the question becomes one of whether rm -rf, or whatever other actual command was used to do the deletes, called fsync, or not. If the command didn't call fsync, then it would have been the normal btrfs commit mechanism, again, every 30 seconds by default, that would have been in play here, and the btrfs log replay wouldn't have anything to do with it. Which I actually strongly suspect to be the case. It's likely that the last commit wasn't completed, so the btrfs reverted to the last atomic commit. That would also explain why a read-only mount /without/ the nologreplay option still showed the files, since read-only does normally still replay that fsync log, so if the files were caught in it, they shouldn't show up at all. Meanwhile, back to the original scenario, just another demonstration of what every good sysadmin knows, often from hard experience, admin fat- fingering -- the human factor -- PEBKAC -- is as much of a danger to the data and the system, if not more, than device or software failure. If would-be backups can't protect from that, they're not backups. Which is why simple RAID fails as a backup method, even if it can protect against device failure. And of course, there's only two cases for the value of the data, it's either worth the hassle and resources to backup, or it's not, and if it's not backed up, by definition of not having that backup, you're defining it as the latter, no matter any claims to the contrary. In this case, as too many unfortunate people eventually find out, actions, or the lack of them, speak louder than words, and if the data is lost due to not having a backup, well, the only thing to do is to be happy that the thing your actions defined as worth more than that data, the time/hassle/resources necessary to do it, was saved. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-09 20:30 ` Duncan @ 2016-08-21 17:54 ` Ivan Sizov 0 siblings, 0 replies; 12+ messages in thread From: Ivan Sizov @ 2016-08-21 17:54 UTC (permalink / raw) To: Duncan; +Cc: Btrfs BTRFS Duncan, you was right. The commit didn't happen and nothing was deleted except ext4 /boot. the System booted normally after GRUB2 and kernel recovery. Thank you much. P.S. I'm sorry for the late answer. 2016-08-09 23:30 GMT+03:00 Duncan <1i5t5.duncan@cox.net>: > Chris Murphy posted on Tue, 09 Aug 2016 11:10:08 -0600 as excerpted: > >> On Mon, Aug 8, 2016 at 12:38 PM, Ivan Sizov <sivan606@gmail.com> wrote: >>> 2016-08-08 20:13 GMT+03:00 Chris Murphy <lists@colorremedies.com>: >>>> Just a wild guess, the deletions may be in the tree log and haven't >>>> been applied to the other trees (fs tree, extent tree, etc). So yes >>>> I'd expect they get deleted on a rw mount. >>>> >>>> This is what kernel? Because kernel 4.6 offers mount option >>>> "nologreplay" which suggests even if you do mount -r that log replay >>>> happens, so you shouldn't see these deleted files unless you mount ro >>>> *and* use nologreplay mount option. >>> >>> Live USB has kernel 4.5.7. Maybe I should try to run "btrfs rescue >>> zero-log" and then mount RW? Will the files safe in that case? >> >> Depends on what's in the log that you're zeroing out. It's entirely >> possible other things are lost, not just the incomplete deletion. And >> also I have no idea if the deletion is entirely contained in only the >> tree log. > > It's worth noting a critical difference between btrfs replay logs and > conventional filesystem replay logs, however, with the result being that > there's a fair chance the log replay has absolutely nothing to do with > this case at all, and that it's simply commit vs. crash timing. > > Btrfs is copy-on-write, with commits designed to be atomic -- changes > work their way up the tree until a root commit finalizes them, and if a > crash occurs, all changes since the last successful commit (with a commit > every 30 seconds by default, and a mount option to change that) are > normally lost. Because the filesystem is copy-on-write, that means the > filesystem should be consistent at that commit, and changes made after > that will be in different locations that haven't made it into the tree > yet, since the next commit wasn't able to happen due to the crash. Thus, > the stuff that conventional filesystems log simply doesn't apply to btrfs > at all. > > By contrast, conventional filesystems rewrite a lot of data and metadata > in-place, and logging lets them write out to a temporary area the changes > they intend to make before they actually write them to the permanent > location, so that in the event of a crash, any data partially written to > the permanent location will be replayed from the log, while if the crash > happened when writing the log so it's corrupt, that record won't be > replayed, and the old content will remain in place. > > Tho of course writing all data twice tends to hit performance rather > hard, so what most event logging filesystems do is only log the metadata, > not the actual data. This lets them be much faster than if they were > logging the data, and normally protects the filesystem structure, but > there's some chance that files rewritten in-place will be corrupt if a > crash happens at the wrong moment. But it limits the damage to only the > file being written at the time, and does away with the requirement to fsck > the entire filesystem after every crash. > > So what /does/ the btrfs log do, then? Good question! =:^) Rather > simply, keeping in mind that commits only normally trigger every 30 > seconds, the btrfs log tracks fsyncs (individual file syncs, as opposed > to whole filesystem syncs), recording them in a replay log, so the > filesystem can return success on the fsync, that the file was actually > synced to permanent storage (often ssds these days, so not always "disk" > as it used to be), without having to either wait upto 30 seconds for the > next root tree commit, or forcing a full filesystem sync and commit, > possibly including many other files, when only the one was requested. > > So with btrfs, it's *only* fsyncs that are logged to the replay log, and > that only to be able to truthfully return that the file was written to > permanent storage, not normal filesystem operations, which are already > atomic due to the copy-on-write semantics, and thus don't need logged. > > So then, the question becomes one of whether rm -rf, or whatever other > actual command was used to do the deletes, called fsync, or not. If the > command didn't call fsync, then it would have been the normal btrfs > commit mechanism, again, every 30 seconds by default, that would have > been in play here, and the btrfs log replay wouldn't have anything to do > with it. > > Which I actually strongly suspect to be the case. It's likely that the > last commit wasn't completed, so the btrfs reverted to the last atomic > commit. That would also explain why a read-only mount /without/ the > nologreplay option still showed the files, since read-only does normally > still replay that fsync log, so if the files were caught in it, they > shouldn't show up at all. > > > Meanwhile, back to the original scenario, just another demonstration of > what every good sysadmin knows, often from hard experience, admin fat- > fingering -- the human factor -- PEBKAC -- is as much of a danger to the > data and the system, if not more, than device or software failure. If > would-be backups can't protect from that, they're not backups. Which is > why simple RAID fails as a backup method, even if it can protect against > device failure. And of course, there's only two cases for the value of > the data, it's either worth the hassle and resources to backup, or it's > not, and if it's not backed up, by definition of not having that backup, > you're defining it as the latter, no matter any claims to the contrary. > In this case, as too many unfortunate people eventually find out, > actions, or the lack of them, speak louder than words, and if the data is > lost due to not having a backup, well, the only thing to do is to be > happy that the thing your actions defined as worth more than that data, > the time/hassle/resources necessary to do it, was saved. > > -- > Duncan - List replies preferred. No HTML msgs. > "Every nonfree program has a lord, a master -- > and if you use the program, he is your master." Richard Stallman > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Ivan Sizov ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-08 16:30 Strange behavior after "rm -rf //" Ivan Sizov 2016-08-08 17:13 ` Chris Murphy @ 2016-08-08 19:02 ` Duncan 2016-08-09 23:24 ` Christian Kujau 2 siblings, 0 replies; 12+ messages in thread From: Duncan @ 2016-08-08 19:02 UTC (permalink / raw) To: linux-btrfs Ivan Sizov posted on Mon, 08 Aug 2016 19:30:16 +0300 as excerpted: > I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after five > seconds, but some files had been deleted. I'd tried to shutdown the > system, but couldn't (a lot of files in /bin had been deleted and > systemd didn't work). After hard reboot (by reset button) and booting to > a live USB a strange thing was discovered. > > Deleted files are present when I "mount -r" the disk, but btrfs-restore > tells they are deleted ("We have looped trying to restore files too many > times to be making progress"). > > What does it mean? Will those files be deleted after RW mount? Chris is likely correct in your case, but I'd like to point out three things. 1) The looping ... warning in btrfs restore is obviously there for a reason, because under some circumstances the filesystem will be damaged in such a way that restore /can/ loop without making progress, but that's not always the case, and in fact, in my own experience, has /never/ been the case. Far more common, at least from my own experience, is seeing that warning simply due to directories containing a large number of files, even when restore /is/ working properly and restoring the files. I don't know where the cutover is, but there's a reason it's a warning that allows you to say continue, and in every single case from my own experience, continuing /enough/ times eventually resulted in a successful restore with no missing files that I could tell (tho I didn't do a before/after comparison, just never missed anything but symlinks, etc, before the option to restore them too was added). So if you haven't tried it yet, tell restore to continue despite the warning and see if it eventually does make progress. Some people even automate the process using yes | btrfs restore ... or similar, tho I've never needed that here, possibly because I use multiple relatively small partitions (all under 50 GiB each except for my media partition and its backup). I guess if they do decide btrfs restore is in an infinite loop, say after hours with no increase in the total size of the files restored, they'd have to break out of the loop manually, tho I've seen several posts where people were asking for restore to have a built-in continue option, or where they used automation, and none where they had to break the loop manually, so I'd guess it's actually pretty rare that a real infinite loop actually happens. And because btrfs is copy-on-write and the old roots stay around for awhile, provided you take pains not to mount the filesystem writable or if you do not to write too much to it, since the more you write the less likely you are to be able to fully recover older transactions, you can likely use restore manually with the -t <transid> option and btrfs-find- root to find an appropriate transid, to get the files back even if they do otherwise appear to be deleted. See the wiki for instructions on that. If you have a new enough btrfs- progs, the page should be referenced in the btrfs-restore manpage. But here it is anyway, since I have the manpage open ATM: https://btrfs.wiki.kernel.org/index.php/Restore 2) Primarily because you didn't mention it and it can be handy in other circumstance, if you're unaware of it, read up on magic sysrequest, aka sysrq aka srq. $KERNDIR/Documentation/sysrq.txt ... and various googlable articles on the subject. Basically, any time you'd otherwise resort to a hard reboot, try a magic- srq sequence first. Longer version: reisub. Shorter version, just the sub. That's emergency Sync, remoUnt-read-only, reBoot (thus s-u-b). It won't always work, particularly for kernel crashes, but even if it doesn't you can get a feel for how bad the crash was by the response or lack thereof (if the s and u light up the storage device activity LED, the kernel was alive and considered it safe to still write to storage, if they don't show activity but the b still reboots, the kernel was alive but either nothing dirty to write or the kernel considered itself damaged and thus wasn't going to risk writing to storage, if none work, the kernel itself was dead). Because your problem this time was userspace, simply no binaries to run, that should have worked, safely shutting down the filesystem. Altho arguably in this case a hard reboot was the better choice, since that final commit might have been lower risk for the filesystem, but would have likely finalized those deletions that you can now recover. (Tho with btrfs being copy-on-write, there's a fair chance you'd have been able to restore the files anyway, if done right away, using restore and manually pointing it at an earlier root.) So you arguably did the right thing with a hard reboot here anyway, but in other cases, magic-srq is incredibly useful to know and may just save your butt, as I believe it has mine a few times by now. 3) I did something similar a couple years ago. In my case, I was (unwisely) testing a script as root, with a typo in a variable name so it was an empty variable and thus started from / instead of the intended path. Fortunately, I have backups, tho I don't keep them as current as I might, and it took out /bin and /boot and then warned me about /dev, which it couldn't delete due to that being the devfs mountpoint. It proceeded into /etc, but that's where I stopped it after the warning about /dev, so I still had /usr/bin and the libs as well as /home, and could rebuild /bin and /etc from backups. But the point it drove home to me is one I had heard before and fortunately was living by, that an admin has as much to fear from fat- fingering something as he does from device, filesystem or software update failure. And of course I shouldn't have been testing that script as root, and anything that scripts rm -r /$variable/* deletions like that needs at minimum an empty-var test that only proceeds with the rm if the variable isn't empty/null. But the primary point is that if it's not backed up, by the inaction of failing to do that backup, you are in a very real and non-negotiable after-the-fact way, defining that data as worth less than the time and resources required to do the backup. Fortunately I did have a (tested, if it's not tested it's not yet a backup!) backup, tho I don't always keep my backups current. But at least I know the risk is limited to the updates between that backup and the current time, and I recognize that by not doing more regular backups, I am in a very real way defining that data in the gap as of only trivial value, to the point that I recognize the risk and when I start getting uncomfortable with the size of the data in that difference gap, I know it's time to do another backup. And by that definition, it's impossible to lose data more valuable than the cost of an additional level of backup that would have kept it safe, whether that's no backup for data of trivial value, only a single on-site backup for data worth a bit more, or a hundred (or a thousand) levels of backup at 50 sites in 20 countries on 5 continents, because the data really is /that/ valuable. So if you /think/ you value the data, have the backups demonstrating that value, because if you don't, you have a very real possibility of demonstrating that you did /not/ value the data as much as you claimed to, because it wasn't backed up and that lack of backup demonstrated the lie in any claim to the contrary. IOW, backups speak louder than words! -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-08 16:30 Strange behavior after "rm -rf //" Ivan Sizov 2016-08-08 17:13 ` Chris Murphy 2016-08-08 19:02 ` Duncan @ 2016-08-09 23:24 ` Christian Kujau 2016-08-12 3:08 ` Russell Coker 2 siblings, 1 reply; 12+ messages in thread From: Christian Kujau @ 2016-08-09 23:24 UTC (permalink / raw) To: Ivan Sizov; +Cc: Btrfs BTRFS On Mon, 8 Aug 2016, Ivan Sizov wrote: > I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after five Out of curiosity, what version of coreutils is this? The --preserve-root option is the default for quite some time now: > Don't include dirname.h, since system.h does it now. > (usage, main): --preserve-root is now the default. > 2006-09-03 02:53:58 +0000 http://git.savannah.gnu.org/cgit/coreutils.git/commit/src/rm.c?id=89ffaa19909d31dffbcf12fb4498afb72666f6c9 Even coreutils-6.10 from Debian/5 refuses to remove "/": $ sudo rm -rf / rm: cannot remove root directory `/' Christian. -- BOFH excuse #243: The computer fleetly, mouse and all. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-09 23:24 ` Christian Kujau @ 2016-08-12 3:08 ` Russell Coker 2016-08-12 6:15 ` Christian Kujau 0 siblings, 1 reply; 12+ messages in thread From: Russell Coker @ 2016-08-12 3:08 UTC (permalink / raw) To: Christian Kujau, Ivan Sizov; +Cc: Btrfs BTRFS http://selinux.coker.com.au/play.html There are a variety of ways of giving the same result that rm doesn't reject. "/*" Wasn't caught last time I checked. See the above URL if you want to test out various rm operations as root. ;) On 10 August 2016 9:24:23 AM AEST, Christian Kujau <lists@nerdbynature.de> wrote: >On Mon, 8 Aug 2016, Ivan Sizov wrote: >> I'd ran "rm -rf //" by mistake two days ago. I'd stopped it after >five > >Out of curiosity, what version of coreutils is this? The >--preserve-root >option is the default for quite some time now: > >> Don't include dirname.h, since system.h does it now. >> (usage, main): --preserve-root is now the default. >> 2006-09-03 02:53:58 +0000 >http://git.savannah.gnu.org/cgit/coreutils.git/commit/src/rm.c?id=89ffaa19909d31dffbcf12fb4498afb72666f6c9 > >Even coreutils-6.10 from Debian/5 refuses to remove "/": > >$ sudo rm -rf / >rm: cannot remove root directory `/' > >Christian. -- Sent from my Nexus 6P with K-9 Mail. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Strange behavior after "rm -rf //" 2016-08-12 3:08 ` Russell Coker @ 2016-08-12 6:15 ` Christian Kujau 0 siblings, 0 replies; 12+ messages in thread From: Christian Kujau @ 2016-08-12 6:15 UTC (permalink / raw) To: Russell Coker; +Cc: Ivan Sizov, Btrfs BTRFS On Fri, 12 Aug 2016, Russell Coker wrote: > There are a variety of ways of giving the same result that rm > doesn't reject. "/*" Wasn't caught last time I checked. See the above > URL if you want to test out various rm operations as root. ;) Oh, yes - "rm -r /*" would work, even with a current coreutils version. But since the OP stated "rm -rf //" I was curious about the userspace part on that. Thanks for the link to the SELinux playground, nice setup! Christian. -- BOFH excuse #346: Your/our computer(s) had suffered a memory leak, and we are waiting for them to be topped up. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2016-08-21 17:55 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-08-08 16:30 Strange behavior after "rm -rf //" Ivan Sizov 2016-08-08 17:13 ` Chris Murphy 2016-08-08 18:38 ` Ivan Sizov 2016-08-08 18:52 ` Hugo Mills 2016-08-08 19:00 ` Ivan Sizov 2016-08-09 17:10 ` Chris Murphy 2016-08-09 20:30 ` Duncan 2016-08-21 17:54 ` Ivan Sizov 2016-08-08 19:02 ` Duncan 2016-08-09 23:24 ` Christian Kujau 2016-08-12 3:08 ` Russell Coker 2016-08-12 6:15 ` Christian Kujau
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.