subvols and parents - how?

subvols and parents - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 2083 bytes --]

Hey.

I'd have a, mainly administrative, question.

When I use subvolumes than these have always a parent subvolume (except
ID5), so I can basically decide between two ways:

a) make child subvolumes, e.g.
5
|
+-root   (=subvol, mountpoint /)
  +-boot/
  +-root/
  +-lib/
  +-home/ (=subvolume)
and soon on... perhaps the whole thing without the dedicated root-
subovlume (although that's probably not so smart, I guess).

b) place at least some of the subvolumes directly below the top-level
and mount them e.g. via /etc/fstab, e.g.
5
|
+-root   (=subvol, mountpoint /)
| +-boot/
| +-root/
| +-lib/
+-home/ (=subvolume, mountpoint /home)


Now I wondered whether this has any technical implications, but neither
the wiki, nor the manpages seem to explain a lot here.


The "differences", AFAIU, are the follows:
- When I mount a given subvolume,.. it's childs are automatically 
  "there".
  Whereas when I don't have them as childs (as in (b)) I must of
  course mount them somehow manually.
- Analogously for umounting.
- I can move existing subvols to higher/lower levels, and the parent
  IDs will change accordingly.

So basically it makes no difference, right? Or is there anything more
technical going on? E.g. with the ref-links or so?
Right now, there are, AFAIK, neither recursive snapshots (and
especially not atomic ones) nor recursive send/receive, right?
If that should ever be implemented, would I perhaps have problems with 
(a) or (b)?


Thanks,
Chris.

btw, for a developer:
$ btrfs subvolume list /mnt/ -pacguqt
ID      gen     cgen    parent  top level       parent_uuid     uuid    path    
--      ---     ----    ------  ---------       -----------     ----    ----    
257     16      8       5       5               -       64f8a75b-5cb5-6e4d-9f30-d45fe3d9e060    <FS_TREE>/root

There seem to be some colum mis-aglinment issues in the output =)
btrfsprogs 4.3

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Duncan]

Christoph Anton Mitterer posted on Tue, 24 Nov 2015 05:56:00 +0100 as
excerpted:

> When I use subvolumes than these have always a parent subvolume (except
> ID5), so I can basically decide between two ways:
> 
> a) make child subvolumes, e.g.
> 5
> |
> +-root   (=subvol, mountpoint /)
>   +-boot/
>   +-root/
>   +-lib/
>   +-home/ (=subvolume)
> and soon on... perhaps the whole thing without the dedicated root-
> subovlume (although that's probably not so smart, I guess).
> 
> b) place at least some of the subvolumes directly below the top-level
> and mount them e.g. via /etc/fstab, e.g.
> 5
> |
> +-root   (=subvol, mountpoint /)
> | +-boot/
> | +-root/
> | +-lib/
> +-home/ (=subvolume, mountpoint /home)
> 
> 
> Now I wondered whether this has any technical implications, but neither
> the wiki, nor the manpages seem to explain a lot here.

Very astute question! =:^)

Somewhere on the wiki I believe there's a recommendation to use (b) 
layout, but to some extent it depends on why you're actually doing 
subvolumes.  

OK, found it on the wiki.  It wasn't under use-cases, where I initially 
thought to look, but under sysadmin guide.  Specifically, see section 
4.2, managing snapshots, but I'd suggest reading the entire subvolumes 
discussion, section 4, or even most/all of the page.

https://btrfs.wiki.kernel.org/index.php/SysadminGuide

(More below.)

> The "differences", AFAIU, are the follows:
> - When I mount a given subvolume,.. it's childs are automatically
>   "there".
>   Whereas when I don't have them as childs (as in (b)) I must of course
>   mount them somehow manually.
> - Analogously for umounting.
> - I can move existing subvols to higher/lower levels, and the parent
>   IDs will change accordingly.
> 
> So basically it makes no difference, right? Or is there anything more
> technical going on? E.g. with the ref-links or so?
> Right now, there are, AFAIK, neither recursive snapshots (and especially
> not atomic ones) nor recursive send/receive, right?
> If that should ever be implemented, would I perhaps have problems with
> (a) or (b)?

If you're doing subvolumes for snapshotting and potential rollback 
purposes, layout (b) can be preferable as it allows a more direct mix and 
match rollback.

Suppose you only want to rollback /, because some update screwed you up, 
but not /home, which is fine.  If /home is a nested subvolume, then 
you're now mounting the nested home subvolume from some other nesting 
tree entirely, whereas if they're all under top-level, you simply mount 
the /home subvolume under whatever snapshot of / you are currently 
booting.

Of course the reverse applies as well, if / is fine but you want to 
rollback /home.  Again, with nesting you're reaching into some other 
nesting to mount what you want, and it can get a bit unintuitive and 
difficult to track, particularly if you go more than the two levels deep, 
but if all the snapshots are direct children of the top level ID 5, it's 
a lot easier.

Tho I'd actually suggest a variant of the flat layout they suggest in the 
sysadmin's guide.  What I'd do is something like this (using your tree 
drawing style):

5
|
+-roots (dir not subvol, note the s, rootS, plural)
| +-root (subvol, mountpoint /)
| | +-boot/ (dir)
| | +-root/ (dir)
| | +-lib/  (dir)
| | +-home/ (empty dir as mountpoint)
| +-root-snapshot-2015.0301 (dated snapshot of root)
| +-root-snapshot-2015.0601 (dated snapshot of root)
| +-root-snapshot-2015.0901 (dated snapshot of root)
+-homes (dir not subvol)
| +-home (subvol, mountpoint /home)
| +-home-snapshot-2015.0301 (dated snapshot of home)
...


Of course, you might also organize by date instead of subvol...

5
|
+- heads (dir, headS plural)
| +-root (subvol)
| +-home (subvol)
| +-whatever (subvol)
+-snapshots-2015.0301 (dir, snapshotS plural)
| +-root-2015.0301 (snapshot of heads/root)
| +-home-2015.0301 (snapshot of heads/home)
| ...
+-snapshots-2015.0601 (dir)
| +-root-2015.0601 (snapshot)
| ...
+-snapshots-2015.0901 (dir)
| +-root-2015.0901 (snapshot)
...


Either of these would make finding a desired snapshot to rollback to much 
easier than a pure flat subvols/snapshots layout, with the preferred one 
depending on whether you want subvols/snapshots grouped by date or by 
snapshotted mountpoint.

The dates organization would make cleaning up old snapshots by date, and 
visually checking that the snapshot cleanup script (if automated) is 
working as intended, somewhat easier, however.


Meanwhile, if the intention for a subvolume is simply to exclude that 
subtree from snapshotting of the parent, as might be the case for example 
if you have a VMs subvol, with the VM image files set NOCOW to avoid 
fragmentation, since snapshotting nocow files forces cow1 (a cow at the 
first write of that block, before returning to nocow, because a snapshot 
locks the existing extents in place for the snapshot, so initial writes 
to a block after a snapshot /can't be nocow or it'd change the snapshot 
too...), and it's not going to be snapshotted on perhaps a different 
schedule than the parent...

... then it can make more sense to use a nested subvolume, as the primary 
and perhaps only reason for the subvolume at all is to prevent the 
subtree from being included in the parent snapshot, and as such, it's 
unlikely to be separately mounted.

OTOH, if there's a chance you might want to mount that subvolume in a 
roll-back scenario, under the snapshot you're rolling back to, then it 
makes sense to put it directly under ID 5 again, and mount it in any case.


Then there's the security angle to consider.  With the (basically, 
possibly modified as I suggested) flat layout, mounting something doesn't 
automatically give people in-tree access to nested subvolumes (subject to 
normal file permissions, of course), like nested layout does.  And with 
(possibly modified) flat layout, the whole subvolume tree doesn't need to 
be mounted all the time either, only when you're actually working with 
subvolumes.


In general, I believe the nested layout is easier to setup initially, 
because you don't have to worry about actually mounting the subvolumes to 
make them accessible.  But flat (or modified flat, as I suggest) is 
easier for continued administration, particularly if you're doing 
snapshots and potentially rollbacks, because while doing the initial 
mount setup is more work, once the mount layout is established, mixing 
and matching various heads with snapshots of varying ages is a relatively 
simple matter of pointing each mount at the appropriate snapshot instead 
of head.  Additionally, (possibly modified) flat layout is easier to do 
subvolume/snapshot management on, because they're all in the same place, 
instead of scattered at random places within the tree.  In my quarterly 
snapshots example, for instance, suppose 2016 rolls around and you decide 
you don't need 2014's snapshots any longer.  Which layout is going to 
make it easiest to delete them all, while conversely, making it least 
likely to fat-finger things and delete the wrong ones?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: subvols and parents - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 5078 bytes --]

On Tue, 2015-11-24 at 08:29 +0000, Duncan wrote:
> OK, found it on the wiki.  It wasn't under use-cases, where I
> initially 
> thought to look, but under sysadmin guide.  Specifically, see section
> 4.2, managing snapshots, but I'd suggest reading the entire
> subvolumes 
> discussion, section 4, or even most/all of the page.
> 
> https://btrfs.wiki.kernel.org/index.php/SysadminGuide
Well I've had read that, but it's pretty vague and especially doesn't
mentioned any of the filesystem internal implications (if there are
any)... like I wondered before, whether this has effects on ref-links
not being used when e.g. sending/recieving ... or on future planned
extensions like recursive snapshots.


> 
> Suppose you only want to rollback /, because some update screwed you
> up, 
> but not /home, which is fine.  If /home is a nested subvolume, then 
> you're now mounting the nested home subvolume from some other nesting
> tree entirely,
That's a bit unclear to me,... I thought when I make a snapshot, any
nested subvols wouldn't be snapshotted and thus be empty dirs.
So I'd have rather that if I would simply have no /home (if I didn't
move it to the rolled-back subvol manually)


> 5
> > 
> +-roots (dir not subvol, note the s, rootS, plural)
> > +-root (subvol, mountpoint /)
> > > +-boot/ (dir)
> > > +-root/ (dir)
> > > +-lib/  (dir)
> > > +-home/ (empty dir as mountpoint)
> > +-root-snapshot-2015.0301 (dated snapshot of root)
> > +-root-snapshot-2015.0601 (dated snapshot of root)
> > +-root-snapshot-2015.0901 (dated snapshot of root)
> +-homes (dir not subvol)
> > +-home (subvol, mountpoint /home)
> > +-home-snapshot-2015.0301 (dated snapshot of home)
> ...
That's more what I've had in mind...
Actually something like this:
5
|
+-root       (=subvol)
| +-boot
| +-home     (subvo=/home being mounted heron)
| +-lib
+-home       (subvol, the current version)
+-snapshots  (=dir)
  +-root
  | +-2015-01-14 (subvol, snapshot)
  | +-2015-09-30 (subvol, snapshot)
  +-home
    +-2015-06-04 (subvol, snapshot)
    +-2015-09-01 (subvol, snapshot)


And it once more points to the problem of the wiki... anyone can write
(I think even I) and it's totally unclear at the first glance how
"[non-]official" and outdated something may be.
Apart from the problem that many important questions (from the PoV of
an more advanced admin that doesn't just do mkfs.btrfs and then never
again thinks about it) :-(


> Meanwhile, if the intention for a subvolume is simply to exclude that
> subtree from snapshotting of the parent, as might be the case for
> example 
that is in fact also use case.. so in practise I'll probably have a mix
of (a) and (b).


> if you have a VMs subvol, with the VM image files set NOCOW to avoid 
> fragmentation, since snapshotting nocow files forces cow1 (a cow at
> the 
> first write of that block, before returning to nocow, because a
> snapshot 
> locks the existing extents in place for the snapshot, so initial
> writes 
> to a block after a snapshot /can't be nocow or it'd change the
> snapshot 
> too...)
Ah that's good to know how that works (or better said, that it works at
all)... I've already wondered myself several times what happens when I
snapshot NOCOW files, ... something that's I guess also missing from
the (missing ;-) ) btrfs-end-user overview and details documentation.


> OTOH, if there's a chance you might want to mount that subvolume in a
> roll-back scenario, under the snapshot you're rolling back to, then
> it 
> makes sense to put it directly under ID 5 again, and mount it in any
> case.
Yes.


> Then there's the security angle to consider.  With the (basically, 
> possibly modified as I suggested) flat layout, mounting something
> doesn't 
> automatically give people in-tree access to nested subvolumes
> (subject to 
> normal file permissions, of course), like nested layout does.  And
> with 
> (possibly modified) flat layout, the whole subvolume tree doesn't
> need to 
> be mounted all the time either, only when you're actually working
> with 
> subvolumes.
Uhm, I don't get the big security advantage here... whether nested or
manually mounted to a subdir,... if the permissions are insecure I'll
have a problem... if they're secure, than not.

Of course I use insecure permissions and don't mount the subvols then
I'd be safe in flat setup, but not in a nested setup...(where they
subvol is "auto-mounted")...

But that seems pretty awkward.



Mhh I think my main question turns out to be whether the different
layouts would have any technical (i.e. not administrative) effects...
like the aforementioned stuff of recursive snapshots (should such thing
ever come to life).
But at least from the userspace tools it seems that I can move subvols
arbitrarily and they adapt their parent IDs accordingly...

So I guess whatever setup one uses nested/flat/mixed... doesn't rule
out any functionalities for the future?!

thx,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 7601 bytes --]

On Tue, Nov 24, 2015 at 10:25:50PM +0100, Christoph Anton Mitterer wrote:
> On Tue, 2015-11-24 at 08:29 +0000, Duncan wrote:
> > OK, found it on the wiki.  It wasn't under use-cases, where I
> > initially 
> > thought to look, but under sysadmin guide.  Specifically, see section
> > 4.2, managing snapshots, but I'd suggest reading the entire
> > subvolumes 
> > discussion, section 4, or even most/all of the page.
> > 
> > https://btrfs.wiki.kernel.org/index.php/SysadminGuide
> Well I've had read that, but it's pretty vague and especially doesn't
> mentioned any of the filesystem internal implications (if there are
> any)... like I wondered before, whether this has effects on ref-links
> not being used when e.g. sending/recieving ... or on future planned
> extensions like recursive snapshots.

   No, there's no particular implications one way or the other in
terms of reflinks. Obviously, if recursive snapshots get implemented,
it'll remove some of the current awkwardness with nested subvols, but
it won't invalidate any existing setups that use the recommendations
in the Sysadmin Guide.

> > Suppose you only want to rollback /, because some update screwed you
> > up, 
> > but not /home, which is fine.  If /home is a nested subvolume, then 
> > you're now mounting the nested home subvolume from some other nesting
> > tree entirely,
> That's a bit unclear to me,... I thought when I make a snapshot, any
> nested subvols wouldn't be snapshotted and thus be empty dirs.

   Correct. This is actually a use-case for nested subvols, and one
which snapper uses -- the target for snapshots of /foo is
/foo/.snapshots/$date, where /foo/.snapshots is a subvol in its own
right. So, if you have a subdir which you won't want to include in
snapshots of a subvol, make it a subvol itself.

> So I'd have rather that if I would simply have no /home (if I didn't
> move it to the rolled-back subvol manually)

   Yup, this is one of the main reasons for not nesting subvols.

> > 5
> > > 
> > +-roots (dir not subvol, note the s, rootS, plural)
> > > +-root (subvol, mountpoint /)
> > > > +-boot/ (dir)
> > > > +-root/ (dir)
> > > > +-lib/  (dir)
> > > > +-home/ (empty dir as mountpoint)
> > > +-root-snapshot-2015.0301 (dated snapshot of root)
> > > +-root-snapshot-2015.0601 (dated snapshot of root)
> > > +-root-snapshot-2015.0901 (dated snapshot of root)
> > +-homes (dir not subvol)
> > > +-home (subvol, mountpoint /home)
> > > +-home-snapshot-2015.0301 (dated snapshot of home)
> > ...
> That's more what I've had in mind...
> Actually something like this:
> 5
> |
> +-root       (=subvol)
> | +-boot
> | +-home     (subvo=/home being mounted heron)
> | +-lib
> +-home       (subvol, the current version)
> +-snapshots  (=dir)
>   +-root
>   | +-2015-01-14 (subvol, snapshot)
>   | +-2015-09-30 (subvol, snapshot)
>   +-home
>     +-2015-06-04 (subvol, snapshot)
>     +-2015-09-01 (subvol, snapshot)
> 
> 
> And it once more points to the problem of the wiki... anyone can write
> (I think even I) and it's totally unclear at the first glance how
> "[non-]official" and outdated something may be.
> Apart from the problem that many important questions (from the PoV of
> an more advanced admin that doesn't just do mkfs.btrfs and then never
> again thinks about it) :-(

   In practice, new content is checked by a number of people when it's
put in, so the case of someone putting random poorly-thought-out crap
in the wiki isn't particularly likely to happen.

> > Meanwhile, if the intention for a subvolume is simply to exclude that
> > subtree from snapshotting of the parent, as might be the case for
> > example 
> that is in fact also use case.. so in practise I'll probably have a mix
> of (a) and (b).
> 
> 
> > if you have a VMs subvol, with the VM image files set NOCOW to avoid 
> > fragmentation, since snapshotting nocow files forces cow1 (a cow at
> > the 
> > first write of that block, before returning to nocow, because a
> > snapshot 
> > locks the existing extents in place for the snapshot, so initial
> > writes 
> > to a block after a snapshot /can't be nocow or it'd change the
> > snapshot 
> > too...)
> Ah that's good to know how that works (or better said, that it works at
> all)... I've already wondered myself several times what happens when I
> snapshot NOCOW files, ... something that's I guess also missing from
> the (missing ;-) ) btrfs-end-user overview and details documentation.

   Please feel free to add the things you'd like to see. As I said
above, we do check the docs on the wiki as they're changed, so if
you're wrong on some details, it won't be a major issue for very
long. If you want to discuss details before you write something, start
a conversation -- either on here, or on IRC (or even on the Talk pages
of the wiki).

> > OTOH, if there's a chance you might want to mount that subvolume in a
> > roll-back scenario, under the snapshot you're rolling back to, then
> > it 
> > makes sense to put it directly under ID 5 again, and mount it in any
> > case.
> Yes.
> 
> 
> > Then there's the security angle to consider.  With the (basically, 
> > possibly modified as I suggested) flat layout, mounting something
> > doesn't 
> > automatically give people in-tree access to nested subvolumes
> > (subject to 
> > normal file permissions, of course), like nested layout does.  And
> > with 
> > (possibly modified) flat layout, the whole subvolume tree doesn't
> > need to 
> > be mounted all the time either, only when you're actually working
> > with 
> > subvolumes.
> Uhm, I don't get the big security advantage here... whether nested or
> manually mounted to a subdir,... if the permissions are insecure I'll
> have a problem... if they're secure, than not.
> 
> Of course I use insecure permissions and don't mount the subvols then
> I'd be safe in flat setup, but not in a nested setup...(where they
> subvol is "auto-mounted")...
> 
> But that seems pretty awkward.
> 
> 
> 
> Mhh I think my main question turns out to be whether the different
> layouts would have any technical (i.e. not administrative) effects...
> like the aforementioned stuff of recursive snapshots (should such thing
> ever come to life).
> But at least from the userspace tools it seems that I can move subvols
> arbitrarily and they adapt their parent IDs accordingly...

   Pretty much, yes.

   Note that the "parent" of send -p and of snapshots is not the same
relationship as the "parent" (containing subvol) of the tree
structure. This is an awkward nomenclature problem, and I'm not sure
how to fix it.

   The first meaning is fixed between a subvol and its snapshot when
the snapshot is created, and won't change, regardless of how the
snapshots are renamed/moved.

   The second meaning... actually, that doesn't change either, because
you can't rename a subvol across another subvol boundary. (I only just
realised this, halfway through the explanation).

   They're still different things, though. :)

> So I guess whatever setup one uses nested/flat/mixed... doesn't rule
> out any functionalities for the future?!

   Correct, it doesn't.

   Hugo.

-- 
Hugo Mills             | Anyone who claims their cryptographic protocol is
hugo@... carfax.org.uk | secure is either a genius or a fool. Given the
http://carfax.org.uk/  | genius/fool ratio for our species, the odds aren't
PGP: E2AB1DE4          | good.                                  Bruce Schneier

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: subvols and parents - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 2824 bytes --]

On Tue, 2015-11-24 at 21:55 +0000, Hugo Mills wrote:
>    In practice, new content is checked by a number of people when
> it's
> put in, so the case of someone putting random poorly-thought-out crap
> in the wiki isn't particularly likely to happen.
Well... it may work in 99% cases... but there could something slip
through, which isn't as easy the case in manpages, which also tend to
be less messy than the huge pile of wiki pages where similar/related
things are described on different pages.

Imagine a case, a non-experienced user update the wiki saying that --
repair should be used, he may not even doing it in bad faith, perhaps
he had success with it and now writes a recipe.
It may take a while until someone of the more experienced guys notices
that and corrects it.
But if ", in the meantime had some fs corruptions,... I may experience
already severe problems by following that suggestion... (and while I do
have many backups of all my data, others may not, and if their life's
data is concerned, they'd be screwed).

So even if it takes you just a few hours to correct such rubbish, you
know that Murphy's law may still hit n people during that time ;-)


> Please feel free to add the things you'd like to see. As I said
> above, we do check the docs on the wiki as they're changed, so if
> you're wrong on some details, it won't be a major issue for very
> long. If you want to discuss details before you write something,
> start
> a conversation -- either on here, or on IRC (or even on the Talk
> pages
> of the wiki).
Well I can write a list together of things which I think should be part
of some more general documentation (i.e. less documentation about
options of the tools)... questions a complete newcomer to btrfs may
have who needs however more than "just a filesystem".


>    Note that the "parent" of send -p and of snapshots is not the same
> relationship as the "parent" (containing subvol) of the tree
> structure. This is an awkward nomenclature problem, and I'm not sure
> how to fix it.
Yeah, that was clear... :-)
Maybe call the "parent" from send -p "base" or something like that...
IMHO that would fit more as the parent there is more like a
"fundament".

Anyway, it's still not as bad as the usage of "RAID1" ;-)


> because
> you can't rename a subvol across another subvol boundary.
That's not quite clear to me... I had subvols like that:
/top/root/below-root
/top/below-top
and was able to move that to:
/top/root/below-top
/top/below-root

i.e. not just changing names but swapping as in:
mv /top/root/below-top /top/tmp
mv /top/below-root /top/root/below-root
mv /top/tmp /top/below-top

with top, root, below-top and below-root all being the same subvols


Thanks a lot for your explanations :)

Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3901 bytes --]

On Wed, Nov 25, 2015 at 12:20:09AM +0100, Christoph Anton Mitterer wrote:
> On Tue, 2015-11-24 at 21:55 +0000, Hugo Mills wrote:
> >    In practice, new content is checked by a number of people when
> > it's
> > put in, so the case of someone putting random poorly-thought-out crap
> > in the wiki isn't particularly likely to happen.
> Well... it may work in 99% cases... but there could something slip
> through, which isn't as easy the case in manpages, which also tend to
> be less messy than the huge pile of wiki pages where similar/related
> things are described on different pages.
> 
> Imagine a case, a non-experienced user update the wiki saying that --
> repair should be used, he may not even doing it in bad faith, perhaps
> he had success with it and now writes a recipe.
> It may take a while until someone of the more experienced guys notices
> that and corrects it.

   You can get update notifications, with the diffs for each page, via
RSS. (At least, I do, and I think David and a few others monitor it in
the same way). The window of failure is fairly small, particularly in
view of the number of such "dangerous" changes made. The total
vulnerability is measured in hours per year...

> But if ", in the meantime had some fs corruptions,... I may experience
> already severe problems by following that suggestion... (and while I do
> have many backups of all my data, others may not, and if their life's
> data is concerned, they'd be screwed).
> 
> So even if it takes you just a few hours to correct such rubbish, you
> know that Murphy's law may still hit n people during that time ;-)
> 
> 
> > Please feel free to add the things you'd like to see. As I said
> > above, we do check the docs on the wiki as they're changed, so if
> > you're wrong on some details, it won't be a major issue for very
> > long. If you want to discuss details before you write something,
> > start
> > a conversation -- either on here, or on IRC (or even on the Talk
> > pages
> > of the wiki).
> Well I can write a list together of things which I think should be part
> of some more general documentation (i.e. less documentation about
> options of the tools)... questions a complete newcomer to btrfs may
> have who needs however more than "just a filesystem".
> 
> 
> >    Note that the "parent" of send -p and of snapshots is not the same
> > relationship as the "parent" (containing subvol) of the tree
> > structure. This is an awkward nomenclature problem, and I'm not sure
> > how to fix it.
> Yeah, that was clear... :-)
> Maybe call the "parent" from send -p "base" or something like that...
> IMHO that would fit more as the parent there is more like a
> "fundament".

   Yes, that makes sense.

> Anyway, it's still not as bad as the usage of "RAID1" ;-)
> 
> 
> > because
> > you can't rename a subvol across another subvol boundary.
> That's not quite clear to me... I had subvols like that:
> /top/root/below-root
> /top/below-top
> and was able to move that to:
> /top/root/below-top
> /top/below-root
> 
> i.e. not just changing names but swapping as in:
> mv /top/root/below-top /top/tmp
> mv /top/below-root /top/root/below-root
> mv /top/tmp /top/below-top
> 
> with top, root, below-top and below-root all being the same subvols

   With a recent mv, it'll be doing doing reflink copies followed by
delete for all of the contents, which makes it pretty efficient, but I
think you'll find that the UUID of the subvols changes. (At least, I
hope it does. If it doesn't, then my mental model of what the FS is
doing is *really* screwed up).

   Hugo.

> Thanks a lot for your explanations :)
> 
> Chris.



-- 
Hugo Mills             | Never underestimate the bandwidth of a Volvo filled
hugo@... carfax.org.uk | with backup tapes.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: subvols and parents - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 2783 bytes --]

On Tue, 2015-11-24 at 23:30 +0000, Hugo Mills wrote:
>    Yes, that makes sense.
Feel free to shamelessly use my idea (as well as the one to call btrfs'
RAID1 replica2 or something else)
:-O


>    With a recent mv
root@heisenberg:/mnt# mv --version
mv (GNU coreutils) 8.23

=> not recent enough...


> but I think you'll find that the UUID of the subvols changes. (At
> least, I hope it does. If it doesn't, then my mental model of what
> the FS is doing is *really* screwed up).
Well... see below:

root@heisenberg:~# truncate  -s 2G image
root@heisenberg:~# losetup -f image 
root@heisenberg:~# mkfs.btrfs /dev/loop0 
btrfs-progs v4.3
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM (2.00GiB) ...
Label:              (null)
UUID:               10e1a55c-448a-4f37-ae5c-6a7801a7f202
Node size:          16384
Sector size:        4096
Filesystem size:    2.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP             110.38MiB
  System:           DUP              12.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     2.00GiB  /dev/loop0

root@heisenberg:~# mount /dev/loop0 /mnt/
root@heisenberg:/mnt# btrfs subvolume create root
Create subvolume './root'
root@heisenberg:/mnt# btrfs subvolume create below-top
Create subvolume './below-top'
root@heisenberg:/mnt# cd root/
root@heisenberg:/mnt/root# btrfs subvolume create below-root
Create subvolume './below-root'
root@heisenberg:/mnt# btrfs subvolume list /mnt/ -pacguqt
ID	gen	cgen	parent	top level	parent_uuid	uuid	path	
--	---	----	------	---------	-----------	----	----	
257	9	7	5	5		-	8fbf521e-77f9-0d49-9891-87767f98c655	root
258	8	8	5	5		-	b49131e9-4207-aa42-8195-c50de5f06136	below-top
259	9	9	257	257		-	20c042be-ead8-204a-a684-94c1a770e739	<FS_TREE>/root/below-root

root@heisenberg:/mnt# mv root/below-root/ tmp
root@heisenberg:/mnt# mv below-top/ root/
root@heisenberg:/mnt# mv tmp/ below-root
root@heisenberg:/mnt# btrfs subvolume list /mnt/ -pacguqt
ID	gen	cgen	parent	top level	parent_uuid	uuid	path	
--	---	----	------	---------	-----------	----	----	
257	9	7	5	5		-	8fbf521e-77f9-0d49-9891-87767f98c655	root
258	8	8	257	257		-	b49131e9-4207-aa42-8195-c50de5f06136	<FS_TREE>/root/below-top
259	9	9	5	5		-	20c042be-ead8-204a-a684-94c1a770e739	below-root
root@heisenberg:/mnt# 


So the UUIDs seem to stay the same (or are these other UUIDs?)

Hope I haven't ruined your day now ;-)


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Duncan]

Christoph Anton Mitterer posted on Tue, 24 Nov 2015 22:25:50 +0100 as
excerpted:

>> Suppose you only want to rollback /, because some update screwed you
>> up,
>> but not /home, which is fine.  If /home is a nested subvolume, then
>> you're now mounting the nested home subvolume from some other nesting
>> tree entirely,
> That's a bit unclear to me,... I thought when I make a snapshot, any
> nested subvols wouldn't be snapshotted and thus be empty dirs.
> So I'd have rather that if I would simply have no /home (if I didn't
> move it to the rolled-back subvol manually)

What I was intending to convey but apparently failed to be quite clear 
enough, suppose:

5
|
+-+ subvols (dir)
  |
  +-+ root (subvol)
  | |
  | + home (nested subvol)
  |
  +-+ snaps-2015.0901 (dir)
    |
    +-+ root-2015.0901 (subvol)


As long as you're on the working /, then /home is a nested subvol, and 
you don't have to mount it to access, tho you can if you want.

But now, you roll back to snaps-2015.0901/root-2015.0901.

It won't have /home nested underneath, as you correctly pointed out, but 
in ordered to access it, you now MUST mount /home, which...

#1 could be a pain to setup if you weren't actually mounting it 
previously, just relying on the nested tree, AND...

#2 The point I was trying to make, now, to mount it you'll mount not a 
native nested subvol, and not a directly available sibling
5/subvols/home, but you'll actually be reaching into an entirely 
different nesting structure to grab something down inside, mounting
5/subvols/root/home subvolume nesting down inside the direct
5/subvols/root sibling subvol.

With just one level of nesting and one additional mount, it's not too 
hard to keep track of, but if you're dealing with four or five levels of 
subvol nesting and some of them you're mounting the working head copy 
while others you're rolling back, it could get difficult to keep straight 
in your head what's going on.

Consider a layout like this:

5
+-+ subvols (dir)
  |
  +-+ root (subvol)
  | |
  | +-+ home (subvol)
  | | |
  | | +-+ henry (dir, no subvol for henry)
  | | |
  | | +-+ fred (subvol)
  | | | |
  | | | +-+ vms (subvol)
  | | | 
  | | +-+ betty (subvol)
  | |
  | +-+ svr (subvol)
  |   |
  |   +-+ vms (subvol)
  |
  +-+ snaps-2015.0901 (dir)
    |
    +-+ root-2015.0901 (subvol here and below)
    |
    +-+ home-2015.0901
    |
    +-+ fred-2015.0901
    |
    +-+ fred-vms-2015.0901
    |
    +-+ betty-2015.0901
    |
    +-+ svr-2015.0901
    |
    +-+ svr-vms-2015.0901


Now, you were hacked and they encrypted a bunch of stuff, but you were 
lucky and caught them before they got everything.  You need to roll back 
root but not home, fred is fine, but his vms subvol needs rolled back, 
betty needs rolled back, svr needs rolled back, but svr's vms are fine.

Try to sort THAT out along with the nesting, and keep it all straight 
while under the severe pressure of trying to recover from a hack in time 
for those svr things to go live for Black Friday in a few hours, where in 
a single day you expect to make as much as you normally do in a month, 
the rest of the year!  The pressure is on!

Oh, and you weren't actually doing the mounts as you were depending on 
the nested tree, so you have to actually setup the mounts as well, not 
just switch them to point to the appropriate location.

OK, so that's a bit contrived, but server encryption for ransom hacks are 
in the news, black Friday starts in a few hours here, and I think the 
point should be obvious! =:^)

(Some years ago, before btrfs, I had something similar setup but with 
partitions.  Disaster struck and I ended up with / from one backup, /usr 
from another, and /var, with the package database of what was installed 
on the other two, from current, or something like that.  Needless to say 
I learned quite some lessons from that, one of which was that everything 
that the package manager installs should be on the same partition with 
the installed-package database, so if it has to be restored from backup, 
at least if it's all old, at least it's all equally old, and the package 
database actually matches what's on the system because it's in the same 
backup!  That partition and btrfs, along with each of its various 
backups, are now 8 GiB each, so it's not like I'll run out of room with 
several levels of backup.  I went mdraid after that, but after an initial 
experiment with lvm on top of the raid, I decided that was too complex to 
deal with in the pressure of a disaster and redid it to multiple raids on 
parallel partitioned hardware.  In a disaster the raid would be bad 
enough to deal with but tolerable, but I did NOT need the complexity of 
lvm on top of raid, and after dealing with the parts of three different 
installs mess, I had the hard-earned wisdom to realize it!

The same idea applies here.  Once you start reaching into nested subvols 
to get the deeper nested subvols you're trying to mount, it's too much 
and you're just begging to get it wrong under the extreme pressures of a 
disaster recovery.  Do it right and keep it mostly flat, and you'll not 
only know your basic mounts are already setup and you only have to point 
them at the rollbacks if appropriate, but you won't be having to reach 
deep into multiple arbitrary nestings to grab subvols from weird spots.  
Disaster recovery you might well be thanking me, come a few months or a 
few years from now!  =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: subvols and parents - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 2882 bytes --]

On Fri, 2015-11-27 at 01:02 +0000, Duncan wrote:
[snip snap]
> #1 could be a pain to setup if you weren't actually mounting it
> previously, just relying on the nested tree, AND...
> 
> #2 The point I was trying to make, now, to mount it you'll mount not
> a 
> native nested subvol, and not a directly available sibling
> 5/subvols/home, but you'll actually be reaching into an entirely 
> different nesting structure to grab something down inside, mounting
> 5/subvols/root/home subvolume nesting down inside the direct
> 5/subvols/root sibling subvol.

Okay so your main point was basically "keeping things administrable"...


> one of which was that everything 
> that the package manager installs should be on the same partition
> with 
> the installed-package database, so if it has to be restored from
> backup, 
> at least if it's all old, at least it's all equally old, and the
> package 
> database actually matches what's on the system because it's in the
> same 
> backup!
I basically agree, though I'd allow few exceptions, like database-like
data that is stored in /var/ sometimes and that doesn't need to be
consistent with anything but iself... e.g. static web pages
(/var/www)... postgresl DB, or sks keyserver DB... and so on.

btw: What's the proper way for merging / splitting into subvols.
E.g. consider I have:
5
|
+--root (subvol)
   |
   +-- var (no subvol)

And say I would want to split of var/www into a subvol.
Well one obvious way would be with mv (and AFAIU that would keep my
ref-links with clones, if any) but that also means that anything that
accesses /var/www probably needs a downtime.
Is it planned to have a special function that basically says:
"make dir foo and anything below (except nested subvols) a subvol named
foo, immediately and atomically"?

And similar vice-versa... a special function that says:
"make subvol foo and anything below (except nested subvols) a dir of
the parent subvol named foo, immediately and atomically"?

Could be handy for real world administration, especially when one
want's
to avoid downtimes.

btw: Few days ago, either Hugo or your thought that mv'ing a subvol
would change it's UUID, but my try (which was with coreutils 8.3 -> no
reflinked mv) seemed to show it wouldn't but there was no further reply
then... so am I right that the UUID wouldn't change?


> The same idea applies here.  Once you start reaching into nested
> subvols 
> to get the deeper nested subvols you're trying to mount, it's too
> much 
> and you're just begging to get it wrong under the extreme pressures
> of a 
> disaster recovery.
Well apparently you oversaw the extremely simple and reliable solution:
leaving a tiny little note on your desk saying something like: "dear
boss, things are screwed up, I'm on vacation now..." ;-)


Thanks,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Duncan]

Christoph Anton Mitterer posted on Wed, 09 Dec 2015 05:36:37 +0100 as
excerpted:

> On Fri, 2015-11-27 at 01:02 +0000, Duncan wrote:
> [snip snap]
>> #2 The point I was trying to make, now, to mount it you'll mount not a
>> native nested subvol, and not a directly available sibling
>> 5/subvols/home, but you'll actually be reaching into an entirely
>> different nesting structure to grab something down inside, mounting
>> 5/subvols/root/home subvolume nesting down inside the direct
>> 5/subvols/root sibling subvol.
> 
> Okay so your main point was basically "keeping things administrable"...

=:^)

>> one of which was that everything that the package manager installs
>> should be on the same partition with the installed-package database, so
>> if it has to be restored from backup, at least if it's all old, at
>> least it's all equally old, and the package database actually matches
>> what's on the system because it's in the same backup!

> I basically agree, though I'd allow few exceptions, like database-like
> data that is stored in /var/ sometimes and that doesn't need to be
> consistent with anything but iself... e.g. static web pages
> (/var/www)... postgresl DB, or sks keyserver DB... and so on.

Of course.  But such data isn't usually managed by the package manager, 
so splitting off a partition or subvolume or whatever, is fair game.

IOW, /var can and should still be on /, but /var/www and the like can be 
on some other subvolume or filesystem.

> 
> btw: What's the proper way for merging / splitting into subvols.
> E.g. consider I have:
> 5
> |
> +--root (subvol)
>    |
>    +-- var (no subvol)
> 
> And say I would want to split of var/www into a subvol.

> Well one obvious way would be with mv (and AFAIU that would keep my
> ref-links with clones, if any) but that also means that anything that
> accesses /var/www probably needs a downtime.
> Is it planned to have a special function that basically says:
> "make dir foo and anything below (except nested subvols) a subvol named
> foo, immediately and atomically"?

AFAIK, no.

There's a similar, but not identical, question covered on the wiki, use-
cases page (link very likely wrapped as I'm not bothering to jump thru 
the hoops to unwrap while keeping everything else formatted, ATM).

https://btrfs.wiki.kernel.org/index.php/
UseCases#Can_I_take_a_snapshot_of_a_directory.3F

If you use the recipe (subvol create, cp with reflink) it suggests there, 
you'll end up with the reflinked copy in a subvol.

You can then mount that subvol over top of the existing dir, and *new* 
file opens will access the new subvol, tho existing open files will of 
course continue to access the files/reflinks to which they have a 
reference, "underneath" the new mount.

For some services it's possible to signal them to reload their files.  
Where this is possible, you can do the overmount trick and then signal 
them to reload, and they should keep running, otherwise undisturbed (altho 
any changes between the reflink and processing the signal will still go 
to the existing open files, I don't believe there's a way around that).

But AFAIK that's the closest it gets, and nothing more along that line is 
planned.

In general, keep in mind that subvolumes work in most respects very much 
like normal directories do, except where they don't. =:^)  If it can't be 
done with normal directories and it's not specifically part of subvol/
snapshot functionality, it very likely can't be done with subvols 
either.  From my read, that's the general rule that applies here.

> And similar vice-versa... a special function that says:
> "make subvol foo and anything below (except nested subvols) a dir of the
> parent subvol named foo, immediately and atomically"?
> 
> Could be handy for real world administration, especially when one want's
> to avoid downtimes.
> 
> btw: Few days ago, either Hugo or your thought that mv'ing a subvol
> would change it's UUID, but my try (which was with coreutils 8.3 -> no
> reflinked mv) seemed to show it wouldn't but there was no further reply
> then... so am I right that the UUID wouldn't change?

That was Hugo.  I read your reply then with interest, but my own use-case 
doesn't involve subvols, so I'd be unlikely to know that level of 
specific detail, and even more unlikely to post about it unless I was 
quite sure on it (as I am about the above due to the wiki reference and 
previous list discussion).

>> The same idea applies here.  Once you start reaching into nested
>> subvols to get the deeper nested subvols you're trying to mount, it's
>> too much and you're just begging to get it wrong under the extreme
>> pressures of a disaster recovery.

> Well apparently you oversaw the extremely simple and reliable solution:
> leaving a tiny little note on your desk saying something like: "dear
> boss, things are screwed up, I'm on vacation now..." ;-)

=:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: subvols and parents - how?

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2054 bytes --]

On 2015-12-09 05:53, Duncan wrote:
> Christoph Anton Mitterer posted on Wed, 09 Dec 2015 05:36:37 +0100 as
> excerpted:
> 
>> On Fri, 2015-11-27 at 01:02 +0000, Duncan wrote:
>> [snip snap]
>>> #2 The point I was trying to make, now, to mount it you'll mount not a
>>> native nested subvol, and not a directly available sibling
>>> 5/subvols/home, but you'll actually be reaching into an entirely
>>> different nesting structure to grab something down inside, mounting
>>> 5/subvols/root/home subvolume nesting down inside the direct
>>> 5/subvols/root sibling subvol.
>>
>> Okay so your main point was basically "keeping things administrable"...
> 
> =:^)
> 
>>> one of which was that everything that the package manager installs
>>> should be on the same partition with the installed-package database, so
>>> if it has to be restored from backup, at least if it's all old, at
>>> least it's all equally old, and the package database actually matches
>>> what's on the system because it's in the same backup!
> 
>> I basically agree, though I'd allow few exceptions, like database-like
>> data that is stored in /var/ sometimes and that doesn't need to be
>> consistent with anything but iself... e.g. static web pages
>> (/var/www)... postgresl DB, or sks keyserver DB... and so on.
> 
> Of course.  But such data isn't usually managed by the package manager, 
> so splitting off a partition or subvolume or whatever, is fair game.
> 
> IOW, /var can and should still be on /, but /var/www and the like can be 
> on some other subvolume or filesystem.
> 
Agreed.  It's not too bad fixing a Gentoo system (as long as
/var/lib/portage/world is still correct, you can just nuke the installed
package database and emerge world, it'll take time, but it will get your
system in a guaranteed consistent state).  On something using dpkg or
rpm though, it's got the potential to be almost impossible to recover
from without a significant amount of low-level knowledge of the package
manager's installation database.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: subvols and parents - how?

[Duncan]

Austin S Hemmelgarn posted on Wed, 09 Dec 2015 14:04:06 -0500 as
excerpted:

> Agreed.  It's not too bad fixing a Gentoo system (as long as
> /var/lib/portage/world is still correct, you can just nuke the installed
> package database and emerge world, it'll take time, but it will get your
> system in a guaranteed consistent state).

For sufficiently loose values of "consistent", yes, as I found out by 
experience.  But it can be done, and I do have the experience to prove it.

What happens in practice is that while yes, as long as @world is correct 
you can install to current and have all those files tracked again as 
appropriate, if your package installation database is missing or out of 
sync with what's actually on your filesystem(s), where the new version of 
various packages will replace older files as they come across them during 
the install process (subject to CONFIG_PROTECT of course, this part isn't 
the problem), the problem is actually where the files of the actually 
installed but untracked version differ from those of the version you're 
installing.

I think the last bug I tracked down to having a stale file from an old 
version still around, was nearly two years after the initial disaster and 
recovery.  It was some shell script (something to do with eselect IIRC) 
that had changed installed bindir location between package versions, and 
the bug was that due to path ordering, the stale version that hadn't been 
replaced by the reinstall and wasn't removed like it normally would have 
been when the old version was uninstalled, because my installed package 
database was out of sync with the backup I had used, was being executed 
as that bindir came earlier in the path, than the bindir for the updated 
version that was in the current package.

So yes it did work and I was back in business, but I was tracing bugs 
down due to stray orphan files that hadn't been cleaned up by the out of 
sync installation database package uninstall, for two years!

I wouldn't exactly call that "consistent", tho it /was/ consistent 
/enough/ to get back in business, if with lingering bugs to track down to 
still stale orphan files for years afterward.

(FWIW, that'd be less of an issue on my own installation now, even if the 
installed package database got similarly out of sync, because on my 
system I'm running a unified root/usr and bin/sbin, now, with symlinks 
/usr -> . and /sbin -> bin, so all normal executables ultimately end up 
in /bin, and all normal libraries in /lib64, with /lib -> lib64, as 
well.  No-multilib so I don't have to worry about that angle.  Portage 
handles the symlinks just fine, tho I've filed I believe three bugs in 
total due to individual ebuild scripts (1, resolved/fixed) or cmake (2, 
one resolved/obsolete as the new version works, one still open, that I'm 
having to work around) doing the wrong thing.)


> On something using dpkg or
> rpm though, it's got the potential to be almost impossible to recover
> from without a significant amount of low-level knowledge of the package
> manager's installation database.





-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: subvols and parents - how?

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]

On 2015-12-09 22:56, Duncan wrote:
> Austin S Hemmelgarn posted on Wed, 09 Dec 2015 14:04:06 -0500 as
> excerpted:
> 
>> Agreed.  It's not too bad fixing a Gentoo system (as long as
>> /var/lib/portage/world is still correct, you can just nuke the installed
>> package database and emerge world, it'll take time, but it will get your
>> system in a guaranteed consistent state).
> 
> For sufficiently loose values of "consistent", yes, as I found out by 
> experience.  But it can be done, and I do have the experience to prove it.
> 
> What happens in practice is that while yes, as long as @world is correct 
> you can install to current and have all those files tracked again as 
> appropriate, if your package installation database is missing or out of 
> sync with what's actually on your filesystem(s), where the new version of 
> various packages will replace older files as they come across them during 
> the install process (subject to CONFIG_PROTECT of course, this part isn't 
> the problem), the problem is actually where the files of the actually 
> installed but untracked version differ from those of the version you're 
> installing.
Oh, definately, it's a useable system short term, but not something you
should be depending on.  The other big difference though is that it's
then trivial to bootstrap a clean install on the same system if you have
the space for it (which is what I usually end up doing).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: subvols and parents - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1718 bytes --]

On Wed, 2015-12-09 at 10:53 +0000, Duncan wrote:
> If you use the recipe (subvol create, cp with reflink) it suggests
> there, 
> you'll end up with the reflinked copy in a subvol.
> 
> You can then mount that subvol over top of the existing dir, and
> *new* 
> file opens will access the new subvol, tho existing open files will
> of 
> course continue to access the files/reflinks to which they have a 
> reference, "underneath" the new mount.
Sure, which would mean however that a downtime is still necessary.



> For some services it's possible to signal them to reload their
> files.  
> Where this is possible, you can do the overmount trick and then
> signal 
> them to reload, and they should keep running, otherwise undisturbed
> (altho 
> any changes between the reflink and processing the signal will still
> go 
> to the existing open files, I don't believe there's a way around
> that).
Yep,.. but as you say,... it doesn't really help to avoid the
downtime,... rather lead to data corruption (not on the filesystem
level, of course, but within the application).


> But AFAIK that's the closest it gets, and nothing more along that
> line is 
> planned.
I've been so free to add that idea to the project ideas:
https://btrfs.wiki.kernel.org/index.php/Project_ideas#.28Atomically.29_convert_directories_into_subvolumes_and_vice_versa

any developer or more experienced user than me is of course free to
remove that again, if it seems not so important or isn't possible to be
implemented.


> In general, keep in mind that subvolumes work in most respects very
> much 
> like normal directories do, except where they don't. =:^)
:-P :-P 


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Duncan]

Christoph Anton Mitterer posted on Tue, 24 Nov 2015 22:25:50 +0100 as
excerpted:

>> Then there's the security angle to consider.  With the (basically,
>> possibly modified as I suggested) flat layout, mounting something
>> doesn't automatically give people in-tree access to nested subvolumes
>> (subject to normal file permissions, of course), like nested layout
>> does.  And with (possibly modified) flat layout, the whole subvolume
>> tree doesn't need to be mounted all the time either, only when you're
>> actually working with subvolumes.

> Uhm, I don't get the big security advantage here... whether nested or
> manually mounted to a subdir,... if the permissions are insecure I'll
> have a problem... if they're secure, than not.

Consider a setuid-root binary with a recently publicized but patched on 
your system vuln.  But if you have root snapshots from before the patch 
and those snapshots are nested below root, then they're always 
accessible.  If the path to the vulnerable setuid is as user accessible 
as it likely was in its original location, then anyone with login access 
to the system is likely to be able to run it from the snapshot... and 
will be able to get root due to the vuln.

On a flat layout, a snapshot with the vuln would have to be mounted 
before it could be accessed, as otherwise it'd be outside the mounted 
tree.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: subvols and parents - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

On Fri, 2015-11-27 at 02:02 +0000, Duncan wrote:
> Uhm, I don't get the big security advantage here... whether nested
> > or
> > manually mounted to a subdir,... if the permissions are insecure
> > I'll
> > have a problem... if they're secure, than not.
> Consider a setuid-root binary with a recently publicized but patched
> on 
> your system vuln.  But if you have root snapshots from before the
> patch 
> and those snapshots are nested below root, then they're always 
> accessible.  If the path to the vulnerable setuid is as user
> accessible 
> as it likely was in its original location, then anyone with login
> access 
> to the system is likely to be able to run it from the snapshot... and
> will be able to get root due to the vuln.

Hmm good point... I think it would be great if you could add that
scenario somewhere to the documentation. :-)
Based on that one can easily think about more/similar examples...
device file that had too permissive modes set, and where snapshotted
like that... and so on.

I think that's another example why it would be nice if btrfs had
something (per subvolume) like ext4's default mount options (I mean the
ones stored in the superblock).

Not only would it allow the userland tools to do things like "adding
notatime" per default on snapshots (at least ro snapshot), so that one
can have them nested and still doesn't suffer from the previously
discussed writes-on-read-amplifications... it would also allow to set
things like nodev, noexec, nosuid and that like on subvols... and again
it would make the whole thing practically usable with nested subvols.


Where would be the appropriate place to record that as a feature
request?
Simply here on the list?


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Duncan]

Christoph Anton Mitterer posted on Wed, 09 Dec 2015 05:38:02 +0100 as
excerpted:

> On Fri, 2015-11-27 at 02:02 +0000, Duncan wrote:

>> Consider a setuid-root binary with a recently publicized but patched on
>> your system vuln.  But if you have root snapshots from before the patch
>> and those snapshots are nested below root, then they're always
>> accessible.  If the path to the vulnerable setuid is as user accessible
>> as it likely was in its original location, then anyone with login
>> access to the system is likely to be able to run it from the
>> snapshot... and will be able to get root due to the vuln.
> 
> Hmm good point... I think it would be great if you could add that
> scenario somewhere to the documentation. :-)

FWIW, I (personally, not sure if that "you" was singular or plural) am 
much more comfortable on the lists than on wikis, which I tend to treat 
much as I did printed encyclopedias back in the day -- as great 
references, but read-only from my perspective.  So while I know they can 
be edited in theory, I just don't tend to get around to it, and don't 
believe I've ever even gotten a wiki login, on /any/ wiki, let alone the 
btrfs wiki.  But as you've likely noticed, I do find them great 
references to read and to post links to. =:^)

So while I'm not opposed in principle to editing the wiki, in practice 
I'm never going to do it.  I'll reply here with the info, and if it gets 
to the wiki, it's almost certainly because someone else posted it there, 
based on my post here, or not.

> Based on that one can easily think about more/similar examples... device
> file that had too permissive modes set, and where snapshotted like
> that... and so on.

Exactly...

> I think that's another example why it would be nice if btrfs had
> something (per subvolume) like ext4's default mount options (I mean the
> ones stored in the superblock).
> 
> Not only would it allow the userland tools to do things like "adding
> notatime" [...] it would also allow to set things like nodev, noexec,
> nosuid and that like on subvols... and again it would make the whole
> thing practically usable with nested subvols.
> 
> Where would be the appropriate place to record that as a feature
> request?
> Simply here on the list?

While the appropriate place would be on the wiki, where there's a page 
for that (here, the request is likely to be lost to history, while on the 
wiki it's in a nice list that both devs and other requesters can look at, 
a year, five years, a decade... into the future), in this case...

There's already a framework for it and a limited implementation, tho only 
a small subset of possible options are yet available.  The UI exposure is 
as btrfs property (see the btrfs-property manpage), with the properties, 
in general, implemented as extented attributes.

You can browse what's currently available, but as I said, it's pretty 
limited ATM, pretty much just enough to prove the concept and serve as a 
placeholder for more to be added, later.

Of course other informational and sometimes configurable bits are exposed 
in the developing btrfs sysfs tree, which is where most of the work of 
that sort has been focused recently.  But it too is somewhat immature and 
developing, tho at this point it seems to be further along than 
properties, partly because btrfs' own tools are using it at times.

Basically, both properties and the sysfs interface demonstrate btrfs' 
status as stabilizING and maturING, as that sort of thing really can't be 
implemented until the basics have settled down far enough to form a 
stable reference point, but not yet fully stable or mature, as both 
interfaces are as yet not fully developed, with more to be added.

But in both cases there's enough there to browse and get a reasonable 
idea how things are being exposed and how they'll work once they are. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: subvols, ro- and bind mounts - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 2587 bytes --]

Hey.

I'd have an additional question about subvols O:-)

Given the following setup:
5
|
+--root (subvol, /)
   +-- mnt (dir)

with the following done:
- init 1
- remount,ro / (i.e. the subvol root)
- mount /dev/btrfs-device /mnt (i.e. mount the top subvol at /mnt)

The following happened:
- / was ro-mounted (obviously, at least one thing that I had expected
  correctly)
- /mnt was ro-mounted either (and the /mnt/root/ nested subvol then as
  well).
  => why is /mnt (i.e. the top level subvol) mounted ro??
  => I would have expected that, since / (i.e. the subvol "root" is ro
     mounted), it's also ro mounted as the nested subvol below 5, i.e.
     my naive thinking was in terms of logic:
     "/ mounted ro" => "subvol root is mounted ro (everywhere)"
       => "thus /mnt/root/ is mounted ro as well"

However, the later doesn't seem to be true, cause then I did:
- remount,rw /mnt
=> now /mnt/*, including /mnt/root/* was rw moutned



So I guess my assumption of subvols behaving more or less as if they'd
be a fs (and thus mounted at one place ro => everywhere ro) is not
true, is it?

Do, ro,rw (and possibly others) instead only affect the respective
mountpoint?
And automatically any nested subvols of that mountpoint?

So I could have basically:
/mount-point1/subvol-a	; ro, noexec
/mount-point2/subvol-a	; rw, compress=yes
/root			; rw, compress=no
/root/here/it/is/nested/subvol-a ; (no mountpoint)

(with subvol-a being the same subvol)

And when I write via mount-point1 I'd get an error, but via mount-
point2 it works and in addition I get compression, while when writing
via the /root mountpoint, where it is nested, I'd get the rw and
compress=no from the "parent" mountpoint /root


Does that sounds correct?
It seems to make sense actually, though it's a bit unfamiliar... if I'm
not correctly wrong, than e.g. in terms of ext* I cannot have the same
fs mounted with different settings,... of course I cannot have it
mounted twice at all, but speaking of bind mounts.

So I guess, that when I'd do --bind mounts instead, I actually do get
the "old" behaviour, i.e. when the source is ro, then the --bind
mount's target is also forcibly ro.


Still, one unclear thing, why got /mnt mounted ro very above?



Thanks,
Chris.

btw: Not sure if I just missed it, but I guess the above should be more
or less documented, showing people that mounting subvols (especially
when mounting the same several times, either directly or as nested
subvol) has these implications.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols, ro- and bind mounts - how?

[S.J.]

> Hey.
>
> I'd have an additional question about subvols O:-)
>
> Given the following setup:
> 5
> |
> +--root (subvol, /)
>     +-- mnt (dir)
>
> with the following done:
> - init 1
> - remount,ro / (i.e. the subvol root)
> - mount /dev/btrfs-device /mnt (i.e. mount the top subvol at /mnt)
>
> The following happened:
> - / was ro-mounted (obviously, at least one thing that I had expected
>    correctly)
> - /mnt was ro-mounted either (and the /mnt/root/ nested subvol then as
>    well).
>    => why is /mnt (i.e. the top level subvol) mounted ro??
>    => I would have expected that, since / (i.e. the subvol "root" is ro
>       mounted), it's also ro mounted as the nested subvol below 5, i.e.
>       my naive thinking was in terms of logic:
>       "/ mounted ro" => "subvol root is mounted ro (everywhere)"
>         => "thus /mnt/root/ is mounted ro as well"
>
> However, the later doesn't seem to be true, cause then I did:
> - remount,rw /mnt
> => now /mnt/*, including /mnt/root/* was rw moutned
>
>
>
> So I guess my assumption of subvols behaving more or less as if they'd
> be a fs (and thus mounted at one place ro => everywhere ro) is not
> true, is it?
>
> Do, ro,rw (and possibly others) instead only affect the respective
> mountpoint?
> And automatically any nested subvols of that mountpoint?
>
> So I could have basically:
> /mount-point1/subvol-a	; ro, noexec
> /mount-point2/subvol-a	; rw, compress=yes
> /root			; rw, compress=no
> /root/here/it/is/nested/subvol-a ; (no mountpoint)
>
> (with subvol-a being the same subvol)
>
> And when I write via mount-point1 I'd get an error, but via mount-
> point2 it works and in addition I get compression, while when writing
> via the /root mountpoint, where it is nested, I'd get the rw and
> compress=no from the "parent" mountpoint /root
>
>
> Does that sounds correct?
> It seems to make sense actually, though it's a bit unfamiliar... if I'm
> not correctly wrong, than e.g. in terms of ext* I cannot have the same
> fs mounted with different settings,... of course I cannot have it
> mounted twice at all, but speaking of bind mounts.
>
> So I guess, that when I'd do --bind mounts instead, I actually do get
> the "old" behaviour, i.e. when the source is ro, then the --bind
> mount's target is also forcibly ro.
>
>
> Still, one unclear thing, why got /mnt mounted ro very above?
>
>
>
> Thanks,
> Chris.
>
> btw: Not sure if I just missed it, but I guess the above should be more
> or less documented, showing people that mounting subvols (especially
> when mounting the same several times, either directly or as nested
> subvol) has these implications.

Quote:

" Most mount options apply to the whole filesystem, and only the options 
for the first subvolume
to be mounted will take effect. This is due to lack of implementation 
and may change in the future. "

from https://btrfs.wiki.kernel.org/index.php/Mount_options in a red box 
on the top.

Re: subvols, ro- and bind mounts - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

On Thu, 2015-12-10 at 23:36 +0100, S.J. wrote:
> Quote:
> 
> " Most mount options apply to the whole filesystem, and only the
> options 
> for the first subvolume
> to be mounted will take effect. This is due to lack of implementation
> and may change in the future. "
> 
> from https://btrfs.wiki.kernel.org/index.php/Mount_options in a red
> box 
> on the top.

I've had read that, but it doesn't really make clear that that options
can effectively differ for the *same* subvol, when mounted several
times (or when appearing additionally as nested subvolume).

Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols, ro- and bind mounts - how?

[Chris Murphy]

On Thu, Dec 10, 2015 at 3:36 PM, S.J. <sorry@anonym.com> wrote:

> Quote:
>
> " Most mount options apply to the whole filesystem, and only the options for
> the first subvolume
> to be mounted will take effect. This is due to lack of implementation and
> may change in the future. "
>
> from https://btrfs.wiki.kernel.org/index.php/Mount_options in a red box on
> the top.

That seems due for a revision because I do rw, ro, rw, rw, ro mounts
in sequence and they stick fine. In fact they stick with the same
subvolume.

[root@f23m ]# mount /dev/sda7 /mnt/1 -o subvol=home
[root@f23m ]# mount /dev/sda7 /mnt/2 -o subvol=home,ro
[root@f23m ]# mount /dev/sda7 /mnt/3 -o subvol=home
[root@f23m ]# mount
[...snip...]
/dev/sda7 on /mnt/1 type btrfs
(rw,relatime,seclabel,ssd,space_cache,subvolid=258,subvol=/home)
/dev/sda7 on /mnt/2 type btrfs
(ro,relatime,seclabel,ssd,space_cache,subvolid=258,subvol=/home)
/dev/sda7 on /mnt/3 type btrfs
(rw,relatime,seclabel,ssd,space_cache,subvolid=258,subvol=/home)

And Project Atomic, a.k.a. ostree and rpm-ostree etc., depend on
mounting different parts of the same fs volume to different mounts
points with different read and read/write settings (bind mounts), and
that works too. http://projectatomic.io/


-- 
Chris Murphy

Re: subvols, ro- and bind mounts - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1703 bytes --]

On Thu, 2015-12-10 at 19:32 -0700, Chris Murphy wrote:
> That seems due for a revision because I do rw, ro, rw, rw, ro mounts
> in sequence and they stick fine. In fact they stick with the same
> subvolume.
> 
> [root@f23m ]# mount /dev/sda7 /mnt/1 -o subvol=home
> [root@f23m ]# mount /dev/sda7 /mnt/2 -o subvol=home,ro
> [root@f23m ]# mount /dev/sda7 /mnt/3 -o subvol=home
> [root@f23m ]# mount
> [...snip...]
> /dev/sda7 on /mnt/1 type btrfs
> (rw,relatime,seclabel,ssd,space_cache,subvolid=258,subvol=/home)
> /dev/sda7 on /mnt/2 type btrfs
> (ro,relatime,seclabel,ssd,space_cache,subvolid=258,subvol=/home)
> /dev/sda7 on /mnt/3 type btrfs
> (rw,relatime,seclabel,ssd,space_cache,subvolid=258,subvol=/home)


Not sure what you mean with "stick" here... I'd say the above has
simply the following semantics:
- the default for mounts is rw
- thus /mnt/1 and /mnt/3 are rw, and 3 isn't rw, because 1 was

In other words, if you change that to the following:
# mount /dev/sda7 /mnt/1 -o subvol=home,ro
# mount /dev/sda7 /mnt/2 -o subvol=home,ro
# mount /dev/sda7 /mnt/3 -o subvol=home
I'd expect that you get
ro
ro
rw

At least based on how I understood the whole system now.
That was actually my question here:
Q: In other words does mounting the same subvol *again* behave like --
bind mounts, i.e. the further mounts would get the options from the
first mount?

And I guess the A:(nswer) is: no, mount options affect the respective
mountpoint (including any nested subvols below that) only (except of
course it's a --bind mount.


If one of the devs could confirm that semantics, I may find some time
to update the wiki accordingly.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 2773 bytes --]

On Wed, 2015-12-09 at 11:26 +0000, Duncan wrote:
> > Hmm good point... I think it would be great if you could add that
> > scenario somewhere to the documentation. :-)
> FWIW, I (personally, not sure if that "you" was singular or plural)
> am 
> much more comfortable on the lists than on wikis, which I tend to
> treat 
> much as I did printed encyclopedias back in the day -- as great 
> references, but read-only from my perspective.
Well you (since it was your idea and I'd consider you "senior" enough
to do it, at least more senior than myself ;) )... or any developer of
course ;)

I did that now, or better said I largely rewrote:
https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes
which I think should rather be its own wiki article.

One of the devs/experts... please double check it and pick/drop what
you like/dislike.
Especially notice that I've changed what was in the wiki, namely
subvolid=0 would mount the toplevel subovl,... I changed that to 5.
Also I assumed the manpage would be correct and subvol= is always
relative to the top level subvol, thus subvol=/ should mount that.

What's still missing now, IMHO, is:
- the snapshots subchapter itself is not complete (especially ro vs. rw
snapshots)... and implications like not mounting snapshots noatime, and
write amplification effects
- a guide when one should make subvole (e.g. keeping things on the root
fs together, unless it's separate like /var/www is usually, but
/var/lib typically "corresponds" to a state of /etc and /usr.
- what I've asked in another mail,.. about subvols and mountopts.

That rework also contains your security idea... shamelessly not quoting
you there O:-)


Well, I hope me editing such central part of the wiki wasn't to
bold,... but if it was, then I guess Chris Mason placing me on the
pillory would be nearly as big of an honour as being insulted by Linus
;-)



> While the appropriate place would be on the wiki, where there's a
> page 
> for that (here, the request is likely to be lost to history, while on
> the 
> wiki it's in a nice list that both devs and other requesters can look
> at, 
> a year, five years, a decade... into the future), in this case...
I think I'm simply going to add it there, in another bold step... O:-)


> There's already a framework for it and a limited implementation, tho
> only 
> a small subset of possible options are yet available.  The UI
> exposure is 
> as btrfs property (see the btrfs-property manpage), with the
> properties, 
> in general, implemented as extented attributes.
Mhh,... I've only recently learned about that properties... well if
that's technically possible to cleanly solve it with XATTRS... fine for
me.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

On Sat, 2015-12-12 at 03:32 +0100, Christoph Anton Mitterer wrote:
> What's still missing now, IMHO, is:
> - a guide when one should make subvole (e.g. keeping things on the
> root
> fs together, unless it's separate like /var/www is usually, but
> /var/lib typically "corresponds" to a state of /etc and /usr.
I just added that:
https://btrfs.wiki.kernel.org/index.php/SysadminGuide#When_To_Make_Subvolumes

Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]

Re: subvols and parents - how?

[Duncan]

Christoph Anton Mitterer posted on Sat, 12 Dec 2015 03:32:57 +0100 as
excerpted:

> [B]etter said I largely rewrote:
> https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Subvolumes
> which I think should rather be its own wiki article.
> 
> One of the devs/experts... please double check it and pick/drop what
> you like/dislike.

I'd hesitate to call myself a real "expert" on subvolumes because I don't 
run them here (tho I did play with them a bit, before deciding the 
additional benefit for my use-case wasn't worth the additional 
complexity), so nearly all of what I post on the subject is simply 
restatement of other people's posts on the subject[1], but...

Looks quite reasonable to me... and /much/ more filled out than the last 
time I read it. =:^)

> Especially notice that I've changed what was in the wiki, namely
> subvolid=0 would mount the toplevel subovl,... I changed that to 5.

On that particular point, subvolid=0 is a special-case alias to 
subvolid=5, for those for which 0 as reference to the toplevel subvolume 
makes more sense than 5.

So 0 or 5, doesn't particularly matter in practice.  Tho 5 is the actual 
tree ID (AFAIK) with 0 simply an alias to it, so 5 is arguably 
technically slightly more correct, while 0 could be argued to be humanly 
slightly more correct as the logical/expected ID for the toplevel 
subvolume.

You might put that there too, somewhere, if you have a mind to do it...

> Also I assumed the manpage would be correct and subvol= is always
> relative to the top level subvol, thus subvol=/ should mount that.

Haven't the foggiest...

> What's still missing now, IMHO, is:
> - the snapshots subchapter itself is not complete (especially ro vs. rw
> snapshots)... and implications like not mounting snapshots noatime, and
> write amplification effects

Yes.

> - a guide when one should make subvole (e.g. keeping things on the root
> fs together, unless it's separate like /var/www is usually, but
> /var/lib typically "corresponds" to a state of /etc and /usr.

I see you added that (as mentioned in a followup).  Looks good.

> - what I've asked in another mail,.. about subvols and mountopts.
> 
> That rework also contains your security idea... shamelessly not quoting
> you there O:-)

That looks good too... tho to be fair that too was simply reposting 
someone else's idea... which was both *entirely* new to me when I read it 
as I'd simply not considered that before, and equally immediately obvious 
in hindsight, which is why it stuck with me so well, because the 
combination of entirely new viewpoint on something and "Duh, he's 
absolutely right and I should have seen it myself" tends to /be/ pretty 
memorable! =:^)

Anyway, glad it's mentioned on the wiki, now. =:^)

---
[1] No claim of subvolume expert: And also because I've a bit of the 
sysadmin's healthy superstition of ever claiming to be an expert until 
it's so clearly the case it's becoming ridiculous to try to claim 
otherwise, since if there's the slightest doubt, Murphy will surely see/
hear and assert his law shortly thereafter, demonstrating the fallacy of 
the claim, likely with a brown-bag level error! =:^\  Of course, without 
anything but the toplevel subvolumes here, that might make the unwary 
sysadmin unwisely bold, but lest I be unpleasantly reminded of the fact 
that the top-level's a subvolume too, I'm still wary of tempting either 
Murphy or fate with such a claim of expertise! =:^\

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: subvols and parents - how?

[Axel Burri]

On 2015-11-27 03:02, Duncan wrote:
>>> Then there's the security angle to consider.  With the (basically,
>>> possibly modified as I suggested) flat layout, mounting something
>>> doesn't automatically give people in-tree access to nested subvolumes
>>> (subject to normal file permissions, of course), like nested layout
>>> does.  And with (possibly modified) flat layout, the whole subvolume
>>> tree doesn't need to be mounted all the time either, only when you're
>>> actually working with subvolumes.
> 
>> Uhm, I don't get the big security advantage here... whether nested or
>> manually mounted to a subdir,... if the permissions are insecure I'll
>> have a problem... if they're secure, than not.
> 
> Consider a setuid-root binary with a recently publicized but patched on 
> your system vuln.  But if you have root snapshots from before the patch 
> and those snapshots are nested below root, then they're always 
> accessible.  If the path to the vulnerable setuid is as user accessible 
> as it likely was in its original location, then anyone with login access 
> to the system is likely to be able to run it from the snapshot... and 
> will be able to get root due to the vuln.
> 
> On a flat layout, a snapshot with the vuln would have to be mounted 
> before it could be accessed, as otherwise it'd be outside the mounted 
> tree.

Talking about security: If you split the "var" parts from the root, you
get a nice side-effect that the generation of the root subvolume does
not change until you update your system (e.g. by installing/updating
packages).

Consider this setup:

5
|
+--root (subvol)
|  |
|  +--var (subvol, or mounted)
|  +--tmp (subvol, or tmpfs)
|  +--run (subvol, or tmpfs)
|  +--home (subvol, or mounted)
|
+--snapshots (dir, uid=root,mode=700)
   |
   +--root.20150101
   +--root.20150102
   +--...

Now if you take a look at the output of "btrfs sub list -c /", you can
see exactly when your root subvolume has changed. This means if you
always create a new snapshot of "root" after you update your system, you
are able to spot changes to the system easily:

# btrfs sub list -c /
ID 100 gen 4242 cgen 50 top level 5 path root
ID 101 gen 4500 cgen 53 top level 5 path var
[...]
ID 5000 gen 4111 cgen 4111 top level 5 path snapshots/root.20151203
ID 5001 gen 4242 cgen 4242 top level 5 path snapshots/root.20151205
[...]

In this example, I can see that "root" has not been altered since
20151205, as "cgen" of the snapshot matches the "gen" of the root
subvolume. This is the last time I updated this system. If the "gen" of
the root subvolume would be greater than 4242, I know some changes were
made to the system root, and I can start tracking it down using "btrfs
subvol find-new".