I'd like a -r flag on btrfs subvolume delete

I'd like a -r flag on btrfs subvolume delete

[Johannes Ernst]

Rationale: cleaning up after containers, which may have created their own subvolumes.

E.g.
	systemd-nspawn —boot —directory dir

where dir is a subvolume. When done with the container, deleting fir directly doesn’t work, because we now also have a subvolume at
	dir/var/lib/machines
and obviously there may be more that the container might have created.

Cleaning this all up is a bit of pain, and
	btrfs subvolume delete -r dir
would solve it nicely.

Cheers,



Johannes Ernst
Blog: http://upon2020.com/
Twitter: @Johannes_Ernst
GPG key: http://upon2020.com/public/pubkey.txt
Check out UBOS, the Linux distro for personal servers I work on: http://ubos.net/

Re: I'd like a -r flag on btrfs subvolume delete

[Chris Murphy]

On Wed, Jul 15, 2015 at 6:11 PM, Johannes Ernst
<johannes.ernst@gmail.com> wrote:

> Cleaning this all up is a bit of pain, and
>         btrfs subvolume delete -r dir
> would solve it nicely.

It's come up before:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg42455.html
http://lists.freedesktop.org/archives/systemd-devel/2015-April/030297.html

I'm concerned about the interaction of machinectl snapshots of its own
subvolumes, and rpm-ostree, and snapper snapshots.

The only really convincing argument for tested subvolumes I've read is
as an explicit break from being included in the snapshotting above it
in the hierarchy.

So the /var/lib/machines organization burdens other projects or users
with the problem of how / or /var is snapshot and then a rollback
happens, how to properly reassemble the system that includes
/var/lib/machines subvolumes that are now in a different tree. How
does this all get located and assembled properly at boot time?

How is all of this backed up properly? How is it restored properly? I
think recursive snapshotting and subvolume deletion is not a good
idea. I think it's a complicated and inelegant work around for
improper subvolume organization.


-- 
Chris Murphy

Re: I'd like a -r flag on btrfs subvolume delete

[Paul Harvey]

On 16 July 2015 at 11:35, Chris Murphy <lists@colorremedies.com> wrote:
> On Wed, Jul 15, 2015 at 6:11 PM, Johannes Ernst
> <johannes.ernst@gmail.com> wrote:
>
>> Cleaning this all up is a bit of pain, and
>>         btrfs subvolume delete -r dir
>> would solve it nicely.

[snip]

> How is all of this backed up properly? How is it restored properly? I
> think recursive snapshotting and subvolume deletion is not a good
> idea. I think it's a complicated and inelegant work around for
> improper subvolume organization.

I for one would love to see authoritative documentation on "proper"
subvolume organization. I was completely lost when writing snazzer and
have so far received very little guidance or even offers of opinions
on this ML.

I've had to create my own logic in my scripts that automatically walk
all subvolumes on all filesystems for the simple reason that
explicitly enumerating it all for dozens of servers becomes a
significant administration burden.

I have different retention needs for /var (particularly /var/cache)
than I do for /home, for example, so carving up my snapshots so that I
can easily drop them from those parts of my filesystems which have a
high churn rate (= more unique extents, occupying a lot of disk) and
yet aren't as important (I need to retain fewer of them) is very
useful.

Re: I'd like a -r flag on btrfs subvolume delete

[Chris Murphy]

On Wed, Jul 15, 2015 at 9:12 PM, Paul Harvey <csirac2@gmail.com> wrote:
> On 16 July 2015 at 11:35, Chris Murphy <lists@colorremedies.com> wrote:

>> How is all of this backed up properly? How is it restored properly? I
>> think recursive snapshotting and subvolume deletion is not a good
>> idea. I think it's a complicated and inelegant work around for
>> improper subvolume organization.
>
> I for one would love to see authoritative documentation on "proper"
> subvolume organization.

The choice of "improper" wasn't ideal on my part. There's nothing
directly wrong with nested subvolumes. But if you then combine them
with snapshots and rollbacks, there are consequences that include more
complication. If more than one things is doing snapshots and
rollbacks, it requires some rules as to who can snapshot what and
where those things go in order to avoid being snapshot again by some
other tool, and then how things get reassembled. There are different
kinds of rollbacks so that needs some rules or it'll just lead to
confusion.


> I was completely lost when writing snazzer and
> have so far received very little guidance or even offers of opinions
> on this ML.

A couple of developers have suggested the folly of nested subvolumes
several times. Discovering the consequences of organizing subvolumes
is a work in progress. I've mentioned a couple times over the years
that distros are inevitably going to end up with fragmented and
mutually incompatible approaches if they don't actively engage each
other cooperatively. And that's turned out to be correct as Fedora,
Ubuntu and SUSE all do things differently with their Btrfs
organization.



> I've had to create my own logic in my scripts that automatically walk
> all subvolumes on all filesystems for the simple reason that
> explicitly enumerating it all for dozens of servers becomes a
> significant administration burden.
>
> I have different retention needs for /var (particularly /var/cache)
> than I do for /home, for example, so carving up my snapshots so that I
> can easily drop them from those parts of my filesystems which have a
> high churn rate (= more unique extents, occupying a lot of disk) and
> yet aren't as important (I need to retain fewer of them) is very
> useful.

At the moment, I like the idea of subvolumes pretty much only at the
top level of the file system (subvolid 5), and I like the naming
convention suggested here:
http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html
under the section "What We Propose"

I don't really like the colons because those are special character so
now I have to type three characters for each one of those. But anyway
those then get assembled in FHS form via fstab using subvol= or
subvolid= mount option, or whatever replaces fstab eventually.

This way you can snapshot different subvolumes at different rates with
different cleanup policies while keeping all of them out of the
normally mounted FHS path. A side plus is that this also puts old
libraries outside the FHS path, sort of like they're in a jail.



-- 
Chris Murphy

Re: I'd like a -r flag on btrfs subvolume delete

[Paul Harvey]

On 16 July 2015 at 13:44, Chris Murphy <lists@colorremedies.com> wrote:
> The choice of "improper" wasn't ideal on my part. There's nothing
> directly wrong with nested subvolumes. But if you then combine them
> with snapshots and rollbacks, there are consequences that include more
> complication. If more than one things is doing snapshots and
> rollbacks, it requires some rules as to who can snapshot what and
> where those things go in order to avoid being snapshot again by some
> other tool, and then how things get reassembled. There are different
> kinds of rollbacks so that needs some rules or it'll just lead to
> confusion.

You do have good points, but you're mixing quite a few issues. One
that's not clearly addressed is the problem that consensus and
conventions are nice but we need stuff "now" :)

Having multiple snapshotting systems is of course suboptimal, that's
stating the obvious. But it's up to admins to decide how they deal
with it, I've seen environments clumsily mix puppet, ansible and salt
-  nobody got fired. So that can't be a *real* excuse for not
implementing recursive subvolume stuff - can it?

I already have to deal with inventing my own conventions on where
snapshots live and how they're named - the closest thing for guidance
I had was looking closely at OpenSuSE's snapper. Which I didn't like.
And I'm *still* finding tools I have to configure properly to deal
with ".shapshot/*" dirs: mlocate/updatedb and friends, AVs, file
integrity daemons...

> A couple of developers have suggested the folly of nested subvolumes
> several times. Discovering the consequences of organizing subvolumes

I've read these discussions (those that I could find - that's a
separate issue). It seems far from obvious that this is "folly". It's
worth pointing out that ZFS works fine with, and has support in its
tools for nested datasets, cf. "zfs destroy -r
/path/to/nested/dataset".

If it's folly, it really should be documented in something more formal
than a couple of meandering ML threads. And perhaps removed as a
feature from btrfs proper. If I was a contributor to btrfs itself, I'd
even dare to ask that if subvolumes or so broken, but there's still a
need for their functionality, perhaps they need reinventing.

I'd be happy to help create a wiki page that properly enumerate the
problems with nested subvolumes if we can collect everything in one
place.

> is a work in progress. I've mentioned a couple times over the years
> that distros are inevitably going to end up with fragmented and
> mutually incompatible approaches if they don't actively engage each
> other cooperatively. And that's turned out to be correct as Fedora,
> Ubuntu and SUSE all do things differently with their Btrfs
> organization.

I'm afraid you're absolutely correct, but in the meantime...

> At the moment, I like the idea of subvolumes pretty much only at the
> top level of the file system (subvolid 5), and I like the naming
> convention suggested here:
> http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html
> under the section "What We Propose"

That proposal is indeed very nice but at 5000 words (literally 5000
words) it is still a speculative design document and wish-list of
stuff that is so far removed from the problems I have today and from
the operating systems and distributions I am actually using today that
I'm having trouble tying this back to how the proposal actually
answers the question: what about recursive subvolume operations....

... and I'm still wondering why if ZFS can deal with nested datasets,
OpenSuSE's snapper can deal with snapshots/rollbacks with nested
subvolumes - we can't deal with nested subvolumes in btrfs-progs.

I suspect the real reason is something along the lines of: "If we do
this, support for recursive or even multiple subvolume operations
should be implemented properly and atomically in the kernel, but
that's an enormous amount of work for now, so until then we don't want
to accidentally encourage  everyone to use a mediocre partial
implementation emulated in userland btrfs-progs - so we'll leave users
shoot themselves in that particular foot on their own if they insist".

I don't think that's quite the same as asserting that nested
subvolumes are somehow fundamentally folly.

> I don't really like the colons because those are special character so
> now I have to type three characters for each one of those. But anyway
> those then get assembled in FHS form via fstab using subvol= or
> subvolid= mount option, or whatever replaces fstab eventually.

I don't see how nested subvolumes stop you from doing this. In fact I
do this on servers I'm likely to do exploratory rollbacks on. That's
roughly how SuSE's snapper project already supports rollbacks
regardless of whether you're using nested subvolumes or not. Granted,
there's more default-subvolid shenanigans but scripting that out of
the way represents way fewer lines of my own code compared to the
tedium of working around the lack of a recursive subvolume support in
btrfs-progs.

> This way you can snapshot different subvolumes at different rates with
> different cleanup policies while keeping all of them out of the
> normally mounted FHS path. A side plus is that this also puts old
> libraries outside the FHS path, sort of like they're in a jail.

Finally, I've been shown something new in this thread. That's actually
a really good point.

If you don't beat me to it, I'll try to start documenting nested
subvolume issues on the wiki next week.

--
Paul

Re: I'd like a -r flag on btrfs subvolume delete

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1799 bytes --]

On Thu, Jul 16, 2015 at 01:12:03PM +1000, Paul Harvey wrote:
> On 16 July 2015 at 11:35, Chris Murphy <lists@colorremedies.com> wrote:
> > On Wed, Jul 15, 2015 at 6:11 PM, Johannes Ernst
> > <johannes.ernst@gmail.com> wrote:
> >
> >> Cleaning this all up is a bit of pain, and
> >>         btrfs subvolume delete -r dir
> >> would solve it nicely.
> 
> [snip]
> 
> > How is all of this backed up properly? How is it restored properly? I
> > think recursive snapshotting and subvolume deletion is not a good
> > idea. I think it's a complicated and inelegant work around for
> > improper subvolume organization.
> 
> I for one would love to see authoritative documentation on "proper"
> subvolume organization. I was completely lost when writing snazzer and
> have so far received very little guidance or even offers of opinions
> on this ML.

   Advice on this point has been on the btrfs wiki for several years,
at:

https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Managing_snapshots

   Hugo.

> I've had to create my own logic in my scripts that automatically walk
> all subvolumes on all filesystems for the simple reason that
> explicitly enumerating it all for dozens of servers becomes a
> significant administration burden.
> 
> I have different retention needs for /var (particularly /var/cache)
> than I do for /home, for example, so carving up my snapshots so that I
> can easily drop them from those parts of my filesystems which have a
> high churn rate (= more unique extents, occupying a lot of disk) and
> yet aren't as important (I need to retain fewer of them) is very
> useful.

-- 
Hugo Mills             | Hail and greetings. We are a flat-pack invasion
hugo@... carfax.org.uk | force from Planet Ikea. We come in pieces.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: I'd like a -r flag on btrfs subvolume delete

[Paul Harvey]

On 16 July 2015 at 17:59, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Thu, Jul 16, 2015 at 01:12:03PM +1000, Paul Harvey wrote:
>> I for one would love to see authoritative documentation on "proper"
>> subvolume organization. I was completely lost when writing snazzer and
>> have so far received very little guidance or even offers of opinions
>> on this ML.
>
>    Advice on this point has been on the btrfs wiki for several years,
> at:
>
> https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Managing_snapshots
>
>    Hugo.

Which is great, and I've consulted it in the past. But nested
subvolumes aren't exactly discouraged there (in fact, they're not
mentioned in that section at all).

In fact, re-reading it, I'm not even sure that using nested subvolumes
is incompatible with that advice.

What I'm proposing is more thorough and assertive documentation that
matches the wisdom handed out on this list.

--
Paul

Re: I'd like a -r flag on btrfs subvolume delete

[Chris Murphy]

On Thu, Jul 16, 2015 at 4:06 AM, Paul Harvey <csirac2@gmail.com> wrote:
> On Thu, 16 Jul 2015 17:59 Hugo Mills <hugo@carfax.org.uk> wrote:
>
>
>>
>> On Thu, Jul 16, 2015 at 01:12:03PM +1000, Paul Harvey wrote:
>> > On 16 July 2015 at 11:35, Chris Murphy <lists@colorremedies.com> wrote:
>> > > On Wed, Jul 15, 2015 at 6:11 PM, Johannes Ernst
>> > > <johannes.ernst@gmail.com> wrote:
>> > >
>> > >> Cleaning this all up is a bit of pain, and
>> > >>         btrfs subvolume delete -r dir
>> > >> would solve it nicely.
>> >
>> > [snip]
>> >
>> > > How is all of this backed up properly? How is it restored properly? I
>> > > think recursive snapshotting and subvolume deletion is not a good
>> > > idea. I think it's a complicated and inelegant work around for
>> > > improper subvolume organization.
>> >
>> > I for one would love to see authoritative documentation on "proper"
>> > subvolume organization. I was completely lost when writing snazzer and
>> > have so far received very little guidance or even offers of opinions
>> > on this ML.
>>
>>    Advice on this point has been on the btrfs wiki for several years,
>> at:
>>
>> https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Managing_snapshots
>
> Which is great, and I've consulted it in the past. But nested subvolumes
> aren't exactly discouraged there (in fact, they're not mentioned in that
> section at all).
>
> In fact, re-reading it, I'm not even sure that using nested subvolumes is
> incompatible with that advice.

It's not so much an incompatibility problem as we start seeing
consequences that rapidly approach pathological behavior. There's no
solution yet for snapshot aware defrag for example, and there are
still performance problems that happen with many snapshots. So making
it easy to exponentially create many snapshots, with a recursive
snapshot option, seems premature.

For the original poster, machinectl has already implemented a
recursive removal of subvolumes.
http://lists.freedesktop.org/archives/systemd-devel/2015-April/031026.html
http://lists.freedesktop.org/archives/systemd-devel/2015-April/030297.html


-- 
Chris Murphy