Naming of subvolumes

Naming of subvolumes

[Richard Hughes]

I'm planning to use "btrfs subvolume snapshot -r <name>" in the system
upgrade functionality[1] if the user is using btrfs for their root file
system. We've got most of the bits in place already for Fedora 18.

One think that confuses me is the convention for the naming of
snapshots. Is there any conventions or prior art there? Can I add
metadata to the snapshot so that I don't have encode everything in the
snapshot name itself?

Also, being able to do a snapshot using a libbtrfs (or whatever) would
be much preferable to exec'ing the btrfs command and then trying to
parse the exit code and any stderr. I don't know if this kind of thing
is the norm for filesystem utilities, but a versioned shared library to
use would be awesome.

Any advice welcome, thanks.

Richard
(PackageKit author)

[1] https://fedoraproject.org/wiki/Features/OfflineSystemUpdates

Re: Naming of subvolumes

[anand jain]

Richard,

On 25/10/2012 20:30, Richard Hughes wrote:
> I'm planning to use "btrfs subvolume snapshot -r<name>" in the system
> upgrade functionality[1] if the user is using btrfs for their root file
> system. We've got most of the bits in place already for Fedora 18.
>
> One think that confuses me is the convention for the naming of
> snapshots. Is there any conventions or prior art there? Can I add
> metadata to the snapshot so that I don't have encode everything in the
> snapshot name itself?

  I have an experimental-patch to add tag for the snapshot.
  I can send it to you for a review (but I doubt if its any near
  for the integration, since it changes the root_info structure
  and I want to know if there is any better place to store
  the tag for the snapshots (like if its better to allocate
  a chunk to place this and all other properties in a xml format ?))

> Also, being able to do a snapshot using a libbtrfs (or whatever) would
> be much preferable to exec'ing the btrfs command and then trying to
> parse the exit code and any stderr.

   Its planned. At least for the functions surrounding the snapshots.

> I don't know if this kind of thing
> is the norm for filesystem utilities, but a versioned shared library to
> use would be awesome.
>
> Any advice welcome, thanks.
>
> Richard
> (PackageKit author)
>
> [1] https://fedoraproject.org/wiki/Features/OfflineSystemUpdates

   Other things which might be useful are
     - default location for the snapshots which can be reset
       using the btrfs set cli or lib.
     - default name for the snapshots.
     - A pre-defined profile for the snapshots
       (eg. Snapshots taken for backup may need to
        bypass the read-cache, when read)

Thanks, Anand

Re: Naming of subvolumes

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

On Thu, Oct 25, 2012 at 01:30:20PM +0100, Richard Hughes wrote:
> I'm planning to use "btrfs subvolume snapshot -r <name>" in the system
> upgrade functionality[1] if the user is using btrfs for their root file
> system. We've got most of the bits in place already for Fedora 18.
> 
> One think that confuses me is the convention for the naming of
> snapshots. Is there any conventions or prior art there? Can I add
> metadata to the snapshot so that I don't have encode everything in the
> snapshot name itself?

   How about user xattrs? IIRC, that's the user.* namespace.

   The only convention I'm aware of is Ubuntu's use of an @
substitution, where the subvolume to be mounted at / is called @, and
the subvolume to be mounted at /home becomes @home. Both of those
subvolumes are stored in the (otherwise empty) top-level of the
filesystem, which is not mounted in normal operation.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- This chap Anon is writing some perfectly lovely stuff ---      
                             at the moment.                              

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Naming of subvolumes

[Richard Hughes]

On Thu, 2012-10-25 at 14:28 +0100, Hugo Mills wrote:
>    How about user xattrs? IIRC, that's the user.* namespace.

Yes, but we still need some kind of metadata standard. For instance, I
might want to put a list of packages that were changed in the metadata
so that admins know why the snapshot was created. It's highly likely
distros like Ubuntu won't want to do it using the same mechanism (i.e.
using PackageKit) but having some shared specs is probably a good idea.

>    The only convention I'm aware of is Ubuntu's use of an @
> substitution, where the subvolume to be mounted at / is called @, and
> the subvolume to be mounted at /home becomes @home.

That could work I suppose.

>  Both of those
> subvolumes are stored in the (otherwise empty) top-level of the
> filesystem, which is not mounted in normal operation.

If we do one upgrade a week, that's 52 subvolumes cluttering things up.
Should the subvolumes be placed somewhere upgrade specific? Any best
practice ideas on naming? e.g. @system-upgrade-20121025?

Richard

Re: Naming of subvolumes

[Richard Hughes]

On Thu, 2012-10-25 at 16:09 +0100, Leonidas Spyropoulos wrote:
> On another note. How would you implement that logic (make a snapshot
> before upgrade) without the lib? Would you wrap package manager into a
> wrapper or patch package manager to call snapshot and the upgrade.
> In both cases you will need to parse stdout, stderr from the snapshot
> creation.

I'm intending to add the snapshot stuff in the logic described here:
https://gitorious.org/packagekit/packagekit/blobs/master/contrib/systemd-updates/README.txt

> Also would you automatically roll back to before snapshot if something
> goes wrong or give option to user? Can you categorise errors on
> upgrades according ti fatality to the system?

If *anything* in the upgrade fails and we've already started the package
transaction then we'd roll back to the previous known-good snapshot.

Richard.

Re: Naming of subvolumes

[Calvin Walton]

[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]

On Thu, 2012-10-25 at 16:16 +0100, Richard Hughes wrote:
> On Thu, 2012-10-25 at 16:09 +0100, Leonidas Spyropoulos wrote:
> > On another note. How would you implement that logic (make a snapshot
> > before upgrade) without the lib? Would you wrap package manager into a
> > wrapper or patch package manager to call snapshot and the upgrade.
> > In both cases you will need to parse stdout, stderr from the snapshot
> > creation.
> 
> I'm intending to add the snapshot stuff in the logic described here:
> https://gitorious.org/packagekit/packagekit/blobs/master/contrib/systemd-updates/README.txt
> 
> > Also would you automatically roll back to before snapshot if something
> > goes wrong or give option to user? Can you categorise errors on
> > upgrades according ti fatality to the system?
> 
> If *anything* in the upgrade fails and we've already started the package
> transaction then we'd roll back to the previous known-good snapshot.

Rolling back to a previous snapshot would require either a reboot or
using something like rsync to copy the old state back to the currently
mounted subvolume, unfortunately. There's no provision in Linux to allow
switching out the mounted filesystem from under running applications :)
-- 
Calvin Walton <calvin.walton@kepstin.ca>

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 4027 bytes --]

Re: Naming of subvolumes

[Richard Hughes]

On 25 October 2012 16:59, Calvin Walton <calvin.walton@kepstin.ca> wrote:
> Rolling back to a previous snapshot would require either a reboot...

I know, the very last thing the system upgrade does is reboot.

Richard.

Re: Naming of subvolumes

[Chris Murphy]

On Oct 25, 2012, at 6:30 AM, Richard Hughes <hughsient@gmail.com> wrote:

> I'm planning to use "btrfs subvolume snapshot -r <name>" in the system
> upgrade functionality[1] if the user is using btrfs for their root file
> system. We've got most of the bits in place already for Fedora 18.

What's the read-only snapshot of and used for?

> One think that confuses me is the convention for the naming of
> snapshots. Is there any conventions or prior art there? Can I add
> metadata to the snapshot so that I don't have encode everything in the
> snapshot name itself?

> If we do one upgrade a week, that's 52 subvolumes cluttering things up.
> Should the subvolumes be placed somewhere upgrade specific? Any best
> practice ideas on naming? e.g. @system-upgrade-20121025?


I'm curious what is being snapshotted, and what the snapshot organization will be, before getting to the naming convention. And if you're going to apply the upgrade to the snapshot, or to the top level file system?

Anaconda-18.19 in Fedora 18 TC6 presently creates subvolumes: boot, root, home, at the top level of the volume, if the user specifies Device Type BTRFS for those mountpoints.

Top Level
|---
|   |--root
|   |--boot
|   |--home

So I'm going to guess that you will actually create a subvolume named something like @system-upgrade-20121025, and then snapshot root, boot, and home into that subvol?

Top Level
|---
|   |--root
|   |--boot
|   |--home
|   |--@system-upgrade-20121025
|       |--root
|       |--boot
|       |--home

If the upgrade to the original top level is successful, you keep the default subvolume ID as 0. If the upgrade is not successful, you change the default subvolume ID to that of @system-upgrade-20121025. (Or you're upgrading the snapshot, if successful change the default subvol ID to that snapshot, if it fails leave it as is, which is probably the fail safe.)

The organization does kinda matter for grub2, because core.img has a prefix baked into it, so it's looking for boot/grub2. And the grub.cfg uses rootflags=subvol=root, so it's looking for the contents of root to boot. And /etc/fstab is likewise using relative references with the subvol= not subvolid=. So current boot root home can't be renamed without changing things elsewhere (like using subvolid instead).

While grub-mkconfig can figure this out for grub.cfg, grubby doesn't. And grubby right now is having a (small?) problem negotiating subvolumes, so it's currently not correctly updating grub.cfg when there are new kernels installed on btrfs.

Anyway, there are a lot of ways to do this, with different consequences. So I'm just wondering what's being snapshotted and what the organization is.

Also, about 52 snapshots? Small issue for the installer team who are looking for existing installations of other OS's, so that instead of getting a bunch of meaningless partitions, those partitions are grouped together by OS. Well, if there are 52 snapshots of Fedora 18, there are effectively 53 copies of /etc/fedora-release to be found. So how is this going to show up?

And I'm also curious then about gnome-disk-utility and how they will show subvolumes, if at all, and if there are 52 or 5000 subvolumes, what's this UI going to look like?

Maybe in the short term you do something just like with kernels and keep the most recent three and start deleting the older ones?


Chris Murphy

Re: Naming of subvolumes

[Richard Hughes]

On 26 October 2012 06:24, Chris Murphy <lists@colorremedies.com> wrote:
> What's the read-only snapshot of and used for?

/root and /boot

> And if you're going to apply the upgrade to the snapshot, or to the top level file system?

That's a very good question. I was going to apply the upgrade to the
top level file system, and then roll-back to the old snapshot if the
new upgrade state does not boot to a GUI. It would work equally well
applying the update to the snapshot and then switching to that,
although I don't know how well userspace is going to cope switching to
a new subvolume at early-runtime given the fact we're running the
upgrade itself from the filesystem and not in a special initrd or
something. Also, we need to ready out lots of upgrade data (200Mb+?)
from somewhere, so doing this on the snapshot would mean creating the
snapshot and copying the data there *before* we reboot to install
upgrades, and we really want to create the snapshot when we're sure
the system will boot.

> So I'm going to guess that you will actually create a subvolume named something like @system-upgrade-20121025, and then snapshot root, boot, and home into that subvol?

Not /home. Packages shouldn't be installing stuff there anyway, and
/home is going to typically much bigger than /root or /boot.

> If the upgrade is not successful, you change the default subvolume ID to that of @system-upgrade-20121025.

I was actually thinking of using btrfs send | btrfs receive to roll
back the root manually. It would be better if btrfs could swap the
subvolume ID's of @system-upgrade-20121025 and 0, as then we don't get
a snapshot that's useless.

> Well, if there are 52 snapshots of Fedora 18, there are effectively 53 copies of /etc/fedora-release to be found. So how is this going to show up?

I figured we could have some kind of metadata in the snapshot to tell
the installer "this is an upgrade snapshot" and to ignore it being
offered to upgrade. Metadata is the key here to make things just work.

> ...if there are 52 or 5000 subvolumes, what's this UI going to look like?

Also a good question. An unanswered question :)

> Maybe in the short term you do something just like with kernels and keep the most recent three and start deleting the older ones?

Yes, this is a very good idea and probably what we need to do for our use case.

Richard

Re: Naming of subvolumes

[Chris Murphy]

On Oct 26, 2012, at 2:27 AM, Richard Hughes <hughsient@gmail.com> wrote:


>> And if you're going to apply the upgrade to the snapshot, or to the top level file system?
> 
> That's a very good question. I was going to apply the upgrade to the
> top level file system, and then roll-back to the old snapshot if the
> new upgrade state does not boot to a GUI. It would work equally well
> applying the update to the snapshot and then switching to that,
> although I don't know how well userspace is going to cope switching to
> a new subvolume at early-runtime given the fact we're running the
> upgrade itself from the filesystem and not in a special initrd or
> something. Also, we need to ready out lots of upgrade data (200Mb+?)
> from somewhere, so doing this on the snapshot would mean creating the
> snapshot and copying the data there *before* we reboot to install
> upgrades, and we really want to create the snapshot when we're sure
> the system will boot.

a. Upgrade top level:
Maybe download upgrade data to /tmp, upon completion snapshot the current system state, then copy the upgrade data to persistent storage and reboot; upgrade of top level boot root begins. The snapshot is the regressive state in case of failure.

b. Upgrade snapshot:
Create snapshot, mount it somewhere; download upgrade data to a location within the snapshot; reboot from the snapshot, it upgrades itself and cleans up. The top level is the regressive state in case of failure.

Either way, 200MB of downloaded (and undeleted) upgrade data isn't stuck in a snapshot after it's used. And either way the snapshot is bootable.

If you get sufficient metadata in the snapshot, then you can name/rename the snapshots whatever you want. I'd also point out it's valid for the user to prefer a different organization, i.e. instead of Fedora taking over the top level of a btrfs volume, to create subvolumes Fedora 17, Fedora 18, Ubuntu 12X, etc., at the top level, and insert boot and root and possibly home in those. In which case the upgrade mechanism should still work.

> 
>> So I'm going to guess that you will actually create a subvolume named something like @system-upgrade-20121025, and then snapshot root, boot, and home into that subvol?
> 
> Not /home. Packages shouldn't be installing stuff there anyway, and
> /home is going to typically much bigger than /root or /boot.

OK so small problem here is that today /etc/fstab is pointing to the home subvolume in a relative location to the default subvolume. The fstab mount option is subvol=home, not subvol=/home, not subvolid=xxx.

So if you want to use changing default subvolumes to make the switch between the current updated state, and rollback states, (which is milliseconds fast), which also means no changes needed to grub's core.img or grub.cfg (since those use relative references for boot and root), a change is needed for home to use an absolute reference: either subvol=/home or use subvolid= in the fstab.

While a bit more obscure in the /etc/fstab the subvolid= is more reliable. That home can be renamed or moved anywhere and it'll still work. I think it's legitimate for a user to create or want


>> If the upgrade is not successful, you change the default subvolume ID to that of @system-upgrade-20121025.
> 
> I was actually thinking of using btrfs send | btrfs receive to roll
> back the root manually. It would be better if btrfs could swap the
> subvolume ID's of @system-upgrade-20121025 and 0, as then we don't get
> a snapshot that's useless.

I haven't tried btrfs send/receive for this purpose, so I can't compare. But btrfs subvolume set-default is faster than the release of my finger from the return key. And it's easy enough the user could do it themselves if they had reasons for regression to a snapshot that differ than the automagic determination of the upgrade pass/fail.

The one needed change, however, is to get /etc/fstab to use an absolute reference for home.


Chris Murphy

Re: Naming of subvolumes

[cwillu]

On Fri, Oct 26, 2012 at 9:54 AM, Chris Murphy <lists@colorremedies.com> wrote:
>
> On Oct 26, 2012, at 2:27 AM, Richard Hughes <hughsient@gmail.com> wrote:
>
>
>>> And if you're going to apply the upgrade to the snapshot, or to the top level file system?
>>
>> That's a very good question. I was going to apply the upgrade to the
>> top level file system, and then roll-back to the old snapshot if the
>> new upgrade state does not boot to a GUI. It would work equally well
>> applying the update to the snapshot and then switching to that,
>> although I don't know how well userspace is going to cope switching to
>> a new subvolume at early-runtime given the fact we're running the
>> upgrade itself from the filesystem and not in a special initrd or
>> something. Also, we need to ready out lots of upgrade data (200Mb+?)
>> from somewhere, so doing this on the snapshot would mean creating the
>> snapshot and copying the data there *before* we reboot to install
>> upgrades, and we really want to create the snapshot when we're sure
>> the system will boot.
>
> a. Upgrade top level:
> Maybe download upgrade data to /tmp, upon completion snapshot the current system state, then copy the upgrade data to persistent storage and reboot; upgrade of top level boot root begins. The snapshot is the regressive state in case of failure.
>
> b. Upgrade snapshot:
> Create snapshot, mount it somewhere; download upgrade data to a location within the snapshot; reboot from the snapshot, it upgrades itself and cleans up. The top level is the regressive state in case of failure.
>
> Either way, 200MB of downloaded (and undeleted) upgrade data isn't stuck in a snapshot after it's used. And either way the snapshot is bootable.
>
> If you get sufficient metadata in the snapshot, then you can name/rename the snapshots whatever you want. I'd also point out it's valid for the user to prefer a different organization, i.e. instead of Fedora taking over the top level of a btrfs volume, to create subvolumes Fedora 17, Fedora 18, Ubuntu 12X, etc., at the top level, and insert boot and root and possibly home in those. In which case the upgrade mechanism should still work.
>
>>
>>> So I'm going to guess that you will actually create a subvolume named something like @system-upgrade-20121025, and then snapshot root, boot, and home into that subvol?
>>
>> Not /home. Packages shouldn't be installing stuff there anyway, and
>> /home is going to typically much bigger than /root or /boot.
>
> OK so small problem here is that today /etc/fstab is pointing to the home subvolume in a relative location to the default subvolume. The fstab mount option is subvol=home, not subvol=/home, not subvolid=xxx.
>
> So if you want to use changing default subvolumes to make the switch between the current updated state, and rollback states, (which is milliseconds fast), which also means no changes needed to grub's core.img or grub.cfg (since those use relative references for boot and root), a change is needed for home to use an absolute reference: either subvol=/home or use subvolid= in the fstab.
>
> While a bit more obscure in the /etc/fstab the subvolid= is more reliable. That home can be renamed or moved anywhere and it'll still work. I think it's legitimate for a user to create or want
>
>
>>> If the upgrade is not successful, you change the default subvolume ID to that of @system-upgrade-20121025.
>>
>> I was actually thinking of using btrfs send | btrfs receive to roll
>> back the root manually. It would be better if btrfs could swap the
>> subvolume ID's of @system-upgrade-20121025 and 0, as then we don't get
>> a snapshot that's useless.
>
> I haven't tried btrfs send/receive for this purpose, so I can't compare. But btrfs subvolume set-default is faster than the release of my finger from the return key. And it's easy enough the user could do it themselves if they had reasons for regression to a snapshot that differ than the automagic determination of the upgrade pass/fail.
>
> The one needed change, however, is to get /etc/fstab to use an absolute reference for home.
>
>
> Chris Murphy
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

I'd argue that everything should be absolute references to subvolumes
(/@home, /@, etc), and neither set-default nor subvolume id's should
be touched.  There's no need, as you can simply mv those around (even
while mounted).  More importantly, it doesn't result in a case where
the fstab in one snapshot points its mountpoint to a different
snapshot, with all the hilarity that would cause over time, and also
allows multiple distros to be installed on the same filesystem without
having them stomp on each others set-defaults: /@fedora, /@rawhide,
/@ubuntu, /@home, etc.

Re: Naming of subvolumes

[Fajar A. Nugraha]

On Sat, Oct 27, 2012 at 8:58 AM, cwillu <cwillu@cwillu.com> wrote:
>> I haven't tried btrfs send/receive for this purpose, so I can't compare. But btrfs subvolume set-default is faster than the release of my finger from the return key. And it's easy enough the user could do it themselves if they had reasons for regression to a snapshot that differ than the automagic determination of the upgrade pass/fail.
>>
>> The one needed change, however, is to get /etc/fstab to use an absolute reference for home.
>>
>>
>> Chris Murphy

> I'd argue that everything should be absolute references to subvolumes
> (/@home, /@, etc), and neither set-default nor subvolume id's should
> be touched.  There's no need, as you can simply mv those around (even
> while mounted).  More importantly, it doesn't result in a case where
> the fstab in one snapshot points its mountpoint to a different
> snapshot, with all the hilarity that would cause over time, and also
> allows multiple distros to be installed on the same filesystem without
> having them stomp on each others set-defaults: /@fedora, /@rawhide,
> /@ubuntu, /@home, etc.


What I do with zfs, which might also be applicable on btrfs:

- Have a separate dataset to install grub: poolname/boot. This can
also be a dedicated partition, if you want. The sole purpose for this
partition/dataset is to select which dataset's grub.cfg to load next
(using "configfile" directive). The grub.cfg here is edited manually.

- Have different datasets for each versioned OS (e.g. before and after
upgrades): poolname/ROOT/ubuntu-1, poolname/ROOT/ubuntu-2, etc. Each
dataset is independent of each other, contains their own /boot
(complete with grub/grub.cfg, kernel, and initrd). grub.cfg on each
dataset selects its own dataset to boot using "bootfs" kernel command
line.

- Have a common home for all environment: poolname/home

- Have zfs set the mountpoint (or mounted in initramfs, in root case),
so I can get away with an empty fstab.

- Do upgrades/modifications in the currently-booted root environment,
but create a clone of current environment (and give it a different
name) so I can roll back to it if needed.


It works great for me so far, since:
- each boot environment is portable-enough to move around when needed,
with only about four config files needed to be changed (e.g. grub.cfg)
when moving between different computers, or when renaming a root
dataset.

- I can rename each root environment easily, or even move it to
different pool/disk when needed.

- I can move back and forth between multiple versions of the boot
environment (all are ubuntu so far, cause IMHO it currently has best
zfs root support).


So back to the original question, I'd suggest NOT to use either
send/receive or set-default. Instead, setup multiple boot environment
(e.g. old version, current version) and let user choose which one to
boot using a menu. However for this to work, grub (the bootloader, and
the userland programs like "update-grub") needs to be able to refer to
each grub.cfg/kernel/initrd in a global manner regardless of what the
current default subvolume is (zfs' grub code uses something like
/poolname/dataset_name/@/path/to/file/in/dataset).

-- 
Fajar

Re: Naming of subvolumes

[Chris Murphy]

On Oct 26, 2012, at 7:58 PM, cwillu <cwillu@cwillu.com> wrote:
> 
> I'd argue that everything should be absolute references to subvolumes
> (/@home, /@, etc), and neither set-default nor subvolume id's should
> be touched.  

grub's core.img prefix is relative to the current default subvolume. I'm unaware of an absolute means of specifying the prefix.

In order to avoid using set-default, you end up with either the need to make a new core.img and install it, prior to every change in what subvolume to boot from (snapshot or other distribution), or do some fancy mv like you mention next.



> There's no need, as you can simply mv those around (even
> while mounted).

What if Ubuntu wants to use root and boot on the top level, but finds boot and root already exist there from Fedora 18? Will Ubunuto package them up properly into a folder or subvol in such a way that Fedora 18 can ever be used again without significant user knowledge about what likely happened?

I think this is untenable. Since set-default is valuable, and its use cannot correctly be proscribed for all time on boot volumes, it's better to figure out how to leverage it.

I think that the top level subvolumes should be named after the distribution: Fedora 18, Fedora 19, Ubuntu 12.04, Suse 12.x, etc. and possibly home can also be in the top level. And inside, can be whatever organization that distro wants. But negotiating generically named boot and root at the top level I think is long term problematic.

The small problem, presently, with Fedora is that grub is referred to as grub2. So the actual core.img prefix that's baked in is /boot/grub2, meaning that even if you change the set-default subvolume, that core.img is incompatible with finding another grub.cfg of the same version without baking a new core.img (or manually setting the prefix).


>  More importantly, it doesn't result in a case where
> the fstab in one snapshot points its mountpoint to a different
> snapshot, with all the hilarity that would cause over time, and also
> allows multiple distros to be installed on the same filesystem without
> having them stomp on each others set-defaults: /@fedora, /@rawhide,
> /@ubuntu, /@home, etc.

You'd have to use absolute paths if you're going to depend on every distro potentially mv'ing every other distros folders and subvolumes around, yes. If a distribution starts moving and renaming my subvolumes, I will not be using that distribution.


Chris Murphy

Re: Naming of (bootable) subvolumes

[Chris Murphy]

On Oct 26, 2012, at 9:03 PM, Fajar A. Nugraha <list@fajar.net> wrote:

> 
> So back to the original question, I'd suggest NOT to use either
> send/receive or set-default. Instead, setup multiple boot environment
> (e.g. old version, current version) and let user choose which one to
> boot using a menu.

Is it possible to make a functioning symbolic or hard link of a subvolume? 

I'm fine with "current" and "previous" options. More than that seems unnecessary. But then, how does the user choose? What's the UI? Is this properly the domain of GRUB2 or something else?

On BIOS machines, perhaps GRUB. On UEFI, I'd say distinctly not GRUB (I think it's a distinctly bad idea to have a combined boot manager and bootloader in a UEFI context, but that's a separate debate).


> However for this to work, grub (the bootloader, and
> the userland programs like "update-grub") needs to be able to refer to
> each grub.cfg/kernel/initrd in a global manner regardless of what the
> current default subvolume is (zfs' grub code uses something like
> /poolname/dataset_name/@/path/to/file/in/dataset).


Example. The following are all subvolumes, subvolume set-default 0, fstab uses subvol=home, subvol=root, subvol=boot for mount options.

toplevel
├── boot
├── home
├── root
├── fedora18
│   ├── boot
│   └── root

On this system, grub-mkconfig produces a grub.cfg only for the system I'm currently booted from. It does not include any entries for fedora18/boot, fedora18/root, even though they are well within the normal search path. And the reference used is relative,  i.e. the kernel parameter in the grub.cfg is rootflags=subvol=root

If it were to create entries potentially for every snapshotted system, it would be a very messy grub.cfg indeed.

It stands to reason that each distro will continue to have their own grub.cfg. 

For BIOS machines, it could be useful if a single core.img containing a single standardized prefix specifying a grub location could be agreed upon. And then merely changing the set-default subvolume would allow different distro grub.cfg's to be found, read and workable with the relative references now in place, (except for home which likely needs to be mounted using subvolid).

For UEFI machines, the plan needs to work with other bootloaders, including the linux kernel's EFISTUB.



Chris Murphy

Re: Naming of (bootable) subvolumes

[Goffredo Baroncelli]

On 2012-10-27 19:22, Chris Murphy wrote:
> 
> On Oct 26, 2012, at 9:03 PM, Fajar A. Nugraha <list@fajar.net>
> wrote:
> 
>> 
>> So back to the original question, I'd suggest NOT to use either 
>> send/receive or set-default. Instead, setup multiple boot
>> environment (e.g. old version, current version) and let user choose
>> which one to boot using a menu.
> 
> Is it possible to make a functioning symbolic or hard link of a
> subvolume?
> 
> I'm fine with "current" and "previous" options. More than that seems
> unnecessary. But then, how does the user choose? What's the UI? Is
> this properly the domain of GRUB2 or something else?


On my computer, when update-grub is called, it scans for all the
subvolumes and show them in the menu, allowing the user to switch at
boot time.

[...]
> 
> Example. The following are all subvolumes, subvolume set-default 0,
> fstab uses subvol=home, subvol=root, subvol=boot for mount options.
> 
> toplevel ├── boot ├── home ├── root ├── fedora18 │   ├── boot │   └──
> root
> 
> On this system, grub-mkconfig produces a grub.cfg only for the system
> I'm currently booted from. It does not include any entries for
> fedora18/boot, fedora18/root, even though they are well within the
> normal search path. And the reference used is relative,  i.e. the
> kernel parameter in the grub.cfg is rootflags=subvol=root
> 
> If it were to create entries potentially for every snapshotted
> system, it would be a very messy grub.cfg indeed.

I don't think so. If a subvolume has a /sbin/int and and /etc it would
be suitable to be used as root filesystem.

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli (kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Naming of (bootable) subvolumes

[Fajar A. Nugraha]

On Sun, Oct 28, 2012 at 12:22 AM, Chris Murphy <lists@colorremedies.com> wrote:
>
> On Oct 26, 2012, at 9:03 PM, Fajar A. Nugraha <list@fajar.net> wrote:
>
>>
>> So back to the original question, I'd suggest NOT to use either
>> send/receive or set-default. Instead, setup multiple boot environment
>> (e.g. old version, current version) and let user choose which one to
>> boot using a menu.
>
> Is it possible to make a functioning symbolic or hard link of a subvolume?
>

Nope, I don't think so.

> I'm fine with "current" and "previous" options. More than that seems unnecessary. But then, how does the user choose?

WIth up and down arrow :)

> What's the UI?

Grub boot menu.

> Is this properly the domain of GRUB2 or something else?

In my setup I use grub2's "configfile" ability. Which basically does a
"go evaluate this other menu config file".

Each boot environment (BE, the term that solaris uses) has a different
entry on the "main" grub.cfg, which loads the BE's corresponding
grub.cfg.

>
> On BIOS machines, perhaps GRUB. On UEFI, I'd say distinctly not GRUB (I think it's a distinctly bad idea to have a combined boot manager and bootloader in a UEFI context, but that's a separate debate).

I don't use UEFI. But the general idea is to have one bootloader which
can load additional config files. And the location of that additional
config file depends on which BE user wants to boot.

> On this system, grub-mkconfig produces a grub.cfg only for the system I'm currently booted from. It does not include any entries for fedora18/boot, fedora18/root, even though they are well within the normal search path. And the reference used is relative,  i.e. the kernel parameter in the grub.cfg is rootflags=subvol=root
>
> If it were to create entries potentially for every snapshotted system, it would be a very messy grub.cfg indeed.
>
> It stands to reason that each distro will continue to have their own grub.cfg.
>

No arguments there. Even in my setup, when I run "update-grub", it
will only update its own grub.cfg, and leave the "main" grub.cfg
untouched. This is how my "main" grub.cfg looks like:


#===========================
set timeout=2

menuentry 'Ubuntu - 20120905 boot menu' {
	configfile	/ROOT/precise-5/@/boot/grub/grub.cfg
}
menuentry 'Ubuntu - 20120814 boot menu' {
	configfile	/ROOT/precise-4/@/boot/grub/grub.cfg
}
#===========================

each BE's grub cfg (e.g. the one under ROOT/precise-5 dataset) is just
your typical Ubuntu's grub.cfg, with only references to kernel/initrd
under that dataser.

> For BIOS machines, it could be useful if a single core.img containing a single standardized prefix specifying a grub location could be agreed upon. And then merely changing the set-default subvolume would allow different distro grub.cfg's to be found, read and workable with the relative references now in place, (except for home which likely needs to be mounted using subvolid).

IMHO the biggest difference is that grub support for zfsonlinux, even
though it has bootfs pool property, has a way to reference ALL
versions of a file (including grub.cfg/kernel/initrd) during boot
time. This way you don't even need to change bootfs whenever you want
to change to a boot environment, you simply choose (or write) a
different grub stanza to boot.

If we continue to rely on current btrfs grub support, unfortunately we
can't have the same thing. And the closest thing would be
"set-default". Which IMHO is VERY messy.

-- 
Fajar

Re: Naming of (bootable) subvolumes

[Chris Murphy]

On Oct 28, 2012, at 3:25 AM, Goffredo Baroncelli <kreijack@inwind.it> wrote:
> On Sun, Oct 28, 2012 at 12:22 AM, Chris Murphy <lists@colorremedies.com> wrote:


> On my computer, when update-grub is called, it scans for all the
> subvolumes and show them in the menu, allowing the user to switch at
> boot time.

If they can go in a snapshots submenu, fine. But UI wise, messy and not inherently a good idea to expose to regular Joe user in the main menu. What's the kernel parameter rootflags= line they're using? Is it a subvol hard path or subvolid?

>> If it were to create entries potentially for every snapshotted
>> system, it would be a very messy grub.cfg indeed.
> 
> I don't think so. If a subvolume has a /sbin/int and and /etc it would
> be suitable to be used as root filesystem.

I understand it's suitable in that it will work. I'm just saying I don't want to see it by default in a menu. I do want to have to go dig for it, because most people most of the time won't have a need to use them. So in the main menu an option 'Snapshots' or 'Previous Systems' to reveal the snapshot list makes sense.


On Oct 28, 2012, at 3:38 AM, "Fajar A. Nugraha" <list@fajar.net> wrote:

>> What's the UI?
> 
> Grub boot menu.

If the UI for switching between subvols is via grub, that means no dependency on 'subvolume set-default'. While you can call the root subvolume absolutely with rootflags=subvol=/hardpath/here/ or rootflags=subvolid=, what about the boot subvol? That's always a relative path as far as I can tell.

So is the grub boot menu itself, i.e. t he grub.cfg that's used depends on the set-default. Are Ubuntu snapshots based just on snapshotting the root subvolume and keeping all possible kernels in a boot subvolume?

>> Is this properly the domain of GRUB2 or something else?
> 
> In my setup I use grub2's "configfile" ability. Which basically does a
> "go evaluate this other menu config file".

*sigh* two GRUB menus. Twice the pain, three times the mischief. But I understand how this works.

>> On BIOS machines, perhaps GRUB. On UEFI, I'd say distinctly not GRUB (I think it's a distinctly bad idea to have a combined boot manager and bootloader in a UEFI context, but that's a separate debate).
> 
> I don't use UEFI. But the general idea is to have one bootloader which
> can load additional config files. And the location of that additional
> config file depends on which BE user wants to boot.


Therein lies some confusion to answer the question of how to organize snapshots and boot from them. Various boot managers and boot loaders should be taken into account, and UEFI firmware implication should be taken into account as well.


> IMHO the biggest difference is that grub support for zfsonlinux, even
> though it has bootfs pool property, has a way to reference ALL
> versions of a file (including grub.cfg/kernel/initrd) during boot
> time. This way you don't even need to change bootfs whenever you want
> to change to a boot environment, you simply choose (or write) a
> different grub stanza to boot.
> 
> If we continue to rely on current btrfs grub support, unfortunately we
> can't have the same thing. And the closest thing would be
> "set-default". Which IMHO is VERY messy.

I'm not entirely familiar with how ZFS implements snapshots or grub's support for them. But it sonds like the bootfs pool property supports absolute path names everywhere, including from within the GRUB core.img? So you're suggesting a GRUB2 limitation when it comes to btrfs?

Fedora has also a limitation where kernel updates are still using grubby, not grub-mkconfig. That's also a problem IMO. It basically just takes the current entry and makes a kernel modification. If a snapshot is produced with each upgrade, the resulting grub.cfg will not be aware of or point to demoted kernels in snapshots. So if GRUB is a weak link for btrfs support, grubby is a weaker link.

Will btrfs send / receive for snapshotting produce large storage requirements isn't it? If I send a current subvol to another subvol as a snapshot, am I not effectively making a copy of that whole sub-filesystem? Whereas 'subvolume snapshot' just does this with metadata, no meaningful additional space required. Only the updated files consume more space, regardless of which snapshot they're applied to.



Chris Murphy