booting btrfs

booting btrfs

[Chris Murphy]

On Sep 26, 2013, at 4:15 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> On 26.09.2013 22:51, Chris Murphy wrote:
>> 
>> Open question is if on BIOS hardware, if a 1MB BIOSBoot is preferred over the 64KB Btrfs bootloader pad? I don't know off hand if each member disk, or subsequently added disks, each have this 64KB pad or just the first member.
>> 
> This is besides the point. I'm fine to discuss such things but in a
> separate thread.

2nd question is whether additional things are needed for /boot on btrfs. Using qemu/kvm and vbox I've tested maybe 6 virtual disks using btrfs single, raid0, raid1, and raid10 configurations, and consistently GRUB can boot /boot as a subvolume. I don't know how resilient this is as the file system is used and chunks end up on different devices or if this is sufficiently abstracted that it doesn't matter and just works; including if devices are removed and added.

3rd question, related, is if it's needed and planned work for GRUB to know about subvolid such as (hd0,gpt5,sv218) so that it can directly locate a subvolume regardless if it's been moved or renamed or if the default subvolume has been changed.



Chris Murphy

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 2304 bytes --]

On 13.10.2013 20:04, Chris Murphy wrote:
> On Sep 26, 2013, at 4:15 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
> 
>> On 26.09.2013 22:51, Chris Murphy wrote:
>>>
>>> Open question is if on BIOS hardware, if a 1MB BIOSBoot is preferred over the 64KB Btrfs bootloader pad? I don't know off hand if each member disk, or subsequently added disks, each have this 64KB pad or just the first member.
>>>
>> This is besides the point. I'm fine to discuss such things but in a
>> separate thread.
> 
> 2nd question is whether additional things are needed for /boot on btrfs. Using qemu/kvm and vbox I've tested maybe 6 virtual disks using btrfs single, raid0, raid1, and raid10 configurations, and consistently GRUB can boot /boot as a subvolume. I don't know how resilient this is as the file system is used and chunks end up on different devices or if this is sufficiently abstracted that it doesn't matter and just works; including if devices are removed and added.
> 
One thing that we would really need is some stability on the level of
concepts. It happened several times in the past and when it happened
again with mounting by volume id, the stance is that as long as concepts
change so regularly and we get no notification or consultation from
devs, we should keep code as-is and handle, that is handling sane cases
and skip the others, like subvolumes without a name.
> 3rd question, related, is if it's needed and planned work for GRUB to know about subvolid such as (hd0,gpt5,sv218)
> so that it can directly locate a subvolume regardless if it's been moved or renamed or if the default subvolume has been changed.
1) Between brackets is disk names and changing this would result in
utter mess. On other hand something in second part is possible. E.g.
(hd0,gpt5):btrfs,<parameters>/
is possible.
But I don't see why one should handle supporting renaming subvolumes
transparently. It would be like having to handle file renames
transparently. Should one reference all files in all config files in all
software by inode numbers, just in event it gets renamed?
> 
> 
> 
> Chris Murphy
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Oct 13, 2013, at 1:47 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> On 13.10.2013 20:04, Chris Murphy wrote:
>> On Sep 26, 2013, at 4:15 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
>> 
>>> On 26.09.2013 22:51, Chris Murphy wrote:
>>>> 
>>>> Open question is if on BIOS hardware, if a 1MB BIOSBoot is preferred over the 64KB Btrfs bootloader pad?


What about this question?


>> 2nd question is whether additional things are needed for /boot on btrfs. Using qemu/kvm and vbox I've tested maybe 6 virtual disks using btrfs single, raid0, raid1, and raid10 configurations, and consistently GRUB can boot /boot as a subvolume. I don't know how resilient this is as the file system is used and chunks end up on different devices or if this is sufficiently abstracted that it doesn't matter and just works; including if devices are removed and added.
>> 
> One thing that we would really need is some stability on the level of
> concepts. It happened several times in the past and when it happened
> again with mounting by volume id, the stance is that as long as concepts
> change so regularly and we get no notification or consultation from
> devs, we should keep code as-is and handle, that is handling sane cases
> and skip the others, like subvolumes without a name.

How does one create a subvolume without a name? All subvolumes have had ID's since at least 2008, and it's been possible to mount by name or ID for quite a few years at least as well.

What is new since kernel v3.2, almost 2 years ago, is the ability to mount subvolumes by full path, relative to top level subvolume rather than the default subvolume. This didn't change a basic concept, it just made an existing one a lot more flexible (and practical). So I'm uncertain what concepts are changing regularly.



>> 3rd question, related, is if it's needed and planned work for GRUB to know about subvolid such as (hd0,gpt5,sv218)
>> so that it can directly locate a subvolume regardless if it's been moved or renamed or if the default subvolume has been changed.
> 1) Between brackets is disk names and changing this would result in
> utter mess. On other hand something in second part is possible. E.g.
> (hd0,gpt5):btrfs,<parameters>/
> is possible.

That seems reasonable.

> But I don't see why one should handle supporting renaming subvolumes
> transparently. It would be like having to handle file renames
> transparently.

It's more like having to handle partition name changes transparently, which of course is the current behavior. Even if I were to move the partition to different blocks on a disk, as long as its partition number is consistent, it's locatable, including by GRUB, regardless of partition name.


> Should one reference all files in all config files in all
> software by inode numbers, just in event it gets renamed?

I don't think the file analogy works very well for subvolumes. Subvolumes have some behaviors like folders, and some behaviors like LVs, and some behaviors like entirely unique file systems volumes.

It might be  possible to refer to subvolume UUID, which is also unique. I haven't tested if mount works with this.


Chris Murphy

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 4077 bytes --]

On 13.10.2013 22:59, Chris Murphy wrote:
> 
> 
> On Oct 13, 2013, at 1:47 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
> 
>> On 13.10.2013 20:04, Chris Murphy wrote:
>>> On Sep 26, 2013, at 4:15 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
>>>
>>>> On 26.09.2013 22:51, Chris Murphy wrote:
>>>>>
>>>>> Open question is if on BIOS hardware, if a 1MB BIOSBoot is preferred over the 64KB Btrfs bootloader pad?
> 
> 
> What about this question?
> 
BIOS Boot partition is prefered. Currently btrfs 64K part isn't used to
whole potential. It will change.
> 
>>> 2nd question is whether additional things are needed for /boot on btrfs. Using qemu/kvm and vbox I've tested maybe 6 virtual disks using btrfs single, raid0, raid1, and raid10 configurations, and consistently GRUB can boot /boot as a subvolume. I don't know how resilient this is as the file system is used and chunks end up on different devices or if this is sufficiently abstracted that it doesn't matter and just works; including if devices are removed and added.
>>>
>> One thing that we would really need is some stability on the level of
>> concepts. It happened several times in the past and when it happened
>> again with mounting by volume id, the stance is that as long as concepts
>> change so regularly and we get no notification or consultation from
>> devs, we should keep code as-is and handle, that is handling sane cases
>> and skip the others, like subvolumes without a name.
> 
> How does one create a subvolume without a name? All subvolumes have had ID's since at least 2008, and it's been possible to mount by name or ID for quite a few years at least as well.
> 
Then this illusion was created by the /proc/mountinfo listing mounted
subdirectory as "/" when mounted by id (or something like that).
> 
> 
>>> 3rd question, related, is if it's needed and planned work for GRUB to know about subvolid such as (hd0,gpt5,sv218)
>>> so that it can directly locate a subvolume regardless if it's been moved or renamed or if the default subvolume has been changed.
>> 1) Between brackets is disk names and changing this would result in
>> utter mess. On other hand something in second part is possible. E.g.
>> (hd0,gpt5):btrfs,<parameters>/
>> is possible.
> 
> That seems reasonable.
> 
>> But I don't see why one should handle supporting renaming subvolumes
>> transparently. It would be like having to handle file renames
>> transparently.
> 
> It's more like having to handle partition name changes transparently, which of course is the current behavior. Even if I were to move the partition to different blocks on a disk,

> as long as its partition number is consistent, it's locatable, including by GRUB, regardless of partition name.
> 
moving blocks is analogous to defragging a file which is also supported.
Filesystem label works on different level, namely it's part of fs. In
msdos partition scheme partitions have no name other than a numerical ID.
> 
>> Should one reference all files in all config files in all
>> software by inode numbers, just in event it gets renamed?
> 
> I don't think the file analogy works very well for subvolumes. Subvolumes have some behaviors like folders, and some behaviors like LVs,

> and some behaviors like entirely unique file systems volumes.
I feel like subvolumes are glorified folders and would have prefered
directories becoming more powerful rather than having a completely new
concept.
On both ZFS and btrfs as far as GRUB is concerned are directories with
just slightly different structure.
Why would numbers be preferable to names? The rename may very well be
intentional in order to make other OS and/or version to boot.
> 
> It might be  possible to refer to subvolume UUID, which is also unique. I haven't tested if mount works with this.
> 
> 
> Chris Murphy
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Oct 13, 2013, at 5:31 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> On 13.10.2013 22:59, Chris Murphy wrote:
>> 
>> How does one create a subvolume without a name? All subvolumes have had ID's since at least 2008, and it's been possible to mount by name or ID for quite a few years at least as well.
>> 
> Then this illusion was created by the /proc/mountinfo listing mounted
> subdirectory as "/" when mounted by id (or something like that).

The top level subvolume (id 5) is likely reported as "/" just like is the case when mounting any file system.

It's possible that changing the default subvolume causes this same behavior rather than mount reporting the full path. I haven't tested this.

> 
>> and some behaviors like entirely unique file systems volumes.
> I feel like subvolumes are glorified folders and would have prefered
> directories becoming more powerful rather than having a completely new
> concept.

That's not possible. It had to be this way to support snapshotting. A snapshot is a subvolume. The subvolume only appears as a directory, in reality the directory links to the subvolume which are a completely separate btree and analogous to a whole separate file system. And since its a separate file system, the inode numbers start over. Conversely a directory is just an item like a file in the tree, it's not a tree in its own right, which a subvolume is.

> On both ZFS and btrfs as far as GRUB is concerned are directories with
> just slightly different structure.

Understood, but I think this is less ideal. It offers less leverage and usage of  the file system as a boot, even as rootfs, file system.

> Why would numbers be preferable to names?

Without the use of subvolid= we  either have to always use full paths for subvolumes (instead of relative), or we can't use btrfs set-default to change the default subvolume. If the paths to subvolumes are relative to the default subvolume, then changing the default subvolume breaks booting. And full paths could become rather cumbersome, since the hierarchy is effectively unlimited.

Today, maybe it's not the best scenario to have two or three OS's installed on one btrfs volume, in separate subvolumes, each with hundreds of snapshots. But it's designed to enable such a workflow once it's stable. I think using full paths (always relative to the top level subvolume which never changes even if the default subvolume is changed) is fine in the near term. And it may be some workflows simply prefer the (human user) transparency that comes with full paths. But from my perspective, a fixed number means I can completely reorganize a hierarchy with no other changes.

> The rename may very well be
> intentional in order to make other OS and/or version to boot.

Yes, and I'm not suggesting the end to supporting either full path, or relative path referencing of subvolumes. Just that it's also possible to use subvolid.


Chris Murphy

Re: booting btrfs

[Andrey Borzenkov]

В Sun, 13 Oct 2013 17:58:41 -0600
Chris Murphy <lists@colorremedies.com> пишет:

> 
> On Oct 13, 2013, at 5:31 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
> 
> > On 13.10.2013 22:59, Chris Murphy wrote:
> >> 
> >> How does one create a subvolume without a name? All subvolumes have had ID's since at least 2008, and it's been possible to mount by name or ID for quite a few years at least as well.
> >> 
> > Then this illusion was created by the /proc/mountinfo listing mounted
> > subdirectory as "/" when mounted by id (or something like that).
> 
> The top level subvolume (id 5) is likely reported as "/" just like is the case when mounting any file system.
> 
> It's possible that changing the default subvolume causes this same behavior rather than mount reporting the full path. I haven't tested this.
> 

No, it is inconsistent behavior for named subvolume and subvolume ids
mounts.

For named subvolume btrfs internally mounts top level directory
(actually using subvolid=0 for this internal mount) and then performs
equivalent of bind mount on subvolume. For subvolid it actually
sets root of filesystem to subvolume during mount. As mountinfo
displays path relative to filesystem root, it is always "/" if subvolid
was used. I do not know if there are practical differences between the
two.

> > 
> >> and some behaviors like entirely unique file systems volumes.
> > I feel like subvolumes are glorified folders and would have prefered
> > directories becoming more powerful rather than having a completely new
> > concept.
> 
> That's not possible. It had to be this way to support snapshotting. A snapshot is a subvolume. The subvolume only appears as a directory, in reality the directory links to the subvolume which are a completely separate btree and analogous to a whole separate file system. And since its a separate file system, the inode numbers start over. Conversely a directory is just an item like a file in the tree, it's not a tree in its own right, which a subvolume is.
> 
> > On both ZFS and btrfs as far as GRUB is concerned are directories with
> > just slightly different structure.
> 
> Understood, but I think this is less ideal. It offers less leverage and usage of  the file system as a boot, even as rootfs, file system.
> 
> > Why would numbers be preferable to names?
> 
> Without the use of subvolid= we  either have to always use full paths for subvolumes (instead of relative), or we can't use btrfs set-default to change the default subvolume. If the paths to subvolumes are relative to the default subvolume, then changing the default subvolume breaks booting. And full paths could become rather cumbersome, since the hierarchy is effectively unlimited.
> 

Why full paths are cumbersome? The

mount -o subvol=path/to/boot /dev/sdX /mnt
vi /mnt/grub/boot.cfg

is 100% equivalent to

mount -o subvolid=0 /dev/sdX /mnt
vi /mnt/path/to/boot/grub/boot.cfg

And GRUB is concerned only with value of $prefix. So as long as
grub-install can resolve path name of /boot/grub to full path from
btrfs root including subvolumes it is OK. And to use subvolume ID you
will need to detect that path is on subvolume anyway.

> Today, maybe it's not the best scenario to have two or three OS's installed on one btrfs volume, in separate subvolumes, each with hundreds of snapshots. But it's designed to enable such a workflow once it's stable. I think using full paths (always relative to the top level subvolume which never changes even if the default subvolume is changed) is fine in the near term. And it may be some workflows simply prefer the (human user) transparency that comes with full paths. But from my perspective, a fixed number means I can completely reorganize a hierarchy with no other changes.
> 

Could you please provide example of what you mean here?

> > The rename may very well be
> > intentional in order to make other OS and/or version to boot.
> 
> Yes, and I'm not suggesting the end to supporting either full path, or relative path referencing of subvolumes. Just that it's also possible to use subvolid.
> 

What about grub-mkconfig? It does not pass explicit subvolume if it is
not present in mountinfo, which means a) if someone changes default
subvolume it will end up mounting wrong root and b) it breaks if mount
by subvolid is used.

Re: booting btrfs

[Chris Murphy]

On Oct 13, 2013, at 11:28 PM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:

> В Sun, 13 Oct 2013 17:58:41 -0600
> Chris Murphy <lists@colorremedies.com> пишет:
> 
>> 
>> On Oct 13, 2013, at 5:31 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
>> 
>>> On 13.10.2013 22:59, Chris Murphy wrote:
>>>> 
>>>> How does one create a subvolume without a name? All subvolumes have had ID's since at least 2008, and it's been possible to mount by name or ID for quite a few years at least as well.
>>>> 
>>> Then this illusion was created by the /proc/mountinfo listing mounted
>>> subdirectory as "/" when mounted by id (or something like that).
>> 
>> The top level subvolume (id 5) is likely reported as "/" just like is the case when mounting any file system.
>> 
>> It's possible that changing the default subvolume causes this same behavior rather than mount reporting the full path. I haven't tested this.
>> 
> 
> No, it is inconsistent behavior for named subvolume and subvolume ids
> mounts.
> 
> For named subvolume btrfs internally mounts top level directory
> (actually using subvolid=0 for this internal mount) and then performs
> equivalent of bind mount on subvolume. For subvolid it actually
> sets root of filesystem to subvolume during mount. As mountinfo
> displays path relative to filesystem root, it is always "/" if subvolid
> was used. I do not know if there are practical differences between the
> two.

I'm finding partial corroboration with this. Mounting subvolid= does not show either subvolume id or name, which seems like a problem.

At the moment, I'm unable to confirm/deny that mountinfo correctly shows full path because when I change the default subvolume, I'm dropped to a grub rescue prompt on the next boot. This isn't expected because set reveals:

prefix=(hd0,msdos1)/boot/grub2
root=jd0,msdos1

In fact, /boot is a subvolume under top level 5. So if GRUB is using full paths, which it seems to be, it should be able to still resolve this even though the default subvolume is not the top level 5 subvolume. Yet it's not working. So it seems that GRUB is using relative pathnames to the default subvolume. That's a problem also because it effectively means btrfs subvolume set-default can't be used without breaking bootability.


> 
> Why full paths are cumbersome? The
> 
> mount -o subvol=path/to/boot /dev/sdX /mnt
> vi /mnt/grub/boot.cfg
> 
> is 100% equivalent to
> 
> mount -o subvolid=0 /dev/sdX /mnt
> vi /mnt/path/to/boot/grub/boot.cfg

It's not 100% equivalent in behavior. For one, paths can become quite long depending on the snapshotting strategy and complexity of the hierarchy. If that hierarchy ever changes, now the mount option must change and the grub.cfg must change or the system is unbootable. That's not the case with ids; the system remains bootable without having to modify fstab or grub.cfg.


> 
> And GRUB is concerned only with value of $prefix. So as long as
> grub-install can resolve path name of /boot/grub to full path from
> btrfs root including subvolumes it is OK.

I'm uncertain about this, because upon changing the default subvolume, I can't boot. Hence at least core.img appears to treat what looks like a full path, as relative to the default subvolume, whereas full paths should always be treated relative to top level 5 subvolume.

> And to use subvolume ID you
> will need to detect that path is on subvolume anyway.

I don't know what this means. The file system clearly can resolve the location of a subvolume independent of path.

> 
>> Today, maybe it's not the best scenario to have two or three OS's installed on one btrfs volume, in separate subvolumes, each with hundreds of snapshots. But it's designed to enable such a workflow once it's stable. I think using full paths (always relative to the top level subvolume which never changes even if the default subvolume is changed) is fine in the near term. And it may be some workflows simply prefer the (human user) transparency that comes with full paths. But from my perspective, a fixed number means I can completely reorganize a hierarchy with no other changes.
>> 
> 
> Could you please provide example of what you mean here?

If I use subvolid's everywhere, it means I could grab Fedora's default hierarchy: boot root home, which are subvolumes, and move them into a either a folder or subvolume called "Fedora 20" and upon reboot everything still works. Right now, that's not the case. If I move these things, or rename them, boot breaks.

It's sortof like using volume labels or partition names to quality paths. There's a reason why UUID is preferred to labels, because if the user depends on labels, then relabels a volume, they can't boot. So there's a better way to do this that preserves the user's ability to name user domain things like volume labels, while not breaking system domain things like an immutable way of booting the system.

It's about making bootability more robust.



>>> The rename may very well be
>>> intentional in order to make other OS and/or version to boot.
>> 
>> Yes, and I'm not suggesting the end to supporting either full path, or relative path referencing of subvolumes. Just that it's also possible to use subvolid.
>> 
> 
> What about grub-mkconfig? It does not pass explicit subvolume if it is
> not present in mountinfo, which means a) if someone changes default
> subvolume it will end up mounting wrong root and b) it breaks if mount
> by subvolid is used.

Clearly there are some things returned by mountinfo that need fixing and enhancement. The same can be said about the mount command. And tools like gnome-disk-utility need some things about 

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

> So it seems that GRUB is using relative pathnames to the default subvolume.
This is not intentional. When this part of code was written there was no
set-default available at all so this couldn't be tested and I simply
followed the specification. It told to take root_tree and
root_dir_objectid from superblock then go to "default" directory. What
of this needs to be changed? Just remove "default" and make it part of
path? We would need to change grub-mkrelpath to match runtime behaviour.
Is there a way to detect that mountinfo gives garbage and somehow get
where the real root points?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Oct 14, 2013, at 1:29 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

>> So it seems that GRUB is using relative pathnames to the default subvolume.
> This is not intentional. When this part of code was written there was no
> set-default available at all so this couldn't be tested and I simply
> followed the specification. It told to take root_tree and
> root_dir_objectid from superblock then go to "default" directory. What
> of this needs to be changed? Just remove "default" and make it part of
> path? We would need to change grub-mkrelpath to match runtime behaviour.

Somehow the default subvolume needs to be ignored and always start from top level 5 subvolume for full pathnames, which is what GRUB already seems to try to use for prefix. I'll inquire and hopefully get someone more knowledgable to reply here.


> Is there a way to detect that mountinfo gives garbage and somehow get
> where the real root points?

I don't know. I've asked on linux-btrfs@. Instead of rebooting, I merely tried mounting without options after changing the default subvolume to a nested subvolume (one attempt subvolume in a subvolume, another a subvolume in a directory): in both cases /proc/self/mountinfo reports / as the root, not the full path or ID of the subvolume actually mounted.

Somehow it seems like the mountinfo root field should return a block device and full path to the mounted subvolume or its ID. Currently it seems like a problem.


Chris Murphy

Re: booting btrfs

[Chris Murphy]

On Oct 14, 2013, at 1:29 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

>> So it seems that GRUB is using relative pathnames to the default subvolume.
> This is not intentional. When this part of code was written there was no
> set-default available at all so this couldn't be tested and I simply
> followed the specification. It told to take root_tree and
> root_dir_objectid from superblock then go to "default" directory. What
> of this needs to be changed? Just remove "default" and make it part of
> path? We would need to change grub-mkrelpath to match runtime behaviour.
> Is there a way to detect that mountinfo gives garbage and somehow get
> where the real root points?

Here's the response. It seems similar but not identical to what you described above.

http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27955.html


Chris Murphy

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

On 14.10.2013 22:45, Chris Murphy wrote:
> 
> On Oct 14, 2013, at 1:29 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
> 
>>> So it seems that GRUB is using relative pathnames to the default subvolume.
>> This is not intentional. When this part of code was written there was no
>> set-default available at all so this couldn't be tested and I simply
>> followed the specification. It told to take root_tree and
>> root_dir_objectid from superblock then go to "default" directory. What
>> of this needs to be changed? Just remove "default" and make it part of
>> path? We would need to change grub-mkrelpath to match runtime behaviour.
>> Is there a way to detect that mountinfo gives garbage and somehow get
>> where the real root points?
> 
> Here's the response. It seems similar but not identical to what you described above.
> 
> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27955.html
> 
I don't see what is the difference between that and what GRUB currently
does. Will look tomorrow morning.
> 
> Chris Murphy
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

On 14.10.2013 22:45, Chris Murphy wrote:
> 
> On Oct 14, 2013, at 1:29 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
> 
>>> So it seems that GRUB is using relative pathnames to the default subvolume.
>> This is not intentional. When this part of code was written there was no
>> set-default available at all so this couldn't be tested and I simply
>> followed the specification. It told to take root_tree and
>> root_dir_objectid from superblock then go to "default" directory. What
>> of this needs to be changed? Just remove "default" and make it part of
>> path? We would need to change grub-mkrelpath to match runtime behaviour.
>> Is there a way to detect that mountinfo gives garbage and somehow get
>> where the real root points?
> 
> Here's the response. It seems similar but not identical to what you described above.
> 
> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27955.html
> 
Possibly this is the answer:
=== modified file 'grub-core/fs/btrfs.c'
--- grub-core/fs/btrfs.c	2013-01-21 01:33:46 +0000
+++ grub-core/fs/btrfs.c	2013-10-14 21:00:53 +0000
@@ -1217,7 +1217,7 @@

   *type = GRUB_BTRFS_DIR_ITEM_TYPE_DIRECTORY;
   *tree = data->sblock.root_tree;
-  key->object_id = data->sblock.root_dir_objectid;
+  key->object_id = grub_cpu_to_le64_compile_time (5);
   key->type = GRUB_BTRFS_ITEM_TYPE_DIR_ITEM;
   key->offset = 0;
   skip_default = 1;




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Oct 14, 2013, at 3:01 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> On 14.10.2013 22:45, Chris Murphy wrote:
>> 
>> On Oct 14, 2013, at 1:29 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
>> 
>>>> So it seems that GRUB is using relative pathnames to the default subvolume.
>>> This is not intentional. When this part of code was written there was no
>>> set-default available at all so this couldn't be tested and I simply
>>> followed the specification. It told to take root_tree and
>>> root_dir_objectid from superblock then go to "default" directory. What
>>> of this needs to be changed? Just remove "default" and make it part of
>>> path? We would need to change grub-mkrelpath to match runtime behaviour.
>>> Is there a way to detect that mountinfo gives garbage and somehow get
>>> where the real root points?
>> 
>> Here's the response. It seems similar but not identical to what you described above.
>> 
>> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27955.html
>> 
> Possibly this is the answer:
> === modified file 'grub-core/fs/btrfs.c'
> --- grub-core/fs/btrfs.c	2013-01-21 01:33:46 +0000
> +++ grub-core/fs/btrfs.c	2013-10-14 21:00:53 +0000
> @@ -1217,7 +1217,7 @@
> 
>   *type = GRUB_BTRFS_DIR_ITEM_TYPE_DIRECTORY;
>   *tree = data->sblock.root_tree;
> -  key->object_id = data->sblock.root_dir_objectid;
> +  key->object_id = grub_cpu_to_le64_compile_time (5);
>   key->type = GRUB_BTRFS_ITEM_TYPE_DIR_ITEM;
>   key->offset = 0;
>   skip_default = 1;
> 

I applied this to current bzr and built it. With the default default subvolume, i.e set-default is ID 5, I get:

GRUB loading ..
Welcome to GRUB!

error: file '/' not found.
Entering rescue mode…
grub rescue>


Chris Murphy

Re: booting btrfs

[Chris Murphy]

On Oct 14, 2013, at 5:09 PM, Chris Murphy <lists@colorremedies.com> wrote:

> 
> On Oct 14, 2013, at 3:01 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
> 
>> On 14.10.2013 22:45, Chris Murphy wrote:
>>> 
>>> On Oct 14, 2013, at 1:29 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
>>> 
>>>>> So it seems that GRUB is using relative pathnames to the default subvolume.
>>>> This is not intentional. When this part of code was written there was no
>>>> set-default available at all so this couldn't be tested and I simply
>>>> followed the specification. It told to take root_tree and
>>>> root_dir_objectid from superblock then go to "default" directory. What
>>>> of this needs to be changed? Just remove "default" and make it part of
>>>> path? We would need to change grub-mkrelpath to match runtime behaviour.
>>>> Is there a way to detect that mountinfo gives garbage and somehow get
>>>> where the real root points?
>>> 
>>> Here's the response. It seems similar but not identical to what you described above.
>>> 
>>> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27955.html
>>> 
>> Possibly this is the answer:
>> === modified file 'grub-core/fs/btrfs.c'
>> --- grub-core/fs/btrfs.c	2013-01-21 01:33:46 +0000
>> +++ grub-core/fs/btrfs.c	2013-10-14 21:00:53 +0000
>> @@ -1217,7 +1217,7 @@
>> 
>>  *type = GRUB_BTRFS_DIR_ITEM_TYPE_DIRECTORY;
>>  *tree = data->sblock.root_tree;
>> -  key->object_id = data->sblock.root_dir_objectid;
>> +  key->object_id = grub_cpu_to_le64_compile_time (5);
>>  key->type = GRUB_BTRFS_ITEM_TYPE_DIR_ITEM;
>>  key->offset = 0;
>>  skip_default = 1;
>> 
> 
> I applied this to current bzr and built it. With the default default subvolume, i.e set-default is ID 5, I get:
> 
> GRUB loading ..
> Welcome to GRUB!
> 
> error: file '/' not found.
> Entering rescue mode…
> grub rescue>
> 


Nevermind. The /etc/fstab was technically not valid at the time I ran grub-install, because I was using subvolid= there, and grub-install probably picked that up.

So with your change applied, I can still boot with 'btrfs subvol set-default 5 /'.  However if I change the default subvolume, I get the same results as without the change:

error: file 'boot/grub/i386-pc/normal.mod' not found
Entering rescue mode…

grub rescue> set
prefix=(hd0,msdos1)/boot/grub
root=hd0,msdos1
grub rescue> ls (hd0,msdos1)/
youfoundme


So instead of it listing the subvolumes in the top level ID 5 subvolume, which includes /boot, it lists 'youfoundme' which is a file in a nested subvolume with ID 266 that has been 'btrfs subvol set-default 266 /'


Chris Murphy

Re: booting btrfs

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 1466 bytes --]

В Mon, 14 Oct 2013 22:50:10 +0200
Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:

> On 14.10.2013 22:45, Chris Murphy wrote:
> > 
> > On Oct 14, 2013, at 1:29 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
> > 
> >>> So it seems that GRUB is using relative pathnames to the default subvolume.
> >> This is not intentional. When this part of code was written there was no
> >> set-default available at all so this couldn't be tested and I simply
> >> followed the specification. It told to take root_tree and
> >> root_dir_objectid from superblock then go to "default" directory. What
> >> of this needs to be changed? Just remove "default" and make it part of
> >> path? We would need to change grub-mkrelpath to match runtime behaviour.
> >> Is there a way to detect that mountinfo gives garbage and somehow get
> >> where the real root points?
> > 
> > Here's the response. It seems similar but not identical to what you described above.
> > 
> > http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27955.html
> > 
> I don't see what is the difference between that and what GRUB currently
> does. Will look tomorrow morning.


I have a feeling that there was some misunderstanding about the
problem. Or btrfs developers do not really intend to have multiple
"active filesystems" at the same time and see "set-default" as the only
legal way to switch between filesystem instances.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: booting btrfs

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 2454 bytes --]

В Mon, 14 Oct 2013 23:01:45 +0200
Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:

> On 14.10.2013 22:45, Chris Murphy wrote:
> > 
> > On Oct 14, 2013, at 1:29 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
> > 
> >>> So it seems that GRUB is using relative pathnames to the default subvolume.
> >> This is not intentional. When this part of code was written there was no
> >> set-default available at all so this couldn't be tested and I simply
> >> followed the specification. It told to take root_tree and
> >> root_dir_objectid from superblock then go to "default" directory. What
> >> of this needs to be changed? Just remove "default" and make it part of
> >> path? We would need to change grub-mkrelpath to match runtime behaviour.
> >> Is there a way to detect that mountinfo gives garbage and somehow get
> >> where the real root points?
> > 
> > Here's the response. It seems similar but not identical to what you described above.
> > 
> > http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27955.html
> > 
> Possibly this is the answer:
> === modified file 'grub-core/fs/btrfs.c'
> --- grub-core/fs/btrfs.c	2013-01-21 01:33:46 +0000
> +++ grub-core/fs/btrfs.c	2013-10-14 21:00:53 +0000
> @@ -1217,7 +1217,7 @@
> 
>    *type = GRUB_BTRFS_DIR_ITEM_TYPE_DIRECTORY;
>    *tree = data->sblock.root_tree;
> -  key->object_id = data->sblock.root_dir_objectid;
> +  key->object_id = grub_cpu_to_le64_compile_time (5);
>    key->type = GRUB_BTRFS_ITEM_TYPE_DIR_ITEM;
>    key->offset = 0;
>    skip_default = 1;
> 
> 
> 

Nope, the actual problem here is that find_path starts with looking up
"default". Which arguably it should not do, but simply interpret
path name as starting from top level.

If btrfs filesystem has subvolumes /sub1 and /sub2, even we have set
default to /sub2, it is still possible to mount /sub1 using explicit
"mount -o subvol=/sub1 ...". Given current grub implementation any
access outside of default subvolume is impossible.

But this also means that grub user level tools have to resolve all path
names to be absolute. It is possible - "btrfs subvolume list" gives you
subvolume paths, so this information is available.

And, BTW, subvolume can be inside of normal directory as well. I.e.

mkdir /dir
btrfs subvolume create /dir/sub

is legal. And you can still mount /dir/sub using option -o
subvol=/dir/sub.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: booting btrfs

[Chris Murphy]

On Oct 14, 2013, at 8:33 PM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:

> В Mon, 14 Oct 2013 22:50:10 +0200
> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
> 
>> On 14.10.2013 22:45, Chris Murphy wrote:
>>> 
>>> On Oct 14, 2013, at 1:29 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
>>> 
>>>>> So it seems that GRUB is using relative pathnames to the default subvolume.
>>>> This is not intentional. When this part of code was written there was no
>>>> set-default available at all so this couldn't be tested and I simply
>>>> followed the specification. It told to take root_tree and
>>>> root_dir_objectid from superblock then go to "default" directory. What
>>>> of this needs to be changed? Just remove "default" and make it part of
>>>> path? We would need to change grub-mkrelpath to match runtime behaviour.
>>>> Is there a way to detect that mountinfo gives garbage and somehow get
>>>> where the real root points?
>>> 
>>> Here's the response. It seems similar but not identical to what you described above.
>>> 
>>> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27955.html
>>> 
>> I don't see what is the difference between that and what GRUB currently
>> does. Will look tomorrow morning.
> 
> 
> I have a feeling that there was some misunderstanding about the
> problem. Or btrfs developers do not really intend to have multiple
> "active filesystems" at the same time and see "set-default" as the only
> legal way to switch between filesystem instances.


There are a number of use cases, and already for a while Fedora and openSuse leverage it, where boot, root, and home, are each subvolumes and each is mounted at the proper mount points. In effect three file systems are mounted at the same time.

A possible use of set-default would be as follows, each of the items is a subvolume:

[top level 5]
	home ID 274
	Fedora 19 ID 259
		boot ID 260
		root ID 261
		home ID 262
	Fedora 20 ID 270
		boot ID 271
		root ID 272
		home ID 273
	openSuse ID 265
		boot ID 266
		home ID 267
		usr ID 268
		var ID 269


By merely changing the set-default subvolume from 259 to 270, I'm booting a different version of Fedora. The same principle can work for making a subvolume "Fedora19-snapshot<date>" and then snapshotting the subvolumes in the Fedora 19 subvolume. Now I can use set-default to rollback to the snapshotted version.

This works because at least Fedora's kernel command line, and fstab use relative subvolume  paths. So by changing the default subvolume, the relative paths are valid for a different version/snapshot of Fedora. And actually it works because of the unintended behavior I've found, where what looks like an absolute path for prefix, is relative. If it were followed absolutely regardless of default subvolume, prefix= would always point to the exact same grub folder and hence the same grub.cfg.

So actually, this is a bit more than somewhat messy, whether the prefix should be formatted relative to the default subvolume, or absolute from top level 5 subvolume. It depends on the use case.

So just like '-o subvol=root' has a different meaning than '-o subvol=/root' there is a difference between prefix=(hd0,msdos1)boot/grub/ and prefix=(hd0,msdos1)/boot/grub/

Chris Murphy

Re: booting btrfs

[Andrey Borzenkov]

В Mon, 14 Oct 2013 21:12:06 -0600
Chris Murphy <lists@colorremedies.com> пишет:

> 
> So actually, this is a bit more than somewhat messy, whether the prefix should be formatted relative to the default subvolume, or absolute from top level 5 subvolume. It depends on the use case.
> 
> So just like '-o subvol=root' has a different meaning than '-o subvol=/root' there is a difference between prefix=(hd0,msdos1)boot/grub/ and prefix=(hd0,msdos1)/boot/grub/
> 

I do not know whether it was the case in the past, but today there is
*no* difference between using absolute or relative form. As I already
said, internally btrfs mounts top level and starts from there anyway.


linux-xtd6:/mnt # btrfs subvolume list /mnt
ID 256 gen 7 top level 5 path dir1/sub1
ID 257 gen 7 top level 5 path dir2/sub2
linux-xtd6:/mnt # btrfs subvolume get-default /mnt
ID 256 gen 7 top level 5 path dir1/sub1
linux-xtd6:~ # umount /mnt
linux-xtd6:~ # mount /dev/sdb /mnt
linux-xtd6:~ # ls -l /mnt
total 0
-rw-r--r-- 1 root root 0 Oct 15 16:50 I am subvolume 1
linux-xtd6:~ # umount /mnt
linux-xtd6:~ # mount -o subvol=/dir2/sub2 /dev/sdb /mnt
linux-xtd6:~ # ls -l /mnt
total 0
-rw-r--r-- 1 root root 0 Oct 15 16:50 I am subvolume 2
linux-xtd6:~ # umount /mnt
linux-xtd6:~ # mount -o subvol=dir2/sub2 /dev/sdb /mnt
linux-xtd6:~ # ls -l /mnt
total 0
-rw-r--r-- 1 root root 0 Oct 15 16:50 I am subvolume 2

I'm not sure when and how top level may become != 5.

Re: booting btrfs

[Chris Murphy]

On Oct 15, 2013, at 10:58 AM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:

> 
> I do not know whether it was the case in the past, but today there is
> *no* difference between using absolute or relative form.

There is a difference because I have a case where one works and the other doesn't. But I think some regression has occurred, because this case is a subvol that won't mount relative to its top level subvolume set as the default subvolume; it can still be mounted with absolute path.

http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27979.html

The FAQ and changelogs still indicate a distinction between full path names and relative ones. But it might be related to a different regression where I can't move subvols into subvols.


> I'm not sure when and how top level may become != 5.

starting where you left off with the sub2 subvolume mounted

# btrfs subvol create /mnt/nested
# btrfs subvol list /mnt
ID 262 gen 135 top level 5 path dir1/sub1
ID 263 gen 140 top level 5 path dir2/sub2
ID 264 gen 140 top level 263 path nested


Chris Murphy

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

On 15.10.2013 21:47, Chris Murphy wrote:
> 
> On Oct 15, 2013, at 10:58 AM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:
> 
>>
>> I do not know whether it was the case in the past, but today there is
>> *no* difference between using absolute or relative form.
> 
> There is a difference because I have a case where one works and the other doesn't. But I think some regression has occurred, because this case is a subvol that won't mount relative to its top level subvolume set as the default subvolume; it can still be mounted with absolute path.
> 
> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27979.html
> 
This is about kernel. Not GRUB. GRUB doesn't follow over-fancy features
like those.
> The FAQ and changelogs still indicate a distinction between full path names and relative ones. But it might be related to a different regression where I can't move subvols into subvols.
> 
> 
>> I'm not sure when and how top level may become != 5.
> 
> starting where you left off with the sub2 subvolume mounted
> 
> # btrfs subvol create /mnt/nested
> # btrfs subvol list /mnt
> ID 262 gen 135 top level 5 path dir1/sub1
> ID 263 gen 140 top level 5 path dir2/sub2
> ID 264 gen 140 top level 263 path nested
> 
> 
> Chris Murphy
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Oct 15, 2013, at 2:02 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> On 15.10.2013 21:47, Chris Murphy wrote:
>> 
>> On Oct 15, 2013, at 10:58 AM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:
>> 
>>> 
>>> I do not know whether it was the case in the past, but today there is
>>> *no* difference between using absolute or relative form.
>> 
>> There is a difference because I have a case where one works and the other doesn't. But I think some regression has occurred, because this case is a subvol that won't mount relative to its top level subvolume set as the default subvolume; it can still be mounted with absolute path.
>> 
>> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg27979.html
>> 
> This is about kernel. Not GRUB. GRUB doesn't follow over-fancy features
> like those.

I understand that, but as a matter of follow-up to Andrey's post which was not directly related to GRUB, I thought it was appropriate to point out the unexpected behavior I'm seeing. And btrfs-progs is discussed at linux-btrfs@, not just kernel issues.

It's an open question for me, now, whether set-default is really working the way it's supposed to. For all I know it changes things on-disk in a way that also affects GRUB behavior contrary to expectations. So I think it's reasonable to be sure the expectations are properly aligned, and go from there.

The regression testing I've done is not close to exhaustive but the behavior I'm seeing in GRUB and in booted linux, are different from behaviors maybe two or three years ago. So the fact both have changed behaviors might be a bug that's in common and worth exploring.


Chris Murphy

Re: booting btrfs

[Chris Murphy]

On Oct 15, 2013, at 10:58 AM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:
> 
> I do not know whether it was the case in the past, but today there is
> *no* difference between using absolute or relative form.

OK so it's confirmed that with kernel 3.2 the subvol= behavior changed from relative to set-default subvolume to always being absolute whether it starts with / or not.

And subvolume set-default is intended to affect the mount command absent usage of subvol option.

So the change needed for GRUB is it should ignore the set-default and always treat prefix paths as absolute starting from top level 5 subvolume. That makes things more clear and less complicated and also won't break bootability if the user wants to e.g. always mount their current home subvolume by default, then nothing else is affected.


Chris Murphy

Re: booting btrfs

[Andrey Borzenkov]

В Tue, 15 Oct 2013 13:47:14 -0600
Chris Murphy <lists@colorremedies.com> пишет:

> 
> > I'm not sure when and how top level may become != 5.
> 
> starting where you left off with the sub2 subvolume mounted
> 
> # btrfs subvol create /mnt/nested
> # btrfs subvol list /mnt
> ID 262 gen 135 top level 5 path dir1/sub1
> ID 263 gen 140 top level 5 path dir2/sub2
> ID 264 gen 140 top level 263 path nested
> 

Now try to remount with subvolid=0 and you will see that all subvolumes
have top level == 5. So it appears to be some run-time property. I do
not know why it is exposed (i.e. how user space can or should make use
of it).

Now "parent" is more interesting because it really tells us parent
subvolume and it permanent on-disk property.

Re: booting btrfs

[Andrey Borzenkov]

В Mon, 14 Oct 2013 14:20:14 -0600
Chris Murphy <lists@colorremedies.com> пишет:

> 
> > Is there a way to detect that mountinfo gives garbage and somehow get
> > where the real root points?
> 
> I don't know. I've asked on linux-btrfs@. Instead of rebooting, I merely tried mounting without options after changing the default subvolume to a nested subvolume (one attempt subvolume in a subvolume, another a subvolume in a directory): in both cases /proc/self/mountinfo reports / as the root, not the full path or ID of the subvolume actually mounted.
> 
> Somehow it seems like the mountinfo root field should return a block device and full path to the mounted subvolume or its ID. Currently it seems like a problem.
> 

To quote one of btrfs developer (I had unrelated discussion on openSUSE
list):

--><--
This is a known problem, on my todo list, with a few non-working
solutions.

If you mount via subvol=/subvol then /proc/self/mountinfo will show
'subvol' as the mounted subvolume (4th column)

4 19 0:17 /subvol /mnt/ rw,relatime - btrfs /dev/sda15 rw,space_cache

but if the subvolume is set-default and then implicitly mounted,
mountinfo will not show that (that's the bug).
--><--

That said, information can be obtained also using different means
(btrfs utility or directly btrfs IOCTL). The question is to which
extent we want to depend on existence of btrfsprogs.

Re: booting btrfs

[Chris Murphy]

On Oct 15, 2013, at 8:45 PM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:

> В Tue, 15 Oct 2013 13:47:14 -0600
> Chris Murphy <lists@colorremedies.com> пишет:
> 
>> 
>>> I'm not sure when and how top level may become != 5.
>> 
>> starting where you left off with the sub2 subvolume mounted
>> 
>> # btrfs subvol create /mnt/nested
>> # btrfs subvol list /mnt
>> ID 262 gen 135 top level 5 path dir1/sub1
>> ID 263 gen 140 top level 5 path dir2/sub2
>> ID 264 gen 140 top level 263 path nested
>> 
> 
> Now try to remount with subvolid=0 and you will see that all subvolumes
> have top level == 5.

But the path is also different:

ID 264 gen 165 top level 5 path dir2/sub2/nested

So this is consistent behavior. From the top level 5 the path is dir2/sub2/nested, whereas from top level 263 the path is nested. Means the same thing.


> Now "parent" is more interesting because it really tells us parent
> subvolume and it permanent on-disk property.

Yes I wish there were some metadata support so that it's possible for the application layer to understand the relationship between subvolumes better; e.g. so that all snapshots for an OS can be co-located in a listing, and sorted by recency. That way it's possible to do rollbacks at boot time to the next most recent "snapshot" of the system. It looks like right now this is up to naming and hierarchy strategy.

Presently, at least one OS installer becomes confused and lists, without distinction, every OS snapshot regardless of location as if it were a unique installation. So a file system with 50 snapshots of rootfs, looks in the installer as if there are 50 unique installations of the OS. Confusing.

However, an explicit "parent" concept might be difficult because a snapshot of a subvolume is a subvolume. I can delete either the original "parent" or the new snapshot and the other remains intact. The "parent" does not need to be retained as a backing for the snapshot. So really a snapshot is not as much a specific thing as it is an activity, and this same behavior applies to the new LVM thin provisioning snapshots as well.

Chris Murphy

Re: booting btrfs

[Chris Murphy]

On Oct 15, 2013, at 8:50 PM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:

> В Mon, 14 Oct 2013 14:20:14 -0600
> Chris Murphy <lists@colorremedies.com> пишет:
> 
>> 
>>> Is there a way to detect that mountinfo gives garbage and somehow get
>>> where the real root points?
>> 
>> I don't know. I've asked on linux-btrfs@. Instead of rebooting, I merely tried mounting without options after changing the default subvolume to a nested subvolume (one attempt subvolume in a subvolume, another a subvolume in a directory): in both cases /proc/self/mountinfo reports / as the root, not the full path or ID of the subvolume actually mounted.
>> 
>> Somehow it seems like the mountinfo root field should return a block device and full path to the mounted subvolume or its ID. Currently it seems like a problem.
>> 
> 
> To quote one of btrfs developer (I had unrelated discussion on openSUSE
> list):
> 
> --><--
> This is a known problem, on my todo list, with a few non-working
> solutions.
> 
> If you mount via subvol=/subvol then /proc/self/mountinfo will show
> 'subvol' as the mounted subvolume (4th column)
> 
> 4 19 0:17 /subvol /mnt/ rw,relatime - btrfs /dev/sda15 rw,space_cache
> 
> but if the subvolume is set-default and then implicitly mounted,
> mountinfo will not show that (that's the bug).

If mounted with subvolid= the same thing happens. Mountinfo doesn't show the subvolume name, it's just /. It's also a problem understanding from mountinfo all of the devices that make up a btrfs volume. This can be learned from btrfsprogs.
> --><--
> 
> That said, information can be obtained also using different means
> (btrfs utility or directly btrfs IOCTL). The question is to which
> extent we want to depend on existence of btrfsprogs.

Yeah at the moment if I use subvolid= to mount, I then have no idea how to find out what subvolume is mounted. As far as I know btrfsprogs doesn't a way to determine what subvols are mounted. It must be inferred (by mountpoint or by contents of the mountpoint).

Anyway, as for support for subvolid in GRUB, I still think it would be nice as it's shorter than full paths. But this is not an enhancement hill I'm willing to die on by any means.


Chris Murphy

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 2473 bytes --]

On 16.10.2013 05:37, Chris Murphy wrote:
> 
> On Oct 15, 2013, at 8:50 PM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:
> 
>> В Mon, 14 Oct 2013 14:20:14 -0600
>> Chris Murphy <lists@colorremedies.com> пишет:
>>
>>>
>>>> Is there a way to detect that mountinfo gives garbage and somehow get
>>>> where the real root points?
>>>
>>> I don't know. I've asked on linux-btrfs@. Instead of rebooting, I merely tried mounting without options after changing the default subvolume to a nested subvolume (one attempt subvolume in a subvolume, another a subvolume in a directory): in both cases /proc/self/mountinfo reports / as the root, not the full path or ID of the subvolume actually mounted.
>>>
>>> Somehow it seems like the mountinfo root field should return a block device and full path to the mounted subvolume or its ID. Currently it seems like a problem.
>>>
>>
>> To quote one of btrfs developer (I had unrelated discussion on openSUSE
>> list):
>>
>> --><--
>> This is a known problem, on my todo list, with a few non-working
>> solutions.
>>
>> If you mount via subvol=/subvol then /proc/self/mountinfo will show
>> 'subvol' as the mounted subvolume (4th column)
>>
>> 4 19 0:17 /subvol /mnt/ rw,relatime - btrfs /dev/sda15 rw,space_cache
>>
>> but if the subvolume is set-default and then implicitly mounted,
>> mountinfo will not show that (that's the bug).
> 
> If mounted with subvolid= the same thing happens. Mountinfo doesn't show the subvolume name, it's just /. It's also a problem understanding from mountinfo all of the devices that make up a btrfs volume. This can be learned from btrfsprogs.
>> --><--
>>
>> That said, information can be obtained also using different means
>> (btrfs utility or directly btrfs IOCTL). The question is to which
>> extent we want to depend on existence of btrfsprogs.
> 
> Yeah at the moment if I use subvolid= to mount, I then have no idea how to find out what subvolume is mounted. As far as I know btrfsprogs doesn't a way to determine what subvols are mounted. It must be inferred (by mountpoint or by contents of the mountpoint).
> 
> Anyway, as for support for subvolid in GRUB, I still think it would be nice as it's shorter than full paths. But this is not an enhancement hill I'm willing to die on by any means.
> 
I changed in trunk to make / refer to real root and modified
grub-mkrelpath to follow the same convention, even if disk is mounted
with subvolid.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 672 bytes --]

В Mon, 28 Oct 2013 01:44:26 +0100
Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:

> I changed in trunk to make / refer to real root and modified
> grub-mkrelpath to follow the same convention, even if disk is mounted
> with subvolid.
> 

Can it cause compatibility issues? It means if config file that works
for grub before this change will stop working after. So e.g. attempt to
"configfile /file/from/partition/with/old/grub-mkconfig" will fail.

May be subvolume support should use different syntax. Something like

(hd0,1){sv=subvolume}/xxx
(hd0,1){svid=NNN}/yyy

And continue to interpret old syntax as relative to default.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1322 bytes --]

On 19.12.2013 17:13, Andrey Borzenkov wrote:
> В Mon, 28 Oct 2013 01:44:26 +0100
> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
> 
>> I changed in trunk to make / refer to real root and modified
>> grub-mkrelpath to follow the same convention, even if disk is mounted
>> with subvolid.
>>
> 
> Can it cause compatibility issues? It means if config file that works
> for grub before this change will stop working after. So e.g. attempt to
> "configfile /file/from/partition/with/old/grub-mkconfig" will fail.
> 
Normally I'd consider this a problem but the current behaviour is the
intended one, just back when the code was written thre were no changing
default yes
> May be subvolume support should use different syntax. Something like
> 
> (hd0,1){sv=subvolume}/xxx
> (hd0,1){svid=NNN}/yyy
> 
This would complicate the code a lot and commit us to maintaining it
long-term. Given that btrfs isn't clasified as stable, I consider this
behaviour change acceptable and it's better to handle this now rather
than perpetuating the issue.
> And continue to interpret old syntax as relative to default.
> 
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Dec 19, 2013, at 11:14 AM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> On 19.12.2013 17:13, Andrey Borzenkov wrote:
>> В Mon, 28 Oct 2013 01:44:26 +0100
>> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
>> 
>>> I changed in trunk to make / refer to real root and modified
>>> grub-mkrelpath to follow the same convention, even if disk is mounted
>>> with subvolid.
>>> 
>> 
>> Can it cause compatibility issues? It means if config file that works
>> for grub before this change will stop working after. So e.g. attempt to
>> "configfile /file/from/partition/with/old/grub-mkconfig" will fail.
>> 
> Normally I'd consider this a problem but the current behaviour is the
> intended one, just back when the code was written thre were no changing
> default yes
>> May be subvolume support should use different syntax. Something like
>> 
>> (hd0,1){sv=subvolume}/xxx
>> (hd0,1){svid=NNN}/yyy
>> 
> This would complicate the code a lot and commit us to maintaining it
> long-term. Given that btrfs isn't clasified as stable, I consider this
> behaviour change acceptable and it's better to handle this now rather
> than perpetuating the issue.

I tend to agree. It'd also be a really small population of possible breakage, because it also requires the default subvolume to have been changed for breakage to happen.

I'm also going to opine that knowledge of configfile, is about as obscure as changing the btrfs default subvolume. I think if someone knows about configfile, even though it's orthogonal to btrfs, that they will have some idea how to troubleshoot or where to go to get help if boot breaks as a result of this change.


Chris Murphy

Re: booting btrfs

[Michael Chang]

2013/12/20 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
> On 19.12.2013 17:13, Andrey Borzenkov wrote:
>> В Mon, 28 Oct 2013 01:44:26 +0100
>> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
>>
>>> I changed in trunk to make / refer to real root and modified
>>> grub-mkrelpath to follow the same convention, even if disk is mounted
>>> with subvolid.
>>>
>>
>> Can it cause compatibility issues? It means if config file that works
>> for grub before this change will stop working after. So e.g. attempt to
>> "configfile /file/from/partition/with/old/grub-mkconfig" will fail.
>>
> Normally I'd consider this a problem but the current behaviour is the
> intended one, just back when the code was written thre were no changing
> default yes
>> May be subvolume support should use different syntax. Something like
>>
>> (hd0,1){sv=subvolume}/xxx
>> (hd0,1){svid=NNN}/yyy
>>
> This would complicate the code a lot and commit us to maintaining it
> long-term. Given that btrfs isn't clasified as stable, I consider this
> behaviour change acceptable and it's better to handle this now rather
> than perpetuating the issue.

Please consider the case if a snapshot was taken against real root fs
tree to a subvolume with SNAPSHOT_ID. With syntax above providing
mount option support we can possibly boot that snapshot with this
config.

  set root=(hd0,1){svid=<SNAPSHOT_ID>}
  set prefix=($root)/boot/grub2
  normal

Without the syntax support it's almost impossible to do that. At lease
I can't figure out any.

Besides we may leverage that mount option support in grub-mount to
test/develop and so on. For modern innovative file systems the mount
option support is becoming necessary for dealing many different use
cases.

Thanks,
Michael

>> And continue to interpret old syntax as relative to default.
>>
>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
>
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 2687 bytes --]

On 20.12.2013 10:46, Michael Chang wrote:
> 2013/12/20 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
>> On 19.12.2013 17:13, Andrey Borzenkov wrote:
>>> В Mon, 28 Oct 2013 01:44:26 +0100
>>> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
>>>
>>>> I changed in trunk to make / refer to real root and modified
>>>> grub-mkrelpath to follow the same convention, even if disk is mounted
>>>> with subvolid.
>>>>
>>>
>>> Can it cause compatibility issues? It means if config file that works
>>> for grub before this change will stop working after. So e.g. attempt to
>>> "configfile /file/from/partition/with/old/grub-mkconfig" will fail.
>>>
>> Normally I'd consider this a problem but the current behaviour is the
>> intended one, just back when the code was written thre were no changing
>> default yes
>>> May be subvolume support should use different syntax. Something like
>>>
>>> (hd0,1){sv=subvolume}/xxx
>>> (hd0,1){svid=NNN}/yyy
>>>
>> This would complicate the code a lot and commit us to maintaining it
>> long-term. Given that btrfs isn't clasified as stable, I consider this
>> behaviour change acceptable and it's better to handle this now rather
>> than perpetuating the issue.
> 
> Please consider the case if a snapshot was taken against real root fs
> tree to a subvolume with SNAPSHOT_ID. With syntax above providing
> mount option support we can possibly boot that snapshot with this
> config.
> 
>   set root=(hd0,1){svid=<SNAPSHOT_ID>}
>   set prefix=($root)/boot/grub2
>   normal
> 
> Without the syntax support it's almost impossible to do that. At lease
> I can't figure out any.
> 
Every volume has a name, even if you don't know it. Use grub-mkrelpath
to find out.
> Besides we may leverage that mount option support in grub-mount to
> test/develop and so on. For modern innovative file systems the mount
> option support is becoming necessary for dealing many different use
> cases.
> 
This is feature request without usecase. As such it's rejected
automatically.
> Thanks,
> Michael
> 
>>> And continue to interpret old syntax as relative to default.
>>>
>>>
>>>
>>> _______________________________________________
>>> Grub-devel mailing list
>>> Grub-devel@gnu.org
>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>
>>
>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Michael Chang]

2013/12/20 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
> On 20.12.2013 10:46, Michael Chang wrote:
>> 2013/12/20 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
>>> On 19.12.2013 17:13, Andrey Borzenkov wrote:
>>>> В Mon, 28 Oct 2013 01:44:26 +0100
>>>> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
>>>>
>>>>> I changed in trunk to make / refer to real root and modified
>>>>> grub-mkrelpath to follow the same convention, even if disk is mounted
>>>>> with subvolid.
>>>>>
>>>>
>>>> Can it cause compatibility issues? It means if config file that works
>>>> for grub before this change will stop working after. So e.g. attempt to
>>>> "configfile /file/from/partition/with/old/grub-mkconfig" will fail.
>>>>
>>> Normally I'd consider this a problem but the current behaviour is the
>>> intended one, just back when the code was written thre were no changing
>>> default yes
>>>> May be subvolume support should use different syntax. Something like
>>>>
>>>> (hd0,1){sv=subvolume}/xxx
>>>> (hd0,1){svid=NNN}/yyy
>>>>
>>> This would complicate the code a lot and commit us to maintaining it
>>> long-term. Given that btrfs isn't clasified as stable, I consider this
>>> behaviour change acceptable and it's better to handle this now rather
>>> than perpetuating the issue.
>>
>> Please consider the case if a snapshot was taken against real root fs
>> tree to a subvolume with SNAPSHOT_ID. With syntax above providing
>> mount option support we can possibly boot that snapshot with this
>> config.
>>
>>   set root=(hd0,1){svid=<SNAPSHOT_ID>}
>>   set prefix=($root)/boot/grub2
>>   normal
>>
>> Without the syntax support it's almost impossible to do that. At lease
>> I can't figure out any.
>>
> Every volume has a name, even if you don't know it. Use grub-mkrelpath
> to find out.

That means we need to modify the grub,cfg in snapshot to make files
used by config refer to new path output by grub-mkrelpath (relative to
real root), right ? That could be difficult to manage especially if
you have a lot of snapshots and the update takes time to finish.

Compare to use path relative to snapshot's fs root, we can leave the
grub.cfg in snapshot unmodified and by setting snapshot id or name in
a master config to switch the snapshot we want to boot. That will make
things a lot easier.

>> Besides we may leverage that mount option support in grub-mount to
>> test/develop and so on. For modern innovative file systems the mount
>> option support is becoming necessary for dealing many different use
>> cases.
>>
> This is feature request without usecase. As such it's rejected
> automatically.

Probably a case is in os-prober, we can use that feature to have
grub-mount test subvolumes without resorting to linux mount. But I
admit that the gain is little.

Regards,
Michael

>> Thanks,
>> Michael
>>
>>>> And continue to interpret old syntax as relative to default.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Grub-devel mailing list
>>>> Grub-devel@gnu.org
>>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Grub-devel mailing list
>>> Grub-devel@gnu.org
>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
>
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 4750 bytes --]

On 20.12.2013 15:54, Michael Chang wrote:
> 2013/12/20 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
>> On 20.12.2013 10:46, Michael Chang wrote:
>>> 2013/12/20 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
>>>> On 19.12.2013 17:13, Andrey Borzenkov wrote:
>>>>> В Mon, 28 Oct 2013 01:44:26 +0100
>>>>> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
>>>>>
>>>>>> I changed in trunk to make / refer to real root and modified
>>>>>> grub-mkrelpath to follow the same convention, even if disk is mounted
>>>>>> with subvolid.
>>>>>>
>>>>>
>>>>> Can it cause compatibility issues? It means if config file that works
>>>>> for grub before this change will stop working after. So e.g. attempt to
>>>>> "configfile /file/from/partition/with/old/grub-mkconfig" will fail.
>>>>>
>>>> Normally I'd consider this a problem but the current behaviour is the
>>>> intended one, just back when the code was written thre were no changing
>>>> default yes
>>>>> May be subvolume support should use different syntax. Something like
>>>>>
>>>>> (hd0,1){sv=subvolume}/xxx
>>>>> (hd0,1){svid=NNN}/yyy
>>>>>
>>>> This would complicate the code a lot and commit us to maintaining it
>>>> long-term. Given that btrfs isn't clasified as stable, I consider this
>>>> behaviour change acceptable and it's better to handle this now rather
>>>> than perpetuating the issue.
>>>
>>> Please consider the case if a snapshot was taken against real root fs
>>> tree to a subvolume with SNAPSHOT_ID. With syntax above providing
>>> mount option support we can possibly boot that snapshot with this
>>> config.
>>>
>>>   set root=(hd0,1){svid=<SNAPSHOT_ID>}
>>>   set prefix=($root)/boot/grub2
>>>   normal
>>>
>>> Without the syntax support it's almost impossible to do that. At lease
>>> I can't figure out any.
>>>
>> Every volume has a name, even if you don't know it. Use grub-mkrelpath
>> to find out.
> 
> That means we need to modify the grub,cfg in snapshot to make files
> used by config refer to new path output by grub-mkrelpath (relative to
> real root), right ? That could be difficult to manage especially if
> you have a lot of snapshots and the update takes time to finish.
> 
> Compare to use path relative to snapshot's fs root, we can leave the
> grub.cfg in snapshot unmodified and by setting snapshot id or name in
> a master config to switch the snapshot we want to boot. That will make
> things a lot easier.
> 
This is not a reason to force part of path name to become part of device
name. Also it leaves problem of passing right options to kernel to mount
right root open.
Because generated config in snapshot will reset $root, any attempt to
alter its behaviour by setting $root will fail anyway.
Sounds like this needs additional complications in generated grub.cfg
when on btrfs (E.g. overrideable $subvolume variable) and attempts to
change device naming schemes don't really solve any of problems but
create bunch of new ones.
The real solution for your problem has to involve sth like:
if [ x$bootdir = x ]; then
   bootdir=<precomputed bootdir>
fi
if [ x$rootdir = x ]; then
   rootdir=<precomputed rootdir>
fi

...
search ....
linux $bootdir/vmlinuz-.... root=.. subvol=$rootdir
initrd $bootdir/initrd.img
> Probably a case is in os-prober, we can use that feature to have
> grub-mount test subvolumes without resorting to linux mount. But I
> admit that the gain is little.
> 
Why isn't the use of subvolume name appropriate?
One of these days, just for the people who insist un numeric ids rather
than names I should write a patch to Linux which ones files only by
inode id and no file name.
> Regards,
> Michael
> 
>>> Thanks,
>>> Michael
>>>
>>>>> And continue to interpret old syntax as relative to default.
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Grub-devel mailing list
>>>>> Grub-devel@gnu.org
>>>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Grub-devel mailing list
>>>> Grub-devel@gnu.org
>>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>>
>>>
>>> _______________________________________________
>>> Grub-devel mailing list
>>> Grub-devel@gnu.org
>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>
>>
>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Dec 20, 2013, at 7:54 AM, Michael Chang <mchang@suse.com> wrote:

>> Every volume has a name, even if you don't know it. Use grub-mkrelpath
>> to find out.
> 
> That means we need to modify the grub,cfg in snapshot to make files
> used by config refer to new path output by grub-mkrelpath (relative to
> real root), right ? That could be difficult to manage especially if
> you have a lot of snapshots and the update takes time to finish.

This isn't just a problem for core.img looking for the wrong grub.cfg for a /boot on Btrfs subvolume snapshot. It's a problem for that grub.cfg pointing to the wrong root snapshot. And it's a problem for the /etc/fstab on that root snapshot, which is likewise incorrect and will be asking for the wrong subvolumes to be mounted.

I really don't think snapshot management is GRUB's job. I think all of this snapshot management is a userspace tool, and it's going to have to figure out a way to deal with this. And probably the simplest solution in the short term is for this user space tool to rename the subvolumes. So e.g. subvolumes:

boot
root
home

And their read only snapshots:

boot_ro.1
boot_ro.2
root_ro.1
root_ro.2
home_ro.1
home_ro.2

The user uses a tool to indicate they now want to boot "the most recent snapshot", and the tool does:

mv boot boot_ro.0
mv root root_ro.0
mv home home_ro.0
btrfs subvol snapshot boot_ro.1 boot
btrfs subvol snapshot root_ro.1 root
btrfs subvol snapshot home_ro.1 root

The lack of -r makes the snapshots rw, the file system metadata contains relationship information: each snapshot has a uuid, and a parent uuid. And the parent contains information about each snapshot made of it. But all of this is domain of the snapshot tool. That's a lot easier than having to go find fstab, grub.cfg, and figure out how to get core.img to know what boot subvolume was intended, etc.


> Compare to use path relative to snapshot's fs root, we can leave the
> grub.cfg in snapshot unmodified and by setting snapshot id or name in
> a master config to switch the snapshot we want to boot. That will make
> things a lot easier.

That sounds something like the Bootloaderspec, which I like in principle in that it recognizes how hostile the distributions are at breaking the boot behavior of the prior OS, in multiboot contexts. But there's some other things that just don't seem workable, and it's also not even adopted upstream yet by GRUB and I don't know what the status of this whole idea is.

I think the idea of snapshots in the domain of a boot manager/boot loader is really overly complicated. For another thing, it's not really appropriate to do a rollback and then immediately start modifying it by booting from it. What you'd want to do is snapshot the rollback, and then use that "cloned" copy of the rollback, leaving the original rollback in place. Otherwise the provenance of that <inserttime> snapshot is obliterated.

And with all of these snapshots being created, something to clean up all these bouquets is necessary. Do we really want GRUB doing that also? I just this this is out of scope for GRUB.

An FHS for Btrfs installation locations and shapshot behaviors is possibly needed, so the distributions aren't stepping all over each other in an even worse way that booting already is.

Chris Murphy

Re: booting btrfs

[Andrey Borzenkov]

В Fri, 20 Dec 2013 21:38:47 -0700
Chris Murphy <lists@colorremedies.com> пишет:

> 
> On Dec 20, 2013, at 7:54 AM, Michael Chang <mchang@suse.com> wrote:
> 
> >> Every volume has a name, even if you don't know it. Use grub-mkrelpath
> >> to find out.
> > 
> > That means we need to modify the grub,cfg in snapshot to make files
> > used by config refer to new path output by grub-mkrelpath (relative to
> > real root), right ? That could be difficult to manage especially if
> > you have a lot of snapshots and the update takes time to finish.
> 
> This isn't just a problem for core.img looking for the wrong grub.cfg for a /boot on Btrfs subvolume snapshot. It's a problem for that grub.cfg pointing to the wrong root snapshot. And it's a problem for the /etc/fstab on that root snapshot, which is likewise incorrect and will be asking for the wrong subvolumes to be mounted.
> 
> I really don't think snapshot management is GRUB's job. I think all of this snapshot management is a userspace tool, and it's going to have to figure out a way to deal with this.

Yes I completely agree here. Expecting to be able to boot from pure
btrfs snapshot is naïve. Michael, here is what openSUSE does by default
when you tell it to use btrfs:

linux-dwhw:~ # btrfs subvolume list /
ID 256 gen 47 top level 5 path boot/grub2/i386-pc
ID 258 gen 18 top level 5 path home
ID 259 gen 18 top level 5 path opt
ID 260 gen 18 top level 5 path srv
ID 261 gen 65 top level 5 path tmp
ID 262 gen 52 top level 5 path usr/local
ID 263 gen 18 top level 5 path var/crash
ID 264 gen 67 top level 5 path var/log
ID 265 gen 18 top level 5 path var/opt
ID 266 gen 62 top level 5 path var/spool
ID 267 gen 57 top level 5 path var/tmp
ID 269 gen 59 top level 5 path .snapshots
ID 270 gen 58 top level 5 path .snapshots/1/snapshot
ID 271 gen 78 top level 5 path test
linux-dwhw:~ # grep btrfs /proc/self/mountinfo
21 1 0:17 / / rw,relatime shared:1 - btrfs /dev/sda2 rw,space_cache

"test" is snapshot of / which I set as default and am currently booted
with it.

linux-dwhw:~ # btrfs subvolume get-default /
ID 271 gen 78 top level 5 path test

And if I now try to access any other subvolumes ...

linux-dwhw:~ # ls -l /boot/grub2/i386-pc/
total 0
linux-dwhw:~ # touch /boot/grub2/i386-pc/x
touch: cannot touch ‘/boot/grub2/i386-pc/x’: Permission denied
linux-dwhw:~ # ls -l /var/spool
total 0
linux-dwhw:~ # touch /var/spool/x
touch: cannot touch ‘/var/spool/x’: Permission denied
linux-dwhw:~ # ls -l /var/log
total 0
linux-dwhw:~ # touch /var/log/x
touch: cannot touch ‘/var/log/x’: Permission denied

So booting from this snapshot is rather useless.

The point here is - creating of fully functional alternate boot
environment involves a bit more than single "btrfs subvolume snapshot"
invocation. Adding "grub-mkconfig" (or even grub-mkimage to record
correct prefix) is really just the minor part of it.

> And probably the simplest solution in the short term is for this user
> space tool to rename the subvolumes. So e.g. subvolumes:
> 
> boot
> root
> home
>
> And their read only snapshots:
> 
> boot_ro.1
> boot_ro.2
> root_ro.1
> root_ro.2
> home_ro.1
> home_ro.2
> 
> The user uses a tool to indicate they now want to boot "the most recent snapshot", and the tool does:
> 
> mv boot boot_ro.0
> mv root root_ro.0
> mv home home_ro.0
> btrfs subvol snapshot boot_ro.1 boot
> btrfs subvol snapshot root_ro.1 root
> btrfs subvol snapshot home_ro.1 root
>

Do you need to reinvent the wheel?

/Boot-Environments
  /Boot_Environment_1
    /root
    /boot
    ...
  /Boot_Environment_2
    ...

The only thing you need to do to switch is equivalent of "btrfs
set-default /Boot-Environments/Boot_Envirnment_2 ... except it is
not that straightforward in current btrfs because path names are
resolved relative to current root :)

> The lack of -r makes the snapshots rw, the file system metadata contains relationship information: each snapshot has a uuid, and a parent uuid. And the parent contains information about each snapshot made of it. But all of this is domain of the snapshot tool. That's a lot easier than having to go find fstab, grub.cfg, and figure out how to get core.img to know what boot subvolume was intended, etc.
> 
> 
> > Compare to use path relative to snapshot's fs root, we can leave the
> > grub.cfg in snapshot unmodified and by setting snapshot id or name in
> > a master config to switch the snapshot we want to boot. That will make
> > things a lot easier.
>

Michael, snapshot of *what*? Whatever means you will use (set-default,
environment variable, mount options) can set only one single property
- root of filesystem. You *STILL* need to describe relationships
between different (snapshots of) multiple subvolumes. I.e. *which*
snapshot of /boot/grub2/i386-pc are you going to access?

Having grub to always use full pathnames makes it unambiguous. Otherwise
it is unmanageable on grub level (*any* directory you access may
potentially have multiple versions). This must really be solved on OS
level by either

- mandating single subvolume for your snapshottable OS, or
- adding support to grafting individual subvolumes inside of btrfs.
  Normal mount will not solve this on bootloader level

 
> That sounds something like the Bootloaderspec, which I like in principle in that it recognizes how hostile the distributions are at breaking the boot behavior of the prior OS, in multiboot contexts. But there's some other things that just don't seem workable, and it's also not even adopted upstream yet by GRUB and I don't know what the status of this whole idea is.
> 
> I think the idea of snapshots in the domain of a boot manager/boot loader is really overly complicated. For another thing, it's not really appropriate to do a rollback and then immediately start modifying it by booting from it. What you'd want to do is snapshot the rollback, and then use that "cloned" copy of the rollback, leaving the original rollback in place. Otherwise the provenance of that <inserttime> snapshot is obliterated.
> 
> And with all of these snapshots being created, something to clean up all these bouquets is necessary. Do we really want GRUB doing that also? I just this this is out of scope for GRUB.
> 
> An FHS for Btrfs installation locations and shapshot behaviors is possibly needed, so the distributions aren't stepping all over each other in an even worse way that booting already is.
> 
> Chris Murphy
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: booting btrfs

[Chris Murphy]

On Dec 21, 2013, at 12:18 AM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:
> 
> Do you need to reinvent the wheel?
> 
> /Boot-Environments
>  /Boot_Environment_1
>    /root
>    /boot
>    ...
>  /Boot_Environment_2
>    ...
> 
> The only thing you need to do to switch is equivalent of "btrfs
> set-default /Boot-Environments/Boot_Envirnment_2 …

I don't need to, but distros do reinvent wheels. If they will cooperate on the usage of set-default then sure, that's preferred. But historically they put next to no effort in preserving the bootability of a previously working distro install.

So I'd only be in favor of opening up the set-default can of worms if distros can agree on some parameters for multiboot on Btrfs and how to share the set-default state. Otherwise I feel like this becomes yet another way for them to step on each other.

As for grub's part in the solution, it seems like it would need to understand absolute paths to subvol 5, as well as relative paths to the default subvol (set with the 'btrfs subvol set-default' command). I think syntax wise it's simplest to use / for absolute and no-/ for relative to default. What do you think?

I still don't think it needs syntax of subvolids. 


> except it is
> not that straightforward in current btrfs because path names are
> resolved relative to current root :)

Navigation of a currently mounted subvolume, yes, I can only navigate below that point. It's no different than being in chroot, or mounting a partition and being unable to navigate into other partitions. 'btrfs subvol set-default' requires both <subvolid> and a <mountpoint>.


Chris Murphy

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

On 23.12.2013 05:45, Chris Murphy wrote:
> As for grub's part in the solution, it seems like it would need to understand absolute paths to subvol 5, as well as relative paths to the default subvol
> (set with the 'btrfs subvol set-default' command).
It doesn't seem to me at all that "relative" paths is a necessity and you don't provide a usecase

> I think syntax wise it's simplest to use / for absolute and no-/ for relative to default. What do you think?
> 
> I still don't think it needs syntax of subvolids. 
> 
> 
>> > except it is
>> > not that straightforward in current btrfs because path names are
>> > resolved relative to current root :)
> Navigation of a currently mounted subvolume, yes, I can only navigate below that point. It's no different than being in chroot, or mounting a partition and being unable to navigate into other partitions.
This analogies all share a common crucial point: GRUB neither supports nor needs any of them.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Dec 22, 2013, at 9:52 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> On 23.12.2013 05:45, Chris Murphy wrote:
>> As for grub's part in the solution, it seems like it would need to understand absolute paths to subvol 5, as well as relative paths to the default subvol
>> (set with the 'btrfs subvol set-default' command).
> It doesn't seem to me at all that "relative" paths is a necessity and you don't provide a usecase

OK it was Andrey who suggested booting snapshots by changing set-default rather than my more complicated example, which actually was a simple one. I don't think either of us is yet saying this is a necessity, that there's no other way. We're just discussing the possible approaches, and figuring out how fragile they are, and how much and where certain work would need to be done. Clearly it's not just a grub concern.

The use case is rolling back from failed system updates for example, to do that means booting a snapshot. Either /boot or rootfs on Btrfs (or both).

If relative paths are used, neither core.img nor grub.cfg needs to be modified, because merely a user space program changing the default subvolume would cause the existing core.img and grub.cfg to boot the snapshot.

If grub only supports absolute paths, then something must at least modify grub.cfg to point to the correct snapshot by changing root=<fullpathtosnapshotwenowwanttoboot>. If /boot is on Btrfs then additionally it means core.img needs to be replaced/modified so it will find the correct grub.cfg. That's a lot more complicated, enough that my crazy idea of just shuffling things around with mv and more snapshots like the simple example I posted previously is probably better.




>>>> except it is
>>>> not that straightforward in current btrfs because path names are
>>>> resolved relative to current root :)
>> Navigation of a currently mounted subvolume, yes, I can only navigate below that point. It's no different than being in chroot, or mounting a partition and being unable to navigate into other partitions.
> This analogies all share a common crucial point: GRUB neither supports nor needs any of them.


That whole part is really off topic to grub and therefore isn't a good reason to categorically reject the idea of supporting paths relative to the default subvolume. 

Re: booting btrfs

[Michael Chang]

2013/12/20 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
> On 20.12.2013 15:54, Michael Chang wrote:
>> 2013/12/20 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
>>> On 20.12.2013 10:46, Michael Chang wrote:
>>>> 2013/12/20 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
>>>>> On 19.12.2013 17:13, Andrey Borzenkov wrote:
>>>>>> В Mon, 28 Oct 2013 01:44:26 +0100
>>>>>> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
>>>>>>
>>>>>>> I changed in trunk to make / refer to real root and modified
>>>>>>> grub-mkrelpath to follow the same convention, even if disk is mounted
>>>>>>> with subvolid.
>>>>>>>
>>>>>>
>>>>>> Can it cause compatibility issues? It means if config file that works
>>>>>> for grub before this change will stop working after. So e.g. attempt to
>>>>>> "configfile /file/from/partition/with/old/grub-mkconfig" will fail.
>>>>>>
>>>>> Normally I'd consider this a problem but the current behaviour is the
>>>>> intended one, just back when the code was written thre were no changing
>>>>> default yes
>>>>>> May be subvolume support should use different syntax. Something like
>>>>>>
>>>>>> (hd0,1){sv=subvolume}/xxx
>>>>>> (hd0,1){svid=NNN}/yyy
>>>>>>
>>>>> This would complicate the code a lot and commit us to maintaining it
>>>>> long-term. Given that btrfs isn't clasified as stable, I consider this
>>>>> behaviour change acceptable and it's better to handle this now rather
>>>>> than perpetuating the issue.
>>>>
>>>> Please consider the case if a snapshot was taken against real root fs
>>>> tree to a subvolume with SNAPSHOT_ID. With syntax above providing
>>>> mount option support we can possibly boot that snapshot with this
>>>> config.
>>>>
>>>>   set root=(hd0,1){svid=<SNAPSHOT_ID>}
>>>>   set prefix=($root)/boot/grub2
>>>>   normal
>>>>
>>>> Without the syntax support it's almost impossible to do that. At lease
>>>> I can't figure out any.
>>>>
>>> Every volume has a name, even if you don't know it. Use grub-mkrelpath
>>> to find out.
>>
>> That means we need to modify the grub,cfg in snapshot to make files
>> used by config refer to new path output by grub-mkrelpath (relative to
>> real root), right ? That could be difficult to manage especially if
>> you have a lot of snapshots and the update takes time to finish.
>>
>> Compare to use path relative to snapshot's fs root, we can leave the
>> grub.cfg in snapshot unmodified and by setting snapshot id or name in
>> a master config to switch the snapshot we want to boot. That will make
>> things a lot easier.
>>
> This is not a reason to force part of path name to become part of device
> name. Also it leaves problem of passing right options to kernel to mount
> right root open.

OK.

> Because generated config in snapshot will reset $root, any attempt to
> alter its behaviour by setting $root will fail anyway.
> Sounds like this needs additional complications in generated grub.cfg
> when on btrfs (E.g. overrideable $subvolume variable) and attempts to
> change device naming schemes don't really solve any of problems but
> create bunch of new ones.
> The real solution for your problem has to involve sth like:
> if [ x$bootdir = x ]; then
>    bootdir=<precomputed bootdir>
> fi
> if [ x$rootdir = x ]; then
>    rootdir=<precomputed rootdir>
> fi
>
> ...
> search ....
> linux $bootdir/vmlinuz-.... root=.. subvol=$rootdir
> initrd $bootdir/initrd.img

Thanks. I will give this config a try.

>> Probably a case is in os-prober, we can use that feature to have
>> grub-mount test subvolumes without resorting to linux mount. But I
>> admit that the gain is little.
>>
> Why isn't the use of subvolume name appropriate?

OK. I realized that can use subvolume name in path and therefore don't
need the mount options.

Thanks,
Michael

> One of these days, just for the people who insist un numeric ids rather
> than names I should write a patch to Linux which ones files only by
> inode id and no file name.
>> Regards,
>> Michael
>>
>>>> Thanks,
>>>> Michael
>>>>
>>>>>> And continue to interpret old syntax as relative to default.
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Grub-devel mailing list
>>>>>> Grub-devel@gnu.org
>>>>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Grub-devel mailing list
>>>>> Grub-devel@gnu.org
>>>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>>>
>>>>
>>>> _______________________________________________
>>>> Grub-devel mailing list
>>>> Grub-devel@gnu.org
>>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Grub-devel mailing list
>>> Grub-devel@gnu.org
>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
>
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: booting btrfs

[Michael Chang]

2013/12/21 Chris Murphy <lists@colorremedies.com>:
>
> On Dec 20, 2013, at 7:54 AM, Michael Chang <mchang@suse.com> wrote:
>
>>> Every volume has a name, even if you don't know it. Use grub-mkrelpath
>>> to find out.
>>
>> That means we need to modify the grub,cfg in snapshot to make files
>> used by config refer to new path output by grub-mkrelpath (relative to
>> real root), right ? That could be difficult to manage especially if
>> you have a lot of snapshots and the update takes time to finish.
>
> This isn't just a problem for core.img looking for the wrong grub.cfg for a /boot on Btrfs subvolume snapshot. It's a problem for that grub.cfg pointing to the wrong root snapshot. And it's a problem for the /etc/fstab on that root snapshot, which is likewise incorrect and will be asking for the wrong subvolumes to be mounted.

I can't deny these problems. The snapshot booting is not possbile be
done solely by bootloader, some additional works in systems, be it
refining /etc/fstab or initrd moutning subvolues and whatever .. could
be necessary.

Now I tend to agree that supporting config for snapshot booting
shouldn't be upstream's consideration due to it's compliexity and
dependency to system, Despite on this, I still like to ask : Did
upstream think about any patch trying to provide relative path support
for btrfs subvolume name or id's a worthy work or not?

>
> I really don't think snapshot management is GRUB's job. I think all of this snapshot management is a userspace tool, and it's going to have to figure out a way to deal with this. And probably the simplest solution in the short term is for this user space tool to rename the subvolumes. So e.g. subvolumes:
>
> boot
> root
> home
>
> And their read only snapshots:
>
> boot_ro.1
> boot_ro.2
> root_ro.1
> root_ro.2
> home_ro.1
> home_ro.2
>
> The user uses a tool to indicate they now want to boot "the most recent snapshot", and the tool does:
>
> mv boot boot_ro.0
> mv root root_ro.0
> mv home home_ro.0
> btrfs subvol snapshot boot_ro.1 boot
> btrfs subvol snapshot root_ro.1 root
> btrfs subvol snapshot home_ro.1 root
>
> The lack of -r makes the snapshots rw, the file system metadata contains relationship information: each snapshot has a uuid, and a parent uuid. And the parent contains information about each snapshot made of it. But all of this is domain of the snapshot tool. That's a lot easier than having to go find fstab, grub.cfg, and figure out how to get core.img to know what boot subvolume was intended, etc.

Thanks.  It can be a solution for some use cases but for the one I
want to achieve it's insufficent. What I'm looking for is to list the
available snapshots in the boot menu for selecting and then booting to
it.

>
>
>> Compare to use path relative to snapshot's fs root, we can leave the
>> grub.cfg in snapshot unmodified and by setting snapshot id or name in
>> a master config to switch the snapshot we want to boot. That will make
>> things a lot easier.
>
> That sounds something like the Bootloaderspec, which I like in principle in that it recognizes how hostile the distributions are at breaking the boot behavior of the prior OS, in multiboot contexts. But there's some other things that just don't seem workable, and it's also not even adopted upstream yet by GRUB and I don't know what the status of this whole idea is.

It's not a good idea for me either that making os installation to a
subvolume, and use set-default to make it default root. That's indeed
overly complicated. Meanwhile the snapshot is not for os installation,
people occasionally want to boot into it for special reasons like he
want to find from when his system performance start downgrading. I'd
consider such multiboot should not consider os installtion be in
snapshots .. because it makes no sense.

>
> I think the idea of snapshots in the domain of a boot manager/boot loader is really overly complicated. For another thing, it's not really appropriate to do a rollback and then immediately start modifying it by booting from it. What you'd want to do is snapshot the rollback, and then use that "cloned" copy of the rollback, leaving the original rollback in place. Otherwise the provenance of that <inserttime> snapshot is obliterated.

I think that's like seed device and yes there should have such
handling in initrd to use a clone copy of read-only snapshots when
your kernel parameters are asking to mount it as /.

>
> And with all of these snapshots being created, something to clean up all these bouquets is necessary. Do we really want GRUB doing that also? I just this this is out of scope for GRUB.

That's definitely not GRUB should care about. :)

>
> An FHS for Btrfs installation locations and shapshot behaviors is possibly needed, so the distributions aren't stepping all over each other in an even worse way that booting already is.

Thanks for your ideas and feedbacks.

Regards,
Michael

>
> Chris Murphy

Re: booting btrfs

[Michael Chang]

2013/12/21 Andrey Borzenkov <arvidjaar@gmail.com>:
> В Fri, 20 Dec 2013 21:38:47 -0700
> Chris Murphy <lists@colorremedies.com> пишет:
>
>>
>> On Dec 20, 2013, at 7:54 AM, Michael Chang <mchang@suse.com> wrote:
>>
>> >> Every volume has a name, even if you don't know it. Use grub-mkrelpath
>> >> to find out.
>> >
>> > That means we need to modify the grub,cfg in snapshot to make files
>> > used by config refer to new path output by grub-mkrelpath (relative to
>> > real root), right ? That could be difficult to manage especially if
>> > you have a lot of snapshots and the update takes time to finish.
>>
>> This isn't just a problem for core.img looking for the wrong grub.cfg for a /boot on Btrfs subvolume snapshot. It's a problem for that grub.cfg pointing to the wrong root snapshot. And it's a problem for the /etc/fstab on that root snapshot, which is likewise incorrect and will be asking for the wrong subvolumes to be mounted.
>>
>> I really don't think snapshot management is GRUB's job. I think all of this snapshot management is a userspace tool, and it's going to have to figure out a way to deal with this.
>
> Yes I completely agree here. Expecting to be able to boot from pure
> btrfs snapshot is naïve. Michael, here is what openSUSE does by default
> when you tell it to use btrfs:
>
> linux-dwhw:~ # btrfs subvolume list /
> ID 256 gen 47 top level 5 path boot/grub2/i386-pc
> ID 258 gen 18 top level 5 path home
> ID 259 gen 18 top level 5 path opt
> ID 260 gen 18 top level 5 path srv
> ID 261 gen 65 top level 5 path tmp
> ID 262 gen 52 top level 5 path usr/local
> ID 263 gen 18 top level 5 path var/crash
> ID 264 gen 67 top level 5 path var/log
> ID 265 gen 18 top level 5 path var/opt
> ID 266 gen 62 top level 5 path var/spool
> ID 267 gen 57 top level 5 path var/tmp
> ID 269 gen 59 top level 5 path .snapshots
> ID 270 gen 58 top level 5 path .snapshots/1/snapshot
> ID 271 gen 78 top level 5 path test
> linux-dwhw:~ # grep btrfs /proc/self/mountinfo
> 21 1 0:17 / / rw,relatime shared:1 - btrfs /dev/sda2 rw,space_cache
>
> "test" is snapshot of / which I set as default and am currently booted
> with it.
>
> linux-dwhw:~ # btrfs subvolume get-default /
> ID 271 gen 78 top level 5 path test
>
> And if I now try to access any other subvolumes ...
>
> linux-dwhw:~ # ls -l /boot/grub2/i386-pc/
> total 0
> linux-dwhw:~ # touch /boot/grub2/i386-pc/x
> touch: cannot touch ‘/boot/grub2/i386-pc/x’: Permission denied
> linux-dwhw:~ # ls -l /var/spool
> total 0
> linux-dwhw:~ # touch /var/spool/x
> touch: cannot touch ‘/var/spool/x’: Permission denied
> linux-dwhw:~ # ls -l /var/log
> total 0
> linux-dwhw:~ # touch /var/log/x
> touch: cannot touch ‘/var/log/x’: Permission denied
>
> So booting from this snapshot is rather useless.
>
> The point here is - creating of fully functional alternate boot
> environment involves a bit more than single "btrfs subvolume snapshot"
> invocation. Adding "grub-mkconfig" (or even grub-mkimage to record
> correct prefix) is really just the minor part of it.

Well .. yes, I've agreed on that system has to work to support it, not
just bootloader.

>
>> And probably the simplest solution in the short term is for this user
>> space tool to rename the subvolumes. So e.g. subvolumes:
>>
>> boot
>> root
>> home
>>
>> And their read only snapshots:
>>
>> boot_ro.1
>> boot_ro.2
>> root_ro.1
>> root_ro.2
>> home_ro.1
>> home_ro.2
>>
>> The user uses a tool to indicate they now want to boot "the most recent snapshot", and the tool does:
>>
>> mv boot boot_ro.0
>> mv root root_ro.0
>> mv home home_ro.0
>> btrfs subvol snapshot boot_ro.1 boot
>> btrfs subvol snapshot root_ro.1 root
>> btrfs subvol snapshot home_ro.1 root
>>
>
> Do you need to reinvent the wheel?
>
> /Boot-Environments
>   /Boot_Environment_1
>     /root
>     /boot
>     ...
>   /Boot_Environment_2
>     ...
>
> The only thing you need to do to switch is equivalent of "btrfs
> set-default /Boot-Environments/Boot_Envirnment_2 ... except it is
> not that straightforward in current btrfs because path names are
> resolved relative to current root :)
>
>> The lack of -r makes the snapshots rw, the file system metadata contains relationship information: each snapshot has a uuid, and a parent uuid. And the parent contains information about each snapshot made of it. But all of this is domain of the snapshot tool. That's a lot easier than having to go find fstab, grub.cfg, and figure out how to get core.img to know what boot subvolume was intended, etc.
>>
>>
>> > Compare to use path relative to snapshot's fs root, we can leave the
>> > grub.cfg in snapshot unmodified and by setting snapshot id or name in
>> > a master config to switch the snapshot we want to boot. That will make
>> > things a lot easier.
>>
>
> Michael, snapshot of *what*?

Only the toplevel subvolume 5 (os installation's /) and all other
subvolume mount points should be defined in /etc/fstab for those
whatever wont be snapshotted.

> Whatever means you will use (set-default,
> environment variable, mount options) can set only one single property
> - root of filesystem.

Not use set default, but use rootflags=subvol=<SNAPSHOT_NAME> (or
rootflags=suvolid=<SNAPSHOT_NAME>...) kernel command line options to
set root of file system.

> You *STILL* need to describe relationships
> between different (snapshots of) multiple subvolumes. I.e. *which*
> snapshot of /boot/grub2/i386-pc are you going to access?

/boot/grub2/i386-pc shouldn't be snapshotted and is a subvolume
mounted by /etc/fstab. It's for the reason that we have to keep
(embedded) grub2 image and module's version in sync.

>
> Having grub to always use full pathnames makes it unambiguous. Otherwise
> it is unmanageable on grub level (*any* directory you access may
> potentially have multiple versions). This must really be solved on OS
> level by either
>
> - mandating single subvolume for your snapshottable OS, or
> - adding support to grafting individual subvolumes inside of btrfs.
>   Normal mount will not solve this on bootloader level

Single subvolume is already horrible enough. :) We should mandate it.

Thanks for your suggestions and feedback,

Regards,
Michael

>
>
>> That sounds something like the Bootloaderspec, which I like in principle in that it recognizes how hostile the distributions are at breaking the boot behavior of the prior OS, in multiboot contexts. But there's some other things that just don't seem workable, and it's also not even adopted upstream yet by GRUB and I don't know what the status of this whole idea is.
>>
>> I think the idea of snapshots in the domain of a boot manager/boot loader is really overly complicated. For another thing, it's not really appropriate to do a rollback and then immediately start modifying it by booting from it. What you'd want to do is snapshot the rollback, and then use that "cloned" copy of the rollback, leaving the original rollback in place. Otherwise the provenance of that <inserttime> snapshot is obliterated.
>>
>> And with all of these snapshots being created, something to clean up all these bouquets is necessary. Do we really want GRUB doing that also? I just this this is out of scope for GRUB.
>>
>> An FHS for Btrfs installation locations and shapshot behaviors is possibly needed, so the distributions aren't stepping all over each other in an even worse way that booting already is.
>>
>> Chris Murphy
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: booting btrfs

[Chris Murphy]

On Dec 22, 2013, at 10:32 PM, Chris Murphy <lists@colorremedies.com> wrote:
> 
> If relative paths are used, neither core.img nor grub.cfg needs to be modified, because merely a user space program changing the default subvolume would cause the existing core.img and grub.cfg to boot the snapshot.

This seems wrong on second thought. The grub.cfg boot param rootflags=subvol=<path> is interpreted by the kernel, not grub. And for Btrfs subvolume paths are always absolute. Same as fstab.

In that case, getting grub to do paths relative to the default subvolume only helps core.img point to the desired snapshot where /boot/grub/grub.cfg is located. The alternative is to write out a new core.img. Another is to ensure /boot is not being snapshot. So we should be clear on the benefits of snapshotting /boot vs the added complexity.


Chris Murphy

Re: booting btrfs

[Chris Murphy]

On Dec 23, 2013, at 7:26 PM, Michael Chang <mchang@suse.com> wrote:

> Now I tend to agree that supporting config for snapshot booting
> shouldn't be upstream's consideration due to it's compliexity and
> dependency to system, Despite on this, I still like to ask : Did
> upstream think about any patch trying to provide relative path support
> for btrfs subvolume name or id's a worthy work or not?

My vague recollection is that it did used to work this way before 2.00, but maybe was unintended?

> Thanks.  It can be a solution for some use cases but for the one I
> want to achieve it's insufficent. What I'm looking for is to list the
> available snapshots in the boot menu for selecting and then booting to
> it.

OK in that case the tool creates the snapshots, modifies its fstab, and updates grub.cfg so they're displayed.

Direct editing of grub.cfg is fragile in that the user could run grub-mkconfig at any time. It's better for the tool to use an existing entry as a template, and add snapshot entries to 40_custom, then run grub-mkconfig. 

Another thing to note is that there are UEFI computers shipping that don't have USB initialized by the firmware by default at boot time. So the user couldn't use this menu on such systems - which some have suggested will become increasingly common. This implies a snapshot tool that provides the UI to choose snapshots prior to rebooting into that snapshot.

> Meanwhile the snapshot is not for os installation,
> people occasionally want to boot into it for special reasons like he
> want to find from when his system performance start downgrading. I'd
> consider such multiboot should not consider os installtion be in
> snapshots .. because it makes no sense.

I understand. I mean it's eventually feasible for a single Btrfs volume to contain multiple distributions in their own subvolumes, because Btrfs volumes are designed to be large, and easily extend to multiple devices.

It's essentially the same as multiboot without Btrfs at all, the core.img would point to a stable "master" grub.cfg that in turn uses configfile to point to the grub.cfg for each distribution. Each distribution maintains its own grub.cfg without modifying the "master" that core.img is pointing to. It would be nice if distributions made this configuration easy for users to implement.

I think this might be better than doing it with set-default and leave that strictly a user domain function/short cut to being able to mount without -o subvol= option.

> 
>> 
>> I think the idea of snapshots in the domain of a boot manager/boot loader is really overly complicated. For another thing, it's not really appropriate to do a rollback and then immediately start modifying it by booting from it. What you'd want to do is snapshot the rollback, and then use that "cloned" copy of the rollback, leaving the original rollback in place. Otherwise the provenance of that <inserttime> snapshot is obliterated.
> 
> I think that's like seed device and yes there should have such
> handling in initrd to use a clone copy of read-only snapshots when
> your kernel parameters are asking to mount it as /.

Yes good point. Your snapshot tool could first create a read only snapshot, then for no space cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg. The tool could give the user the option to always "revert" the changes caused by booting a snapshot - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.


Chris Murphy

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 466 bytes --]

On 24.12.2013 04:43, Chris Murphy wrote:
> Another thing to note is that there are UEFI computers shipping that don't have USB initialized by the firmware by default at boot time. So the user couldn't use this menu on such systems - which some have suggested will become increasingly common. This implies a snapshot tool that provides the UI to choose snapshots prior to rebooting into that snapshot.
You just have to use GRUB own USB drivers on those systems.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Dec 23, 2013, at 8:46 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> On 24.12.2013 04:43, Chris Murphy wrote:
>> Another thing to note is that there are UEFI computers shipping that don't have USB initialized by the firmware by default at boot time. So the user couldn't use this menu on such systems - which some have suggested will become increasingly common. This implies a snapshot tool that provides the UI to choose snapshots prior to rebooting into that snapshot.
> You just have to use GRUB own USB drivers on those systems.

Good to know. Thanks.


Chris Murphy

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]

On 24.12.2013 04:43, Chris Murphy wrote:
> d point. Your snapshot tool could first create a read only snapshot, then for no space
> cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg.
> The tool could give the user the option to always "revert" the changes caused by booting a snapshot
> - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.
I don't like the idea of constantly modifying grub.cfg.
Points to consider:
- core of GRUB be it in embedding area or efi executable isn't snapshottable
- core and modules version have to match.
- translations should match originating strings.
Three together imply that snapshotting $prefix/$cpu-$platform is useless
if not outright harmful. modules should reside either in .efi
(mkstandalone way) or in a separate volume, never to be snapshotted.
The path to this volume would be baked in core, so default volume
changes won't create core/module mismatch.

The configuration of master GRUB could have a list of all
snapshots/distros/w/e (alternatively they could be listed at runtime)
and source a grub.cfg from this snapshot (either directly or after user
has chosen the submenu) setting some variable to indicate the path to
snapshot. This slave grub.cfg would contain only entries.

Configuration like themes and timeouts would be set on master level.
In case of submenu it's possible to change resolution/theme/font and so
on but it seems like only waste of time.

Init scripts will take care of creating rw clone of snapshot if necessarry.

In this scenario you don't care what the default volume is, and that's
the way it should be as single btrfs may contain several distributions
but only one can own the default.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Chris Murphy]

On Dec 23, 2013, at 9:20 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> On 24.12.2013 04:43, Chris Murphy wrote:
>> d point. Your snapshot tool could first create a read only snapshot, then for no space
>> cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg.
>> The tool could give the user the option to always "revert" the changes caused by booting a snapshot
>> - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.
> I don't like the idea of constantly modifying grub.cfg.

OK. But in any case, is it valid that we want grub-mkconfig to still be able to produce complete and valid grub.cfgs? We don't want it to revert to a snapshot incapable grub.cfg. If the grub.cfg is corrupt or accidentally deleted, or /boot must be restored, we'd probably want grub-mkconfig to produce a fully correct and capable grub.cfg, yes?

> Points to consider:
> - core of GRUB be it in embedding area or efi executable isn't snapshottable
> - core and modules version have to match.
> - translations should match originating strings.
> Three together imply that snapshotting $prefix/$cpu-$platform is useless
> if not outright harmful. modules should reside either in .efi
> (mkstandalone way) or in a separate volume, never to be snapshotted.
> The path to this volume would be baked in core, so default volume
> changes won't create core/module mismatch.

Yeah I agree. There's a possible work around if someone can think of why /boot should be snapshotable: /boot is a subvolume and /boot/grub is also a subvolume; if a snapshot is made of /boot it will not contain /boot/grub at all (the creation of a snapshot does not recursively create snapshots of subvolumes within a subvolume). So in effect if /boot/grub is a subvolume that will make it immune to being dragged along in a snapshot unintentionally. *shrug* But I'm still not imagining a significant advantage to snapshotting /boot.


> The configuration of master GRUB could have a list of all
> snapshots/distros/w/e (alternatively they could be listed at runtime)
> and source a grub.cfg from this snapshot (either directly or after user
> has chosen the submenu) setting some variable to indicate the path to
> snapshot. This slave grub.cfg would contain only entries.
> 
> Configuration like themes and timeouts would be set on master level.
> In case of submenu it's possible to change resolution/theme/font and so
> on but it seems like only waste of time.
> 
> Init scripts will take care of creating rw clone of snapshot if necessarry.

The user space tool that manages these snapshots, and whatever modifications need to be made to make them bootable, should be able to give grub whatever it needs to boot these snapshots. If it's possible that grub, via a module or grub.cfg, can dynamically adjust the menu to show available snapshots to boot from, without constantly modifying grub.cfg, I think that sounds much more stable.


> 
> In this scenario you don't care what the default volume is, and that's
> the way it should be as single btrfs may contain several distributions
> but only one can own the default.

Yes, I'm strongly leaning toward the user alone should own the default subvolume. Consider that the user can still change the default subvolume, and this can't be taken away from them. If a distro uses it, and successful boot depends on the correct subvolume being set as default, the user can inadvertently break boot by changing the set-default. It doesn't sound OK to put the user in that situation.


Chris Murphy

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 4849 bytes --]

On 24.12.2013 07:12, Chris Murphy wrote:
> 
> On Dec 23, 2013, at 9:20 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
> 
>> On 24.12.2013 04:43, Chris Murphy wrote:
>>> d point. Your snapshot tool could first create a read only snapshot, then for no space
>>> cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg.
>>> The tool could give the user the option to always "revert" the changes caused by booting a snapshot
>>> - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.
>> I don't like the idea of constantly modifying grub.cfg.
> 
> OK. But in any case, is it valid that we want grub-mkconfig to still be able to produce complete and valid grub.cfgs? We don't want it to revert to a snapshot incapable grub.cfg. If the grub.cfg is corrupt or accidentally deleted, or /boot must be restored, we'd probably want grub-mkconfig to produce a fully correct and capable grub.cfg, yes?
> 
you can use source_extract / configfile_extract to take only entries.
>> Points to consider:
>> - core of GRUB be it in embedding area or efi executable isn't snapshottable
>> - core and modules version have to match.
>> - translations should match originating strings.
>> Three together imply that snapshotting $prefix/$cpu-$platform is useless
>> if not outright harmful. modules should reside either in .efi
>> (mkstandalone way) or in a separate volume, never to be snapshotted.
>> The path to this volume would be baked in core, so default volume
>> changes won't create core/module mismatch.
> 
> Yeah I agree. There's a possible work around if someone can think of why /boot should be snapshotable:
Did I mention /boot at all? I spoke only about stuff under $prefix.
> /boot is a subvolume and /boot/grub is also a subvolume;

> if a snapshot is made of /boot it will not contain /boot/grub at all (the creation of a snapshot does not

> recursively create snapshots of subvolumes within a subvolume). So in effect if /boot/grub is a subvolume

> that will make it immune to being dragged along in a snapshot unintentionally. *shrug* But I'm

> still not imagining a significant advantage to snapshotting /boot.
>
/boot has to be snapshotted together with / to ensure coherency between
kernel, modules and userland. Only $prefix needs exclusion with grub.cfg
requiring special handling.

> 
>> The configuration of master GRUB could have a list of all
>> snapshots/distros/w/e (alternatively they could be listed at runtime)
>> and source a grub.cfg from this snapshot (either directly or after user
>> has chosen the submenu) setting some variable to indicate the path to
>> snapshot. This slave grub.cfg would contain only entries.
>>
>> Configuration like themes and timeouts would be set on master level.
>> In case of submenu it's possible to change resolution/theme/font and so
>> on but it seems like only waste of time.
>>
>> Init scripts will take care of creating rw clone of snapshot if necessarry.
> 
> The user space tool that manages these snapshots, and whatever modifications
> need to be made to make them bootable,
You need special init handling as you need ability to revert several
times to a given snapshot every time branching to a new series.
> should be able to give grub whatever
> it needs to boot these snapshots. If it's possible that grub, via a module or
> grub.cfg, can dynamically adjust the menu to show available snapshots to boot
> from, without constantly modifying grub.cfg, I think that sounds much more stable.
>
insmod regexp
for x in /debian/*; do
  if [ -f $x/boot/grub/grub.cfg ]; then
    submenu "Debian (snapshot at $x)" "$x" {
      configfile_extract $1/boot/grub/grub.cfg
    }
done

> 
>>
>> In this scenario you don't care what the default volume is, and that's
>> the way it should be as single btrfs may contain several distributions
>> but only one can own the default.
> 
> Yes, I'm strongly leaning toward the user alone should own the default
> subvolume. Consider that the user can still change the default subvolume,
> and this can't be taken away from them. If a distro uses it, and successful
> boot depends on the correct subvolume being set as default, the user can
> inadvertently break boot by changing the set-default. It doesn't sound OK
> to put the user in that situation.
>
I don't see any usefullness in default subvolume for fstab-ed disks.
Every fstab entry should contain explicit subvolume name, possibly
derived from boot parameters. Default subvolume is mainly interesting
for removable media.

> 
> Chris Murphy
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Michael Chang]

2013/12/24 Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com>:
> On 24.12.2013 07:12, Chris Murphy wrote:
>>
>> On Dec 23, 2013, at 9:20 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
>>
>>> On 24.12.2013 04:43, Chris Murphy wrote:
>>>> d point. Your snapshot tool could first create a read only snapshot, then for no space
>>>> cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg.
>>>> The tool could give the user the option to always "revert" the changes caused by booting a snapshot
>>>> - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.
>>> I don't like the idea of constantly modifying grub.cfg.
>>
>> OK. But in any case, is it valid that we want grub-mkconfig to still be able to produce complete and valid grub.cfgs? We don't want it to revert to a snapshot incapable grub.cfg. If the grub.cfg is corrupt or accidentally deleted, or /boot must be restored, we'd probably want grub-mkconfig to produce a fully correct and capable grub.cfg, yes?
>>
> you can use source_extract / configfile_extract to take only entries.
>>> Points to consider:
>>> - core of GRUB be it in embedding area or efi executable isn't snapshottable
>>> - core and modules version have to match.
>>> - translations should match originating strings.
>>> Three together imply that snapshotting $prefix/$cpu-$platform is useless
>>> if not outright harmful. modules should reside either in .efi
>>> (mkstandalone way) or in a separate volume, never to be snapshotted.
>>> The path to this volume would be baked in core, so default volume
>>> changes won't create core/module mismatch.
>>
>> Yeah I agree. There's a possible work around if someone can think of why /boot should be snapshotable:
> Did I mention /boot at all? I spoke only about stuff under $prefix.
>> /boot is a subvolume and /boot/grub is also a subvolume;
>
>> if a snapshot is made of /boot it will not contain /boot/grub at all (the creation of a snapshot does not
>
>> recursively create snapshots of subvolumes within a subvolume). So in effect if /boot/grub is a subvolume
>
>> that will make it immune to being dragged along in a snapshot unintentionally. *shrug* But I'm
>
>> still not imagining a significant advantage to snapshotting /boot.
>>
> /boot has to be snapshotted together with / to ensure coherency between
> kernel, modules and userland. Only $prefix needs exclusion with grub.cfg
> requiring special handling.
>
>>
>>> The configuration of master GRUB could have a list of all
>>> snapshots/distros/w/e (alternatively they could be listed at runtime)
>>> and source a grub.cfg from this snapshot (either directly or after user
>>> has chosen the submenu) setting some variable to indicate the path to
>>> snapshot. This slave grub.cfg would contain only entries.
>>>
>>> Configuration like themes and timeouts would be set on master level.
>>> In case of submenu it's possible to change resolution/theme/font and so
>>> on but it seems like only waste of time.
>>>
>>> Init scripts will take care of creating rw clone of snapshot if necessarry.
>>
>> The user space tool that manages these snapshots, and whatever modifications
>> need to be made to make them bootable,
> You need special init handling as you need ability to revert several
> times to a given snapshot every time branching to a new series.
>> should be able to give grub whatever
>> it needs to boot these snapshots. If it's possible that grub, via a module or
>> grub.cfg, can dynamically adjust the menu to show available snapshots to boot
>> from, without constantly modifying grub.cfg, I think that sounds much more stable.
>>
> insmod regexp
> for x in /debian/*; do
>   if [ -f $x/boot/grub/grub.cfg ]; then
>     submenu "Debian (snapshot at $x)" "$x" {
>       configfile_extract $1/boot/grub/grub.cfg
>     }
> done

As a follow up to your previous example. If we want the grub.cfg
unmodfied from it's snapshot source, we need to dynamically set it's
new subvolume to boot. I suppose a more complete config would
therefore like this.

insmod regexp
for x in /debian/*; do
    if [ -f $x/boot/grub/grub.cfg ]; then
       submenu "Debian (snapshot at $x)" "$x" {
       bootdir="$1"
       rootdir="$1"
       export bootdir
       export rootdir
       configfile_extract $1/boot/grub/grub.cfg
    }
done

Having a way to retain unmodfied copy of grub.cfg in snapshot is
pretty important imho, not only because the snapshot could be
read-only, but also we can diff with snapshots to track it's changes.

And very thankful for the "$1" magic, it heals my headache of variable
assignment per each (sub)menu entries.

Regards,
Michael

>
>>
>>>
>>> In this scenario you don't care what the default volume is, and that's
>>> the way it should be as single btrfs may contain several distributions
>>> but only one can own the default.
>>
>> Yes, I'm strongly leaning toward the user alone should own the default
>> subvolume. Consider that the user can still change the default subvolume,
>> and this can't be taken away from them. If a distro uses it, and successful
>> boot depends on the correct subvolume being set as default, the user can
>> inadvertently break boot by changing the set-default. It doesn't sound OK
>> to put the user in that situation.
>>
> I don't see any usefullness in default subvolume for fstab-ed disks.
> Every fstab entry should contain explicit subvolume name, possibly
> derived from boot parameters. Default subvolume is mainly interesting
> for removable media.
>
>>
>> Chris Murphy
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
>
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: booting btrfs

[Andrey Borzenkov]

On Tue, Dec 24, 2013 at 11:28 AM, Michael Chang <mchang@suse.com> wrote:
>
> And very thankful for the "$1" magic, it heals my headache of variable
> assignment per each (sub)menu entries.
>

"$1" is always menu entry title, you probably mean "$2" :)

Re: booting btrfs

[Michael Chang]

On Mon, Dec 23, 2013 at 08:43:34PM -0700, Chris Murphy wrote:
> 
> On Dec 23, 2013, at 7:26 PM, Michael Chang <mchang@suse.com> wrote:
> 
> > Now I tend to agree that supporting config for snapshot booting
> > shouldn't be upstream's consideration due to it's compliexity and
> > dependency to system, Despite on this, I still like to ask : Did
> > upstream think about any patch trying to provide relative path support
> > for btrfs subvolume name or id's a worthy work or not?
> 
> My vague recollection is that it did used to work this way before 2.00, but maybe was unintended?

It used to follow relative path of set-default volume, but was reverted
to always use absolute path of real root. It's similar to my question
but mine is to have a path intepretation per any subvolume set via
environment variable or so.

It will work like this way.

set btrfs_subvol=.snapshot_1
<All path intepretation by the .snapshot_1 subvolume ..>

set btrfs_subvol=.snapshot_2
<All path intepretation by the .snapshot_2 subvolume ..>

But this would bring ambiguous path back that I'm not sure a good idea
or not to have such feature.

> 
> > Thanks.  It can be a solution for some use cases but for the one I
> > want to achieve it's insufficent. What I'm looking for is to list the
> > available snapshots in the boot menu for selecting and then booting to
> > it.
> 
> OK in that case the tool creates the snapshots, modifies its fstab, and updates grub.cfg so they're displayed.
> 
> Direct editing of grub.cfg is fragile in that the user could run grub-mkconfig at any time. It's better for the tool to use an existing entry as a template, and add snapshot entries to 40_custom, then run grub-mkconfig.

Yes. I think this is suggested approch for modifying grub configs.
What bothers me in hooking into grub-mkconfig is it takes time to
finish the "entire" config and will slow down snapshot tools in
creating the snapshot if we hook grub-mkconfig into it's post
processing scripts.

Does offer an option like `--run-script=90_btrfs_snapshot` to
grub-mkconfig feasible or not? My apologies if this is off topic
here.

Regards,
Michael


> 
> Another thing to note is that there are UEFI computers shipping that don't have USB initialized by the firmware by default at boot time. So the user couldn't use this menu on such systems - which some have suggested will become increasingly common. This implies a snapshot tool that provides the UI to choose snapshots prior to rebooting into that snapshot.
> 
> > Meanwhile the snapshot is not for os installation,
> > people occasionally want to boot into it for special reasons like he
> > want to find from when his system performance start downgrading. I'd
> > consider such multiboot should not consider os installtion be in
> > snapshots .. because it makes no sense.
> 
> I understand. I mean it's eventually feasible for a single Btrfs volume to contain multiple distributions in their own subvolumes, because Btrfs volumes are designed to be large, and easily extend to multiple devices.
> 
> It's essentially the same as multiboot without Btrfs at all, the core.img would point to a stable "master" grub.cfg that in turn uses configfile to point to the grub.cfg for each distribution. Each distribution maintains its own grub.cfg without modifying the "master" that core.img is pointing to. It would be nice if distributions made this configuration easy for users to implement.
> 
> I think this might be better than doing it with set-default and leave that strictly a user domain function/short cut to being able to mount without -o subvol= option.
> 
> > 
> >> 
> >> I think the idea of snapshots in the domain of a boot manager/boot loader is really overly complicated. For another thing, it's not really appropriate to do a rollback and then immediately start modifying it by booting from it. What you'd want to do is snapshot the rollback, and then use that "cloned" copy of the rollback, leaving the original rollback in place. Otherwise the provenance of that <inserttime> snapshot is obliterated.
> > 
> > I think that's like seed device and yes there should have such
> > handling in initrd to use a clone copy of read-only snapshots when
> > your kernel parameters are asking to mount it as /.
> 
> Yes good point. Your snapshot tool could first create a read only snapshot, then for no space cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg. The tool could give the user the option to always "revert" the changes caused by booting a snapshot - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.
> 
> 
> Chris Murphy
> 

-- 
Michael Chang
Software Engineer
Rm. B, 26F, No.216, Tun Hwa S. Rd., Sec.2
Taipei 106, Taiwan, R.O.C
+886223760030
mchang@suse.com
SUSE

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

On 30.12.2013 11:18, Michael Chang wrote:
> On Mon, Dec 23, 2013 at 08:43:34PM -0700, Chris Murphy wrote:
>>
>> On Dec 23, 2013, at 7:26 PM, Michael Chang <mchang@suse.com> wrote:
>>
>>> Now I tend to agree that supporting config for snapshot booting
>>> shouldn't be upstream's consideration due to it's compliexity and
>>> dependency to system, Despite on this, I still like to ask : Did
>>> upstream think about any patch trying to provide relative path support
>>> for btrfs subvolume name or id's a worthy work or not?
>>
>> My vague recollection is that it did used to work this way before 2.00, but maybe was unintended?
> 
> It used to follow relative path of set-default volume, but was reverted
> to always use absolute path of real root. It's similar to my question
> but mine is to have a path intepretation per any subvolume set via
> environment variable or so.
> 
> It will work like this way.
> 
> set btrfs_subvol=.snapshot_1
> <All path intepretation by the .snapshot_1 subvolume ..>
> 
> set btrfs_subvol=.snapshot_2
> <All path intepretation by the .snapshot_2 subvolume ..>
> 
> But this would bring ambiguous path back that I'm not sure a good idea
> or not to have such feature.
> 
No. Just add $btrfs_subvol into paths that you want modified.
> 
> Yes. I think this is suggested approch for modifying grub configs.
> What bothers me in hooking into grub-mkconfig is it takes time to
> finish the "entire" config and will slow down snapshot tools in
> creating the snapshot if we hook grub-mkconfig into it's post
> processing scripts.
> 
> Does offer an option like `--run-script=90_btrfs_snapshot` to
> grub-mkconfig feasible or not? My apologies if this is off topic
> here.
> 
Not necessarry.Read my e-mail for explanation on how to do sanely and
raise any problems you see with it.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 291 bytes --]

Re: booting btrfs

[Andrey Borzenkov]

On Mon, Dec 30, 2013 at 3:28 PM, Vladimir 'φ-coder/phcoder' Serbinenko
<phcoder@gmail.com> wrote:
> On 30.12.2013 11:18, Michael Chang wrote:
>> On Mon, Dec 23, 2013 at 08:43:34PM -0700, Chris Murphy wrote:
>>>
>>> On Dec 23, 2013, at 7:26 PM, Michael Chang <mchang@suse.com> wrote:
>>>
>>>> Now I tend to agree that supporting config for snapshot booting
>>>> shouldn't be upstream's consideration due to it's compliexity and
>>>> dependency to system, Despite on this, I still like to ask : Did
>>>> upstream think about any patch trying to provide relative path support
>>>> for btrfs subvolume name or id's a worthy work or not?
>>>
>>> My vague recollection is that it did used to work this way before 2.00, but maybe was unintended?
>>
>> It used to follow relative path of set-default volume, but was reverted
>> to always use absolute path of real root. It's similar to my question
>> but mine is to have a path intepretation per any subvolume set via
>> environment variable or so.
>>
>> It will work like this way.
>>
>> set btrfs_subvol=.snapshot_1
>> <All path intepretation by the .snapshot_1 subvolume ..>
>>
>> set btrfs_subvol=.snapshot_2
>> <All path intepretation by the .snapshot_2 subvolume ..>
>>
>> But this would bring ambiguous path back that I'm not sure a good idea
>> or not to have such feature.
>>
> No. Just add $btrfs_subvol into paths that you want modified.

I do not think that even this is required. We already have
${config_directory} so configuraton simply can be done relative to it.

>>
>> Yes. I think this is suggested approch for modifying grub configs.
>> What bothers me in hooking into grub-mkconfig is it takes time to
>> finish the "entire" config and will slow down snapshot tools in
>> creating the snapshot if we hook grub-mkconfig into it's post
>> processing scripts.
>>
>> Does offer an option like `--run-script=90_btrfs_snapshot` to
>> grub-mkconfig feasible or not? My apologies if this is off topic
>> here.
>>
> Not necessarry.Read my e-mail for explanation on how to do sanely and
> raise any problems you see with it.
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: booting btrfs

[Michael Chang]

On Mon, Dec 30, 2013 at 12:28:44PM +0100, Vladimir 'φ-coder/phcoder' Serbinenko wrote:
> On 30.12.2013 11:18, Michael Chang wrote:
> > On Mon, Dec 23, 2013 at 08:43:34PM -0700, Chris Murphy wrote:
> >>
> >> On Dec 23, 2013, at 7:26 PM, Michael Chang <mchang@suse.com> wrote:
> >>
> >>> Now I tend to agree that supporting config for snapshot booting
> >>> shouldn't be upstream's consideration due to it's compliexity and
> >>> dependency to system, Despite on this, I still like to ask : Did
> >>> upstream think about any patch trying to provide relative path support
> >>> for btrfs subvolume name or id's a worthy work or not?
> >>
> >> My vague recollection is that it did used to work this way before 2.00, but maybe was unintended?
> > 
> > It used to follow relative path of set-default volume, but was reverted
> > to always use absolute path of real root. It's similar to my question
> > but mine is to have a path intepretation per any subvolume set via
> > environment variable or so.
> > 
> > It will work like this way.
> > 
> > set btrfs_subvol=.snapshot_1
> > <All path intepretation by the .snapshot_1 subvolume ..>
> > 
> > set btrfs_subvol=.snapshot_2
> > <All path intepretation by the .snapshot_2 subvolume ..>
> > 
> > But this would bring ambiguous path back that I'm not sure a good idea
> > or not to have such feature.
> > 
> No. Just add $btrfs_subvol into paths that you want modified.

OK. Good to know. Though I think it could provide usefulness in some
cases but now I am also convinced that having consistent path
outweights them.

> > 
> > Yes. I think this is suggested approch for modifying grub configs.
> > What bothers me in hooking into grub-mkconfig is it takes time to
> > finish the "entire" config and will slow down snapshot tools in
> > creating the snapshot if we hook grub-mkconfig into it's post
> > processing scripts.
> > 
> > Does offer an option like `--run-script=90_btrfs_snapshot` to
> > grub-mkconfig feasible or not? My apologies if this is off topic
> > here.

Sorry I didn't make this clear. In openSUSE, we manage snapshots with
one tool called snapper. And it uses it's own metadata to describe the
snapshots (creation date, types, etc ..) and that script is actually
intended to get the snapshots named and organized from metadata
provided by snapper.

I thought this is not interested by upstream as it's not general and is
customed feature of downstream distribution, also that it would be
installed and maintained by snapper or any other package according to
feature requirments. It's submenu will be created side-by-side with
the default one we're discussing here (which's dynamically created) as
add-on to it to offer more choice.

That's why this is off topic in my mind as it seems to deviate the
discussion. :) I bumped to the issue of running grub-mkconfig slowing
down the creation process, although there could use other way to get
around it, I'd like to poke if any one has different ideas.

> > 
> Not necessarry.Read my e-mail for explanation on how to do sanely and
> raise any problems you see with it.
> 

Overall it works for me with your recipe config of dynamic entries.
The problem I had was extract_entries_configfile cannot execute
commnads due to they are jailed[1], so no entry could boot under it. 
OTOH the menu was correctly displayed.

I have no idea how to unjail the commands, subsequent tests using
configfile was successful, but not as ideal as having
extract_entries_configfile as it offers less risk on different menu
contexts, among other things.

[1]
error: setparams isn't allowed to execute in an extractor.
error: insmod isn't allowed to execute in an extractor.
error: insmod isn't allowed to execute in an extractor.
error: insmod isn't allowed to execute in an extractor.
error: insmod isn't allowed to execute in an extractor.
error: echo isn't allowed to execute in an extractor.
error: linux isn't allowed to execute in an extractor.
error: echo isn't allowed to execute in an extractor.
error: initrd isn't allowed to execute in an extractor.

Anyone can help to pointer me this?

Thanks,
Michael

> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: booting btrfs

[Michael Chang]

On Tue, Dec 24, 2013 at 11:46:00AM +0400, Andrey Borzenkov wrote:
> On Tue, Dec 24, 2013 at 11:28 AM, Michael Chang <mchang@suse.com> wrote:
> >
> > And very thankful for the "$1" magic, it heals my headache of variable
> > assignment per each (sub)menu entries.
> >
> 
> "$1" is always menu entry title, you probably mean "$2" :)

You're right. $2 is correct. 

I think people would intutively map $0 to argv0 and $1 to argv1 and
is wrong here. :)

Thanks,
Michael

> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: booting btrfs

[Michael Chang]

On Mon, Dec 30, 2013 at 03:52:36PM +0400, Andrey Borzenkov wrote:
> On Mon, Dec 30, 2013 at 3:28 PM, Vladimir 'φ-coder/phcoder' Serbinenko
> <phcoder@gmail.com> wrote:
> > On 30.12.2013 11:18, Michael Chang wrote:
> >> On Mon, Dec 23, 2013 at 08:43:34PM -0700, Chris Murphy wrote:
> >>>
> >>> On Dec 23, 2013, at 7:26 PM, Michael Chang <mchang@suse.com> wrote:
> >>>
> >>>> Now I tend to agree that supporting config for snapshot booting
> >>>> shouldn't be upstream's consideration due to it's compliexity and
> >>>> dependency to system, Despite on this, I still like to ask : Did
> >>>> upstream think about any patch trying to provide relative path support
> >>>> for btrfs subvolume name or id's a worthy work or not?
> >>>
> >>> My vague recollection is that it did used to work this way before 2.00, but maybe was unintended?
> >>
> >> It used to follow relative path of set-default volume, but was reverted
> >> to always use absolute path of real root. It's similar to my question
> >> but mine is to have a path intepretation per any subvolume set via
> >> environment variable or so.
> >>
> >> It will work like this way.
> >>
> >> set btrfs_subvol=.snapshot_1
> >> <All path intepretation by the .snapshot_1 subvolume ..>
> >>
> >> set btrfs_subvol=.snapshot_2
> >> <All path intepretation by the .snapshot_2 subvolume ..>
> >>
> >> But this would bring ambiguous path back that I'm not sure a good idea
> >> or not to have such feature.
> >>
> > No. Just add $btrfs_subvol into paths that you want modified.
> 
> I do not think that even this is required. We already have
> ${config_directory} so configuraton simply can be done relative to it.

Thanks for pointing me this. I think we can try to figure out
subvolume dynamically by combining it with ${prefix}. The script would
look like this.

regexp --set=prefix_path "(/.*)" "$prefix"
regexp --set=config_directory_path "(/.*)" "$config_directory"
regexp --set=subvol "(.*)$prefix_path" "$config_directory_path"

if [ -n "$subvol" -a -d "$subvol" ]; then
  rootflags="rootflags=subvol=${subvol}"
else
  rootflags=""
  subvol=""
fi

linux ${subvol}/boot/kernel $rootflags
initrd ${subvol}/boot/initrd

Thanks,
Michael

> 
> >>
> >> Yes. I think this is suggested approch for modifying grub configs.
> >> What bothers me in hooking into grub-mkconfig is it takes time to
> >> finish the "entire" config and will slow down snapshot tools in
> >> creating the snapshot if we hook grub-mkconfig into it's post
> >> processing scripts.
> >>
> >> Does offer an option like `--run-script=90_btrfs_snapshot` to
> >> grub-mkconfig feasible or not? My apologies if this is off topic
> >> here.
> >>
> > Not necessarry.Read my e-mail for explanation on how to do sanely and
> > raise any problems you see with it.
> >
> >
> > _______________________________________________
> > Grub-devel mailing list
> > Grub-devel@gnu.org
> > https://lists.gnu.org/mailman/listinfo/grub-devel
> >
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

-- 
Michael Chang
Software Engineer
Rm. B, 26F, No.216, Tun Hwa S. Rd., Sec.2
Taipei 106, Taiwan, R.O.C
+886223760030
mchang@suse.com
SUSE

Re: booting btrfs

[Chris Murphy]

On Dec 31, 2013, at 12:50 AM, Michael Chang <mchang@suse.com> wrote:

> On Mon, Dec 30, 2013 at 03:52:36PM +0400, Andrey Borzenkov wrote:
>> On Mon, Dec 30, 2013 at 3:28 PM, Vladimir 'φ-coder/phcoder' Serbinenko
>> <phcoder@gmail.com> wrote:
>>> On 30.12.2013 11:18, Michael Chang wrote:
>>>> On Mon, Dec 23, 2013 at 08:43:34PM -0700, Chris Murphy wrote:
>>>>> 
>>>>> On Dec 23, 2013, at 7:26 PM, Michael Chang <mchang@suse.com> wrote:
>>>>> 
>>>>>> Now I tend to agree that supporting config for snapshot booting
>>>>>> shouldn't be upstream's consideration due to it's compliexity and
>>>>>> dependency to system, Despite on this, I still like to ask : Did
>>>>>> upstream think about any patch trying to provide relative path support
>>>>>> for btrfs subvolume name or id's a worthy work or not?
>>>>> 
>>>>> My vague recollection is that it did used to work this way before 2.00, but maybe was unintended?
>>>> 
>>>> It used to follow relative path of set-default volume, but was reverted
>>>> to always use absolute path of real root. It's similar to my question
>>>> but mine is to have a path intepretation per any subvolume set via
>>>> environment variable or so.
>>>> 
>>>> It will work like this way.
>>>> 
>>>> set btrfs_subvol=.snapshot_1
>>>> <All path intepretation by the .snapshot_1 subvolume ..>
>>>> 
>>>> set btrfs_subvol=.snapshot_2
>>>> <All path intepretation by the .snapshot_2 subvolume ..>
>>>> 
>>>> But this would bring ambiguous path back that I'm not sure a good idea
>>>> or not to have such feature.
>>>> 
>>> No. Just add $btrfs_subvol into paths that you want modified.
>> 
>> I do not think that even this is required. We already have
>> ${config_directory} so configuraton simply can be done relative to it.
> 
> Thanks for pointing me this. I think we can try to figure out
> subvolume dynamically by combining it with ${prefix}. The script would
> look like this.
> 
> regexp --set=prefix_path "(/.*)" "$prefix"
> regexp --set=config_directory_path "(/.*)" "$config_directory"
> regexp --set=subvol "(.*)$prefix_path" "$config_directory_path"
> 
> if [ -n "$subvol" -a -d "$subvol" ]; then
>  rootflags="rootflags=subvol=${subvol}"
> else
>  rootflags=""
>  subvol=""
> fi
> 
> linux ${subvol}/boot/kernel $rootflags
> initrd ${subvol}/boot/initrd

Don't those last two lines hardwire a /boot directory? To me this implies /boot is assumed to be a directory on a Btrfs rootfs, and will be snapshot along with rootfs.

To keep /boot stable it seems it should either be in a boot subvolume so that it's separate from rootfs, or maybe even in the short term could be an e.g. ext4, partition.


Chris Murphy

Re: booting btrfs

[Michael Chang]

2014/1/1 Chris Murphy <lists@colorremedies.com>:
>
> On Dec 31, 2013, at 12:50 AM, Michael Chang <mchang@suse.com> wrote:
>
>> On Mon, Dec 30, 2013 at 03:52:36PM +0400, Andrey Borzenkov wrote:
>>> On Mon, Dec 30, 2013 at 3:28 PM, Vladimir 'φ-coder/phcoder' Serbinenko
>>> <phcoder@gmail.com> wrote:
>>>> On 30.12.2013 11:18, Michael Chang wrote:
>>>>> On Mon, Dec 23, 2013 at 08:43:34PM -0700, Chris Murphy wrote:
>>>>>>
>>>>>> On Dec 23, 2013, at 7:26 PM, Michael Chang <mchang@suse.com> wrote:
>>>>>>
>>>>>>> Now I tend to agree that supporting config for snapshot booting
>>>>>>> shouldn't be upstream's consideration due to it's compliexity and
>>>>>>> dependency to system, Despite on this, I still like to ask : Did
>>>>>>> upstream think about any patch trying to provide relative path support
>>>>>>> for btrfs subvolume name or id's a worthy work or not?
>>>>>>
>>>>>> My vague recollection is that it did used to work this way before 2.00, but maybe was unintended?
>>>>>
>>>>> It used to follow relative path of set-default volume, but was reverted
>>>>> to always use absolute path of real root. It's similar to my question
>>>>> but mine is to have a path intepretation per any subvolume set via
>>>>> environment variable or so.
>>>>>
>>>>> It will work like this way.
>>>>>
>>>>> set btrfs_subvol=.snapshot_1
>>>>> <All path intepretation by the .snapshot_1 subvolume ..>
>>>>>
>>>>> set btrfs_subvol=.snapshot_2
>>>>> <All path intepretation by the .snapshot_2 subvolume ..>
>>>>>
>>>>> But this would bring ambiguous path back that I'm not sure a good idea
>>>>> or not to have such feature.
>>>>>
>>>> No. Just add $btrfs_subvol into paths that you want modified.
>>>
>>> I do not think that even this is required. We already have
>>> ${config_directory} so configuraton simply can be done relative to it.
>>
>> Thanks for pointing me this. I think we can try to figure out
>> subvolume dynamically by combining it with ${prefix}. The script would
>> look like this.
>>
>> regexp --set=prefix_path "(/.*)" "$prefix"
>> regexp --set=config_directory_path "(/.*)" "$config_directory"
>> regexp --set=subvol "(.*)$prefix_path" "$config_directory_path"
>>
>> if [ -n "$subvol" -a -d "$subvol" ]; then
>>  rootflags="rootflags=subvol=${subvol}"
>> else
>>  rootflags=""
>>  subvol=""
>> fi
>>
>> linux ${subvol}/boot/kernel $rootflags
>> initrd ${subvol}/boot/initrd
>
> Don't those last two lines hardwire a /boot directory? To me this implies /boot is assumed to be a directory on a Btrfs rootfs, and will be snapshot along with rootfs.
>
> To keep /boot stable it seems it should either be in a boot subvolume so that it's separate from rootfs, or maybe even in the short term could be an e.g. ext4, partition.

We snapshot /boot for kernel and initrd, otherwise the rollback would
encounter problem of incompatible userland and kernel/kernel modules.
And we need the ability to rollback them in terms of usefulness.

Regards,
Michael

>
>
> Chris Murphy
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: booting btrfs

[Chris Murphy]

On Jan 1, 2014, at 10:17 PM, Michael Chang <mchang@suse.com> wrote:
> 
> We snapshot /boot for kernel and initrd, otherwise the rollback would
> encounter problem of incompatible userland and kernel/kernel modules.
> And we need the ability to rollback them in terms of usefulness.

Of course, understood.

core.img is only going to point to one /boot, which may not be the /boot snapshot needed for the kernel and initrd. This will be really confusing for mortal users. They'd be unlikely to figure it out and understand it, without documentation.

If core.img points to the "current" /boot, which it should, that boot has the accumulated knowledge of all snapshots, and any recently updated grub modules. Choosing to boot a snapshot means using a different /boot for kernel/initramfs than what grub is using for its root. I don't off hand see a problem with this because it's literally just two files that grub loads from a different boot subvolume, found with an absolute path to that snapshot. And it also uses rootflags=subvol= to use the matching root snapshot.

A separate issue that's not grub's problem is how to deal with the (now wrong) fstab entries. systemd looks at fstab and generates mount jobs from that; if taught to understand it's booting a snapshot it could second guess parts of the fstab. Based on the name of the currently booting root snapshot, which systemd definitely knows, it could mount that subvol= instead of what's in fstab. It can use name substitution to do the same thing with the other subvolume-snapshots that match the root one. Meaning all of the snapshots for a system have the same base (re)naming convention.


Chris Murphy

Re: booting btrfs

[Chris Murphy]

On Jan 7, 2014, at 10:55 AM, Chris Murphy <lists@colorremedies.com> wrote:

> 
> On Jan 1, 2014, at 10:17 PM, Michael Chang <mchang@suse.com> wrote:
>> 
>> We snapshot /boot for kernel and initrd, otherwise the rollback would
>> encounter problem of incompatible userland and kernel/kernel modules.
>> And we need the ability to rollback them in terms of usefulness.
> 
> Of course, understood.
> 
> core.img is only going to point to one /boot, which may not be the /boot snapshot needed for the kernel and initrd. This will be really confusing for mortal users. They'd be unlikely to figure it out and understand it, without documentation.
> 
> If core.img points to the "current" /boot, which it should, that boot has the accumulated knowledge of all snapshots, and any recently updated grub modules. Choosing to boot a snapshot means using a different /boot for kernel/initramfs than what grub is using for its root. I don't off hand see a problem with this because it's literally just two files that grub loads from a different boot subvolume, found with an absolute path to that snapshot. And it also uses rootflags=subvol= to use the matching root snapshot.
> 
> A separate issue that's not grub's problem is how to deal with the (now wrong) fstab entries. systemd looks at fstab and generates mount jobs from that; if taught to understand it's booting a snapshot it could second guess parts of the fstab. Based on the name of the currently booting root snapshot, which systemd definitely knows, it could mount that subvol= instead of what's in fstab. It can use name substitution to do the same thing with the other subvolume-snapshots that match the root one. Meaning all of the snapshots for a system have the same base (re)naming convention.

Another hiccup. Maybe it's a silly use case. Consider /boot on Btrfs, multiple-device, raid1 data/metadata profile, UEFI Secure Boot. A drive dies, and the system needs to be rebooted before a rebuild occurs.

This works today with /boot on raid1/10 Btrfs. Yes, I manually have to add a degraded mount option as this isn't automatically done by Btrfs yet. But the GRUB boot part works. Even degraded, the path to grub.cfg is valid. And the file system itself keeps multiple copies so there's no work keeping it current.

With Secure Boot it's a problem. The signed grubx64.efi has a fixed prefix location, valid on any computer, to search for grub.cfg, which is on the ESP. So now we need to have multiple copies of grub.cfg, somehow synced, on each ESP. Or another solution. If we had a Btrfs subvolumetypeGUID, analogous to the GPT partitiontypeGUID, and specify that as the baked in partuuid for a signed grubx64.efi to search for /boot/grub/grub.cfg.

What do you think? Total edge case?


Chris Murphy

Re: booting btrfs

[Michael Chang]

2014/1/9 Chris Murphy <lists@colorremedies.com>:
>
> On Jan 7, 2014, at 10:55 AM, Chris Murphy <lists@colorremedies.com> wrote:
>
>>
>> On Jan 1, 2014, at 10:17 PM, Michael Chang <mchang@suse.com> wrote:
>>>
>>> We snapshot /boot for kernel and initrd, otherwise the rollback would
>>> encounter problem of incompatible userland and kernel/kernel modules.
>>> And we need the ability to rollback them in terms of usefulness.
>>
>> Of course, understood.
>>
>> core.img is only going to point to one /boot, which may not be the /boot snapshot needed for the kernel and initrd. This will be really confusing for mortal users. They'd be unlikely to figure it out and understand it, without documentation.
>>
>> If core.img points to the "current" /boot, which it should, that boot has the accumulated knowledge of all snapshots, and any recently updated grub modules. Choosing to boot a snapshot means using a different /boot for kernel/initramfs than what grub is using for its root. I don't off hand see a problem with this because it's literally just two files that grub loads from a different boot subvolume, found with an absolute path to that snapshot. And it also uses rootflags=subvol= to use the matching root snapshot.
>>
>> A separate issue that's not grub's problem is how to deal with the (now wrong) fstab entries. systemd looks at fstab and generates mount jobs from that; if taught to understand it's booting a snapshot it could second guess parts of the fstab. Based on the name of the currently booting root snapshot, which systemd definitely knows, it could mount that subvol= instead of what's in fstab. It can use name substitution to do the same thing with the other subvolume-snapshots that match the root one. Meaning all of the snapshots for a system have the same base (re)naming convention.
>
> Another hiccup. Maybe it's a silly use case. Consider /boot on Btrfs, multiple-device, raid1 data/metadata profile, UEFI Secure Boot. A drive dies, and the system needs to be rebooted before a rebuild occurs.
>
> This works today with /boot on raid1/10 Btrfs. Yes, I manually have to add a degraded mount option as this isn't automatically done by Btrfs yet. But the GRUB boot part works. Even degraded, the path to grub.cfg is valid. And the file system itself keeps multiple copies so there's no work keeping it current.
>
> With Secure Boot it's a problem. The signed grubx64.efi has a fixed prefix location, valid on any computer, to search for grub.cfg, which is on the ESP. So now we need to have multiple copies of grub.cfg, somehow synced, on each ESP. Or another solution. If we had a Btrfs subvolumetypeGUID, analogous to the GPT partitiontypeGUID, and specify that as the baked in partuuid for a signed grubx64.efi to search for /boot/grub/grub.cfg.

Isn't search --fs-uuid sufficient for this task? Or I'm afraid that I
didn't understand your problem enough.

  cat <<EOF > /boot/efi/EFI/<DISTRIBUTION>/grub.cfg
  search --fs-uuid --set=root `grub-probe --target=fs_uuid /boot/grub`
  set prefix=(\$root)/boot/grub
  configfile \$prefix/grub.cfg
  EOF

  grub-mkconfig -o /boot/grub/grub.cfg

This way we can avoid calling "grub-mkconfig -o
/boot/efi/EFI/<DISTRIBUTION>/grub.cfg" and hopefully can get rid of
the problem you have.

Regards,
Michael

>
> What do you think? Total edge case?
>
>
> Chris Murphy
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: booting btrfs

[Chris Murphy]

On Jan 9, 2014, at 3:03 AM, Michael Chang <mchang@suse.com> wrote:

> 2014/1/9 Chris Murphy <lists@colorremedies.com>:
>> 
>> On Jan 7, 2014, at 10:55 AM, Chris Murphy <lists@colorremedies.com> wrote:
>> 
>>> 
>>> On Jan 1, 2014, at 10:17 PM, Michael Chang <mchang@suse.com> wrote:
>>>> 
>>>> We snapshot /boot for kernel and initrd, otherwise the rollback would
>>>> encounter problem of incompatible userland and kernel/kernel modules.
>>>> And we need the ability to rollback them in terms of usefulness.
>>> 
>>> Of course, understood.
>>> 
>>> core.img is only going to point to one /boot, which may not be the /boot snapshot needed for the kernel and initrd. This will be really confusing for mortal users. They'd be unlikely to figure it out and understand it, without documentation.
>>> 
>>> If core.img points to the "current" /boot, which it should, that boot has the accumulated knowledge of all snapshots, and any recently updated grub modules. Choosing to boot a snapshot means using a different /boot for kernel/initramfs than what grub is using for its root. I don't off hand see a problem with this because it's literally just two files that grub loads from a different boot subvolume, found with an absolute path to that snapshot. And it also uses rootflags=subvol= to use the matching root snapshot.
>>> 
>>> A separate issue that's not grub's problem is how to deal with the (now wrong) fstab entries. systemd looks at fstab and generates mount jobs from that; if taught to understand it's booting a snapshot it could second guess parts of the fstab. Based on the name of the currently booting root snapshot, which systemd definitely knows, it could mount that subvol= instead of what's in fstab. It can use name substitution to do the same thing with the other subvolume-snapshots that match the root one. Meaning all of the snapshots for a system have the same base (re)naming convention.
>> 
>> Another hiccup. Maybe it's a silly use case. Consider /boot on Btrfs, multiple-device, raid1 data/metadata profile, UEFI Secure Boot. A drive dies, and the system needs to be rebooted before a rebuild occurs.
>> 
>> This works today with /boot on raid1/10 Btrfs. Yes, I manually have to add a degraded mount option as this isn't automatically done by Btrfs yet. But the GRUB boot part works. Even degraded, the path to grub.cfg is valid. And the file system itself keeps multiple copies so there's no work keeping it current.
>> 
>> With Secure Boot it's a problem. The signed grubx64.efi has a fixed prefix location, valid on any computer, to search for grub.cfg, which is on the ESP. So now we need to have multiple copies of grub.cfg, somehow synced, on each ESP. Or another solution. If we had a Btrfs subvolumetypeGUID, analogous to the GPT partitiontypeGUID, and specify that as the baked in partuuid for a signed grubx64.efi to search for /boot/grub/grub.cfg.
> 
> Isn't search --fs-uuid sufficient for this task? Or I'm afraid that I
> didn't understand your problem enough.
> 
>  cat <<EOF > /boot/efi/EFI/<DISTRIBUTION>/grub.cfg
>  search --fs-uuid --set=root `grub-probe --target=fs_uuid /boot/grub`
>  set prefix=(\$root)/boot/grub
>  configfile \$prefix/grub.cfg
>  EOF
> 
>  grub-mkconfig -o /boot/grub/grub.cfg
> 
> This way we can avoid calling "grub-mkconfig -o
> /boot/efi/EFI/<DISTRIBUTION>/grub.cfg" and hopefully can get rid of
> the problem you have.

Yes I think that will work. Do you agree that a GUI installer permitting, e.g. /boot on Btrfs raid1, should implement this? Or should it be a future feature of grub-mkconfig to figure out, that two grub.cfgs are needed?


Chris Murphy

Re: booting btrfs

[Andrey Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 3856 bytes --]

В Tue, 24 Dec 2013 05:20:19 +0100
Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:

> On 24.12.2013 04:43, Chris Murphy wrote:
> > d point. Your snapshot tool could first create a read only snapshot, then for no space
> > cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg.
> > The tool could give the user the option to always "revert" the changes caused by booting a snapshot
> > - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.
> I don't like the idea of constantly modifying grub.cfg.
> Points to consider:
> - core of GRUB be it in embedding area or efi executable isn't snapshottable
> - core and modules version have to match.
> - translations should match originating strings.
> Three together imply that snapshotting $prefix/$cpu-$platform is useless
> if not outright harmful. modules should reside either in .efi
> (mkstandalone way) or in a separate volume, never to be snapshotted.
> The path to this volume would be baked in core, so default volume
> changes won't create core/module mismatch.
>

This is true if we mandate that embedded core image is *the*
bootloader. But it can simply chainload core.img from $prefix which will
guarantee that core.img always matches its modules. This would allow
snapshot $prefix on grub update (or any grub change for that matter) to
have fallback case if anything goes wrong.

So this is similar to stage1.5, which also in principle could be
installed once and never changed.
 
> The configuration of master GRUB could have a list of all
> snapshots/distros/w/e

That exactly means "constantly modifying grub.cfg" on every
snapshot creation, unless I'm mistaken? :)

>                       (alternatively they could be listed at runtime)
> and source a grub.cfg from this snapshot (either directly or after user
> has chosen the submenu) setting some variable to indicate the path to
> snapshot. This slave grub.cfg would contain only entries.
>

This is a bit cumbersome today. Snapshot is done from master; and as
far as I understand from this discussion, most people like to avoid
special steps to prepare snapshot for booting. Which means that
snapshot contains exactly the same entries as master. We need to either
somehow filter entries, or change how grub configuration is generated.

Something like

grub-mkconfig --split

which creates separate configuration file for each script
in /etc/grub.d allowing master to selectively source (extract entries
from) only parts of it. Or always generate two different configs -
"master" and "slave".

BTW 30_os-prober will happily fetch boot entries from every existing
snapshot, presenting them all with identical names and "merging" all
boot entries from all snapshots because it generates the same menu id
(it includes only fs UUID, but no subvolume information).

> Configuration like themes and timeouts would be set on master level.
> In case of submenu it's possible to change resolution/theme/font and so
> on but it seems like only waste of time.
> 

Another possibility would be to a) snapshot /boot/grub together with the
rest of / and b) chainload grub from snapshot. Then nothing needs
changing at all (except some small magic to set BTRFS subvolume at
runtime). The only problem here is to pass $prefix on chainloaded grub.
For EFI we get this almost for free, but for other platforms I'm not
sure. Could we pass this information as parameter when multiboot'ing
core.img? 

> Init scripts will take care of creating rw clone of snapshot if necessarry.
> 
> In this scenario you don't care what the default volume is, and that's
> the way it should be as single btrfs may contain several distributions
> but only one can own the default.
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: booting btrfs

[Michael Chang]

2014/1/11 Andrey Borzenkov <arvidjaar@gmail.com>:
> В Tue, 24 Dec 2013 05:20:19 +0100
> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
>
>> On 24.12.2013 04:43, Chris Murphy wrote:
>> > d point. Your snapshot tool could first create a read only snapshot, then for no space
>> > cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg.
>> > The tool could give the user the option to always "revert" the changes caused by booting a snapshot
>> > - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.
>> I don't like the idea of constantly modifying grub.cfg.
>> Points to consider:
>> - core of GRUB be it in embedding area or efi executable isn't snapshottable
>> - core and modules version have to match.
>> - translations should match originating strings.
>> Three together imply that snapshotting $prefix/$cpu-$platform is useless
>> if not outright harmful. modules should reside either in .efi
>> (mkstandalone way) or in a separate volume, never to be snapshotted.
>> The path to this volume would be baked in core, so default volume
>> changes won't create core/module mismatch.
>>
>
> This is true if we mandate that embedded core image is *the*
> bootloader. But it can simply chainload core.img from $prefix which will
> guarantee that core.img always matches its modules. This would allow
> snapshot $prefix on grub update (or any grub change for that matter) to
> have fallback case if anything goes wrong.
>
> So this is similar to stage1.5, which also in principle could be
> installed once and never changed.

My concern is on chainloading /fs/core.img appropriate ? It looked to
me would likely to fail because the installed lba address has been
hardcoded on it and would go back to the origin (stage1.5 per your
instance.)

Or we need a way to control "boot" by jumping to it's loaded second sector.

>
>> The configuration of master GRUB could have a list of all
>> snapshots/distros/w/e
>
> That exactly means "constantly modifying grub.cfg" on every
> snapshot creation, unless I'm mistaken? :)

For snapshot creation is possible to not update master config if it's
using "dynamic" way of syntax to find the snapshots, as Vladmir has
been demonstrated in it's examples.

>
>>                       (alternatively they could be listed at runtime)
>> and source a grub.cfg from this snapshot (either directly or after user
>> has chosen the submenu) setting some variable to indicate the path to
>> snapshot. This slave grub.cfg would contain only entries.
>>
>
> This is a bit cumbersome today. Snapshot is done from master; and as
> far as I understand from this discussion, most people like to avoid
> special steps to prepare snapshot for booting. Which means that
> snapshot contains exactly the same entries as master. We need to either
> somehow filter entries, or change how grub configuration is generated.
>
> Something like
>
> grub-mkconfig --split
>
> which creates separate configuration file for each script
> in /etc/grub.d allowing master to selectively source (extract entries
> from) only parts of it. Or always generate two different configs -
> "master" and "slave".

It seems like generating a slave config sourced by master in snapshot
is less work and intrusive to current grub-mkconfig.

I like the idea of --split, and would be great to see the extending of
it with ability to update on specified configuration files. For
example, we don't forced to run os-prober or other scripts after new
kernel entries created.

  --split --run=10_linux,20_linux_xen

>
> BTW 30_os-prober will happily fetch boot entries from every existing
> snapshot, presenting them all with identical names and "merging" all
> boot entries from all snapshots because it generates the same menu id
> (it includes only fs UUID, but no subvolume information).

Are you suggesting to use os-prober, instead sourced by master config
directly for inclusion of boot entries of snapshots ? Or I've
misinterpreted it?

>
>> Configuration like themes and timeouts would be set on master level.
>> In case of submenu it's possible to change resolution/theme/font and so
>> on but it seems like only waste of time.
>>
>
> Another possibility would be to a) snapshot /boot/grub together with the
> rest of / and b) chainload grub from snapshot. Then nothing needs
> changing at all (except some small magic to set BTRFS subvolume at
> runtime). The only problem here is to pass $prefix on chainloaded grub.
> For EFI we get this almost for free, but for other platforms I'm not
> sure. Could we pass this information as parameter when multiboot'ing
> core.img?

If the concept could work that's definitely more feasible solution to
me as well, if the problems could be sorted out and fixed.

Regards,
Michael

>
>> Init scripts will take care of creating rw clone of snapshot if necessarry.
>>
>> In this scenario you don't care what the default volume is, and that's
>> the way it should be as single btrfs may contain several distributions
>> but only one can own the default.
>>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: booting btrfs

[Michael Chang]

2014/1/10 Chris Murphy <lists@colorremedies.com>:
>
> On Jan 9, 2014, at 3:03 AM, Michael Chang <mchang@suse.com> wrote:
>
>> 2014/1/9 Chris Murphy <lists@colorremedies.com>:
>>>
>>> On Jan 7, 2014, at 10:55 AM, Chris Murphy <lists@colorremedies.com> wrote:
>>>
>>>>
>>>> On Jan 1, 2014, at 10:17 PM, Michael Chang <mchang@suse.com> wrote:
>>>>>
>>>>> We snapshot /boot for kernel and initrd, otherwise the rollback would
>>>>> encounter problem of incompatible userland and kernel/kernel modules.
>>>>> And we need the ability to rollback them in terms of usefulness.
>>>>
>>>> Of course, understood.
>>>>
>>>> core.img is only going to point to one /boot, which may not be the /boot snapshot needed for the kernel and initrd. This will be really confusing for mortal users. They'd be unlikely to figure it out and understand it, without documentation.
>>>>
>>>> If core.img points to the "current" /boot, which it should, that boot has the accumulated knowledge of all snapshots, and any recently updated grub modules. Choosing to boot a snapshot means using a different /boot for kernel/initramfs than what grub is using for its root. I don't off hand see a problem with this because it's literally just two files that grub loads from a different boot subvolume, found with an absolute path to that snapshot. And it also uses rootflags=subvol= to use the matching root snapshot.
>>>>
>>>> A separate issue that's not grub's problem is how to deal with the (now wrong) fstab entries. systemd looks at fstab and generates mount jobs from that; if taught to understand it's booting a snapshot it could second guess parts of the fstab. Based on the name of the currently booting root snapshot, which systemd definitely knows, it could mount that subvol= instead of what's in fstab. It can use name substitution to do the same thing with the other subvolume-snapshots that match the root one. Meaning all of the snapshots for a system have the same base (re)naming convention.
>>>
>>> Another hiccup. Maybe it's a silly use case. Consider /boot on Btrfs, multiple-device, raid1 data/metadata profile, UEFI Secure Boot. A drive dies, and the system needs to be rebooted before a rebuild occurs.
>>>
>>> This works today with /boot on raid1/10 Btrfs. Yes, I manually have to add a degraded mount option as this isn't automatically done by Btrfs yet. But the GRUB boot part works. Even degraded, the path to grub.cfg is valid. And the file system itself keeps multiple copies so there's no work keeping it current.
>>>
>>> With Secure Boot it's a problem. The signed grubx64.efi has a fixed prefix location, valid on any computer, to search for grub.cfg, which is on the ESP. So now we need to have multiple copies of grub.cfg, somehow synced, on each ESP. Or another solution. If we had a Btrfs subvolumetypeGUID, analogous to the GPT partitiontypeGUID, and specify that as the baked in partuuid for a signed grubx64.efi to search for /boot/grub/grub.cfg.
>>
>> Isn't search --fs-uuid sufficient for this task? Or I'm afraid that I
>> didn't understand your problem enough.
>>
>>  cat <<EOF > /boot/efi/EFI/<DISTRIBUTION>/grub.cfg
>>  search --fs-uuid --set=root `grub-probe --target=fs_uuid /boot/grub`
>>  set prefix=(\$root)/boot/grub
>>  configfile \$prefix/grub.cfg
>>  EOF
>>
>>  grub-mkconfig -o /boot/grub/grub.cfg
>>
>> This way we can avoid calling "grub-mkconfig -o
>> /boot/efi/EFI/<DISTRIBUTION>/grub.cfg" and hopefully can get rid of
>> the problem you have.
>
> Yes I think that will work. Do you agree that a GUI installer permitting, e.g. /boot on Btrfs raid1, should implement this?

Honestly I'm not clear about the benefit of it on your case (/boot on
Btrfs) but it would be helpful if you want managing your config path
more in common and general, so use it if you find it useful. :)

> Or should it be a future feature of grub-mkconfig to figure out, that two grub.cfgs are needed?

No. I don't think grub-mkconfig should be changed at all. Use it in
your UI wrapper scripts to grub-mkcofig would be better.

Regards,
Michael

>
>
> Chris Murphy
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: booting btrfs

[Andrey Borzenkov]

On Mon, Jan 13, 2014 at 9:05 AM, Michael Chang <mchang@suse.com> wrote:
> 2014/1/11 Andrey Borzenkov <arvidjaar@gmail.com>:
>> В Tue, 24 Dec 2013 05:20:19 +0100
>> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
>>
>>> On 24.12.2013 04:43, Chris Murphy wrote:
>>> > d point. Your snapshot tool could first create a read only snapshot, then for no space
>>> > cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg.
>>> > The tool could give the user the option to always "revert" the changes caused by booting a snapshot
>>> > - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.
>>> I don't like the idea of constantly modifying grub.cfg.
>>> Points to consider:
>>> - core of GRUB be it in embedding area or efi executable isn't snapshottable
>>> - core and modules version have to match.
>>> - translations should match originating strings.
>>> Three together imply that snapshotting $prefix/$cpu-$platform is useless
>>> if not outright harmful. modules should reside either in .efi
>>> (mkstandalone way) or in a separate volume, never to be snapshotted.
>>> The path to this volume would be baked in core, so default volume
>>> changes won't create core/module mismatch.
>>>
>>
>> This is true if we mandate that embedded core image is *the*
>> bootloader. But it can simply chainload core.img from $prefix which will
>> guarantee that core.img always matches its modules. This would allow
>> snapshot $prefix on grub update (or any grub change for that matter) to
>> have fallback case if anything goes wrong.
>>
>> So this is similar to stage1.5, which also in principle could be
>> installed once and never changed.
>
> My concern is on chainloading /fs/core.img appropriate ? It looked to
> me would likely to fail because the installed lba address has been
> hardcoded on it and would go back to the origin (stage1.5 per your
> instance.)
>

No. LBA is updated in image that is embedded - it is not the same as
what is kept in core.img. grub-bios-setup modifies it in memory before
writing into embedding area.

> Or we need a way to control "boot" by jumping to it's loaded second sector.
>
>>
>>> The configuration of master GRUB could have a list of all
>>> snapshots/distros/w/e
>>
>> That exactly means "constantly modifying grub.cfg" on every
>> snapshot creation, unless I'm mistaken? :)
>
> For snapshot creation is possible to not update master config if it's
> using "dynamic" way of syntax to find the snapshots, as Vladmir has
> been demonstrated in it's examples.
>

I replied specifically to "configuration could have a list of all
snapshots". It is not dynamic :)

>>
>>>                       (alternatively they could be listed at runtime)
>>> and source a grub.cfg from this snapshot (either directly or after user
>>> has chosen the submenu) setting some variable to indicate the path to
>>> snapshot. This slave grub.cfg would contain only entries.
>>>
>>
>> This is a bit cumbersome today. Snapshot is done from master; and as
>> far as I understand from this discussion, most people like to avoid
>> special steps to prepare snapshot for booting. Which means that
>> snapshot contains exactly the same entries as master. We need to either
>> somehow filter entries, or change how grub configuration is generated.
>>
>> Something like
>>
>> grub-mkconfig --split
>>
>> which creates separate configuration file for each script
>> in /etc/grub.d allowing master to selectively source (extract entries
>> from) only parts of it. Or always generate two different configs -
>> "master" and "slave".
>
> It seems like generating a slave config sourced by master in snapshot
> is less work and intrusive to current grub-mkconfig.
>
> I like the idea of --split, and would be great to see the extending of
> it with ability to update on specified configuration files. For
> example, we don't forced to run os-prober or other scripts after new
> kernel entries created.
>
>   --split --run=10_linux,20_linux_xen
>

Yes, that would be of course interesting extension in this case.

>>
>> BTW 30_os-prober will happily fetch boot entries from every existing
>> snapshot, presenting them all with identical names and "merging" all
>> boot entries from all snapshots because it generates the same menu id
>> (it includes only fs UUID, but no subvolume information).
>
> Are you suggesting to use os-prober, instead sourced by master config
> directly for inclusion of boot entries of snapshots ? Or I've
> misinterpreted it?
>

Oh, sorry for confusion. I simply meant that current 30_os-prober is
buggy and needs fixing. I think I have an idea how.

>>
>>> Configuration like themes and timeouts would be set on master level.
>>> In case of submenu it's possible to change resolution/theme/font and so
>>> on but it seems like only waste of time.
>>>
>>
>> Another possibility would be to a) snapshot /boot/grub together with the
>> rest of / and b) chainload grub from snapshot. Then nothing needs
>> changing at all (except some small magic to set BTRFS subvolume at
>> runtime). The only problem here is to pass $prefix on chainloaded grub.
>> For EFI we get this almost for free, but for other platforms I'm not
>> sure. Could we pass this information as parameter when multiboot'ing
>> core.img?
>
> If the concept could work that's definitely more feasible solution to
> me as well, if the problems could be sorted out and fixed.
>
> Regards,
> Michael
>
>>
>>> Init scripts will take care of creating rw clone of snapshot if necessarry.
>>>
>>> In this scenario you don't care what the default volume is, and that's
>>> the way it should be as single btrfs may contain several distributions
>>> but only one can own the default.
>>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: booting btrfs

[Chris Murphy]

On Jan 12, 2014, at 10:13 PM, Michael Chang <mchang@suse.com> wrote:

> 2014/1/10 Chris Murphy <lists@colorremedies.com>:
>> 
>> On Jan 9, 2014, at 3:03 AM, Michael Chang <mchang@suse.com> wrote:
>>> 
>>> cat <<EOF > /boot/efi/EFI/<DISTRIBUTION>/grub.cfg
>>> search --fs-uuid --set=root `grub-probe --target=fs_uuid /boot/grub`
>>> set prefix=(\$root)/boot/grub
>>> configfile \$prefix/grub.cfg
>>> EOF
>>> 
>>> grub-mkconfig -o /boot/grub/grub.cfg
>>> 
>>> This way we can avoid calling "grub-mkconfig -o
>>> /boot/efi/EFI/<DISTRIBUTION>/grub.cfg" and hopefully can get rid of
>>> the problem you have.
>> 
>> Yes I think that will work. Do you agree that a GUI installer permitting, e.g. /boot on Btrfs raid1, should implement this?
> 
> Honestly I'm not clear about the benefit of it on your case (/boot on
> Btrfs) but it would be helpful if you want managing your config path
> more in common and general, so use it if you find it useful. :)

There are other problems with Fedora that prevent this from being usable now, including grubby which can't update grub.cfg during kernel updates, when /boot is on a Btrfs subvolume. So I have no present implementation, it's a question how to boot from Btrfs raid1 in the future, and have as little duplicative efforts as possible. Also, my argument is that if a GUI installer permits the user to create bootable raid1, that it should also properly configure all drives, and the static ESP grub.cfg, to make the system bootable. Otherwise the user has to do this manually which requires esoteric knowledge beyond most users. If the system really isn't bootable in the face of a disk failure, then I think a GUI installer shouldn't even offer bootable raid1 (Btrfs or otherwise) as an option.


Chris Murphy

Re: booting btrfs

[Michael Chang]

2014/1/13 Andrey Borzenkov <arvidjaar@gmail.com>:
> On Mon, Jan 13, 2014 at 9:05 AM, Michael Chang <mchang@suse.com> wrote:
>> 2014/1/11 Andrey Borzenkov <arvidjaar@gmail.com>:
>>> В Tue, 24 Dec 2013 05:20:19 +0100
>>> Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> пишет:
>>>
>>>> On 24.12.2013 04:43, Chris Murphy wrote:
>>>> > d point. Your snapshot tool could first create a read only snapshot, then for no space
>>>> > cost also create a rw snapshot of the read only one, then add the rw snapshot to the grub.cfg.
>>>> > The tool could give the user the option to always "revert" the changes caused by booting a snapshot
>>>> > - this would cause the rw snapshot being deleted and a new rw snapshot created from the read only one.
>>>> I don't like the idea of constantly modifying grub.cfg.
>>>> Points to consider:
>>>> - core of GRUB be it in embedding area or efi executable isn't snapshottable
>>>> - core and modules version have to match.
>>>> - translations should match originating strings.
>>>> Three together imply that snapshotting $prefix/$cpu-$platform is useless
>>>> if not outright harmful. modules should reside either in .efi
>>>> (mkstandalone way) or in a separate volume, never to be snapshotted.
>>>> The path to this volume would be baked in core, so default volume
>>>> changes won't create core/module mismatch.
>>>>
>>>
>>> This is true if we mandate that embedded core image is *the*
>>> bootloader. But it can simply chainload core.img from $prefix which will
>>> guarantee that core.img always matches its modules. This would allow
>>> snapshot $prefix on grub update (or any grub change for that matter) to
>>> have fallback case if anything goes wrong.
>>>
>>> So this is similar to stage1.5, which also in principle could be
>>> installed once and never changed.
>>
>> My concern is on chainloading /fs/core.img appropriate ? It looked to
>> me would likely to fail because the installed lba address has been
>> hardcoded on it and would go back to the origin (stage1.5 per your
>> instance.)
>>
>
> No. LBA is updated in image that is embedded - it is not the same as
> what is kept in core.img. grub-bios-setup modifies it in memory before
> writing into embedding area.

OK. To be more verbose, the question of mine is that I'm uncertain if
we don't explicitly install to a partition using blocklists, the
/fs/core.img would still automatically get blocklists of itself
updated on it's own diskboot.S header or stay with default?

>
>> Or we need a way to control "boot" by jumping to it's loaded second sector.
>>
>>>
>>>> The configuration of master GRUB could have a list of all
>>>> snapshots/distros/w/e
>>>
>>> That exactly means "constantly modifying grub.cfg" on every
>>> snapshot creation, unless I'm mistaken? :)
>>
>> For snapshot creation is possible to not update master config if it's
>> using "dynamic" way of syntax to find the snapshots, as Vladmir has
>> been demonstrated in it's examples.
>>
>
> I replied specifically to "configuration could have a list of all
> snapshots". It is not dynamic :)

Oh. OK.

>
>>>
>>>>                       (alternatively they could be listed at runtime)
>>>> and source a grub.cfg from this snapshot (either directly or after user
>>>> has chosen the submenu) setting some variable to indicate the path to
>>>> snapshot. This slave grub.cfg would contain only entries.
>>>>
>>>
>>> This is a bit cumbersome today. Snapshot is done from master; and as
>>> far as I understand from this discussion, most people like to avoid
>>> special steps to prepare snapshot for booting. Which means that
>>> snapshot contains exactly the same entries as master. We need to either
>>> somehow filter entries, or change how grub configuration is generated.
>>>
>>> Something like
>>>
>>> grub-mkconfig --split
>>>
>>> which creates separate configuration file for each script
>>> in /etc/grub.d allowing master to selectively source (extract entries
>>> from) only parts of it. Or always generate two different configs -
>>> "master" and "slave".
>>
>> It seems like generating a slave config sourced by master in snapshot
>> is less work and intrusive to current grub-mkconfig.
>>
>> I like the idea of --split, and would be great to see the extending of
>> it with ability to update on specified configuration files. For
>> example, we don't forced to run os-prober or other scripts after new
>> kernel entries created.
>>
>>   --split --run=10_linux,20_linux_xen
>>
>
> Yes, that would be of course interesting extension in this case.

For your reference, there's reports on openSUSE that os-prober entries
should only be created once and not updated with every grub-mkconfig.
They wanted to update os-prober entries manually since no notification
about foreign os changes thus manual work is more reasonable (to
them). They wanted to avoid the unneeded calls form being taken place
on purpose that is ideally isolated. I think the problem could be
solved more generally by introducing this change.

>
>>>
>>> BTW 30_os-prober will happily fetch boot entries from every existing
>>> snapshot, presenting them all with identical names and "merging" all
>>> boot entries from all snapshots because it generates the same menu id
>>> (it includes only fs UUID, but no subvolume information).
>>
>> Are you suggesting to use os-prober, instead sourced by master config
>> directly for inclusion of boot entries of snapshots ? Or I've
>> misinterpreted it?
>>
>
> Oh, sorry for confusion. I simply meant that current 30_os-prober is
> buggy and needs fixing. I think I have an idea how.

You meant fixing 30_os-prober or getting btrfs snapshot booting to
work ? You already know I've been working on the latter. :)

Thanks,
Michael

>
>>>
>>>> Configuration like themes and timeouts would be set on master level.
>>>> In case of submenu it's possible to change resolution/theme/font and so
>>>> on but it seems like only waste of time.
>>>>
>>>
>>> Another possibility would be to a) snapshot /boot/grub together with the
>>> rest of / and b) chainload grub from snapshot. Then nothing needs
>>> changing at all (except some small magic to set BTRFS subvolume at
>>> runtime). The only problem here is to pass $prefix on chainloaded grub.
>>> For EFI we get this almost for free, but for other platforms I'm not
>>> sure. Could we pass this information as parameter when multiboot'ing
>>> core.img?
>>
>> If the concept could work that's definitely more feasible solution to
>> me as well, if the problems could be sorted out and fixed.
>>
>> Regards,
>> Michael
>>
>>>
>>>> Init scripts will take care of creating rw clone of snapshot if necessarry.
>>>>
>>>> In this scenario you don't care what the default volume is, and that's
>>>> the way it should be as single btrfs may contain several distributions
>>>> but only one can own the default.
>>>>
>>>
>>> _______________________________________________
>>> Grub-devel mailing list
>>> Grub-devel@gnu.org
>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: booting btrfs

[Andrey Borzenkov]

On Mon, Jan 13, 2014 at 1:12 PM, Michael Chang <mchang@suse.com> wrote:
>
> OK. To be more verbose, the question of mine is that I'm uncertain if
> we don't explicitly install to a partition using blocklists, the
> /fs/core.img would still automatically get blocklists of itself
> updated on it's own diskboot.S header or stay with default?
>

On i386-pc you boot core.img by using "multiboot". It bypasses
blocklists completely so it is irrelevant. On EFI you get standard EFI
binary that you can "chainload".

>>>>
>>>> BTW 30_os-prober will happily fetch boot entries from every existing
>>>> snapshot, presenting them all with identical names and "merging" all
>>>> boot entries from all snapshots because it generates the same menu id
>>>> (it includes only fs UUID, but no subvolume information).
>>>
>>> Are you suggesting to use os-prober, instead sourced by master config
>>> directly for inclusion of boot entries of snapshots ? Or I've
>>> misinterpreted it?
>>>
>>
>> Oh, sorry for confusion. I simply meant that current 30_os-prober is
>> buggy and needs fixing. I think I have an idea how.
>
> You meant fixing 30_os-prober or getting btrfs snapshot booting to
> work ?

I mean fixing 30_os-prober

Re: booting btrfs

[Michael Chang]

On Mon, Jan 13, 2014 at 05:08:49PM +0400, Andrey Borzenkov wrote:
> On Mon, Jan 13, 2014 at 1:12 PM, Michael Chang <mchang@suse.com> wrote:
> >
> > OK. To be more verbose, the question of mine is that I'm uncertain if
> > we don't explicitly install to a partition using blocklists, the
> > /fs/core.img would still automatically get blocklists of itself
> > updated on it's own diskboot.S header or stay with default?
> >
> 
> On i386-pc you boot core.img by using "multiboot". It bypasses
> blocklists completely so it is irrelevant. On EFI you get standard EFI
> binary that you can "chainload".

Now I understand it, thanks for the explaination.

Regards,
Michael

> 
> >>>>
> >>>> BTW 30_os-prober will happily fetch boot entries from every existing
> >>>> snapshot, presenting them all with identical names and "merging" all
> >>>> boot entries from all snapshots because it generates the same menu id
> >>>> (it includes only fs UUID, but no subvolume information).
> >>>
> >>> Are you suggesting to use os-prober, instead sourced by master config
> >>> directly for inclusion of boot entries of snapshots ? Or I've
> >>> misinterpreted it?
> >>>
> >>
> >> Oh, sorry for confusion. I simply meant that current 30_os-prober is
> >> buggy and needs fixing. I think I have an idea how.
> >
> > You meant fixing 30_os-prober or getting btrfs snapshot booting to
> > work ?
> 
> I mean fixing 30_os-prober
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: booting btrfs

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 261 bytes --]

On 13.01.2014 06:05, Michael Chang wrote:
>   --split --run=10_linux,20_linux_xen
You shouldn't assume that only files you installed are the ones
available. E.g. memtest adds its own config to /etc/grub.d. To disable
os-prober use GRUB_DISABLE_OS_PROBER


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 274 bytes --]

Re: booting btrfs

[Michael Chang]

On Tue, Jan 21, 2014 at 09:09:34AM +0100, Vladimir 'φ-coder/phcoder' Serbinenko wrote:
> On 13.01.2014 06:05, Michael Chang wrote:
> >   --split --run=10_linux,20_linux_xen
> You shouldn't assume that only files you installed are the ones
> available. E.g. memtest adds its own config to /etc/grub.d. To disable
> os-prober use GRUB_DISABLE_OS_PROBER
> 

My idea is that some scripts under /etc/grub.d/ could be rather
isolated with others and would benefit from this feature that could
update it's own changes transparent to others. OTOH bulk update seems
to be more safe and consistent, but sometimes it's cumbersome.

Thanks,
Michael
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel