RFC: A partition for grubenv, etc.

RFC: A partition for grubenv, etc.

[Chris Murphy]

Hi,

It's not possible for GRUB pre-boot environment to write to grubenv
when it's on Btrfs, ZFS, LVM, mdadm raid, or LUKS. Also, at least XFS
upstream is super skeptical of anything except kernel code making any
kind of modification inside the file system region, and I suspect it's
the same concern with ext4 developers too. While there are file system
specific locations for bootloader usage, they're all different and
quite small. XFS has none. ext4 has 512 bytes. Btrfs has maybe 1 or 2
MiB, ZFS (?), mdadm (?) and LVM (?).

Proposal: A new partition type for MBR and GPT, functionally a
replacement for the BIOS Boot partition, but it would be a partition
owned by the bootloader for whatever it wants to use it for. It'd be
up to the bootloader to figure out how to segment it for bootloader
and environmental portions. We definitely need both MBR and GPT
partition types, it should be a partition exclusively reserved for the
bootloader. This effectively deprecates the use of the MBR gap, and
BIOS Boot partition types, and further it deprecates the use of file
systems (all of them, for consistency sake) for the use of grubenv.

Variation: Keep BIOS Boot and repurpose it to include grubenv, while
also specifying an MBR type code for its equivalent.


Use case: For example, Fedora has a "hidden grub menu" feature where
by a variable in grubenv is reset (written to) by grub pre-boot. And
then a systemd unit changes that variable to indicate a successful
boot once some time and/or tests have been met. If grubenv indicates
successful boot, the next boot's grub menu is hidden. If it wasn't
successful the next boot's grub menu is displayed. It's only possible
to achieve this with some reliable bidirectional way of communicating
between the preboot and booted environments, which is the point of
grubenv, but it can't work much of the time due to the above
limitations.

-- 
Chris Murphy

Re: RFC: A partition for grubenv, etc.

[Konrad Rzeszutek Wilk]

On Tue, May 25, 2021 at 04:58:23PM -0600, Chris Murphy wrote:
> Hi,
> 
> It's not possible for GRUB pre-boot environment to write to grubenv
> when it's on Btrfs, ZFS, LVM, mdadm raid, or LUKS. Also, at least XFS
> upstream is super skeptical of anything except kernel code making any
> kind of modification inside the file system region, and I suspect it's
> the same concern with ext4 developers too. While there are file system
> specific locations for bootloader usage, they're all different and
> quite small. XFS has none. ext4 has 512 bytes. Btrfs has maybe 1 or 2
> MiB, ZFS (?), mdadm (?) and LVM (?).

Aren't most distro setups using EFI which in effect means GRUB has to
deal most of the time with FAT32 file system? Does that work?

> 
> Proposal: A new partition type for MBR and GPT, functionally a
> replacement for the BIOS Boot partition, but it would be a partition
> owned by the bootloader for whatever it wants to use it for. It'd be
> up to the bootloader to figure out how to segment it for bootloader
> and environmental portions. We definitely need both MBR and GPT
> partition types, it should be a partition exclusively reserved for the
> bootloader. This effectively deprecates the use of the MBR gap, and
> BIOS Boot partition types, and further it deprecates the use of file
> systems (all of them, for consistency sake) for the use of grubenv.
> 
> Variation: Keep BIOS Boot and repurpose it to include grubenv, while
> also specifying an MBR type code for its equivalent.
> 
> 
> Use case: For example, Fedora has a "hidden grub menu" feature where
> by a variable in grubenv is reset (written to) by grub pre-boot. And
> then a systemd unit changes that variable to indicate a successful
> boot once some time and/or tests have been met. If grubenv indicates
> successful boot, the next boot's grub menu is hidden. If it wasn't
> successful the next boot's grub menu is displayed. It's only possible
> to achieve this with some reliable bidirectional way of communicating
> between the preboot and booted environments, which is the point of
> grubenv, but it can't work much of the time due to the above
> limitations.
> 
> -- 
> Chris Murphy
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: RFC: A partition for grubenv, etc.

[Toomas Soome]

[-- Attachment #1: Type: text/plain, Size: 3205 bytes --]



> On 26. May 2021, at 11:07, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> On Tue, May 25, 2021 at 04:58:23PM -0600, Chris Murphy wrote:
>> Hi,
>> 
>> It's not possible for GRUB pre-boot environment to write to grubenv
>> when it's on Btrfs, ZFS, LVM, mdadm raid, or LUKS. Also, at least XFS
>> upstream is super skeptical of anything except kernel code making any
>> kind of modification inside the file system region, and I suspect it's
>> the same concern with ext4 developers too. While there are file system
>> specific locations for bootloader usage, they're all different and
>> quite small. XFS has none. ext4 has 512 bytes. Btrfs has maybe 1 or 2
>> MiB, ZFS (?), mdadm (?) and LVM (?).
> 
> Aren't most distro setups using EFI which in effect means GRUB has to
> deal most of the time with FAT32 file system? Does that work?


Also OpenZFS does have space and API to access area called “bootenv”. Data is stored there in nvlist structures, there is userland library (libzfsbootenv). From boot loader point of view, zfs module already does have ability to read nvlist packed data, to implement ability to read this data is not that hard. Updates are a bit harder because nvlist encoding needs to be implemented too.

wrapping this as storage based backend callbacks and implementing generic front end is a nice excercise;)

rgds,
toomas

> 
>> 
>> Proposal: A new partition type for MBR and GPT, functionally a
>> replacement for the BIOS Boot partition, but it would be a partition
>> owned by the bootloader for whatever it wants to use it for. It'd be
>> up to the bootloader to figure out how to segment it for bootloader
>> and environmental portions. We definitely need both MBR and GPT
>> partition types, it should be a partition exclusively reserved for the
>> bootloader. This effectively deprecates the use of the MBR gap, and
>> BIOS Boot partition types, and further it deprecates the use of file
>> systems (all of them, for consistency sake) for the use of grubenv.
>> 
>> Variation: Keep BIOS Boot and repurpose it to include grubenv, while
>> also specifying an MBR type code for its equivalent.
>> 
>> 
>> Use case: For example, Fedora has a "hidden grub menu" feature where
>> by a variable in grubenv is reset (written to) by grub pre-boot. And
>> then a systemd unit changes that variable to indicate a successful
>> boot once some time and/or tests have been met. If grubenv indicates
>> successful boot, the next boot's grub menu is hidden. If it wasn't
>> successful the next boot's grub menu is displayed. It's only possible
>> to achieve this with some reliable bidirectional way of communicating
>> between the preboot and booted environments, which is the point of
>> grubenv, but it can't work much of the time due to the above
>> limitations.
>> 
>> -- 
>> Chris Murphy
>> 
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel


[-- Attachment #2: Type: text/html, Size: 5074 bytes --]

Re: RFC: A partition for grubenv, etc.

[Michael Chang]

On Tue, May 25, 2021 at 04:58:23PM -0600, Chris Murphy wrote:
> Hi,
> 
> It's not possible for GRUB pre-boot environment to write to grubenv
> when it's on Btrfs, ZFS, LVM, mdadm raid, or LUKS. Also, at least XFS
> upstream is super skeptical of anything except kernel code making any
> kind of modification inside the file system region, and I suspect it's
> the same concern with ext4 developers too. While there are file system
> specific locations for bootloader usage, they're all different and
> quite small. XFS has none. ext4 has 512 bytes. Btrfs has maybe 1 or 2
> MiB, ZFS (?), mdadm (?) and LVM (?).
> 
> Proposal: A new partition type for MBR and GPT, functionally a
> replacement for the BIOS Boot partition, but it would be a partition
> owned by the bootloader for whatever it wants to use it for. It'd be
> up to the bootloader to figure out how to segment it for bootloader
> and environmental portions. We definitely need both MBR and GPT
> partition types, it should be a partition exclusively reserved for the
> bootloader. This effectively deprecates the use of the MBR gap, and
> BIOS Boot partition types, and further it deprecates the use of file
> systems (all of them, for consistency sake) for the use of grubenv.
> 
> Variation: Keep BIOS Boot and repurpose it to include grubenv, while
> also specifying an MBR type code for its equivalent.

The grubenv should be treated as property per grub.cfg, not per image.
This means that the grubenv partition may have to be divided into slots
to store settings for each and every grub.cfg, and then additional data
to record the change (add/removal) and identification - which sounds
like to work out a filesystem.

So the question is why not use filesystem writable by grub for the
proposed grubenv partition ? For UEFI we already have one, the EFI
System Partition.

Thanks,
Michael

> 
> 
> Use case: For example, Fedora has a "hidden grub menu" feature where
> by a variable in grubenv is reset (written to) by grub pre-boot. And
> then a systemd unit changes that variable to indicate a successful
> boot once some time and/or tests have been met. If grubenv indicates
> successful boot, the next boot's grub menu is hidden. If it wasn't
> successful the next boot's grub menu is displayed. It's only possible
> to achieve this with some reliable bidirectional way of communicating
> between the preboot and booted environments, which is the point of
> grubenv, but it can't work much of the time due to the above
> limitations.
> 
> -- 
> Chris Murphy
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: RFC: A partition for grubenv, etc.

[Chris Murphy]

On Wed, May 26, 2021 at 2:08 AM Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>
> On Tue, May 25, 2021 at 04:58:23PM -0600, Chris Murphy wrote:
> > Hi,
> >
> > It's not possible for GRUB pre-boot environment to write to grubenv
> > when it's on Btrfs, ZFS, LVM, mdadm raid, or LUKS. Also, at least XFS
> > upstream is super skeptical of anything except kernel code making any
> > kind of modification inside the file system region, and I suspect it's
> > the same concern with ext4 developers too. While there are file system
> > specific locations for bootloader usage, they're all different and
> > quite small. XFS has none. ext4 has 512 bytes. Btrfs has maybe 1 or 2
> > MiB, ZFS (?), mdadm (?) and LVM (?).
>
> Aren't most distro setups using EFI which in effect means GRUB has to
> deal most of the time with FAT32 file system? Does that work?

I'm not sure what most distros do, but grub-install creates a
grubx64.efi that expects to find /boot/grub/grubenv and
/boot/grub/grub.cfg and at least on Fedora we've unified UEFI and BIOS
GRUBs so that they use /boot/grub2/ for configuration files, not least
of which is for user sanity, but for future cloud images that are dual
bootloader thus a single image can boot UEFI or BIOS VMs, with a
single consistent configuration location.


-- 
Chris Murphy

Re: RFC: A partition for grubenv, etc.

[Chris Murphy]

On Wed, May 26, 2021 at 3:17 AM Michael Chang via Grub-devel
<grub-devel@gnu.org> wrote:
>
> On Tue, May 25, 2021 at 04:58:23PM -0600, Chris Murphy wrote:
> > Hi,
> >
> > It's not possible for GRUB pre-boot environment to write to grubenv
> > when it's on Btrfs, ZFS, LVM, mdadm raid, or LUKS. Also, at least XFS
> > upstream is super skeptical of anything except kernel code making any
> > kind of modification inside the file system region, and I suspect it's
> > the same concern with ext4 developers too. While there are file system
> > specific locations for bootloader usage, they're all different and
> > quite small. XFS has none. ext4 has 512 bytes. Btrfs has maybe 1 or 2
> > MiB, ZFS (?), mdadm (?) and LVM (?).
> >
> > Proposal: A new partition type for MBR and GPT, functionally a
> > replacement for the BIOS Boot partition, but it would be a partition
> > owned by the bootloader for whatever it wants to use it for. It'd be
> > up to the bootloader to figure out how to segment it for bootloader
> > and environmental portions. We definitely need both MBR and GPT
> > partition types, it should be a partition exclusively reserved for the
> > bootloader. This effectively deprecates the use of the MBR gap, and
> > BIOS Boot partition types, and further it deprecates the use of file
> > systems (all of them, for consistency sake) for the use of grubenv.
> >
> > Variation: Keep BIOS Boot and repurpose it to include grubenv, while
> > also specifying an MBR type code for its equivalent.
>
> The grubenv should be treated as property per grub.cfg, not per image.
> This means that the grubenv partition may have to be divided into slots
> to store settings for each and every grub.cfg, and then additional data
> to record the change (add/removal) and identification - which sounds
> like to work out a filesystem.
>
> So the question is why not use filesystem writable by grub for the
> proposed grubenv partition ? For UEFI we already have one, the EFI
> System Partition.

Upstream GRUB doesn't use this location. Distros who create and sign
their own grubx64.efi (et al) OSLoaders may assume the grub.cfg and
grubenv are in the same path as the *efi OSLoader, but I can't say
whether this is common. Certainly it leaves out sufficiently common
cases including all of BIOS which still represents most virtual
machines even if not most new hardware, and suggests abandoning the
entire "/boot on $nameyourfilesystem" approach. Fedora no longer keeps
the primary grub.cfg on the ESP, it's moved to /boot with a static
grub.cfg on the ESP that fowards to the real one. I thought (open)SUSE
did this too. *shrug*

So is the next era going to be we recommend /boot on FAT?


-- 
Chris Murphy

Re: RFC: A partition for grubenv, etc.

[Michael Chang]

On Wed, May 26, 2021 at 08:36:05PM -0600, Chris Murphy wrote:
> On Wed, May 26, 2021 at 3:17 AM Michael Chang via Grub-devel
> <grub-devel@gnu.org> wrote:
> >
> > On Tue, May 25, 2021 at 04:58:23PM -0600, Chris Murphy wrote:
> > > Hi,
> > >
> > > It's not possible for GRUB pre-boot environment to write to grubenv
> > > when it's on Btrfs, ZFS, LVM, mdadm raid, or LUKS. Also, at least XFS
> > > upstream is super skeptical of anything except kernel code making any
> > > kind of modification inside the file system region, and I suspect it's
> > > the same concern with ext4 developers too. While there are file system
> > > specific locations for bootloader usage, they're all different and
> > > quite small. XFS has none. ext4 has 512 bytes. Btrfs has maybe 1 or 2
> > > MiB, ZFS (?), mdadm (?) and LVM (?).
> > >
> > > Proposal: A new partition type for MBR and GPT, functionally a
> > > replacement for the BIOS Boot partition, but it would be a partition
> > > owned by the bootloader for whatever it wants to use it for. It'd be
> > > up to the bootloader to figure out how to segment it for bootloader
> > > and environmental portions. We definitely need both MBR and GPT
> > > partition types, it should be a partition exclusively reserved for the
> > > bootloader. This effectively deprecates the use of the MBR gap, and
> > > BIOS Boot partition types, and further it deprecates the use of file
> > > systems (all of them, for consistency sake) for the use of grubenv.
> > >
> > > Variation: Keep BIOS Boot and repurpose it to include grubenv, while
> > > also specifying an MBR type code for its equivalent.
> >
> > The grubenv should be treated as property per grub.cfg, not per image.
> > This means that the grubenv partition may have to be divided into slots
> > to store settings for each and every grub.cfg, and then additional data
> > to record the change (add/removal) and identification - which sounds
> > like to work out a filesystem.
> >
> > So the question is why not use filesystem writable by grub for the
> > proposed grubenv partition ? For UEFI we already have one, the EFI
> > System Partition.
> 
> Upstream GRUB doesn't use this location. Distros who create and sign
> their own grubx64.efi (et al) OSLoaders may assume the grub.cfg and
> grubenv are in the same path as the *efi OSLoader, but I can't say
> whether this is common. Certainly it leaves out sufficiently common
> cases including all of BIOS which still represents most virtual
> machines even if not most new hardware, and suggests abandoning the
> entire "/boot on $nameyourfilesystem" approach. Fedora no longer keeps
> the primary grub.cfg on the ESP, it's moved to /boot with a static
> grub.cfg on the ESP that fowards to the real one. I thought (open)SUSE
> did this too. *shrug*
> 
> So is the next era going to be we recommend /boot on FAT?

No. I meant a new partition type only for grubenv files and keep
everything else as is. The filesystem can be extX or FAT whichever is
writable by grub.  For efi, we probably don't need to define new
partition, as we can reuse the efi system partition.

It is easier to handle in script as we only need to point to the new
writable location if available, the concern to admin is that we also
have to define mount point for linux (eg /boot/grubenv).

Thanks,
Michael

> 
> 
> -- 
> Chris Murphy
> 

Re: RFC: A partition for grubenv, etc.

[Luiz Angelo Daros de Luca]

This was already discussed in this ML a couple of times.

It is not uncommon for some FS to have an unused header space. This
happens for btrfs and SUSE patches grub2 to use it:
https://build.opensuse.org/package/view_file/openSUSE:Factory/grub2/grub2-grubenv-in-btrfs-header.patch?expand=1.
It is a "workaround", specific for a single FS that needs to be
redesigned for each FS where grub cannot write to.
Its complexity grows with each new FS.

I think that grub should offer an alternative way to store grubenv
independently from FS in use. Grub already requires some spare space
for stage 1.5 and grub could use the remaining space after that for
grubenv. For BIOS with DOS partitions, it uses the blocks between
MBR and the first partition (sector 63 or 2048 for recent disks). For
BIOS with GPT, grub already requires a grub-bios partition (normally
way bigger than needed). And for UEFI, as it was already mentioned,
there is an EFI partition. I know I'm not considering anything
but x86 but it is where this issue happens. A new dedicated partition
might be overkill and only mostly only useful for GPT, where there
are already partitions dedicated for booting (grub bios boot or EFI).

My proposal would be an additional source for grubenv. SUSE still uses
the FS /boot/grub2/grubenv to store a
env_block variable (as well as other variables) that, if present, will
point to where an additional grubenv should be. My proposal is quite
the same,
but not limited to BTRFS.

grub2-mkconfig could be opportunistic, using only /boot/grub2/grubenv
when it is writable by grub, /boot/EFI when present, btrfs first
blocks or stage1.5 data when everything
else fails. Grub and userland tools would only need to check if the
"normal" grubenv does define a "$grubenv_device", "$grubenv_block" or
"$grubenv_include".

As it was mentioned, grubenv should be grub.cfg specific, not image
specific. As /boot/grub2/grubenv is still in use, each grub.conf would
still point to a different grubenv file that would include
a different grubenv_{device,block,include}. grubenv could contain an
uuid variable to help grub2-mkconfig and grub iteratively look for a
matching grub_block or allocate a new slot.
Something like this:

/boot/grub2/grubenv
grubenv_device=(hd0,msdos4)
grubenv_block=0 # for btrfs
grubenv_maxblock=64 # for btrfs 0x10000 (btrfs header location) /
0x400 (grubenv size)
# grubenv_block=1234 for stage1.5 area where 1234 is the stage1.5 size in blocks
grubenv_uuid=3dfd5d2a-7314-47dc-ba5a-e02aa2b00749

(hd0,msdos4)@0
grubenv_uuid=3dfd5d2a-7314-47dc-ba5a-e02aa2b00749
next_boot=2

With a simple grub.cfg scriptlet (using not valid syntax):

if [ "$grubenv_include" ]; then
   # for EFI
   load_env $grubenv_include
fi
if [ "$grubenv_device" ] && [ "$grubenv_block" ]; then
  for n in {1..$grubenv_maxblock}; do
     if [ "$grubenv_uuid" ]; then
       expected_uuid=$grubenv_uuid
       load_env -d $grubend_device -b $grubenv_block grubenv_uuid
       if [ "$grubenv_uuid" != "$expected_uuid" ]; then
         grubenv_uuid=$expected_uuid
         grubenv_block++
         continue
       fi
     fi
     # I hope it checks the header...
     if ! load_env -d $grubend_device -b $grubenv_block; then
       # not found or failed
       grubenv_device=
       grubenv_block=
     fi
     break
  fi
...
if [ "$grubenv_device" ] && [ "$grubenv_block" ]; then
   save_env -d $grubend_device -b $grubenv_block grubenv_uuid
fi

The problem is that the location of every single /boot/grub2/grubenv
is not deterministic. It would be hard to safely garbage-collect
unused grubenv blocks.
I think that whenever grub2-editenv writes a block, it should
rightshift the available area ($grubenv_maxblock) and add or move the
modified grubenv to position 0
(the hottest block). Eventually all unused grubenv blocks will be
dropped during the rightshift.

I'm not trying to solve every single corner case but I'm suggesting
changes that would empower the OS to solve the issue (requiring a
partition with writable FS for grubenv).
For most cases where grubenv is not writable today, it could
automatically solve the issue. For others where multiple complex
grub.conf/grubenv are in use,
it would give tools to the advanced user to manually solve the issue.
I'm not worried about advanced users because they can today create a
dedicated fat32 partition with
manually defined grubenv path. I'm thinking about distributions and
the general user that marginally knows what grub is.

Regards,

---
     Luiz Angelo Daros de Luca
            luizluca@gmail.com

Re: RFC: A partition for grubenv, etc.

[Chris Murphy]

On Thu, May 27, 2021 at 2:59 AM Michael Chang <mchang@suse.com> wrote:
>
> On Wed, May 26, 2021 at 08:36:05PM -0600, Chris Murphy wrote:
> > So is the next era going to be we recommend /boot on FAT?
>
> No. I meant a new partition type only for grubenv files and keep
> everything else as is. The filesystem can be extX or FAT whichever is
> writable by grub.  For efi, we probably don't need to define new
> partition, as we can reuse the efi system partition.

Or use the same partition for consistency instead of yet another exception?

But also, GRUB strictly speaking doesn't write to any file system.
It's overwriting a sector or two, no file system metadata is updated,
hence the problem.

> It is easier to handle in script as we only need to point to the new
> writable location if available, the concern to admin is that we also
> have to define mount point for linux (eg /boot/grubenv).

That'd be three partitions needed by BIOS GRUB: BIOS Boot, grubenv,
and /boot. Plus ~440 bytes on LBA 0, quasi-partition. Seems like a
lot.

What about a FAT formatted file system at /boot/grub and everything
just lives there? Deprecate BIOS Boot, use blocklists instead. Grubenv
partition proposal isn't needed, it can just stay where it is,
unchanged and can be reliably written to. And whether UEFI or BIOS,
grub looks to /boot/grub/ for all modules and configuration files?



-- 
Chris Murphy

Re: RFC: A partition for grubenv, etc.

[Chris Murphy]

On Thu, May 27, 2021 at 12:49 PM Luiz Angelo Daros de Luca
<luizluca@gmail.com> wrote:
>
> My proposal would be an additional source for grubenv. SUSE still uses
> the FS /boot/grub2/grubenv to store a
> env_block variable (as well as other variables) that, if present, will
> point to where an additional grubenv should be. My proposal is quite
> the same,
> but not limited to BTRFS.
>
> grub2-mkconfig could be opportunistic, using only /boot/grub2/grubenv
> when it is writable by grub, /boot/EFI when present, btrfs first
> blocks or stage1.5 data when everything
> else fails. Grub and userland tools would only need to check if the
> "normal" grubenv does define a "$grubenv_device", "$grubenv_block" or
> "$grubenv_include".

I think it's better than the original proposal, and better than a file
system mounted at /boot/grubenv only for that purpose. I'd like to see
more consistency, fewer exceptions.

And that also means the old argument about the MBR gap. GRUB should
have its own partition here too.



Chris Murphy

Re: RFC: A partition for grubenv, etc.

[Michael Chang]

On Thu, May 27, 2021 at 03:49:14PM -0300, Luiz Angelo Daros de Luca wrote:
> This was already discussed in this ML a couple of times.
> 
> It is not uncommon for some FS to have an unused header space. This
> happens for btrfs and SUSE patches grub2 to use it:
> https://build.opensuse.org/package/view_file/openSUSE:Factory/grub2/grub2-grubenv-in-btrfs-header.patch?expand=1.
> It is a "workaround", specific for a single FS that needs to be
> redesigned for each FS where grub cannot write to.
> Its complexity grows with each new FS.
> 
> I think that grub should offer an alternative way to store grubenv
> independently from FS in use. Grub already requires some spare space
> for stage 1.5 and grub could use the remaining space after that for
> grubenv. For BIOS with DOS partitions, it uses the blocks between
> MBR and the first partition (sector 63 or 2048 for recent disks). For
> BIOS with GPT, grub already requires a grub-bios partition (normally
> way bigger than needed). And for UEFI, as it was already mentioned,
> there is an EFI partition. I know I'm not considering anything
> but x86 but it is where this issue happens. A new dedicated partition
> might be overkill and only mostly only useful for GPT, where there
> are already partitions dedicated for booting (grub bios boot or EFI).
> 
> My proposal would be an additional source for grubenv. SUSE still uses
> the FS /boot/grub2/grubenv to store a
> env_block variable (as well as other variables) that, if present, will
> point to where an additional grubenv should be. My proposal is quite
> the same,
> but not limited to BTRFS.
> 
> grub2-mkconfig could be opportunistic, using only /boot/grub2/grubenv
> when it is writable by grub, /boot/EFI when present, btrfs first
> blocks or stage1.5 data when everything
> else fails. Grub and userland tools would only need to check if the
> "normal" grubenv does define a "$grubenv_device", "$grubenv_block" or
> "$grubenv_include".
> 
> As it was mentioned, grubenv should be grub.cfg specific, not image
> specific. As /boot/grub2/grubenv is still in use, each grub.conf would
> still point to a different grubenv file that would include
> a different grubenv_{device,block,include}. grubenv could contain an
> uuid variable to help grub2-mkconfig and grub iteratively look for a
> matching grub_block or allocate a new slot.
> Something like this:
> 
> /boot/grub2/grubenv
> grubenv_device=(hd0,msdos4)
> grubenv_block=0 # for btrfs
> grubenv_maxblock=64 # for btrfs 0x10000 (btrfs header location) /
> 0x400 (grubenv size)
> # grubenv_block=1234 for stage1.5 area where 1234 is the stage1.5 size in blocks
> grubenv_uuid=3dfd5d2a-7314-47dc-ba5a-e02aa2b00749
> 
> (hd0,msdos4)@0
> grubenv_uuid=3dfd5d2a-7314-47dc-ba5a-e02aa2b00749
> next_boot=2
> 
> With a simple grub.cfg scriptlet (using not valid syntax):
> 
> if [ "$grubenv_include" ]; then
>    # for EFI
>    load_env $grubenv_include
> fi
> if [ "$grubenv_device" ] && [ "$grubenv_block" ]; then
>   for n in {1..$grubenv_maxblock}; do
>      if [ "$grubenv_uuid" ]; then
>        expected_uuid=$grubenv_uuid
>        load_env -d $grubend_device -b $grubenv_block grubenv_uuid
>        if [ "$grubenv_uuid" != "$expected_uuid" ]; then
>          grubenv_uuid=$expected_uuid
>          grubenv_block++
>          continue
>        fi
>      fi
>      # I hope it checks the header...
>      if ! load_env -d $grubend_device -b $grubenv_block; then
>        # not found or failed
>        grubenv_device=
>        grubenv_block=
>      fi
>      break
>   fi
> ...
> if [ "$grubenv_device" ] && [ "$grubenv_block" ]; then
>    save_env -d $grubend_device -b $grubenv_block grubenv_uuid
> fi
> 
> The problem is that the location of every single /boot/grub2/grubenv
> is not deterministic. It would be hard to safely garbage-collect
> unused grubenv blocks.
> I think that whenever grub2-editenv writes a block, it should
> rightshift the available area ($grubenv_maxblock) and add or move the
> modified grubenv to position 0
> (the hottest block). Eventually all unused grubenv blocks will be
> dropped during the rightshift.
> 
> I'm not trying to solve every single corner case but I'm suggesting
> changes that would empower the OS to solve the issue (requiring a
> partition with writable FS for grubenv).
> For most cases where grubenv is not writable today, it could
> automatically solve the issue. For others where multiple complex
> grub.conf/grubenv are in use,
> it would give tools to the advanced user to manually solve the issue.
> I'm not worried about advanced users because they can today create a
> dedicated fat32 partition with
> manually defined grubenv path. I'm thinking about distributions and
> the general user that marginally knows what grub is.

This looks good to me, and the best thing to have is that this opens the
possibility to do in place upgrade and then you will have the writable
grubenv. :)

However I'd like to add one thing to the list. As long as this will have
to deal with very sensible block areas. The tool (grub-editenv?) should
be enhanced to understand the usable blocks of the filesystem or
partition table in use, and therefore prohitbiting those who tries to
set out any settings of writing out of bound. It is important to make
sure no data corruptio will happen before usability.

Thanks,
Michael

> 
> Regards,
> 
> ---
>      Luiz Angelo Daros de Luca
>             luizluca@gmail.com
> 

Re: RFC: A partition for grubenv, etc.

[Daniel Kiper]

CC-ing a few folks who may be interested in this thread...

First of all, sorry for late reply but I am really busy. I hope I will
be able to reply in more timely manner starting from now.

On Thu, May 27, 2021 at 04:08:45PM -0600, Chris Murphy wrote:
> On Thu, May 27, 2021 at 2:59 AM Michael Chang <mchang@suse.com> wrote:
> >
> > On Wed, May 26, 2021 at 08:36:05PM -0600, Chris Murphy wrote:
> > > So is the next era going to be we recommend /boot on FAT?
> >
> > No. I meant a new partition type only for grubenv files and keep
> > everything else as is. The filesystem can be extX or FAT whichever is
> > writable by grub.  For efi, we probably don't need to define new
> > partition, as we can reuse the efi system partition.
>
> Or use the same partition for consistency instead of yet another exception?
>
> But also, GRUB strictly speaking doesn't write to any file system.
> It's overwriting a sector or two, no file system metadata is updated,
> hence the problem.
>
> > It is easier to handle in script as we only need to point to the new
> > writable location if available, the concern to admin is that we also
> > have to define mount point for linux (eg /boot/grubenv).
>
> That'd be three partitions needed by BIOS GRUB: BIOS Boot, grubenv,
> and /boot. Plus ~440 bytes on LBA 0, quasi-partition. Seems like a
> lot.
>
> What about a FAT formatted file system at /boot/grub and everything
> just lives there? Deprecate BIOS Boot, use blocklists instead. Grubenv
> partition proposal isn't needed, it can just stay where it is,
> unchanged and can be reliably written to. And whether UEFI or BIOS,
> grub looks to /boot/grub/ for all modules and configuration files?

At this point I am not fully convinced the /boot/grub should be on
a separate filesystem. Though after going through this thread it seems
to me we should add full write support to the GRUB for a simple file
system. The FAT looks like a good candidate. Does not it? This way we
will be able to support writable grubenv without any limitations and have
flexibility to add other writable files without bigger hassle if needed.

Additionally, it is worth mentioning the [1] work which adds a save_env
redirection layer. It allows us to choose target for grubenv. However,
if we want make it more generic then we could redirect any writes to any
target which can be potentially useful.

Last but not least, I think we do not need to change anything in current
partitions usage. In particular it seems to me the BIOS Boot Partition
usage should stay as is.

Daniel

[1] https://lists.gnu.org/archive/html/grub-devel/2020-03/msg00126.html

Re: RFC: A partition for grubenv, etc.

[Chris Murphy]

On Wed, Jul 28, 2021 at 6:23 AM Daniel Kiper <dkiper@net-space.pl> wrote:
>
> At this point I am not fully convinced the /boot/grub should be on
> a separate filesystem. Though after going through this thread it seems
> to me we should add full write support to the GRUB for a simple file
> system. The FAT looks like a good candidate. Does not it? This way we
> will be able to support writable grubenv without any limitations and have
> flexibility to add other writable files without bigger hassle if needed.

On UEFI, the firmware's FAT driver should be write capable, but I
don't know if GRUB can make use of it directly rather than using
GRUB's own FAT driver?

On non-UEFI systems, we still need GRUB to have its own read-write
capable driver. I think parity between BIOS and UEFI is important, but
I'd also say if there's a faster/easier/more maintainable path with
some other file system for other bootloaders, that'd be even more
preferred. I don't know of such a file system. But UDF is in the same
ballpark, long since supported in the kernel, supports optical,
removable, hard drive use cases.

> Additionally, it is worth mentioning the [1] work which adds a save_env
> redirection layer. It allows us to choose target for grubenv. However,
> if we want make it more generic then we could redirect any writes to any
> target which can be potentially useful.

Btrfs has two fairly large areas for exclusive bootloader usage,
~64KiB from partition start to the superblock. And ~1-2 MiB after the
first superblock and before the first block group. But no other file
system I'm aware of has such a large reservation for the bootloader.
XFS has none. ext4 might have 512 bytes. I think it's valid to ask for
an explicit reservation for bootloader usage, but this is just more of
the same issue as grubenv and BIOS Boot. There is no file system in
such a location so in effect it's up to GRUB to internally implement
something, in which case it might as well just have a dedicated
partition for grubfs or whatever it'd be.

> Last but not least, I think we do not need to change anything in current
> partitions usage. In particular it seems to me the BIOS Boot Partition
> usage should stay as is.

I don't mind just retasking BIOS Boot into a  general "bootloader"
partition that bootloaders can use. Or even make it explicitly a GRUB
bootloader partition type GUID. It's better if each bootloader has its
own? Originally BIOS Boot wasn't "owned" by GRUB, any bootloader can
use it, but insofar as I'm aware it's the only bootloader that uses
this partition type GUID.





--
Chris Murphy

Re: RFC: A partition for grubenv, etc.

[Daniel Kiper]

On Thu, Sep 23, 2021 at 08:59:14PM -0600, Chris Murphy wrote:
> On Wed, Jul 28, 2021 at 6:23 AM Daniel Kiper <dkiper@net-space.pl> wrote:
> >
> > At this point I am not fully convinced the /boot/grub should be on
> > a separate filesystem. Though after going through this thread it seems
> > to me we should add full write support to the GRUB for a simple file
> > system. The FAT looks like a good candidate. Does not it? This way we
> > will be able to support writable grubenv without any limitations and have
> > flexibility to add other writable files without bigger hassle if needed.
>
> On UEFI, the firmware's FAT driver should be write capable, but I
> don't know if GRUB can make use of it directly rather than using
> GRUB's own FAT driver?

IIRC it uses own driver. I am not sure at this point we can piggyback
on the UEFI FAT driver. If not we have to be sure both drivers,
the GRUB and UEFI ones, do not conflict with each other in some ways.

> On non-UEFI systems, we still need GRUB to have its own read-write
> capable driver. I think parity between BIOS and UEFI is important, but
> I'd also say if there's a faster/easier/more maintainable path with
> some other file system for other bootloaders, that'd be even more
> preferred. I don't know of such a file system. But UDF is in the same
> ballpark, long since supported in the kernel, supports optical,
> removable, hard drive use cases.

We can consider UDF too. Though FAT is much more common and I think we
should take it into account.

> > Additionally, it is worth mentioning the [1] work which adds a save_env
> > redirection layer. It allows us to choose target for grubenv. However,
> > if we want make it more generic then we could redirect any writes to any
> > target which can be potentially useful.
>
> Btrfs has two fairly large areas for exclusive bootloader usage,
> ~64KiB from partition start to the superblock. And ~1-2 MiB after the
> first superblock and before the first block group. But no other file
> system I'm aware of has such a large reservation for the bootloader.
> XFS has none. ext4 might have 512 bytes. I think it's valid to ask for
> an explicit reservation for bootloader usage, but this is just more of
> the same issue as grubenv and BIOS Boot. There is no file system in
> such a location so in effect it's up to GRUB to internally implement
> something, in which case it might as well just have a dedicated
> partition for grubfs or whatever it'd be.

I think we should use well know simple filesystem to write/store the
GRUB data. Coming up with own structure to store the data requires
maintaining this custom structure and write e.g. tools which checks and
fix their integrity. If we choose filesystem solution we have these
tools out of the box and they are maintained by others. So, (potentially)
less work for the GRUB folks... :-)

> > Last but not least, I think we do not need to change anything in current
> > partitions usage. In particular it seems to me the BIOS Boot Partition
> > usage should stay as is.
>
> I don't mind just retasking BIOS Boot into a  general "bootloader"
> partition that bootloaders can use. Or even make it explicitly a GRUB
> bootloader partition type GUID. It's better if each bootloader has its
> own? Originally BIOS Boot wasn't "owned" by GRUB, any bootloader can
> use it, but insofar as I'm aware it's the only bootloader that uses
> this partition type GUID.

I think the BIOS boot partition should stay as is and we should
introduce separate partition type called e.g. Bootloader data partition.
Then it should be formatted as FAT and have relevant directory structure,
e.g. the GRUB should store its data in /bootloader/grub directory.

Daniel