GRUB writing to grubenv outside of kernel fs code

GRUB writing to grubenv outside of kernel fs code

[Chris Murphy]

https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_btrfs.3F

Does anyone know if this is still a problem on Btrfs if grubenv has
xattr +C set? In which case it should be possible to overwrite and
there's no csums that are invalidated.

I kinda wonder if in 2018 it's specious for, effectively out of tree
code, to be making modifications to the file system, outside of the
file system.


-- 
Chris Murphy

Re: GRUB writing to grubenv outside of kernel fs code

[Chris Murphy]

On Mon, Sep 17, 2018 at 9:44 PM, Chris Murphy <lists@colorremedies.com> wrote:
> https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_btrfs.3F
>
> Does anyone know if this is still a problem on Btrfs if grubenv has
> xattr +C set? In which case it should be possible to overwrite and
> there's no csums that are invalidated.
>
> I kinda wonder if in 2018 it's specious for, effectively out of tree
> code, to be making modifications to the file system, outside of the
> file system.

a. The bootloader code (pre-boot, not user space setup stuff) would
have to know how to read xattr and refuse to overwrite a grubenv
lacking xattr +C.
b. The bootloader code, would have to have sophisticated enough Btrfs
knowledge to know if the grubenv has been reflinked or snapshot,
because even if +C, it may not be valid to overwrite, and COW must
still happen, and there's no way the code in GRUB can do full blow COW
and update a bunch of metadata.

So answering my own question, this isn't workable. And it seems the
same problem for dm-thin.

There are a couple of reserve locations in Btrfs at the start and I
think after the first superblock, for bootloader embedding. Possibly
one or both of those areas could be used for this so it's outside the
file system. But other implementations are going to run into this
problem too.

I'm not sure how else to describe state. If NVRAM is sufficiently wear
resilient enough to have writes to it possibly every day, for every
boot, to indicate boot success/fail.

-- 
Chris Murphy

Re: GRUB writing to grubenv outside of kernel fs code

[Andrei Borzenkov]

18.09.2018 07:21, Chris Murphy пишет:
> On Mon, Sep 17, 2018 at 9:44 PM, Chris Murphy <lists@colorremedies.com> wrote:
>> https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_btrfs.3F
>>
>> Does anyone know if this is still a problem on Btrfs if grubenv has
>> xattr +C set? In which case it should be possible to overwrite and
>> there's no csums that are invalidated.
>>
>> I kinda wonder if in 2018 it's specious for, effectively out of tree
>> code, to be making modifications to the file system, outside of the
>> file system.
> 
> a. The bootloader code (pre-boot, not user space setup stuff) would
> have to know how to read xattr and refuse to overwrite a grubenv
> lacking xattr +C.
> b. The bootloader code, would have to have sophisticated enough Btrfs
> knowledge to know if the grubenv has been reflinked or snapshot,
> because even if +C, it may not be valid to overwrite, and COW must
> still happen, and there's no way the code in GRUB can do full blow COW
> and update a bunch of metadata.
> 
> So answering my own question, this isn't workable. And it seems the
> same problem for dm-thin.
> 
> There are a couple of reserve locations in Btrfs at the start and I
> think after the first superblock, for bootloader embedding. Possibly
> one or both of those areas could be used for this so it's outside the
> file system. But other implementations are going to run into this
> problem too.
> 

That's what SUSE grub2 version does - it includes patches to redirect
writes on btrfs to reserved area. I am not sure how it behaves in case
of multi-device btrfs though.

> I'm not sure how else to describe state. If NVRAM is sufficiently wear
> resilient enough to have writes to it possibly every day, for every
> boot, to indicate boot success/fail.
> 

Re: GRUB writing to grubenv outside of kernel fs code

[Chris Murphy]

On Mon, Sep 17, 2018 at 11:24 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> 18.09.2018 07:21, Chris Murphy пишет:
>> On Mon, Sep 17, 2018 at 9:44 PM, Chris Murphy <lists@colorremedies.com> wrote:
>>> https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_btrfs.3F
>>>
>>> Does anyone know if this is still a problem on Btrfs if grubenv has
>>> xattr +C set? In which case it should be possible to overwrite and
>>> there's no csums that are invalidated.
>>>
>>> I kinda wonder if in 2018 it's specious for, effectively out of tree
>>> code, to be making modifications to the file system, outside of the
>>> file system.
>>
>> a. The bootloader code (pre-boot, not user space setup stuff) would
>> have to know how to read xattr and refuse to overwrite a grubenv
>> lacking xattr +C.
>> b. The bootloader code, would have to have sophisticated enough Btrfs
>> knowledge to know if the grubenv has been reflinked or snapshot,
>> because even if +C, it may not be valid to overwrite, and COW must
>> still happen, and there's no way the code in GRUB can do full blow COW
>> and update a bunch of metadata.
>>
>> So answering my own question, this isn't workable. And it seems the
>> same problem for dm-thin.
>>
>> There are a couple of reserve locations in Btrfs at the start and I
>> think after the first superblock, for bootloader embedding. Possibly
>> one or both of those areas could be used for this so it's outside the
>> file system. But other implementations are going to run into this
>> problem too.
>>
>
> That's what SUSE grub2 version does - it includes patches to redirect
> writes on btrfs to reserved area. I am not sure how it behaves in case
> of multi-device btrfs though.

The patches aren't upstream yet? Will they be?

They redirect writes to grubenv specifically? Or do they use the
reserved areas like a hidden and fixed location for what grubenv would
contain?

I guess the user space grub-editenv could write to grubenv, which even
if COW, GRUB can pick up that change. But GRUB itself writes its
changes to a reserved area.

Hmmm. Complicated.

-- 
Chris Murphy

Re: GRUB writing to grubenv outside of kernel fs code

[Goffredo Baroncelli]

On 18/09/2018 06.21, Chris Murphy wrote:
> b. The bootloader code, would have to have sophisticated enough Btrfs
> knowledge to know if the grubenv has been reflinked or snapshot,
> because even if +C, it may not be valid to overwrite, and COW must
> still happen, and there's no way the code in GRUB can do full blow COW
> and update a bunch of metadata.

And what if GRUB ignore the possibility of COWing and overwrite the data ? Is it a so big problem that the data is changed in all the snapshots ? 
It would be interested if the same problem happens for a swap file.....


BR
G.Baroncelli

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: GRUB writing to grubenv outside of kernel fs code

[Andrei Borzenkov]

18.09.2018 08:37, Chris Murphy пишет:
> On Mon, Sep 17, 2018 at 11:24 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>> 18.09.2018 07:21, Chris Murphy пишет:
>>> On Mon, Sep 17, 2018 at 9:44 PM, Chris Murphy <lists@colorremedies.com> wrote:
>>>> https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_btrfs.3F
>>>>
>>>> Does anyone know if this is still a problem on Btrfs if grubenv has
>>>> xattr +C set? In which case it should be possible to overwrite and
>>>> there's no csums that are invalidated.
>>>>
>>>> I kinda wonder if in 2018 it's specious for, effectively out of tree
>>>> code, to be making modifications to the file system, outside of the
>>>> file system.
>>>
>>> a. The bootloader code (pre-boot, not user space setup stuff) would
>>> have to know how to read xattr and refuse to overwrite a grubenv
>>> lacking xattr +C.
>>> b. The bootloader code, would have to have sophisticated enough Btrfs
>>> knowledge to know if the grubenv has been reflinked or snapshot,
>>> because even if +C, it may not be valid to overwrite, and COW must
>>> still happen, and there's no way the code in GRUB can do full blow COW
>>> and update a bunch of metadata.
>>>
>>> So answering my own question, this isn't workable. And it seems the
>>> same problem for dm-thin.
>>>
>>> There are a couple of reserve locations in Btrfs at the start and I
>>> think after the first superblock, for bootloader embedding. Possibly
>>> one or both of those areas could be used for this so it's outside the
>>> file system. But other implementations are going to run into this
>>> problem too.
>>>
>>
>> That's what SUSE grub2 version does - it includes patches to redirect
>> writes on btrfs to reserved area. I am not sure how it behaves in case
>> of multi-device btrfs though.
> 
> The patches aren't upstream yet? Will they be?
> 

I do not know. Personally I think much easier is to make grub location
independent of /boot, allowing grub be installed in separate partition.
This automatically covers all other cases (like MD, LVM etc).

> They redirect writes to grubenv specifically? Or do they use the
> reserved areas like a hidden and fixed location for what grubenv would
> contain?
> 
> I guess the user space grub-editenv could write to grubenv, which even
> if COW, GRUB can pick up that change. But GRUB itself writes its
> changes to a reserved area.
> 
> Hmmm. Complicated.
> 

Re: GRUB writing to grubenv outside of kernel fs code

[Austin S. Hemmelgarn]

On 2018-09-18 14:16, Andrei Borzenkov wrote:
> 18.09.2018 08:37, Chris Murphy пишет:
>> On Mon, Sep 17, 2018 at 11:24 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>>> 18.09.2018 07:21, Chris Murphy пишет:
>>>> On Mon, Sep 17, 2018 at 9:44 PM, Chris Murphy <lists@colorremedies.com> wrote:
>>>>> https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_btrfs.3F
>>>>>
>>>>> Does anyone know if this is still a problem on Btrfs if grubenv has
>>>>> xattr +C set? In which case it should be possible to overwrite and
>>>>> there's no csums that are invalidated.
>>>>>
>>>>> I kinda wonder if in 2018 it's specious for, effectively out of tree
>>>>> code, to be making modifications to the file system, outside of the
>>>>> file system.
>>>>
>>>> a. The bootloader code (pre-boot, not user space setup stuff) would
>>>> have to know how to read xattr and refuse to overwrite a grubenv
>>>> lacking xattr +C.
>>>> b. The bootloader code, would have to have sophisticated enough Btrfs
>>>> knowledge to know if the grubenv has been reflinked or snapshot,
>>>> because even if +C, it may not be valid to overwrite, and COW must
>>>> still happen, and there's no way the code in GRUB can do full blow COW
>>>> and update a bunch of metadata.
>>>>
>>>> So answering my own question, this isn't workable. And it seems the
>>>> same problem for dm-thin.
>>>>
>>>> There are a couple of reserve locations in Btrfs at the start and I
>>>> think after the first superblock, for bootloader embedding. Possibly
>>>> one or both of those areas could be used for this so it's outside the
>>>> file system. But other implementations are going to run into this
>>>> problem too.
>>>>
>>>
>>> That's what SUSE grub2 version does - it includes patches to redirect
>>> writes on btrfs to reserved area. I am not sure how it behaves in case
>>> of multi-device btrfs though.
>>
>> The patches aren't upstream yet? Will they be?
>>
> 
> I do not know. Personally I think much easier is to make grub location
> independent of /boot, allowing grub be installed in separate partition.
> This automatically covers all other cases (like MD, LVM etc).
It actually is independent of /boot already.  I've got it running just 
fine on my laptop off of the EFI system partition (which is independent 
of my /boot partition), and thus have no issues with handling of the 
grubenv file.  The problem is that all the big distros assume you want 
it in /boot, so they have no option for putting it anywhere else.

Actually installing it elsewhere is not hard though, you just pass 
`--boot-directory=/wherever` to the `grub-install` script and turn off 
your distributions automatic reinstall mechanism so it doesn't get 
screwed up by the package manager when the GRUB package gets updated. 
You can also make `/boot/grub` a symbolic link pointing to the real GRUB 
directory, so that you don't have to pass any extra options to tools 
like grub-reboot or grub-set-default.

Re: GRUB writing to grubenv outside of kernel fs code

[Andrei Borzenkov]

18.09.2018 21:25, Austin S. Hemmelgarn пишет:
> On 2018-09-18 14:16, Andrei Borzenkov wrote:
>> 18.09.2018 08:37, Chris Murphy пишет:
>>> On Mon, Sep 17, 2018 at 11:24 PM, Andrei Borzenkov
>>> <arvidjaar@gmail.com> wrote:
>>>> 18.09.2018 07:21, Chris Murphy пишет:
>>>>> On Mon, Sep 17, 2018 at 9:44 PM, Chris Murphy
>>>>> <lists@colorremedies.com> wrote:
...
>>>>>
>>>>> There are a couple of reserve locations in Btrfs at the start and I
>>>>> think after the first superblock, for bootloader embedding. Possibly
>>>>> one or both of those areas could be used for this so it's outside the
>>>>> file system. But other implementations are going to run into this
>>>>> problem too.
>>>>>
>>>>
>>>> That's what SUSE grub2 version does - it includes patches to redirect
>>>> writes on btrfs to reserved area. I am not sure how it behaves in case
>>>> of multi-device btrfs though.
>>>
>>> The patches aren't upstream yet? Will they be?
>>>
>>
>> I do not know. Personally I think much easier is to make grub location
>> independent of /boot, allowing grub be installed in separate partition.
>> This automatically covers all other cases (like MD, LVM etc).
> It actually is independent of /boot already.  I've got it running just
> fine on my laptop off of the EFI system partition (which is independent
> of my /boot partition), and thus have no issues with handling of the
> grubenv file.  The problem is that all the big distros assume you want
> it in /boot, so they have no option for putting it anywhere else.
> 

This requires more than just explicit --boot-directory. With current
monolithic configuration file listing all available kernels this file
cannot be in the same location, it must be together with kernels (think
about rollback to snapshot with completely different content). Or some
different, more flexible configuration is needed.

As is now grub silently assumes everything is under /boot. This turned
out to be oversimplified.

> Actually installing it elsewhere is not hard though, you just pass
> `--boot-directory=/wherever` to the `grub-install` script and turn off
> your distributions automatic reinstall mechanism so it doesn't get
> screwed up by the package manager when the GRUB package gets updated.
> You can also make `/boot/grub` a symbolic link pointing to the real GRUB
> directory, so that you don't have to pass any extra options to tools
> like grub-reboot or grub-set-default.

Re: GRUB writing to grubenv outside of kernel fs code

[Chris Murphy]

On Tue, Sep 18, 2018 at 11:15 AM, Goffredo Baroncelli
<kreijack@libero.it> wrote:
> On 18/09/2018 06.21, Chris Murphy wrote:
>> b. The bootloader code, would have to have sophisticated enough Btrfs
>> knowledge to know if the grubenv has been reflinked or snapshot,
>> because even if +C, it may not be valid to overwrite, and COW must
>> still happen, and there's no way the code in GRUB can do full blow COW
>> and update a bunch of metadata.
>
> And what if GRUB ignore the possibility of COWing and overwrite the data ? Is it a so big problem that the data is changed in all the snapshots ?
> It would be interested if the same problem happens for a swap file.....

I think it's an abomination :-) It totally perverts the idea of
reflinks and snapshots and blurs the line between domains. Is it a
user file or not and are these user space commands or not and are they
reliable or do they have exceptions?

I have a boot subvolume mounted at /boot, and this boot subvolume gets
snapshot, and if GRUB can overwrite grubenv, it overwrites the
purported GRUB state information in every one of those boots, going
back maybe months, even when these are read only subvolumes.

I think it's a problem, and near as I can tell it'll be a problem for
all kinds of complex storage. I don't see how the bootloader itself
can do an overwrite onto raid5 or raid6. That's certainly supported by
GRUB for reading, but is the bootloader overwrite of gruvenv going to
recompute parity and write to multiple devices? Eek!


-- 
Chris Murphy

Re: GRUB writing to grubenv outside of kernel fs code

[Chris Murphy]

On Tue, Sep 18, 2018 at 12:16 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> 18.09.2018 08:37, Chris Murphy пишет:

>> The patches aren't upstream yet? Will they be?
>>
>
> I do not know. Personally I think much easier is to make grub location
> independent of /boot, allowing grub be installed in separate partition.
> This automatically covers all other cases (like MD, LVM etc).

The only case where I'm aware of this happens is Fedora on UEFI where
they write grubenv and grub.cfg on the FAT ESP. I'm pretty sure
upstream expects grubenv and grub.cfg at /boot/grub and I haven't ever
seen it elsewhere (except Fedora on UEFI).

I'm not sure this is much easier. Yet another volume that would be
persistently mounted? Where? A nested mount at /boot/grub? I'm not
liking that at all. Even Windows and macOS have saner and simpler to
understand booting methods than this.


-- 
Chris Murphy

Re: GRUB writing to grubenv outside of kernel fs code

[Chris Murphy]

On Tue, Sep 18, 2018 at 12:25 PM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:

> It actually is independent of /boot already.  I've got it running just fine
> on my laptop off of the EFI system partition (which is independent of my
> /boot partition), and thus have no issues with handling of the grubenv file.
> The problem is that all the big distros assume you want it in /boot, so they
> have no option for putting it anywhere else.
>
> Actually installing it elsewhere is not hard though, you just pass
> `--boot-directory=/wherever` to the `grub-install` script and turn off your
> distributions automatic reinstall mechanism so it doesn't get screwed up by
> the package manager when the GRUB package gets updated. You can also make
> `/boot/grub` a symbolic link pointing to the real GRUB directory, so that
> you don't have to pass any extra options to tools like grub-reboot or
> grub-set-default.

This is how Fedora builds their signed grubx64.efi to behave. But you
cannot ever run grub-install on a Secure Boot enabled computer, or you
now have to learn all about signing your own binaries. I don't even
like doing that, let alone saner users.

So for those distros that support Secure Boot, in practice you're
stuck with the behavior of their prebuilt GRUB binary that goes on the
ESP.


-- 
Chris Murphy

Re: GRUB writing to grubenv outside of kernel fs code

[Andrei Borzenkov]

18.09.2018 21:57, Chris Murphy пишет:
> On Tue, Sep 18, 2018 at 12:16 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>> 18.09.2018 08:37, Chris Murphy пишет:
> 
>>> The patches aren't upstream yet? Will they be?
>>>
>>
>> I do not know. Personally I think much easier is to make grub location
>> independent of /boot, allowing grub be installed in separate partition.
>> This automatically covers all other cases (like MD, LVM etc).
> 
> The only case where I'm aware of this happens is Fedora on UEFI where
> they write grubenv and grub.cfg on the FAT ESP. I'm pretty sure
> upstream expects grubenv and grub.cfg at /boot/grub and I haven't ever
> seen it elsewhere (except Fedora on UEFI).
> 
> I'm not sure this is much easier. Yet another volume that would be
> persistently mounted? Where? A nested mount at /boot/grub? I'm not
> liking that at all. Even Windows and macOS have saner and simpler to
> understand booting methods than this.
> 
> 
That's exactly what Windows ended up with - separate boot volume with
bootloader related files.

Re: GRUB writing to grubenv outside of kernel fs code

[Goffredo Baroncelli]

On 18/09/2018 20.52, Chris Murphy wrote:
> On Tue, Sep 18, 2018 at 11:15 AM, Goffredo Baroncelli
> <kreijack@libero.it> wrote:
>> On 18/09/2018 06.21, Chris Murphy wrote:
>>> b. The bootloader code, would have to have sophisticated enough Btrfs
>>> knowledge to know if the grubenv has been reflinked or snapshot,
>>> because even if +C, it may not be valid to overwrite, and COW must
>>> still happen, and there's no way the code in GRUB can do full blow COW
>>> and update a bunch of metadata.
>>
>> And what if GRUB ignore the possibility of COWing and overwrite the data ? Is it a so big problem that the data is changed in all the snapshots ?
>> It would be interested if the same problem happens for a swap file.....
> 
> I think it's an abomination :-) It totally perverts the idea of
> reflinks and snapshots and blurs the line between domains. 

:-)

> Is it a
> user file or not and are these user space commands or not and are they
> reliable or do they have exceptions?

On this statement I fully agree, on the one below a bit less
> 
> I have a boot subvolume mounted at /boot, and this boot subvolume gets
> snapshot, and if GRUB can overwrite grubenv, it overwrites the
> purported GRUB state information in every one of those boots, going
> back maybe months, even when these are read only subvolumes.

Also the 'suse' behavior have the same issue: storing the data somewhere in the storage reserved area suffers of the same problem. We should be realistic, without implement a full btrfs filesystem engine, it is near impossible to have a grubenv file visible by the filesystem and snapshot-able.


> 
> I think it's a problem, and near as I can tell it'll be a problem for
> all kinds of complex storage. I don't see how the bootloader itself
> can do an overwrite onto raid5 or raid6. 


> That's certainly supported by GRUB for reading
Not yet, I am working on that [1]

> but is the bootloader overwrite of gruvenv going to
> recompute parity and write to multiple devices? Eek!

Recompute the parity should not be a big deal. Updating all the (b)trees would be a too complex goal.
> 
> 

[1] http://lists.gnu.org/archive/html/grub-devel/2018-06/msg00064.html
-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: GRUB writing to grubenv outside of kernel fs code

[Austin S. Hemmelgarn]

On 2018-09-18 14:38, Andrei Borzenkov wrote:
> 18.09.2018 21:25, Austin S. Hemmelgarn пишет:
>> On 2018-09-18 14:16, Andrei Borzenkov wrote:
>>> 18.09.2018 08:37, Chris Murphy пишет:
>>>> On Mon, Sep 17, 2018 at 11:24 PM, Andrei Borzenkov
>>>> <arvidjaar@gmail.com> wrote:
>>>>> 18.09.2018 07:21, Chris Murphy пишет:
>>>>>> On Mon, Sep 17, 2018 at 9:44 PM, Chris Murphy
>>>>>> <lists@colorremedies.com> wrote:
> ...
>>>>>>
>>>>>> There are a couple of reserve locations in Btrfs at the start and I
>>>>>> think after the first superblock, for bootloader embedding. Possibly
>>>>>> one or both of those areas could be used for this so it's outside the
>>>>>> file system. But other implementations are going to run into this
>>>>>> problem too.
>>>>>>
>>>>>
>>>>> That's what SUSE grub2 version does - it includes patches to redirect
>>>>> writes on btrfs to reserved area. I am not sure how it behaves in case
>>>>> of multi-device btrfs though.
>>>>
>>>> The patches aren't upstream yet? Will they be?
>>>>
>>>
>>> I do not know. Personally I think much easier is to make grub location
>>> independent of /boot, allowing grub be installed in separate partition.
>>> This automatically covers all other cases (like MD, LVM etc).
>> It actually is independent of /boot already.  I've got it running just
>> fine on my laptop off of the EFI system partition (which is independent
>> of my /boot partition), and thus have no issues with handling of the
>> grubenv file.  The problem is that all the big distros assume you want
>> it in /boot, so they have no option for putting it anywhere else.
>>
> 
> This requires more than just explicit --boot-directory. With current
> monolithic configuration file listing all available kernels this file
> cannot be in the same location, it must be together with kernels (think
> about rollback to snapshot with completely different content). Or some
> different, more flexible configuration is needed.
Uh, no, it doesn't need to be with the kernels.  Fedora stores it on the 
ESP separate from the kernels (which are still on the boot partition) if 
you use Secure Boot, and I'm doing the same (without secure boot) 
without issue.  You do have to explicitly set the `root` variable 
correctly in the config though to get it to work though, and the default 
upstream 'easy configuration' arrangement does not do this consistently. 
  It's not too hard to hack in though, and it's positively trivial if 
you just write your own configuration files by hand like I do (no, I'm 
not crazy, the default configuration generator just produces a 
brobdingnagian monstrosity of a config that has tons of stuff I don't 
need and makes invalid assumptions about how I want things invoked, and 
the config syntax is actually not that hard).
> 
> As is now grub silently assumes everything is under /boot. This turned
> out to be oversimplified.
No, it assumes everything is under whatever you told GRUB to set the 
default value of the `prefix` variable to when you built the GRUB image, 
which is automatically set to the path you pass to `--boot-directory` 
when you use grub-install.  This persists until you explicitly set that 
variable to a different location, or change the `root` variable (but 
GRUB still uses `prefix` for module look-ups if you just change the 
`root` variable).

Re: GRUB writing to grubenv outside of kernel fs code

[Austin S. Hemmelgarn]

On 2018-09-18 14:57, Chris Murphy wrote:
> On Tue, Sep 18, 2018 at 12:16 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>> 18.09.2018 08:37, Chris Murphy пишет:
> 
>>> The patches aren't upstream yet? Will they be?
>>>
>>
>> I do not know. Personally I think much easier is to make grub location
>> independent of /boot, allowing grub be installed in separate partition.
>> This automatically covers all other cases (like MD, LVM etc).
> 
> The only case where I'm aware of this happens is Fedora on UEFI where
> they write grubenv and grub.cfg on the FAT ESP. I'm pretty sure
> upstream expects grubenv and grub.cfg at /boot/grub and I haven't ever
> seen it elsewhere (except Fedora on UEFI).
> 
> I'm not sure this is much easier. Yet another volume that would be
> persistently mounted? Where? A nested mount at /boot/grub? I'm not
> liking that at all. Even Windows and macOS have saner and simpler to
> understand booting methods than this.
On this front maybe, but Windows' boot sequence is insane in it's own 
way (fun fact, if you have the Windows 8/8.1/10 boot-loader set up to 
multi-boot and want it to boot to something other than the default, it 
has to essentially _reboot the machine_ to actually boot that 
alternative entry).

Re: GRUB writing to grubenv outside of kernel fs code

[Austin S. Hemmelgarn]

On 2018-09-18 15:00, Chris Murphy wrote:
> On Tue, Sep 18, 2018 at 12:25 PM, Austin S. Hemmelgarn
> <ahferroin7@gmail.com> wrote:
> 
>> It actually is independent of /boot already.  I've got it running just fine
>> on my laptop off of the EFI system partition (which is independent of my
>> /boot partition), and thus have no issues with handling of the grubenv file.
>> The problem is that all the big distros assume you want it in /boot, so they
>> have no option for putting it anywhere else.
>>
>> Actually installing it elsewhere is not hard though, you just pass
>> `--boot-directory=/wherever` to the `grub-install` script and turn off your
>> distributions automatic reinstall mechanism so it doesn't get screwed up by
>> the package manager when the GRUB package gets updated. You can also make
>> `/boot/grub` a symbolic link pointing to the real GRUB directory, so that
>> you don't have to pass any extra options to tools like grub-reboot or
>> grub-set-default.
> 
> This is how Fedora builds their signed grubx64.efi to behave. But you
> cannot ever run grub-install on a Secure Boot enabled computer, or you
> now have to learn all about signing your own binaries. I don't even
> like doing that, let alone saner users.
> 
> So for those distros that support Secure Boot, in practice you're
> stuck with the behavior of their prebuilt GRUB binary that goes on the
> ESP.
Agreed, but that avoids the issues we're talking about here completely 
because the grubenv file ends up on the ESP too.

Re: GRUB writing to grubenv outside of kernel fs code

[Chris Murphy]

On Tue, Sep 18, 2018 at 1:01 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> 18.09.2018 21:57, Chris Murphy пишет:
>> On Tue, Sep 18, 2018 at 12:16 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>>> 18.09.2018 08:37, Chris Murphy пишет:
>>
>>>> The patches aren't upstream yet? Will they be?
>>>>
>>>
>>> I do not know. Personally I think much easier is to make grub location
>>> independent of /boot, allowing grub be installed in separate partition.
>>> This automatically covers all other cases (like MD, LVM etc).
>>
>> The only case where I'm aware of this happens is Fedora on UEFI where
>> they write grubenv and grub.cfg on the FAT ESP. I'm pretty sure
>> upstream expects grubenv and grub.cfg at /boot/grub and I haven't ever
>> seen it elsewhere (except Fedora on UEFI).
>>
>> I'm not sure this is much easier. Yet another volume that would be
>> persistently mounted? Where? A nested mount at /boot/grub? I'm not
>> liking that at all. Even Windows and macOS have saner and simpler to
>> understand booting methods than this.
>>
>>
> That's exactly what Windows ended up with - separate boot volume with
> bootloader related files.

The OEM installer will absolutely install to a single partition. If
you point it to a blank drive on BIOS it will preferentially create a
"system" volume that's used for booting. But it's not mandatory. On
UEFI, it doesn't create a "system" volume, just "recovery" is ~500M
and "reserved" 16M. The reserved partition is blank unless you've done
some resizing on the main volume. The recovery volume contains
Winre.wim which is used for doing resets. If you blow away that
partition, you can still boot, but you can't do resets.

-- 
Chris Murphy

Re: GRUB writing to grubenv outside of kernel fs code

[Chris Murphy]

On Tue, Sep 18, 2018 at 1:11 PM, Goffredo Baroncelli <kreijack@inwind.it> wrote:


>> I think it's a problem, and near as I can tell it'll be a problem for
>> all kinds of complex storage. I don't see how the bootloader itself
>> can do an overwrite onto raid5 or raid6.
>
>
>> That's certainly supported by GRUB for reading
> Not yet, I am working on that [1]


Sorry! I meant mdadm raid56. It definitely can read that format for
some time and even degraded! It's pretty cool. But I see no way that
it's sane to have the bootloader write to such a volume.

I've run into some issue where grub2-mkconfig and grubby, can change
the grub.cfg, and then do a really fast reboot without cleanly
unmounting the volume - and what happens? Can't boot. The bootloader
can't do log replay so it doesn't see the new grub.cfg at all. If all
you do is mount the volume and unmount, log replay happens, the fs
metadata is all fixed up just fine, and now the bootloader can see it.
This same problem can happen with the kernel and initramfs
installations.

(Hilariously the reason why this can happen is because of a process
exempting itself from being forcibly killed by systemd *against* the
documented advice of systemd devs that you should only do this for
processes not on rootfs; but as a consequence of this process doing
the wrong thing, systemd at reboot time ends up doing an unclean
unmount and reboot because it won't kill the kill exempt process.)

So *already* we have file systems that are becoming too complicated
for the bootloader to reliably read, because they cannot do journal
relay, let alone have any chance of modifying (nor would I want them
to do this). So yeah I'm, very rapidly becoming opposed to grubenv on
anything but super simple volumes like maybe ext4 without a journal
(extents are nice); or even perhaps GRUB should just implement its own
damn file system and we give it its own partition - similar to BIOS
Boot - but probably a little bigger


>
>> but is the bootloader overwrite of gruvenv going to
>> recompute parity and write to multiple devices? Eek!
>
> Recompute the parity should not be a big deal. Updating all the (b)trees would be a too complex goal.

I think it's just asking for trouble. Sometimes the best answer ends
up being no, no and definitely no.

-- 
Chris Murphy

Re: GRUB writing to grubenv outside of kernel fs code

[Chris Murphy]

On Mon, Sep 17, 2018 at 9:44 PM, Chris Murphy <lists@colorremedies.com> wrote:
> https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_btrfs.3F
>
> Does anyone know if this is still a problem on Btrfs if grubenv has
> xattr +C set? In which case it should be possible to overwrite and
> there's no csums that are invalidated.

I'm wrong.

$ sudo grub2-editenv --verbose grubenv create
[sudo] password for chris:
[chris@f29h ~]$ ll
-rw-r--r--. 1 root  root     1024 Sep 18 13:37 grubenv
[chris@f29h ~]$ stat -f grubenv
  File: "grubenv"
    ID: ac9ba8ecdce5b017 Namelen: 255     Type: btrfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 46661632   Free: 37479747   Available: 37422535
Inodes: Total: 0          Free: 0
[chris@f29h ~]$ sudo filefrag -v grubenv
Filesystem type is: 9123683e
File size of grubenv is 1024 (1 block of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..    4095:          0..      4095:   4096:
last,not_aligned,inline,eof
grubenv: 1 extent found
[chris@f29h ~]$

So it's an inline extent, which means nocow doesn't apply. It's
metadata so it *must* be COW. And any overwrite would trigger a
metadata checksum error.

First I'd argue it should refuse to create the file on Btrfs. But if
it does create grubenv, instead it should know that on Btrfs it must
redirect it to the appropriate btrfs reserved area (no idea how this
works) rather than to a file.



-- 
Chris Murphy

Re: GRUB writing to grubenv outside of kernel fs code

[Duncan]

Chris Murphy posted on Tue, 18 Sep 2018 13:34:14 -0600 as excerpted:

> I've run into some issue where grub2-mkconfig and grubby, can change the
> grub.cfg, and then do a really fast reboot without cleanly unmounting
> the volume - and what happens? Can't boot. The bootloader can't do log
> replay so it doesn't see the new grub.cfg at all. If all you do is mount
> the volume and unmount, log replay happens, the fs metadata is all fixed
> up just fine, and now the bootloader can see it.
> This same problem can happen with the kernel and initramfs
> installations.
> 
> (Hilariously the reason why this can happen is because of a process
> exempting itself from being forcibly killed by systemd *against* the
> documented advice of systemd devs that you should only do this for
> processes not on rootfs; but as a consequence of this process doing the
> wrong thing, systemd at reboot time ends up doing an unclean unmount and
> reboot because it won't kill the kill exempt process.)

That's... interesting!

FWIW here I use grub2, but as many admins I'm quite comfortable with 
bash, and the high-level grub2 config mechanisms simply didn't let me do 
what I needed to do.  So I had to learn the lower-level grub bash-like 
scripting language to do what I wanted to do, and I even go so far as to 
install-mask some of the higher level stuff so it doesn't get installed 
at all, and thus can't somehow run and screw up my config.

So I edit my grub scripts (and grubenv) much like I'd edit any other 
system script (and its separate config file where I have them) I might 
need to update, then save my work, and with both a bios-boot partition 
setup for grub-core and an entirely separate /boot that's not routinely 
mounted unless I'm updating it, I normally unmount it when I'm done, 
before I actually reboot.

So I've never had systemd interfere.

(And of course I have backups.  In fact, on my main personal system, with 
both the working root and its primary backup being btrfs pair-device 
raid1 on separate devices, I have four physical ssds installed, with a 
bios-boot partition with grub installed and a separate dedicated (btrfs 
dup mode) /boot on each of all four, so I have a working grub and /boot 
and three backups, each of which I can point the bios at and have tested 
separately as bootable.  So if upgrading grub or anything on /boot goes 
wrong I find that out testing the working copy, and boot one of the 
backups to resolve the problem before eventually upgrading all three 
backups after the working copy upgrade is well tested.)

> So *already* we have file systems that are becoming too complicated for
> the bootloader to reliably read, because they cannot do journal relay,
> let alone have any chance of modifying (nor would I want them to do
> this). So yeah I'm, very rapidly becoming opposed to grubenv on anything
> but super simple volumes like maybe ext4 without a journal (extents are
> nice); or even perhaps GRUB should just implement its own damn file
> system and we give it its own partition - similar to BIOS Boot - but
> probably a little bigger

You realize that solution is already standardized as EFI and its standard 
FAT filesystem, right?

=:^)

>>> but is the bootloader overwrite of gruvenv going to recompute parity
>>> and write to multiple devices? Eek!
>>
>> Recompute the parity should not be a big deal. Updating all the
>> (b)trees would be a too complex goal.
> 
> I think it's just asking for trouble. Sometimes the best answer ends up
> being no, no and definitely no.

Agreed.  I actually /like/ the fact that at the grub prompt I can rely on 
everything being read-only, and if that SuSE patch to put grubenv in the 
reserved space and make it writable gets upstreamed, I really hope 
there's a build-time configure option to disable the feature, because IMO 
grub doesn't /need/ to save state at that point, and allowing it to do so 
is effectively needlessly playing a risky Russian Roulette game with my 
storage devices.  Were it actually needed that'd be different, but it's 
not needed, so any risk is too much risk.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: GRUB writing to grubenv outside of kernel fs code

[Andrei Borzenkov]

18.09.2018 22:11, Austin S. Hemmelgarn пишет:
> On 2018-09-18 14:38, Andrei Borzenkov wrote:
>> 18.09.2018 21:25, Austin S. Hemmelgarn пишет:
>>> On 2018-09-18 14:16, Andrei Borzenkov wrote:
>>>> 18.09.2018 08:37, Chris Murphy пишет:
>>>>> On Mon, Sep 17, 2018 at 11:24 PM, Andrei Borzenkov
>>>>> <arvidjaar@gmail.com> wrote:
>>>>>> 18.09.2018 07:21, Chris Murphy пишет:
>>>>>>> On Mon, Sep 17, 2018 at 9:44 PM, Chris Murphy
>>>>>>> <lists@colorremedies.com> wrote:
>> ...
>>>>>>>
>>>>>>> There are a couple of reserve locations in Btrfs at the start and I
>>>>>>> think after the first superblock, for bootloader embedding. Possibly
>>>>>>> one or both of those areas could be used for this so it's outside
>>>>>>> the
>>>>>>> file system. But other implementations are going to run into this
>>>>>>> problem too.
>>>>>>>
>>>>>>
>>>>>> That's what SUSE grub2 version does - it includes patches to redirect
>>>>>> writes on btrfs to reserved area. I am not sure how it behaves in
>>>>>> case
>>>>>> of multi-device btrfs though.
>>>>>
>>>>> The patches aren't upstream yet? Will they be?
>>>>>
>>>>
>>>> I do not know. Personally I think much easier is to make grub location
>>>> independent of /boot, allowing grub be installed in separate partition.
>>>> This automatically covers all other cases (like MD, LVM etc).
>>> It actually is independent of /boot already.  I've got it running just
>>> fine on my laptop off of the EFI system partition (which is independent
>>> of my /boot partition), and thus have no issues with handling of the
>>> grubenv file.  The problem is that all the big distros assume you want
>>> it in /boot, so they have no option for putting it anywhere else.
>>>
>>
>> This requires more than just explicit --boot-directory. With current
>> monolithic configuration file listing all available kernels this file
>> cannot be in the same location, it must be together with kernels (think
>> about rollback to snapshot with completely different content). Or some
>> different, more flexible configuration is needed.
> Uh, no, it doesn't need to be with the kernels.

It does not need to be *with* kernels but it must match content of /boot
if you want to allow booting from multiple subvolumes (or even
partitions) using the same grub instance. The most obvious case is
snapper rollback used by SUSE. You still have single instance of
bootloader, but multiple subvolumes with different kernels. So somehow
bootloader must know which kernels to offer depending on which subvolume
you select.

Re: GRUB writing to grubenv outside of kernel fs code

[Goffredo Baroncelli]

On 18/09/2018 19.15, Goffredo Baroncelli wrote:
>> b. The bootloader code, would have to have sophisticated enough Btrfs
>> knowledge to know if the grubenv has been reflinked or snapshot,
>> because even if +C, it may not be valid to overwrite, and COW must
>> still happen, and there's no way the code in GRUB can do full blow COW
>> and update a bunch of metadata.

> And what if GRUB ignore the possibility of COWing and overwrite the data ? Is it a so big problem that the data is changed in all the snapshots ? 
> It would be interested if the same problem happens for a swap file.....

I gave a look to the Sandoval's patches about implementing swap on BTRFS. This patch set
prevents the subvolume containing the swapfile to be snapshot-ted (and the file to be balanced and so on...); what if we would add the same constraint to the grubenv file ?


BR
G.Baroncelli
-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: GRUB writing to grubenv outside of kernel fs code

[Austin S. Hemmelgarn]

On 2018-09-19 15:08, Goffredo Baroncelli wrote:
> On 18/09/2018 19.15, Goffredo Baroncelli wrote:
>>> b. The bootloader code, would have to have sophisticated enough Btrfs
>>> knowledge to know if the grubenv has been reflinked or snapshot,
>>> because even if +C, it may not be valid to overwrite, and COW must
>>> still happen, and there's no way the code in GRUB can do full blow COW
>>> and update a bunch of metadata.
> 
>> And what if GRUB ignore the possibility of COWing and overwrite the data ? Is it a so big problem that the data is changed in all the snapshots ?
>> It would be interested if the same problem happens for a swap file.....
> 
> I gave a look to the Sandoval's patches about implementing swap on BTRFS. This patch set
> prevents the subvolume containing the swapfile to be snapshot-ted (and the file to be balanced and so on...); what if we would add the same constraint to the grubenv file ?
We would need to have a generalized mechanism of doing this then, 
because there's no way in hell a patch special-casing a single filename 
is going to make it into mainline.

Whatever mechanism is used, it should also:

* Force the file to not be inlined in metadata.
* Enforce the file having the NOCOW attribute being set.