x86_64 32-bit EFI mixed mode boot broken

x86_64 32-bit EFI mixed mode boot broken

[Clayton Craft]

[-- Attachment #1: Type: text/plain, Size: 1284 bytes --]

I've been chasing a problem with 32-bit EFI mixed mode booting on two different
(x86_64) Intel Bay Trail platforms, where the system reboots or hangs seemingly
very early somewhere before or after loading the kernel. I've not been able to
get any output from the kernel or stub over efifb when the issue happens[0], and
do not have serial console access on these systems. 

v6.8 fails for me, and presumably so does everything back to v6.2. v6.1 is able
to boot OK on these platforms with mixed mode, and it looks like there are a lot
of changes from 6.1..6.2 for EFI/mixed mode booting. I did managed to bisect the
issue to: 

        commit e2ab9eab324cdf240de89741e4a1aa79919f0196
        Author: Ard Biesheuvel <ardb@kernel.org>
        Date:   Tue Nov 22 17:10:02 2022 +0100
        
            x86/boot/compressed: Move 32-bit entrypoint code into .text section

However I'm not sure how to proceed from here, or if my bisect is all that
useful since the commit seems to be in the middle of a bunch of changes I do not
understand. I've been using systemd-boot to test this (both the full bootloader
and UKI w/ the sd-boot stub). Is 32-bit mixed mode on x86_64 working for others?

Thanks,
Clayton

0. Using "debug ignore_loglevel earlycon=efifb earlyprintk=efi,keep efi=debug"

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: x86_64 32-bit EFI mixed mode boot broken

[Ard Biesheuvel]

On Tue, 2 Apr 2024 at 01:44, Clayton Craft <clayton@craftyguy.net> wrote:
>
> On Sun, 24 Mar 2024 22:53:47 +0200 Ard Biesheuvel <ardb@kernel.org> wrote:
> > > > Thanks.
> > > >
> > > > I pushed another branch
> > > >
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton-3
> > > >
> > > > which has a proper fix for the issue that you found.
> > > >
> > > > As it turns out, the compat mixed mode (with handover protocol) was
> > > >
> > > > broken from the beginning, and the change you identified just happened
> > > >
> > > > to trigger it on your hardware.
> > >
> > >
> > > Tested and fixes booting on my Bay Trail tablet and NUC. Thanks for fixing this!
> > >
> > > Tested-by: Clayton Craft <clayton@craftyguy.net>
>
> So... maybe my testing wasn't as thorough as I thought, OR I am experiencing an
> unrelated problem. In any case, I'm having some difficulty figuring out what to
> blame. When using this patch on the 6.6 LTS:
>
> 1) kernel seems to panic right after displaying `disabling bootcon [efifb0]`. I
> determined that it's panicking by setting `panic=-1` and seeing it reboot after
> showing that message. I can work around it by setting `keep_bootcon`, but that's
> not ideal.
>

First of all, if you make it all the way to this point, it is unlikely
that this is the same issue. The issue that was fixed was preventing
the boot from proceeding past the very early 32-bit boot stage in the
decompressor, and by the time you hit this panic, the kernel is up an
running.


> 2) kernel complains about no root (from initrd). I can work around this by
> passing `initrd=my-initramfs` on the kernel cmdline.
>

So why is systemd-boot not passing this directly? AFAIK this is the
default method for systemd-boot, and I don't think it implements any
other methods.

> I haven't tried any newer kernels yet. The second issue above makes me wonder if
> your patch related to args might be to blame, but I'm not sure. Any help poking
> around further would be greatly appreciated :)
>

You might try this stable -rc

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/log/?h=linux-6.6.y

which should get released any day now. It has a couple of EFI related
fixes, although none of them seem related in particular.

Another thing you might try is to pass efi=noruntime to the boot, to
check whether EFI is implicated in this to begin with.

Re: x86_64 32-bit EFI mixed mode boot broken

[Clayton Craft]

[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]

On Sun, 24 Mar 2024 22:53:47 +0200 Ard Biesheuvel <ardb@kernel.org> wrote:
> > > Thanks.
> > >
> > > I pushed another branch
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton-3
> > >
> > > which has a proper fix for the issue that you found.
> > >
> > > As it turns out, the compat mixed mode (with handover protocol) was
> > >
> > > broken from the beginning, and the change you identified just happened
> > >
> > > to trigger it on your hardware.
> >
> >
> > Tested and fixes booting on my Bay Trail tablet and NUC. Thanks for fixing this!
> >
> > Tested-by: Clayton Craft <clayton@craftyguy.net>

So... maybe my testing wasn't as thorough as I thought, OR I am experiencing an
unrelated problem. In any case, I'm having some difficulty figuring out what to
blame. When using this patch on the 6.6 LTS:

1) kernel seems to panic right after displaying `disabling bootcon [efifb0]`. I
determined that it's panicking by setting `panic=-1` and seeing it reboot after
showing that message. I can work around it by setting `keep_bootcon`, but that's
not ideal.

2) kernel complains about no root (from initrd). I can work around this by
passing `initrd=my-initramfs` on the kernel cmdline.

I haven't tried any newer kernels yet. The second issue above makes me wonder if
your patch related to args might be to blame, but I'm not sure. Any help poking
around further would be greatly appreciated :)

-Clayton

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: x86_64 32-bit EFI mixed mode boot broken

[Ard Biesheuvel]

On Sun, 24 Mar 2024 at 22:46, Clayton Craft <clayton@craftyguy.net> wrote:
>
> March 24, 2024 at 10:43 AM, "Ard Biesheuvel" <ardb@kernel.org> wrote:
>
>
> > Thanks.
> >
> > I pushed another branch
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton-3
> >
> > which has a proper fix for the issue that you found.
> >
> > As it turns out, the compat mixed mode (with handover protocol) was
> >
> > broken from the beginning, and the change you identified just happened
> >
> > to trigger it on your hardware.
>
>
> Tested and fixes booting on my Bay Trail tablet and NUC. Thanks for fixing this!
>
> Tested-by: Clayton Craft <clayton@craftyguy.net>
>

Thanks!

Re: x86_64 32-bit EFI mixed mode boot broken

[Clayton Craft]

March 24, 2024 at 10:43 AM, "Ard Biesheuvel" <ardb@kernel.org> wrote:


> Thanks.
> 
> I pushed another branch
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton-3
> 
> which has a proper fix for the issue that you found.
> 
> As it turns out, the compat mixed mode (with handover protocol) was
> 
> broken from the beginning, and the change you identified just happened
> 
> to trigger it on your hardware.


Tested and fixes booting on my Bay Trail tablet and NUC. Thanks for fixing this!

Tested-by: Clayton Craft <clayton@craftyguy.net>

-Clayton

Re: x86_64 32-bit EFI mixed mode boot broken

[Ard Biesheuvel]

On Sun, 24 Mar 2024 at 16:49, Hans de Goede <hdegoede@redhat.com> wrote:
>
> Hi,
>
> On 3/21/24 11:48 PM, Ard Biesheuvel wrote:
> > (cc Hans)
> >
> > On Thu, 21 Mar 2024 at 23:05, Clayton Craft <clayton@craftyguy.net> wrote:
> >>
> >> I've been chasing a problem with 32-bit EFI mixed mode booting on two different
> >> (x86_64) Intel Bay Trail platforms, where the system reboots or hangs seemingly
> >> very early somewhere before or after loading the kernel. I've not been able to
> >> get any output from the kernel or stub over efifb when the issue happens[0], and
> >> do not have serial console access on these systems.
> >>
> >> v6.8 fails for me, and presumably so does everything back to v6.2. v6.1 is able
> >> to boot OK on these platforms with mixed mode, and it looks like there are a lot
> >> of changes from 6.1..6.2 for EFI/mixed mode booting.
> >
> > v6.1 just received some EFI related backports, so please check the
> > latest v6.1.y as well.
> >
> >> I did managed to bisect the
> >> issue to:
> >>
> >>         commit e2ab9eab324cdf240de89741e4a1aa79919f0196
> >>         Author: Ard Biesheuvel <ardb@kernel.org>
> >>         Date:   Tue Nov 22 17:10:02 2022 +0100
> >>
> >>             x86/boot/compressed: Move 32-bit entrypoint code into .text section
> >>
> >> However I'm not sure how to proceed from here, or if my bisect is all that
> >> useful since the commit seems to be in the middle of a bunch of changes I do not
> >> understand. I've been using systemd-boot to test this (both the full bootloader
> >> and UKI w/ the sd-boot stub). Is 32-bit mixed mode on x86_64 working for others?
> >>
> >
> > I usually test on 32-bit OVMF built with LOAD_X64_ON_IA32_ENABLE,
> > which allows the use of the compat entry point. This is different from
> > the EFI handover protocol, and I am not sure which one you are using.
> >
> > I have never had any reports, or noticed any issues myself. Last time
> > I tried (some weeks ago) it was working for me.
> > CC'ing Hans who may have more data points.
>
> I've been offline for most of the week and I see that in the mean time
> you seem to have found a fix, great.
>
> FWIW I have been booting everything up to 6.8.0 on my own mixed-mode
> Bay Trail tablets without issues, so the problem seems to be specific to
> certain BIOS-es.
>
> Please Cc me on the final fix, then I can test that early and double check
> that things don't regress on other mixed-mode Bay Trail devices.
>

Thanks.

I pushed another branch

https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton-3

which has a proper fix for the issue that you found.

As it turns out, the compat mixed mode (with handover protocol) was
broken from the beginning, and the change you identified just happened
to trigger it on your hardware.

Re: x86_64 32-bit EFI mixed mode boot broken

[Hans de Goede]

Hi,

On 3/21/24 11:48 PM, Ard Biesheuvel wrote:
> (cc Hans)
> 
> On Thu, 21 Mar 2024 at 23:05, Clayton Craft <clayton@craftyguy.net> wrote:
>>
>> I've been chasing a problem with 32-bit EFI mixed mode booting on two different
>> (x86_64) Intel Bay Trail platforms, where the system reboots or hangs seemingly
>> very early somewhere before or after loading the kernel. I've not been able to
>> get any output from the kernel or stub over efifb when the issue happens[0], and
>> do not have serial console access on these systems.
>>
>> v6.8 fails for me, and presumably so does everything back to v6.2. v6.1 is able
>> to boot OK on these platforms with mixed mode, and it looks like there are a lot
>> of changes from 6.1..6.2 for EFI/mixed mode booting.
> 
> v6.1 just received some EFI related backports, so please check the
> latest v6.1.y as well.
> 
>> I did managed to bisect the
>> issue to:
>>
>>         commit e2ab9eab324cdf240de89741e4a1aa79919f0196
>>         Author: Ard Biesheuvel <ardb@kernel.org>
>>         Date:   Tue Nov 22 17:10:02 2022 +0100
>>
>>             x86/boot/compressed: Move 32-bit entrypoint code into .text section
>>
>> However I'm not sure how to proceed from here, or if my bisect is all that
>> useful since the commit seems to be in the middle of a bunch of changes I do not
>> understand. I've been using systemd-boot to test this (both the full bootloader
>> and UKI w/ the sd-boot stub). Is 32-bit mixed mode on x86_64 working for others?
>>
> 
> I usually test on 32-bit OVMF built with LOAD_X64_ON_IA32_ENABLE,
> which allows the use of the compat entry point. This is different from
> the EFI handover protocol, and I am not sure which one you are using.
> 
> I have never had any reports, or noticed any issues myself. Last time
> I tried (some weeks ago) it was working for me.
> CC'ing Hans who may have more data points.

I've been offline for most of the week and I see that in the mean time
you seem to have found a fix, great.

FWIW I have been booting everything up to 6.8.0 on my own mixed-mode
Bay Trail tablets without issues, so the problem seems to be specific to
certain BIOS-es.

Please Cc me on the final fix, then I can test that early and double check
that things don't regress on other mixed-mode Bay Trail devices.

Regards,

Hans

Re: x86_64 32-bit EFI mixed mode boot broken

[Clayton Craft]

March 23, 2024 at 10:18 AM, "Ard Biesheuvel" <ardb@kernel.org> wrote:
> On Sat, 23 Mar 2024 at 14:39, Ard Biesheuvel <ardb@kernel.org> wrote:
> > 
> >  OK.
> > 
> >  I have reshuffled the branch and put the patch you identified as the
> > 
> >  one fixing the boot first. Please double check whether this change
> > 
> >  still fixes the boot for you.
> > 
> >  https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton-2
> > 
> >  If so, we can try applying it to mainline, and merge it if it works there too.

This branch boots on my Bay Trail systems.


> I have a mainline branch with the same change applied, please try that one too.

I assume this is the `efi-mixed-mode-boot-hack` branch, if so then this branch also boots on these systems.

-Clayton

Re: x86_64 32-bit EFI mixed mode boot broken

[Ard Biesheuvel]

On Sat, 23 Mar 2024 at 14:39, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Fri, 22 Mar 2024 at 21:34, Clayton Craft <clayton@craftyguy.net> wrote:
> >
> > March 22, 2024 at 11:30 AM, "Ard Biesheuvel" <ardb@kernel.org> wrote:
> >
> >
> > >
> > > On Fri, 22 Mar 2024 at 19:57, Clayton Craft <clayton@craftyguy.net> wrote:
> > >
> > > I have pushed a branch below that reverts the patch you identified in
> > >
> > > 4 separate steps. Could you please check which step makes your system
> > >
> > > boot again?
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton
> > >
> >
> > Thanks a lot for doing this, I really appreciate the help!
> >
> > It looks like if I build from 868a7245, booting breaks again on my Bay Trail systems. If I put back 00e85ab5, they boot again.
> >
>
> OK.
>
> I have reshuffled the branch and put the patch you identified as the
> one fixing the boot first. Please double check whether this change
> still fixes the boot for you.
>
> https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton-2
>
> If so, we can try applying it to mainline, and merge it if it works there too.
>

I have a mainline branch with the same change applied, please try that one too.

Re: x86_64 32-bit EFI mixed mode boot broken

[Ard Biesheuvel]

On Fri, 22 Mar 2024 at 21:34, Clayton Craft <clayton@craftyguy.net> wrote:
>
> March 22, 2024 at 11:30 AM, "Ard Biesheuvel" <ardb@kernel.org> wrote:
>
>
> >
> > On Fri, 22 Mar 2024 at 19:57, Clayton Craft <clayton@craftyguy.net> wrote:
> >
> > I have pushed a branch below that reverts the patch you identified in
> >
> > 4 separate steps. Could you please check which step makes your system
> >
> > boot again?
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton
> >
>
> Thanks a lot for doing this, I really appreciate the help!
>
> It looks like if I build from 868a7245, booting breaks again on my Bay Trail systems. If I put back 00e85ab5, they boot again.
>

OK.

I have reshuffled the branch and put the patch you identified as the
one fixing the boot first. Please double check whether this change
still fixes the boot for you.

https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton-2

If so, we can try applying it to mainline, and merge it if it works there too.

If not, we will need better debugging to figure out what the hell is going on.

Re: x86_64 32-bit EFI mixed mode boot broken

[Clayton Craft]

March 22, 2024 at 11:30 AM, "Ard Biesheuvel" <ardb@kernel.org> wrote:


> 
> On Fri, 22 Mar 2024 at 19:57, Clayton Craft <clayton@craftyguy.net> wrote:
> 
> I have pushed a branch below that reverts the patch you identified in
> 
> 4 separate steps. Could you please check which step makes your system
> 
> boot again?
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton
>

Thanks a lot for doing this, I really appreciate the help!

It looks like if I build from 868a7245, booting breaks again on my Bay Trail systems. If I put back 00e85ab5, they boot again.

-Clayton

Re: x86_64 32-bit EFI mixed mode boot broken

[Ard Biesheuvel]

On Fri, 22 Mar 2024 at 19:57, Clayton Craft <clayton@craftyguy.net> wrote:
>
> March 22, 2024 at 9:51 AM, "Ard Biesheuvel" <ardb@kernel.org> wrote:
>
>
> > > As I mentioned initially, I bisected my failure to e2ab9eab32. Does that give
> > >
> > >  any hint about what might be the problem?
> > >
> >
> > Not at all, unfortunately. What we might try is to back out the
> >
> > changes step by step. I can prepare a branch for you if you like.
>
>
> Yeah sure, if you don't mind! I'd be happy to test changes in a branch to try and figure out what is causing this.
>

I have pushed a branch below that reverts the patch you identified in
4 separate steps. Could you please check which step makes your system
boot again?

https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=efi-clayton

Re: x86_64 32-bit EFI mixed mode boot broken

[Clayton Craft]

March 22, 2024 at 9:51 AM, "Ard Biesheuvel" <ardb@kernel.org> wrote:


> > As I mentioned initially, I bisected my failure to e2ab9eab32. Does that give
> > 
> >  any hint about what might be the problem?
> > 
> 
> Not at all, unfortunately. What we might try is to back out the
> 
> changes step by step. I can prepare a branch for you if you like.


Yeah sure, if you don't mind! I'd be happy to test changes in a branch to try and figure out what is causing this.

-Clayton

Re: x86_64 32-bit EFI mixed mode boot broken

[Ard Biesheuvel]

On Fri, 22 Mar 2024 at 18:19, Clayton Craft <clayton@craftyguy.net> wrote:
>
> On Fri, 22 Mar 2024 13:52:05 +0100 Ard Biesheuvel <ardb@kernel.org> wrote:
> > That depends on the bootloader. One of the changes around that time is
> > the introduction of this Kconfig symbol: before that, the EFI handover
> > protocol was always supported but now it can be compiled out. So the
> > safe choice is to enable it.
>
> I am using systemd-boot, and based on my understanding of its source it seems to
> only use handover on old kernels that don't support LINUX_INITRD_MEDIA_GUID.

That does ring a bell. So that means you must be using the .compat
entry point, which is what 32-bit OVMF uses too.

> >
> > However, while looking more deeply into this, I noticed that we are
> > running quite low own stack space. Mixed mode is different because it
> > calls into the boot services using the decompressor's boot stack,
> > rather than using the one that was provided by firmware at entry.
> > (Note that the UEFI spec mandates 128k of stack space)
> >
> > In my case, I bisected the regression to
> >
> > commit 5c4feadb0011983bbc4587bc61056c7b379d9969 (HEAD)
> > Author: Ard Biesheuvel <ardb@kernel.org>
> > Date:   Mon Aug 7 18:27:16 2023 +0200
> >
> >     x86/decompressor: Move global symbol references to C code
> >
> > which moves the boot stack into a different memory region. Formerly,
> > we'd end up at the far end of the heap when overrunning the stack but
> > now, we end up crashing. Of course, overwriting the heap can cause
> > problems of its own, so we'll need to bump this in any case.
> >
> > Could you give this a try please?
> >
> >
> > --- a/arch/x86/include/asm/boot.h
> > +++ b/arch/x86/include/asm/boot.h
> > @@ -38,7 +38,7 @@
> >  #endif
> >
> >  #ifdef CONFIG_X86_64
> > -# define BOOT_STACK_SIZE       0x4000
> > +# define BOOT_STACK_SIZE       0x10000
> >
> >  /*
> >   * Used by decompressor's startup_32() to allocate page tables for identity
>
> Just gave this a try, on 6.1.82, and the system still reboots after selecting
> the kernel in the bootloader. So it seems like my problem is different.
>

Yeah, it was a long shot.

> As I mentioned initially, I bisected my failure to e2ab9eab32. Does that give
> any hint about what might be the problem?
>

Not at all, unfortunately. What we might try is to back out the
changes step by step. I can prepare a branch for you if you like.

Re: x86_64 32-bit EFI mixed mode boot broken

[Clayton Craft]

[-- Attachment #1: Type: text/plain, Size: 1981 bytes --]

On Fri, 22 Mar 2024 13:52:05 +0100 Ard Biesheuvel <ardb@kernel.org> wrote:
> That depends on the bootloader. One of the changes around that time is
> the introduction of this Kconfig symbol: before that, the EFI handover
> protocol was always supported but now it can be compiled out. So the
> safe choice is to enable it.

I am using systemd-boot, and based on my understanding of its source it seems to
only use handover on old kernels that don't support LINUX_INITRD_MEDIA_GUID.
> 
> However, while looking more deeply into this, I noticed that we are
> running quite low own stack space. Mixed mode is different because it
> calls into the boot services using the decompressor's boot stack,
> rather than using the one that was provided by firmware at entry.
> (Note that the UEFI spec mandates 128k of stack space)
> 
> In my case, I bisected the regression to
> 
> commit 5c4feadb0011983bbc4587bc61056c7b379d9969 (HEAD)
> Author: Ard Biesheuvel <ardb@kernel.org>
> Date:   Mon Aug 7 18:27:16 2023 +0200
> 
>     x86/decompressor: Move global symbol references to C code
> 
> which moves the boot stack into a different memory region. Formerly,
> we'd end up at the far end of the heap when overrunning the stack but
> now, we end up crashing. Of course, overwriting the heap can cause
> problems of its own, so we'll need to bump this in any case.
> 
> Could you give this a try please?
> 
> 
> --- a/arch/x86/include/asm/boot.h
> +++ b/arch/x86/include/asm/boot.h
> @@ -38,7 +38,7 @@
>  #endif
> 
>  #ifdef CONFIG_X86_64
> -# define BOOT_STACK_SIZE       0x4000
> +# define BOOT_STACK_SIZE       0x10000
> 
>  /*
>   * Used by decompressor's startup_32() to allocate page tables for identity

Just gave this a try, on 6.1.82, and the system still reboots after selecting
the kernel in the bootloader. So it seems like my problem is different.

As I mentioned initially, I bisected my failure to e2ab9eab32. Does that give
any hint about what might be the problem?

-Clayton

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: x86_64 32-bit EFI mixed mode boot broken

[Ard Biesheuvel]

On Fri, 22 Mar 2024 at 01:06, Clayton Craft <clayton@craftyguy.net> wrote:
>
> On Thu, 21 Mar 2024 23:48:09 +0100 Ard Biesheuvel <ardb@kernel.org> wrote:
> > > v6.8 fails for me, and presumably so does everything back to v6.2. v6.1 is able
> > > to boot OK on these platforms with mixed mode, and it looks like there are a lot
> > > of changes from 6.1..6.2 for EFI/mixed mode booting.
> >
> > v6.1 just received some EFI related backports, so please check the
> > latest v6.1.y as well.
>
> I just gave v6.1.82 a try, and it fails to boot for me. That seems to be a
> regression from the 6.1.0 that I tested previously.
>
> > I usually test on 32-bit OVMF built with LOAD_X64_ON_IA32_ENABLE,
> > which allows the use of the compat entry point. This is different from
> > the EFI handover protocol, and I am not sure which one you are using.
>
> I should have mentioned this previously, here's the EFI-related kconfig that I
> am using. If there's anything missing then please let me know:
>
>         CONFIG_EFI=y
>         CONFIG_EFI_EARLYCON=y
>         CONFIG_EFI_ESRT=y
>         # CONFIG_EFI_HANDOVER_PROTOCOL is not set
>         CONFIG_EFI_MIXED=y
>         CONFIG_EFI_RUNTIME_WRAPPERS=y
>         CONFIG_EFI_STUB=y
>         CONFIG_EFI_VARS_PSTORE=m
>         CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE=y
>
> Note that the EFI handover protocol support is disabled, I was under the
> impression that it's not required for mixed mode.
>

That depends on the bootloader. One of the changes around that time is
the introduction of this Kconfig symbol: before that, the EFI handover
protocol was always supported but now it can be compiled out. So the
safe choice is to enable it.

However, while looking more deeply into this, I noticed that we are
running quite low own stack space. Mixed mode is different because it
calls into the boot services using the decompressor's boot stack,
rather than using the one that was provided by firmware at entry.
(Note that the UEFI spec mandates 128k of stack space)

In my case, I bisected the regression to

commit 5c4feadb0011983bbc4587bc61056c7b379d9969 (HEAD)
Author: Ard Biesheuvel <ardb@kernel.org>
Date:   Mon Aug 7 18:27:16 2023 +0200

    x86/decompressor: Move global symbol references to C code

which moves the boot stack into a different memory region. Formerly,
we'd end up at the far end of the heap when overrunning the stack but
now, we end up crashing. Of course, overwriting the heap can cause
problems of its own, so we'll need to bump this in any case.

Could you give this a try please?


--- a/arch/x86/include/asm/boot.h
+++ b/arch/x86/include/asm/boot.h
@@ -38,7 +38,7 @@
 #endif

 #ifdef CONFIG_X86_64
-# define BOOT_STACK_SIZE       0x4000
+# define BOOT_STACK_SIZE       0x10000

 /*
  * Used by decompressor's startup_32() to allocate page tables for identity

Re: x86_64 32-bit EFI mixed mode boot broken

[Clayton Craft]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

On Thu, 21 Mar 2024 23:48:09 +0100 Ard Biesheuvel <ardb@kernel.org> wrote:
> > v6.8 fails for me, and presumably so does everything back to v6.2. v6.1 is able
> > to boot OK on these platforms with mixed mode, and it looks like there are a lot
> > of changes from 6.1..6.2 for EFI/mixed mode booting.
> 
> v6.1 just received some EFI related backports, so please check the
> latest v6.1.y as well.

I just gave v6.1.82 a try, and it fails to boot for me. That seems to be a
regression from the 6.1.0 that I tested previously.

> I usually test on 32-bit OVMF built with LOAD_X64_ON_IA32_ENABLE,
> which allows the use of the compat entry point. This is different from
> the EFI handover protocol, and I am not sure which one you are using.

I should have mentioned this previously, here's the EFI-related kconfig that I
am using. If there's anything missing then please let me know:

        CONFIG_EFI=y
        CONFIG_EFI_EARLYCON=y
        CONFIG_EFI_ESRT=y
        # CONFIG_EFI_HANDOVER_PROTOCOL is not set
        CONFIG_EFI_MIXED=y
        CONFIG_EFI_RUNTIME_WRAPPERS=y
        CONFIG_EFI_STUB=y
        CONFIG_EFI_VARS_PSTORE=m
        CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE=y

Note that the EFI handover protocol support is disabled, I was under the
impression that it's not required for mixed mode.

Please let me know if there's any more information I can provide about my
testing / setup.

-Clayton

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: x86_64 32-bit EFI mixed mode boot broken

[Ard Biesheuvel]

(cc Hans)

On Thu, 21 Mar 2024 at 23:05, Clayton Craft <clayton@craftyguy.net> wrote:
>
> I've been chasing a problem with 32-bit EFI mixed mode booting on two different
> (x86_64) Intel Bay Trail platforms, where the system reboots or hangs seemingly
> very early somewhere before or after loading the kernel. I've not been able to
> get any output from the kernel or stub over efifb when the issue happens[0], and
> do not have serial console access on these systems.
>
> v6.8 fails for me, and presumably so does everything back to v6.2. v6.1 is able
> to boot OK on these platforms with mixed mode, and it looks like there are a lot
> of changes from 6.1..6.2 for EFI/mixed mode booting.

v6.1 just received some EFI related backports, so please check the
latest v6.1.y as well.

> I did managed to bisect the
> issue to:
>
>         commit e2ab9eab324cdf240de89741e4a1aa79919f0196
>         Author: Ard Biesheuvel <ardb@kernel.org>
>         Date:   Tue Nov 22 17:10:02 2022 +0100
>
>             x86/boot/compressed: Move 32-bit entrypoint code into .text section
>
> However I'm not sure how to proceed from here, or if my bisect is all that
> useful since the commit seems to be in the middle of a bunch of changes I do not
> understand. I've been using systemd-boot to test this (both the full bootloader
> and UKI w/ the sd-boot stub). Is 32-bit mixed mode on x86_64 working for others?
>

I usually test on 32-bit OVMF built with LOAD_X64_ON_IA32_ENABLE,
which allows the use of the compat entry point. This is different from
the EFI handover protocol, and I am not sure which one you are using.

I have never had any reports, or noticed any issues myself. Last time
I tried (some weeks ago) it was working for me.
CC'ing Hans who may have more data points.

x86_64 32-bit EFI mixed mode boot broken

[Clayton Craft]

[-- Attachment #1: Type: text/plain, Size: 1274 bytes --]

I've been chasing a problem with 32-bit EFI mixed mode booting on two different
(x86_64) Intel Bay Trail platforms, where the system reboots or hangs seemingly
very early somewhere before or after loading the kernel. I've not been able to
get any output from the kernel or stub over efifb when the issue happens[0], and
do not have serial console access on these systems.

v6.8 fails for me, and presumably so does everything back to v6.2. v6.1 is able
to boot OK on these platforms with mixed mode, and it looks like there are a lot
of changes from 6.1..6.2 for EFI/mixed mode booting. I did managed to bisect the
issue to:

        commit e2ab9eab324cdf240de89741e4a1aa79919f0196
        Author: Ard Biesheuvel <ardb@kernel.org>
        Date:   Tue Nov 22 17:10:02 2022 +0100

            x86/boot/compressed: Move 32-bit entrypoint code into .text section

However I'm not sure how to proceed from here, or if my bisect is all that
useful since the commit seems to be in the middle of a bunch of changes I do not
understand. I've been using systemd-boot to test this (both the full bootloader
and UKI w/ the sd-boot stub). Is 32-bit mixed mode on x86_64 working for others?

Thanks,
Clayton

0. Using "debug ignore_loglevel earlycon=efifb earlyprintk=efi,keep efi=debug"

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]