LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Qu Wenruo]

Hi,

When backporting some btrfs specific patches to all LTS kernels, I found
v4.14.290 kernel unable to boot as a KVM guest with edk2-ovmf
(edk2-ovmf: 202205, qemu 7.0.0, libvirt 1:8.6.0).

While all other LTS/stable branches (4.19.x, 5.4.x, 5.10.x, 5.15.x,
5.18.x, 5.19.x) can boot without a hipccup.

I tried the following configs, but none of them can even provide an
early output:

- CONFIG_X86_VERBOSE_BOOTUP
- CONFIG_EARLY_PRINTK
- CONFIG_EARLY_PRINTK_EFI

Is this a known bug or something new?

Thanks,
Qu

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Greg KH]

On Mon, Aug 22, 2022 at 09:15:59AM +0800, Qu Wenruo wrote:
> Hi,
> 
> When backporting some btrfs specific patches to all LTS kernels, I found
> v4.14.290 kernel unable to boot as a KVM guest with edk2-ovmf
> (edk2-ovmf: 202205, qemu 7.0.0, libvirt 1:8.6.0).
> 
> While all other LTS/stable branches (4.19.x, 5.4.x, 5.10.x, 5.15.x,
> 5.18.x, 5.19.x) can boot without a hipccup.
> 
> I tried the following configs, but none of them can even provide an
> early output:
> 
> - CONFIG_X86_VERBOSE_BOOTUP
> - CONFIG_EARLY_PRINTK
> - CONFIG_EARLY_PRINTK_EFI
> 
> Is this a known bug or something new?

Has this ever worked properly on this very old kernel tree?  If so, can
you use 'git bisect' to find the offending commit?

thanks,

greg k-h

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Qu Wenruo]

On 2022/8/22 14:25, Greg KH wrote:
> On Mon, Aug 22, 2022 at 09:15:59AM +0800, Qu Wenruo wrote:
>> Hi,
>>
>> When backporting some btrfs specific patches to all LTS kernels, I found
>> v4.14.290 kernel unable to boot as a KVM guest with edk2-ovmf
>> (edk2-ovmf: 202205, qemu 7.0.0, libvirt 1:8.6.0).
>>
>> While all other LTS/stable branches (4.19.x, 5.4.x, 5.10.x, 5.15.x,
>> 5.18.x, 5.19.x) can boot without a hipccup.
>>
>> I tried the following configs, but none of them can even provide an
>> early output:
>>
>> - CONFIG_X86_VERBOSE_BOOTUP
>> - CONFIG_EARLY_PRINTK
>> - CONFIG_EARLY_PRINTK_EFI
>>
>> Is this a known bug or something new?
>
> Has this ever worked properly on this very old kernel tree?  If so, can
> you use 'git bisect' to find the offending commit?

Unfortunately the initial v4.14 from upstream can not even be compiled.

I'll try some more recent v4.14.x tags to see if I can get a working
release to do the bisect.

Thanks,
Qu

>
> thanks,
>
> greg k-h

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Greg KH]

On Mon, Aug 22, 2022 at 03:24:53PM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/8/22 14:25, Greg KH wrote:
> > On Mon, Aug 22, 2022 at 09:15:59AM +0800, Qu Wenruo wrote:
> > > Hi,
> > > 
> > > When backporting some btrfs specific patches to all LTS kernels, I found
> > > v4.14.290 kernel unable to boot as a KVM guest with edk2-ovmf
> > > (edk2-ovmf: 202205, qemu 7.0.0, libvirt 1:8.6.0).
> > > 
> > > While all other LTS/stable branches (4.19.x, 5.4.x, 5.10.x, 5.15.x,
> > > 5.18.x, 5.19.x) can boot without a hipccup.
> > > 
> > > I tried the following configs, but none of them can even provide an
> > > early output:
> > > 
> > > - CONFIG_X86_VERBOSE_BOOTUP
> > > - CONFIG_EARLY_PRINTK
> > > - CONFIG_EARLY_PRINTK_EFI
> > > 
> > > Is this a known bug or something new?
> > 
> > Has this ever worked properly on this very old kernel tree?  If so, can
> > you use 'git bisect' to find the offending commit?
> 
> Unfortunately the initial v4.14 from upstream can not even be compiled.

Really?  Try using an older version of gcc and you should be fine.  It
did build properly back in 2017 when it was released :)

thanks,

greg k-h

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Qu Wenruo]

On 2022/8/22 15:33, Greg KH wrote:
> On Mon, Aug 22, 2022 at 03:24:53PM +0800, Qu Wenruo wrote:
>>
>>
>> On 2022/8/22 14:25, Greg KH wrote:
>>> On Mon, Aug 22, 2022 at 09:15:59AM +0800, Qu Wenruo wrote:
>>>> Hi,
>>>>
>>>> When backporting some btrfs specific patches to all LTS kernels, I found
>>>> v4.14.290 kernel unable to boot as a KVM guest with edk2-ovmf
>>>> (edk2-ovmf: 202205, qemu 7.0.0, libvirt 1:8.6.0).
>>>>
>>>> While all other LTS/stable branches (4.19.x, 5.4.x, 5.10.x, 5.15.x,
>>>> 5.18.x, 5.19.x) can boot without a hipccup.
>>>>
>>>> I tried the following configs, but none of them can even provide an
>>>> early output:
>>>>
>>>> - CONFIG_X86_VERBOSE_BOOTUP
>>>> - CONFIG_EARLY_PRINTK
>>>> - CONFIG_EARLY_PRINTK_EFI
>>>>
>>>> Is this a known bug or something new?
>>>
>>> Has this ever worked properly on this very old kernel tree?  If so, can
>>> you use 'git bisect' to find the offending commit?
>>
>> Unfortunately the initial v4.14 from upstream can not even be compiled.
>
> Really?  Try using an older version of gcc and you should be fine.  It
> did build properly back in 2017 when it was released :)

Yeah, I'm pretty sure my toolchain is too new for v4.14.0. But my distro
only provides the latest and mostly upstream packages.

It may be a even worse disaster to find a way to rollback to older
toolchains using my distro...

Also my hardware may not be well suited for older kernels either.
(Zen 3 CPU used here)

In fact, I even find it hard just to locate a v4.14.x tag that can compile.
After some bisection between v4.14.x tags, only v4.14.268 and newer tags
can even be compiled using latest toolchain.
(But still tons of warning, and tons of objdump warnings against
insn_get_length()).

I'm not sure what's the normal practice for backports to such old branch.

Do you stable guys keep dedicated VMs loaded with older distro just for
these old branches?
If so, any recommendation on those kinda retro distro?

Thanks,
Qu

>
> thanks,
>
> greg k-h

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Greg KH]

On Mon, Aug 22, 2022 at 03:49:51PM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/8/22 15:33, Greg KH wrote:
> > On Mon, Aug 22, 2022 at 03:24:53PM +0800, Qu Wenruo wrote:
> > > 
> > > 
> > > On 2022/8/22 14:25, Greg KH wrote:
> > > > On Mon, Aug 22, 2022 at 09:15:59AM +0800, Qu Wenruo wrote:
> > > > > Hi,
> > > > > 
> > > > > When backporting some btrfs specific patches to all LTS kernels, I found
> > > > > v4.14.290 kernel unable to boot as a KVM guest with edk2-ovmf
> > > > > (edk2-ovmf: 202205, qemu 7.0.0, libvirt 1:8.6.0).
> > > > > 
> > > > > While all other LTS/stable branches (4.19.x, 5.4.x, 5.10.x, 5.15.x,
> > > > > 5.18.x, 5.19.x) can boot without a hipccup.
> > > > > 
> > > > > I tried the following configs, but none of them can even provide an
> > > > > early output:
> > > > > 
> > > > > - CONFIG_X86_VERBOSE_BOOTUP
> > > > > - CONFIG_EARLY_PRINTK
> > > > > - CONFIG_EARLY_PRINTK_EFI
> > > > > 
> > > > > Is this a known bug or something new?
> > > > 
> > > > Has this ever worked properly on this very old kernel tree?  If so, can
> > > > you use 'git bisect' to find the offending commit?
> > > 
> > > Unfortunately the initial v4.14 from upstream can not even be compiled.
> > 
> > Really?  Try using an older version of gcc and you should be fine.  It
> > did build properly back in 2017 when it was released :)
> 
> Yeah, I'm pretty sure my toolchain is too new for v4.14.0. But my distro
> only provides the latest and mostly upstream packages.
> 
> It may be a even worse disaster to find a way to rollback to older
> toolchains using my distro...
> 
> Also my hardware may not be well suited for older kernels either.
> (Zen 3 CPU used here)
> 
> In fact, I even find it hard just to locate a v4.14.x tag that can compile.
> After some bisection between v4.14.x tags, only v4.14.268 and newer tags
> can even be compiled using latest toolchain.
> (But still tons of warning, and tons of objdump warnings against
> insn_get_length()).
> 
> I'm not sure what's the normal practice for backports to such old branch.
> 
> Do you stable guys keep dedicated VMs loaded with older distro just for
> these old branches?

I don't, that's why those kernels can be built with newer versions of
gcc.

Your distro should have a version of gcc-10 or gcc-9 that can be
installed, right?  Or maybe use the gcc versions on kernel.org that only
work for kernel builds?

> If so, any recommendation on those kinda retro distro?

Try installing an old version of Debian, or better yet, use a distro
that provides old versions of gcc :)

good luck!

greg k-h

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Willy Tarreau]

On Mon, Aug 22, 2022 at 03:49:51PM +0800, Qu Wenruo wrote:
> Yeah, I'm pretty sure my toolchain is too new for v4.14.0. But my distro
> only provides the latest and mostly upstream packages.

Then there's something odd in your use case. Old kernels are aimed at
those who need to have systems on which nothing changes. It's particularly
strange to be stuck in the past for the kernel but to continually upgrade
userland. Most often it's the opposite that's seen.

Regardless, if you need an older compiler, just use these ones:

   https://mirrors.edge.kernel.org/pub/tools/crosstool/

They go back to 4.9.4 for x86, you'll surely find the right one for your
usage. I've long used 4.7.4 for kernels up to 4.9 and 6.5 for 4.19 and
above, so something within that area will surely match your needs.

> It may be a even worse disaster to find a way to rollback to older
> toolchains using my distro...

No, using a prebuilt toolchain like those above is quite trivial and
it will avoid messing up with your local packages. That's the best
solution I can recommend.

Willy

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Qu Wenruo]

On 2022/8/22 15:58, Greg KH wrote:
> On Mon, Aug 22, 2022 at 03:49:51PM +0800, Qu Wenruo wrote:
>>
>>
>> On 2022/8/22 15:33, Greg KH wrote:
>>> On Mon, Aug 22, 2022 at 03:24:53PM +0800, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2022/8/22 14:25, Greg KH wrote:
>>>>> On Mon, Aug 22, 2022 at 09:15:59AM +0800, Qu Wenruo wrote:
>>>>>> Hi,
>>>>>>
>>>>>> When backporting some btrfs specific patches to all LTS kernels, I found
>>>>>> v4.14.290 kernel unable to boot as a KVM guest with edk2-ovmf
>>>>>> (edk2-ovmf: 202205, qemu 7.0.0, libvirt 1:8.6.0).
>>>>>>
>>>>>> While all other LTS/stable branches (4.19.x, 5.4.x, 5.10.x, 5.15.x,
>>>>>> 5.18.x, 5.19.x) can boot without a hipccup.
>>>>>>
>>>>>> I tried the following configs, but none of them can even provide an
>>>>>> early output:
>>>>>>
>>>>>> - CONFIG_X86_VERBOSE_BOOTUP
>>>>>> - CONFIG_EARLY_PRINTK
>>>>>> - CONFIG_EARLY_PRINTK_EFI
>>>>>>
>>>>>> Is this a known bug or something new?
>>>>>
>>>>> Has this ever worked properly on this very old kernel tree?  If so, can
>>>>> you use 'git bisect' to find the offending commit?
>>>>
>>>> Unfortunately the initial v4.14 from upstream can not even be compiled.
>>>
>>> Really?  Try using an older version of gcc and you should be fine.  It
>>> did build properly back in 2017 when it was released :)
>>
>> Yeah, I'm pretty sure my toolchain is too new for v4.14.0. But my distro
>> only provides the latest and mostly upstream packages.
>>
>> It may be a even worse disaster to find a way to rollback to older
>> toolchains using my distro...
>>
>> Also my hardware may not be well suited for older kernels either.
>> (Zen 3 CPU used here)
>>
>> In fact, I even find it hard just to locate a v4.14.x tag that can compile.
>> After some bisection between v4.14.x tags, only v4.14.268 and newer tags
>> can even be compiled using latest toolchain.
>> (But still tons of warning, and tons of objdump warnings against
>> insn_get_length()).
>>
>> I'm not sure what's the normal practice for backports to such old branch.
>>
>> Do you stable guys keep dedicated VMs loaded with older distro just for
>> these old branches?
>
> I don't, that's why those kernels can be built with newer versions of
> gcc.
>
> Your distro should have a version of gcc-10 or gcc-9 that can be
> installed, right?

This may sounds like a meme, but I'm really using Archlinux for my VM
and host, and it doesn't provide older GCC at all.

>  Or maybe use the gcc versions on kernel.org that only
> work for kernel builds?
>
>> If so, any recommendation on those kinda retro distro?
>
> Try installing an old version of Debian, or better yet, use a distro
> that provides old versions of gcc :)

I guess that's the only way to go.

Thanks for all the advice,
Qu

>
> good luck!
>
> greg k-h

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Qu Wenruo]

On 2022/8/22 16:04, Willy Tarreau wrote:
> On Mon, Aug 22, 2022 at 03:49:51PM +0800, Qu Wenruo wrote:
>> Yeah, I'm pretty sure my toolchain is too new for v4.14.0. But my distro
>> only provides the latest and mostly upstream packages.
>
> Then there's something odd in your use case. Old kernels are aimed at
> those who need to have systems on which nothing changes. It's particularly
> strange to be stuck in the past for the kernel but to continually upgrade
> userland. Most often it's the opposite that's seen.

Yep, that's my bad, just too lazy to get a time-period correct distro,
or learn other package management tools from other distros.

>
> Regardless, if you need an older compiler, just use these ones:
>
>     https://mirrors.edge.kernel.org/pub/tools/crosstool/
>
> They go back to 4.9.4 for x86, you'll surely find the right one for your
> usage. I've long used 4.7.4 for kernels up to 4.9 and 6.5 for 4.19 and
> above, so something within that area will surely match your needs.

BTW, it would be way more awesome if the page can provide some hint on
the initial release date of the compilers.

It would help a lot of choose the toolchain then.

>
>> It may be a even worse disaster to find a way to rollback to older
>> toolchains using my distro...
>
> No, using a prebuilt toolchain like those above is quite trivial and
> it will avoid messing up with your local packages. That's the best
> solution I can recommend.

Thanks for all the info, it really helps a lot,
Qu

>
> Willy

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Willy Tarreau]

On Mon, Aug 22, 2022 at 04:19:49PM +0800, Qu Wenruo wrote:
> > Regardless, if you need an older compiler, just use these ones:
> > 
> >     https://mirrors.edge.kernel.org/pub/tools/crosstool/
> > 
> > They go back to 4.9.4 for x86, you'll surely find the right one for your
> > usage. I've long used 4.7.4 for kernels up to 4.9 and 6.5 for 4.19 and
> > above, so something within that area will surely match your needs.
> 
> BTW, it would be way more awesome if the page can provide some hint on
> the initial release date of the compilers.
> 
> It would help a lot of choose the toolchain then.

It wouldn't help, if you look closely, you'll notice that in the "other
releases" section you have the most recent version of each of them. That
does not preclude the existence of the branch earlier. For example gcc-9
was released in 2019 and 9.5 was emitted 3 years later. That's quite an
amplitude that doesn't help.

Willy

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Greg KH]

On Mon, Aug 22, 2022 at 04:13:03PM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/8/22 15:58, Greg KH wrote:
> > On Mon, Aug 22, 2022 at 03:49:51PM +0800, Qu Wenruo wrote:
> > > 
> > > 
> > > On 2022/8/22 15:33, Greg KH wrote:
> > > > On Mon, Aug 22, 2022 at 03:24:53PM +0800, Qu Wenruo wrote:
> > > > > 
> > > > > 
> > > > > On 2022/8/22 14:25, Greg KH wrote:
> > > > > > On Mon, Aug 22, 2022 at 09:15:59AM +0800, Qu Wenruo wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > When backporting some btrfs specific patches to all LTS kernels, I found
> > > > > > > v4.14.290 kernel unable to boot as a KVM guest with edk2-ovmf
> > > > > > > (edk2-ovmf: 202205, qemu 7.0.0, libvirt 1:8.6.0).
> > > > > > > 
> > > > > > > While all other LTS/stable branches (4.19.x, 5.4.x, 5.10.x, 5.15.x,
> > > > > > > 5.18.x, 5.19.x) can boot without a hipccup.
> > > > > > > 
> > > > > > > I tried the following configs, but none of them can even provide an
> > > > > > > early output:
> > > > > > > 
> > > > > > > - CONFIG_X86_VERBOSE_BOOTUP
> > > > > > > - CONFIG_EARLY_PRINTK
> > > > > > > - CONFIG_EARLY_PRINTK_EFI
> > > > > > > 
> > > > > > > Is this a known bug or something new?
> > > > > > 
> > > > > > Has this ever worked properly on this very old kernel tree?  If so, can
> > > > > > you use 'git bisect' to find the offending commit?
> > > > > 
> > > > > Unfortunately the initial v4.14 from upstream can not even be compiled.
> > > > 
> > > > Really?  Try using an older version of gcc and you should be fine.  It
> > > > did build properly back in 2017 when it was released :)
> > > 
> > > Yeah, I'm pretty sure my toolchain is too new for v4.14.0. But my distro
> > > only provides the latest and mostly upstream packages.
> > > 
> > > It may be a even worse disaster to find a way to rollback to older
> > > toolchains using my distro...
> > > 
> > > Also my hardware may not be well suited for older kernels either.
> > > (Zen 3 CPU used here)
> > > 
> > > In fact, I even find it hard just to locate a v4.14.x tag that can compile.
> > > After some bisection between v4.14.x tags, only v4.14.268 and newer tags
> > > can even be compiled using latest toolchain.
> > > (But still tons of warning, and tons of objdump warnings against
> > > insn_get_length()).
> > > 
> > > I'm not sure what's the normal practice for backports to such old branch.
> > > 
> > > Do you stable guys keep dedicated VMs loaded with older distro just for
> > > these old branches?
> > 
> > I don't, that's why those kernels can be built with newer versions of
> > gcc.
> > 
> > Your distro should have a version of gcc-10 or gcc-9 that can be
> > installed, right?
> 
> This may sounds like a meme, but I'm really using Archlinux for my VM
> and host, and it doesn't provide older GCC at all.

Archlinux does provide older gcc versions, that's what I use.

It still supports gcc11 in the main repo, and there is gcc10 in AUR as
well as gcc9.  Try those!

good luck!

greg k-h

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Qu Wenruo]

On 2022/8/22 16:30, Willy Tarreau wrote:
> On Mon, Aug 22, 2022 at 04:19:49PM +0800, Qu Wenruo wrote:
>>> Regardless, if you need an older compiler, just use these ones:
>>>
>>>      https://mirrors.edge.kernel.org/pub/tools/crosstool/
>>>
>>> They go back to 4.9.4 for x86, you'll surely find the right one for your
>>> usage. I've long used 4.7.4 for kernels up to 4.9 and 6.5 for 4.19 and
>>> above, so something within that area will surely match your needs.
>>
>> BTW, it would be way more awesome if the page can provide some hint on
>> the initial release date of the compilers.
>>
>> It would help a lot of choose the toolchain then.
>
> It wouldn't help, if you look closely, you'll notice that in the "other
> releases" section you have the most recent version of each of them. That
> does not preclude the existence of the branch earlier. For example gcc-9
> was released in 2019 and 9.5 was emitted 3 years later. That's quite an
> amplitude that doesn't help.

Maybe I'm totally wrong, but if GCC10.1 is released May 2020, and even
10.4 is released 2022, then shouldn't we expect the kernel releases
around 2020 can be compiled for all GCC 10.x releases?

Thus the initial release date should be a good enough hint for most cases.

If go this method, for v4.14 I guess I should go gcc 7.x, as gcc 7.1 is
released May 2017, even the latest 7.5 is released 2019.

Or is my uneducated guess completely wrong?

Thanks,
Qu

>
> Willy

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Greg KH]

On Mon, Aug 22, 2022 at 07:07:19PM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/8/22 16:30, Willy Tarreau wrote:
> > On Mon, Aug 22, 2022 at 04:19:49PM +0800, Qu Wenruo wrote:
> > > > Regardless, if you need an older compiler, just use these ones:
> > > > 
> > > >      https://mirrors.edge.kernel.org/pub/tools/crosstool/
> > > > 
> > > > They go back to 4.9.4 for x86, you'll surely find the right one for your
> > > > usage. I've long used 4.7.4 for kernels up to 4.9 and 6.5 for 4.19 and
> > > > above, so something within that area will surely match your needs.
> > > 
> > > BTW, it would be way more awesome if the page can provide some hint on
> > > the initial release date of the compilers.
> > > 
> > > It would help a lot of choose the toolchain then.
> > 
> > It wouldn't help, if you look closely, you'll notice that in the "other
> > releases" section you have the most recent version of each of them. That
> > does not preclude the existence of the branch earlier. For example gcc-9
> > was released in 2019 and 9.5 was emitted 3 years later. That's quite an
> > amplitude that doesn't help.
> 
> Maybe I'm totally wrong, but if GCC10.1 is released May 2020, and even
> 10.4 is released 2022, then shouldn't we expect the kernel releases
> around 2020 can be compiled for all GCC 10.x releases?
> 
> Thus the initial release date should be a good enough hint for most cases.
> 
> If go this method, for v4.14 I guess I should go gcc 7.x, as gcc 7.1 is
> released May 2017, even the latest 7.5 is released 2019.
> 
> Or is my uneducated guess completely wrong?

Try it and see!

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Qu Wenruo]

On 2022/8/22 17:00, Greg KH wrote:
> On Mon, Aug 22, 2022 at 04:13:03PM +0800, Qu Wenruo wrote:
>>
>>
>> On 2022/8/22 15:58, Greg KH wrote:
>>> On Mon, Aug 22, 2022 at 03:49:51PM +0800, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2022/8/22 15:33, Greg KH wrote:
>>>>> On Mon, Aug 22, 2022 at 03:24:53PM +0800, Qu Wenruo wrote:
>>>>>>
>>>>>>
>>>>>> On 2022/8/22 14:25, Greg KH wrote:
>>>>>>> On Mon, Aug 22, 2022 at 09:15:59AM +0800, Qu Wenruo wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> When backporting some btrfs specific patches to all LTS kernels, I found
>>>>>>>> v4.14.290 kernel unable to boot as a KVM guest with edk2-ovmf
>>>>>>>> (edk2-ovmf: 202205, qemu 7.0.0, libvirt 1:8.6.0).
>>>>>>>>
>>>>>>>> While all other LTS/stable branches (4.19.x, 5.4.x, 5.10.x, 5.15.x,
>>>>>>>> 5.18.x, 5.19.x) can boot without a hipccup.
>>>>>>>>
>>>>>>>> I tried the following configs, but none of them can even provide an
>>>>>>>> early output:
>>>>>>>>
>>>>>>>> - CONFIG_X86_VERBOSE_BOOTUP
>>>>>>>> - CONFIG_EARLY_PRINTK
>>>>>>>> - CONFIG_EARLY_PRINTK_EFI
>>>>>>>>
>>>>>>>> Is this a known bug or something new?
>>>>>>>
>>>>>>> Has this ever worked properly on this very old kernel tree?  If so, can
>>>>>>> you use 'git bisect' to find the offending commit?
>>>>>>
>>>>>> Unfortunately the initial v4.14 from upstream can not even be compiled.
>>>>>
>>>>> Really?  Try using an older version of gcc and you should be fine.  It
>>>>> did build properly back in 2017 when it was released :)
>>>>
>>>> Yeah, I'm pretty sure my toolchain is too new for v4.14.0. But my distro
>>>> only provides the latest and mostly upstream packages.
>>>>
>>>> It may be a even worse disaster to find a way to rollback to older
>>>> toolchains using my distro...
>>>>
>>>> Also my hardware may not be well suited for older kernels either.
>>>> (Zen 3 CPU used here)
>>>>
>>>> In fact, I even find it hard just to locate a v4.14.x tag that can compile.
>>>> After some bisection between v4.14.x tags, only v4.14.268 and newer tags
>>>> can even be compiled using latest toolchain.
>>>> (But still tons of warning, and tons of objdump warnings against
>>>> insn_get_length()).
>>>>
>>>> I'm not sure what's the normal practice for backports to such old branch.
>>>>
>>>> Do you stable guys keep dedicated VMs loaded with older distro just for
>>>> these old branches?
>>>
>>> I don't, that's why those kernels can be built with newer versions of
>>> gcc.
>>>
>>> Your distro should have a version of gcc-10 or gcc-9 that can be
>>> installed, right?
>>
>> This may sounds like a meme, but I'm really using Archlinux for my VM
>> and host, and it doesn't provide older GCC at all.
>
> Archlinux does provide older gcc versions, that's what I use.
>
> It still supports gcc11 in the main repo, and there is gcc10 in AUR as
> well as gcc9.  Try those!

Thanks a lot, didn't even notice the gcc11 in official repos.

Tried to compile gcc10 from AUR, which failed to compile.


Anyway, thanks to the advice from Willy, I got the pre-built crosstool
(gcc 7.5) set up, with some small tweaks like disabling
CONFIG_RANDOMIZE_BASE to workaround the RELOCS failure, it at least
compiles for v4.14.0.

Although there is still warning from test_gen_len:

  Warning: ffffffff818158cc:	0f ff e9             	ud0    %ecx,%ebp
  Warning: objdump says 3 bytes, but insn_get_length() says 2
  Warning: arch/x86/tools/test_get_len found difference at
<cpu_idle_poll>:ffffffff818159b0

And unfortunately v4.14 still fails to boot, even with GCC 7.5, which
provides an almost perfect (except above wanrings) build.

I also tried to reduce the CPUid, from host-passthru to qemu64, and
rebuild, no change (same test_get_len wanrings, same boot failure).

No clue at all now, would try older debian in a VM then.

If no change, a time period correct host maybe my next try...

Thanks,
Qu
>
> good luck!
>
> greg k-h

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Willy Tarreau]

On Mon, Aug 22, 2022 at 07:07:19PM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/8/22 16:30, Willy Tarreau wrote:
> > On Mon, Aug 22, 2022 at 04:19:49PM +0800, Qu Wenruo wrote:
> > > > Regardless, if you need an older compiler, just use these ones:
> > > > 
> > > >      https://mirrors.edge.kernel.org/pub/tools/crosstool/
> > > > 
> > > > They go back to 4.9.4 for x86, you'll surely find the right one for your
> > > > usage. I've long used 4.7.4 for kernels up to 4.9 and 6.5 for 4.19 and
> > > > above, so something within that area will surely match your needs.
> > > 
> > > BTW, it would be way more awesome if the page can provide some hint on
> > > the initial release date of the compilers.
> > > 
> > > It would help a lot of choose the toolchain then.
> > 
> > It wouldn't help, if you look closely, you'll notice that in the "other
> > releases" section you have the most recent version of each of them. That
> > does not preclude the existence of the branch earlier. For example gcc-9
> > was released in 2019 and 9.5 was emitted 3 years later. That's quite an
> > amplitude that doesn't help.
> 
> Maybe I'm totally wrong, but if GCC10.1 is released May 2020, and even
> 10.4 is released 2022, then shouldn't we expect the kernel releases
> around 2020 can be compiled for all GCC 10.x releases?
> 
> Thus the initial release date should be a good enough hint for most cases.

If you speak about initial release, yes that should generally be a valid
assumption.

> If go this method, for v4.14 I guess I should go gcc 7.x, as gcc 7.1 is
> released May 2017, even the latest 7.5 is released 2019.

Then it should definitely work. But I think you're spending way too much
time comparing dates and discussing on the subject. By the time it took
to check these dates, you could already have downloaded one such compiler
and built a kernel to verify it did build correctly ;-)

Willy

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Willy Tarreau]

On Mon, Aug 22, 2022 at 07:43:18PM +0800, Qu Wenruo wrote:
> Tried to compile gcc10 from AUR, which failed to compile.
> 
> 
> Anyway, thanks to the advice from Willy, I got the pre-built crosstool
> (gcc 7.5) set up, with some small tweaks like disabling
> CONFIG_RANDOMIZE_BASE to workaround the RELOCS failure, it at least
> compiles for v4.14.0.
> 
> Although there is still warning from test_gen_len:
> 
>  Warning: ffffffff818158cc:	0f ff e9             	ud0    %ecx,%ebp
>  Warning: objdump says 3 bytes, but insn_get_length() says 2
>  Warning: arch/x86/tools/test_get_len found difference at
> <cpu_idle_poll>:ffffffff818159b0

Strange, sounds like a binutils issue though I could be wrong.

> And unfortunately v4.14 still fails to boot, even with GCC 7.5, which
> provides an almost perfect (except above wanrings) build.
> 
> I also tried to reduce the CPUid, from host-passthru to qemu64, and
> rebuild, no change (same test_get_len wanrings, same boot failure).
> 
> No clue at all now, would try older debian in a VM then.

I suggest that instead of switching distros you should rather first
try 4.14.0 to verify if there was a regression affecting your system.
And if so, then a bisect will certainly be welcome. If it still does
not work, then maybe a different distro could help, though I doubt it.

Willy

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Qu Wenruo]

On 2022/8/22 19:53, Willy Tarreau wrote:
> On Mon, Aug 22, 2022 at 07:07:19PM +0800, Qu Wenruo wrote:
>>
>>
>> On 2022/8/22 16:30, Willy Tarreau wrote:
>>> On Mon, Aug 22, 2022 at 04:19:49PM +0800, Qu Wenruo wrote:
>>>>> Regardless, if you need an older compiler, just use these ones:
>>>>>
>>>>>       https://mirrors.edge.kernel.org/pub/tools/crosstool/
>>>>>
>>>>> They go back to 4.9.4 for x86, you'll surely find the right one for your
>>>>> usage. I've long used 4.7.4 for kernels up to 4.9 and 6.5 for 4.19 and
>>>>> above, so something within that area will surely match your needs.
>>>>
>>>> BTW, it would be way more awesome if the page can provide some hint on
>>>> the initial release date of the compilers.
>>>>
>>>> It would help a lot of choose the toolchain then.
>>>
>>> It wouldn't help, if you look closely, you'll notice that in the "other
>>> releases" section you have the most recent version of each of them. That
>>> does not preclude the existence of the branch earlier. For example gcc-9
>>> was released in 2019 and 9.5 was emitted 3 years later. That's quite an
>>> amplitude that doesn't help.
>>
>> Maybe I'm totally wrong, but if GCC10.1 is released May 2020, and even
>> 10.4 is released 2022, then shouldn't we expect the kernel releases
>> around 2020 can be compiled for all GCC 10.x releases?
>>
>> Thus the initial release date should be a good enough hint for most cases.
>
> If you speak about initial release, yes that should generally be a valid
> assumption.
>
>> If go this method, for v4.14 I guess I should go gcc 7.x, as gcc 7.1 is
>> released May 2017, even the latest 7.5 is released 2019.
>
> Then it should definitely work. But I think you're spending way too much
> time comparing dates and discussing on the subject. By the time it took
> to check these dates, you could already have downloaded one such compiler
> and built a kernel to verify it did build correctly ;-)

That's completely true.

Except the fact that, even with time period correct tool chain (gcc
7.5), the compiled v4.14.x kernel (tried both 4.14.0 and latest
4.14.290), neither can boot nor cause any early boot message to pop up. :(

Anyway, thank you very much for the toolchain part, it would definitely
help for my future adventure related to stable kernels (looking at tons
of v4.12 related branches with sad face).

Thanks,
Qu

>
> Willy

Re: LTS kernel Linux 4.14.290 unable to boot with edk2-ovmf (x86_64 UEFI runtime)

[Qu Wenruo]

On 2022/8/22 19:58, Willy Tarreau wrote:
> On Mon, Aug 22, 2022 at 07:43:18PM +0800, Qu Wenruo wrote:
>> Tried to compile gcc10 from AUR, which failed to compile.
>>
>>
>> Anyway, thanks to the advice from Willy, I got the pre-built crosstool
>> (gcc 7.5) set up, with some small tweaks like disabling
>> CONFIG_RANDOMIZE_BASE to workaround the RELOCS failure, it at least
>> compiles for v4.14.0.
>>
>> Although there is still warning from test_gen_len:
>>
>>   Warning: ffffffff818158cc:	0f ff e9             	ud0    %ecx,%ebp
>>   Warning: objdump says 3 bytes, but insn_get_length() says 2
>>   Warning: arch/x86/tools/test_get_len found difference at
>> <cpu_idle_poll>:ffffffff818159b0
>
> Strange, sounds like a binutils issue though I could be wrong.

I'm using CROSS_COMPILE= option, which should cover the objdump from the
prebuilt "x86_64-linux-objdump" from that precompiled 7.5 crosstool.

>
>> And unfortunately v4.14 still fails to boot, even with GCC 7.5, which
>> provides an almost perfect (except above wanrings) build.
>>
>> I also tried to reduce the CPUid, from host-passthru to qemu64, and
>> rebuild, no change (same test_get_len wanrings, same boot failure).
>>
>> No clue at all now, would try older debian in a VM then.
>
> I suggest that instead of switching distros you should rather first
> try 4.14.0 to verify if there was a regression affecting your system.

Already tried, the v4.14 above really means v4.14.0 (aka v4.14 tag
directly from upstream, not from stable).

And the latest v4.14.290 can not boot neither, even rebuilt using that
toolchain.

> And if so, then a bisect will certainly be welcome. If it still does
> not work, then maybe a different distro could help, though I doubt it.

Will try debian for now, or even try some older hardware if I could find...

Thanks,
Qu

>
> Willy