Re: gcc-10: kernel stack is corrupted and fails to boot

[Arnd Bergmann <arnd@arndb.de>]

On Wed, May 13, 2020 at 8:50 AM Kalle Valo <kvalo@codeaurora.org> wrote:
>
> Kalle Valo <kvalo@codeaurora.org> writes:
>
> > This motivated me to switch to using GCC 10.x and I noticed that you had
> > already upgraded crosstool so it was a trivial thing to do, awesome :)
> >
> > https://mirrors.edge.kernel.org/pub/tools/crosstool/
>
> And now I have a problem :) I first noticed that my x86 testbox is not
> booting when I compile the kernel with GCC 10.1.0 from crosstool. I
> didn't get any error messages so I just downgraded the compiler and the
> kernel was booting fine again. Next I decided to try GCC 10.1 with my
> x86 laptop and it also failed to boot, but this time I got kernel logs
> and saw this:
>
> Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: start_secodary+0x178/0x180
>
> Call Trace:
> dump_stack
> panic
> ? _raw_spin_unlock_irqrestore
> ? start_secondary
> __stack_chk_fail
> start_secondary
> secondary_startup
>
> (I wrote the above messages manually from a picture so expect typos)
>
> Then also on my x86 laptop I downgraded the compiler to GCC 8.1.0 (from
> crosstool), rebuilt the exactly same kernel version and the kernel
> booted without issues.
>
> I'm using 5.7.0-rc4-wt-ath+ which is basically v5.7-rc4 plus latest
> wireless patches, and I doubt the wireless patches are making any
> difference this early in the boot. All compilers I use are prebuilt
> binaries from kernel.org crosstool repo[1] with addition of ccache
> v3.4.1 to speed up my builds.
>
> Any ideas? How should I debug this further?

At least if it fails reproducibly, it's probably not too hard to drill
down further. Some ideas:

* I'd first try to reproduce it in qemu. Since you don't even need
  any user space or modules, I would simply try
  $ qemu-system-x86_64 -nographic -monitor none -append
"console=ttyS0" -serial stdio -smp 4 -kernel arch/x86/boot/bzImage
  I tried it here with an x86 defconfig linux-next kernel but did not
  run into the problem you described. If you share your .config,
  I can try reproducing with that as well. Once there is a reproducer
  in qemu, it should be trivial to step through it using gdb.

* There are still two prerelease compiler versions on kernel.org,
  from February and from April. You can try each one to see
  if this was a recent regression. It's also possible that there is
  a problem with my specific builds of gcc-10.1, and that the
  compiler is actually fine for others.The gcc-10 packages in
  Fedora/Debian/Ubuntu are probably better tested.

       Arnd