Re: Linux 6.1-rc3

From: Linus Torvalds <torvalds@linux-foundation.org>
To: Guenter Roeck <linux@roeck-us.net>, Arnd Bergmann <arnd@arndb.de>,
	Russell King <rmk+kernel@armlinux.org.uk>,
	Masahiro Yamada <masahiroy@kernel.org>
Cc: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Miguel Ojeda <ojeda@kernel.org>
Subject: Re: Linux 6.1-rc3
Date: Tue, 1 Nov 2022 10:29:01 -0700	[thread overview]
Message-ID: <CAHk-=wi+5sRXL9fTHQRiR9zVEFDPhQceWACBG3QQ=9xoRVmR8g@mail.gmail.com> (raw)
In-Reply-To: <20221101170015.GA1314742@roeck-us.net>

[ Adding Arnd and Masahiro in case they remember what the secret sauce
for finding those cases was ]

On Tue, Nov 1, 2022 at 10:00 AM Guenter Roeck <linux@roeck-us.net> wrote:
>
> There is one spurious build error. It does not happen all the time,
> and even on the same SHA it is not easy to reproduce. If I see it,
> I may see it on one server reliably but not at all on another with
> almost the same hardware configuration.
>
> Building powerpc:allnoconfig ... failed
> --------------
> Error log:
> Inconsistent kallsyms data
> Try make KALLSYMS_EXTRA_PASS=1 as a workaround

The random "Inconsistent kallsyms data" issue is something that we've
had for over a decade.

It plagued the arm people for a long while, and if I recall correctly
rmk was fighting it until he figured out some workaround, and then it
got very rare (but still continued to happen for other cases).

But when it happens, it's some random code or data layout thing, and
then reverting a particular commit will "fix" it - but it tends to
happen with specific configurations and compiler versions, and when
people trigger it it tends to be quite random.

Arnd fixed once case of it (again on ARM) last year, triggered by some
lld behavior: efe6e3068067 ("kallsyms: fix nonconverging kallsyms
table with lld")

And looking around, the fix for the issue that plagued rmk for so long
was fixed by this one: commit 9973290ce20a ("ARM: 7428/1: Prevent
KALLSYM size mismatch on ARM").

And since you see it on powerpc, I suspect it's some variation of
commit 516d980f8541 ("scripts/kallsyms: skip ppc compiler stub
*.long_branch.* / *.plt_branch.*") that you reported a couple of years
ago.

IOW, I expect it is - once again - some random linker-generated stub
that ends up causing problems, where the re-link stage ends up being
unstable because of some subtle alignment issue or other. I think
zero-sized symbols have often been involved.

The warning is extremely annoying, but while that warning *could* be a
sign of some serious problem in general, in this "random linker output
instability" case it should be harmless. Just very very annoying.

                    Linus