From: Masahiro Yamada <masahiroy@kernel.org>
To: Shreeya Patel <shreeya.patel@collabora.com>
Cc: "Greg KH" <gregkh@linuxfoundation.org>,
"Maksim Panchenko" <maks@meta.com>,
"Ricardo Cañuelo" <ricardo.canuelo@collabora.com>,
"Michal Marek" <michal.lkml@markovi.net>,
"Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
clang-built-linux <llvm@lists.linux.dev>,
"Bill Wendling" <morbo@google.com>,
"Nathan Chancellor" <nathan@kernel.org>,
regressions@lists.linux.dev,
"gustavo.padovan@collabora.com" <gustavo.padovan@collabora.com>,
"Guillaume Charles Tucker" <guillaume.tucker@collabora.com>,
denys.f@collabora.com,
"Nick Desaulniers" <ndesaulniers@google.com>,
kernelci@lists.linux.dev,
"Collabora Kernel ML" <kernel@collabora.com>
Subject: Re: [PATCH v4] Makefile.compiler: replace cc-ifversion with compiler-specific macros
Date: Tue, 20 Jun 2023 13:19:01 +0900 [thread overview]
Message-ID: <CAK7LNAS8Y9syCiHMO2r75D6hgCSsDDvJ_=VMKpYqjondnbSZjw@mail.gmail.com> (raw)
In-Reply-To: <a037a08c-44c4-24e8-1cba-7e4e8b21ffaa@collabora.com>
On Mon, Jun 12, 2023 at 7:10 PM Shreeya Patel
<shreeya.patel@collabora.com> wrote:
>
> Hi Masahiro,
>
>
> On 24/05/23 02:57, Nick Desaulniers wrote:
> > On Tue, May 23, 2023 at 3:27 AM Shreeya Patel
> > <shreeya.patel@collabora.com> wrote:
> >> Hi Nick and Masahiro,
> >>
> >> On 23/05/23 01:22, Nick Desaulniers wrote:
> >>> On Mon, May 22, 2023 at 9:52 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >>>> On Mon, May 22, 2023 at 12:09:34PM +0200, Ricardo Cañuelo wrote:
> >>>>> On vie, may 19 2023 at 08:57:24, Nick Desaulniers <ndesaulniers@google.com> wrote:
> >>>>>> It could be; if the link order was changed, it's possible that this
> >>>>>> target may be hitting something along the lines of:
> >>>>>> https://isocpp.org/wiki/faq/ctors#static-init-order i.e. the "static
> >>>>>> initialization order fiasco"
> >>>>>>
> >>>>>> I'm struggling to think of how this appears in C codebases, but I
> >>>>>> swear years ago I had a discussion with GKH (maybe?) about this. I
> >>>>>> think I was playing with converting Kbuild to use Ninja rather than
> >>>>>> Make; the resulting kernel image wouldn't boot because I had modified
> >>>>>> the order the object files were linked in. If you were to randomly
> >>>>>> shuffle the object files in the kernel, I recall some hazard that may
> >>>>>> prevent boot.
> >>>>> I thought that was specifically a C++ problem? But then again, the
> >>>>> kernel docs explicitly say that the ordering of obj-y goals in kbuild is
> >>>>> significant in some instances [1]:
> >>>> Yes, it matters, you can not change it. If you do, systems will break.
> >>>> It is the only way we have of properly ordering our init calls within
> >>>> the same "level".
> >>> Ah, right it was the initcall ordering. Thanks for the reminder.
> >>>
> >>> (There's a joke in there similar to the use of regexes to solve a
> >>> problem resulting in two new problems; initcalls have levels for
> >>> ordering, but we still have (unexpressed) dependencies between calls
> >>> of the same level; brittle!).
> >>>
> >>> +Maksim, since that might be relevant info for the BOLT+Kernel work.
> >>>
> >>> Ricardo,
> >>> https://elinux.org/images/e/e8/2020_ELCE_initcalls_myjosserand.pdf
> >>> mentions that there's a kernel command line param `initcall_debug`.
> >>> Perhaps that can be used to see if
> >>> 5750121ae7382ebac8d47ce6d68012d6cd1d7926 somehow changed initcall
> >>> ordering, resulting in a config that cannot boot?
> >>
> >> Here are the links to Lava jobs ran with initcall_debug added to the
> >> kernel command line.
> >>
> >> 1. Where regression happens (5750121ae7382ebac8d47ce6d68012d6cd1d7926)
> >> https://lava.collabora.dev/scheduler/job/10417706
> >> <https://lava.collabora.dev/scheduler/job/10417706>
> >>
> >> 2. With a revert of the commit 5750121ae7382ebac8d47ce6d68012d6cd1d7926
> >> https://lava.collabora.dev/scheduler/job/10418012
> >> <https://lava.collabora.dev/scheduler/job/10418012>
> > Thanks!
> >
> > Yeah, I can see a diff in the initcall ordering as a result of
> > commit 5750121ae738 ("kbuild: list sub-directories in ./Kbuild")
> >
> > https://gist.github.com/nickdesaulniers/c09db256e42ad06b90842a4bb85cc0f4
> >
> > Not just different orderings, but some initcalls seem unique to the
> > before vs. after, which is troubling. (example init_events and
> > init_fs_sysctls respectively)
> >
> > That isn't conclusive evidence that changes to initcall ordering are
> > to blame, but I suspect confirming that precisely to be very very time
> > consuming.
> >
> > Masahiro, what are your thoughts on reverting 5750121ae738? There are
> > conflicts in Kbuild and Makefile when reverting 5750121ae738 on
> > mainline.
>
> I'm not sure if you followed the conversation but we are still seeing
> this regression with the latest kernel builds and would like to know if
> you plan to revert 5750121ae738?
Reverting 5750121ae738 does not solve the issue
because the issue happens even before 5750121ae738.
multi_v7_defconfig + debug.config + CONFIG_MODULES=n
fails to boot in the same way.
The revert would hide the issue on a particular build setup.
I submitted a patch to more pin-point the issue.
Let's see how it goes.
https://lore.kernel.org/lkml/ZJEni98knMMkU%2Fcl@buildd.core.avm.de/T/#t
(BTW, the initcall order is unrelated)
>
>
> Thanks,
> Shreeya Patel
>
> >>
> >> Thanks,
> >> Shreeya Patel
> >>
> >
--
Best Regards
Masahiro Yamada
next prev parent reply other threads:[~2023-06-20 4:19 UTC|newest]
Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-07 4:59 [PATCH v3 0/5] fix debug info for asm and DEBUG_INFO_SPLIT Nick Desaulniers
2022-09-07 4:59 ` [PATCH v3 1/5] x86/boot/compressed: prefer cc-option for CFLAGS additions Nick Desaulniers
2022-09-09 21:05 ` Masahiro Yamada
2022-09-07 4:59 ` [PATCH v3 2/5] Makefile.compiler: Use KBUILD_AFLAGS for as-option Nick Desaulniers
2022-09-09 21:06 ` Masahiro Yamada
2022-09-07 4:59 ` [PATCH v3 3/5] Makefile.compiler: replace cc-ifversion with compiler-specific macros Nick Desaulniers
2022-09-09 21:59 ` Masahiro Yamada
2022-09-07 4:59 ` [PATCH v3 4/5] Makefile.debug: re-enable debug info for .S files Nick Desaulniers
2022-09-07 4:59 ` [PATCH v3 5/5] Makefile.debug: set -g unconditional on CONFIG_DEBUG_INFO_SPLIT Nick Desaulniers
2022-09-09 20:56 ` Masahiro Yamada
2022-09-09 21:02 ` Masahiro Yamada
2022-09-19 17:08 ` [PATCH v4] Makefile.compiler: replace cc-ifversion with compiler-specific macros Nick Desaulniers
2022-09-23 19:44 ` Masahiro Yamada
2022-09-24 14:28 ` Masahiro Yamada
2022-09-25 1:22 ` Masahiro Yamada
2022-09-26 21:41 ` Nick Desaulniers
2023-04-27 11:53 ` Shreeya Patel
2023-04-28 7:41 ` Thorsten Leemhuis
2023-05-02 9:48 ` Shreeya Patel
2023-05-02 11:58 ` Linux regression tracking (Thorsten Leemhuis)
2023-04-28 17:27 ` Nick Desaulniers
2023-05-03 21:02 ` Shreeya Patel
2023-05-03 21:15 ` Nick Desaulniers
2023-05-03 22:33 ` Shreeya Patel
2023-05-15 23:01 ` Nick Desaulniers
2023-05-17 8:34 ` Shreeya Patel
2023-05-17 15:39 ` Ricardo Cañuelo
2023-05-17 16:27 ` Nick Desaulniers
2023-05-18 14:23 ` Ricardo Cañuelo
2023-05-18 21:12 ` Nick Desaulniers
2023-05-19 8:35 ` Ricardo Cañuelo
2023-05-19 15:57 ` Nick Desaulniers
2023-05-22 10:09 ` Ricardo Cañuelo
2023-05-22 16:52 ` Greg KH
2023-05-22 19:52 ` Nick Desaulniers
2023-05-22 20:01 ` Greg KH
2023-05-22 20:16 ` Nick Desaulniers
2023-05-23 10:27 ` Shreeya Patel
2023-05-23 21:27 ` Nick Desaulniers
2023-06-12 10:10 ` Shreeya Patel
2023-06-20 4:19 ` Masahiro Yamada [this message]
2023-07-10 12:09 ` Linux regression tracking (Thorsten Leemhuis)
2023-07-11 11:16 ` Shreeya Patel
2023-08-29 11:28 ` Linux regression tracking (Thorsten Leemhuis)
2023-09-11 10:05 ` Thorsten Leemhuis
2023-09-15 9:33 ` Shreeya Patel
2023-09-30 10:24 ` Masahiro Yamada
2022-09-19 17:30 ` [PATCH v4] Makefile.debug: set -g unconditional on CONFIG_DEBUG_INFO_SPLIT Nick Desaulniers
2022-09-24 1:34 ` Masahiro Yamada
2022-09-19 17:45 ` [PATCH v4] Makefile.debug: re-enable debug info for .S files Nick Desaulniers
2022-09-24 2:11 ` Masahiro Yamada
2022-09-24 2:20 ` Nick Desaulniers
2022-09-24 6:43 ` Masahiro Yamada
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAK7LNAS8Y9syCiHMO2r75D6hgCSsDDvJ_=VMKpYqjondnbSZjw@mail.gmail.com' \
--to=masahiroy@kernel.org \
--cc=denys.f@collabora.com \
--cc=gregkh@linuxfoundation.org \
--cc=guillaume.tucker@collabora.com \
--cc=gustavo.padovan@collabora.com \
--cc=kernel@collabora.com \
--cc=kernelci@lists.linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=llvm@lists.linux.dev \
--cc=maks@meta.com \
--cc=michal.lkml@markovi.net \
--cc=morbo@google.com \
--cc=nathan@kernel.org \
--cc=ndesaulniers@google.com \
--cc=regressions@lists.linux.dev \
--cc=ricardo.canuelo@collabora.com \
--cc=shreeya.patel@collabora.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).