From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9B3E15AE; Tue, 20 Jun 2023 04:19:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 46C42C433CD; Tue, 20 Jun 2023 04:19:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687234778; bh=NyGsLeW/FztEsf9dVsONe46AWBlbnmJSRV1nyDFVDmM=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=SyYSgxyRvMQ8bpj+1tiIcMRWk9Q70jG/7TLqBfKKy3zECmTd3QV/+M7adHuWfbg5z HswzlZkElK3j8hqijaUg3Ant3XNG94VmIBvI7gEnLscldIiyESkU/2vW97lJfx6pW9 SxSEI8ds3VW0pXcBTMlMC2jBZjnrZmPk+ONpMHyQCTyxE9PpzshdH+Y9yo0Knpkt9x +NlNo4ZmmOKQT4VUiAX6qfPMdu95kzDFiSRFMkvZlAytkmrkTpK1uxBNoyCiWllxUv E1N4G3G7CrmWVXbzHGP4rRuCm9CzTe/J4FboxWm9y+PwWvNDemNb8YVszXybGz+iwK ZP5I3S++JFuTw== Received: by mail-oo1-f51.google.com with SMTP id 006d021491bc7-55b21c3f359so2660717eaf.0; Mon, 19 Jun 2023 21:19:38 -0700 (PDT) X-Gm-Message-State: AC+VfDzA1nk5Vj6tUZwoAfM1PJo1qrQd4epmieO5NMsrJKCIzALprxnH z1RptUcevx+XJr1NgbIBKhdQc0cNtZWWME6O/kk= X-Google-Smtp-Source: ACHHUZ50dBIk80zRdS9T10z5yvKFkNOL2YvbT9WJB+0+jV9iiQvBtez20V9tVjAvd8EezQ0PGvBQj2XapoOMQguh5H0= X-Received: by 2002:a4a:c607:0:b0:558:b793:f10c with SMTP id l7-20020a4ac607000000b00558b793f10cmr5116789ooq.3.1687234777440; Mon, 19 Jun 2023 21:19:37 -0700 (PDT) Precedence: bulk X-Mailing-List: llvm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <17c91d37-7d9c-0df4-2438-2b30ca0b5777@collabora.com> <878rdlk9bi.fsf@rcn-XPS-13-9305.i-did-not-set--mail-host-address--so-tickle-me> <875y8ok9b5.fsf@rcn-XPS-13-9305.i-did-not-set--mail-host-address--so-tickle-me> <87353ok78h.fsf@rcn-XPS-13-9305.i-did-not-set--mail-host-address--so-tickle-me> <2023052247-bobtail-factsheet-d104@gregkh> In-Reply-To: From: Masahiro Yamada Date: Tue, 20 Jun 2023 13:19:01 +0900 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v4] Makefile.compiler: replace cc-ifversion with compiler-specific macros To: Shreeya Patel Cc: Greg KH , Maksim Panchenko , =?UTF-8?Q?Ricardo_Ca=C3=B1uelo?= , Michal Marek , Linux Kernel Mailing List , clang-built-linux , Bill Wendling , Nathan Chancellor , regressions@lists.linux.dev, "gustavo.padovan@collabora.com" , Guillaume Charles Tucker , denys.f@collabora.com, Nick Desaulniers , kernelci@lists.linux.dev, Collabora Kernel ML Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Jun 12, 2023 at 7:10=E2=80=AFPM Shreeya Patel wrote: > > Hi Masahiro, > > > On 24/05/23 02:57, Nick Desaulniers wrote: > > On Tue, May 23, 2023 at 3:27=E2=80=AFAM Shreeya Patel > > wrote: > >> Hi Nick and Masahiro, > >> > >> On 23/05/23 01:22, Nick Desaulniers wrote: > >>> On Mon, May 22, 2023 at 9:52=E2=80=AFAM Greg KH wrote: > >>>> On Mon, May 22, 2023 at 12:09:34PM +0200, Ricardo Ca=C3=B1uelo wrote= : > >>>>> On vie, may 19 2023 at 08:57:24, Nick Desaulniers wrote: > >>>>>> It could be; if the link order was changed, it's possible that thi= s > >>>>>> target may be hitting something along the lines of: > >>>>>> https://isocpp.org/wiki/faq/ctors#static-init-order i.e. the "stat= ic > >>>>>> initialization order fiasco" > >>>>>> > >>>>>> I'm struggling to think of how this appears in C codebases, but I > >>>>>> swear years ago I had a discussion with GKH (maybe?) about this. I > >>>>>> think I was playing with converting Kbuild to use Ninja rather tha= n > >>>>>> Make; the resulting kernel image wouldn't boot because I had modif= ied > >>>>>> the order the object files were linked in. If you were to randoml= y > >>>>>> shuffle the object files in the kernel, I recall some hazard that = may > >>>>>> prevent boot. > >>>>> I thought that was specifically a C++ problem? But then again, the > >>>>> kernel docs explicitly say that the ordering of obj-y goals in kbui= ld is > >>>>> significant in some instances [1]: > >>>> Yes, it matters, you can not change it. If you do, systems will bre= ak. > >>>> It is the only way we have of properly ordering our init calls withi= n > >>>> the same "level". > >>> Ah, right it was the initcall ordering. Thanks for the reminder. > >>> > >>> (There's a joke in there similar to the use of regexes to solve a > >>> problem resulting in two new problems; initcalls have levels for > >>> ordering, but we still have (unexpressed) dependencies between calls > >>> of the same level; brittle!). > >>> > >>> +Maksim, since that might be relevant info for the BOLT+Kernel work. > >>> > >>> Ricardo, > >>> https://elinux.org/images/e/e8/2020_ELCE_initcalls_myjosserand.pdf > >>> mentions that there's a kernel command line param `initcall_debug`. > >>> Perhaps that can be used to see if > >>> 5750121ae7382ebac8d47ce6d68012d6cd1d7926 somehow changed initcall > >>> ordering, resulting in a config that cannot boot? > >> > >> Here are the links to Lava jobs ran with initcall_debug added to the > >> kernel command line. > >> > >> 1. Where regression happens (5750121ae7382ebac8d47ce6d68012d6cd1d7926) > >> https://lava.collabora.dev/scheduler/job/10417706 > >> > >> > >> 2. With a revert of the commit 5750121ae7382ebac8d47ce6d68012d6cd1d792= 6 > >> https://lava.collabora.dev/scheduler/job/10418012 > >> > > Thanks! > > > > Yeah, I can see a diff in the initcall ordering as a result of > > commit 5750121ae738 ("kbuild: list sub-directories in ./Kbuild") > > > > https://gist.github.com/nickdesaulniers/c09db256e42ad06b90842a4bb85cc0f= 4 > > > > Not just different orderings, but some initcalls seem unique to the > > before vs. after, which is troubling. (example init_events and > > init_fs_sysctls respectively) > > > > That isn't conclusive evidence that changes to initcall ordering are > > to blame, but I suspect confirming that precisely to be very very time > > consuming. > > > > Masahiro, what are your thoughts on reverting 5750121ae738? There are > > conflicts in Kbuild and Makefile when reverting 5750121ae738 on > > mainline. > > I'm not sure if you followed the conversation but we are still seeing > this regression with the latest kernel builds and would like to know if > you plan to revert 5750121ae738? Reverting 5750121ae738 does not solve the issue because the issue happens even before 5750121ae738. multi_v7_defconfig + debug.config + CONFIG_MODULES=3Dn fails to boot in the same way. The revert would hide the issue on a particular build setup. I submitted a patch to more pin-point the issue. Let's see how it goes. https://lore.kernel.org/lkml/ZJEni98knMMkU%2Fcl@buildd.core.avm.de/T/#t (BTW, the initcall order is unrelated) > > > Thanks, > Shreeya Patel > > >> > >> Thanks, > >> Shreeya Patel > >> > > -- Best Regards Masahiro Yamada