Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

From: Sedat Dilek <sedat.dilek@gmail.com>
To: Jiri Slaby <jslaby@suse.cz>
Cc: masahiroy@kernel.org, linux-kbuild@vger.kernel.org,
	linux-kernel@vger.kernel.org,
	Michal Marek <michal.lkml@markovi.net>,
	Nick Desaulniers <ndesaulniers@google.com>,
	Nathan Chancellor <nathan@kernel.org>,
	Sami Tolvanen <samitolvanen@google.com>
Subject: Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o
Date: Fri, 17 Jun 2022 12:41:10 +0200	[thread overview]
Message-ID: <CA+icZUXDGdPrPKUnevt99LUpTRPe=ogqF33uHQRYrQ6Kh-iTAw@mail.gmail.com> (raw)
In-Reply-To: <CA+icZUUSTcrJqZB-gwNYt5objVg1J5+Ous6_hof0_A6eVCM-Kg@mail.gmail.com>

On Fri, Jun 17, 2022 at 12:35 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > >
> > > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@suse.cz> wrote:
> > > >
> > > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > > is lost.
> > > >
> > > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > > of it.
> > > >
> > > > Cc: Sedat Dilek <sedat.dilek@gmail.com>
> > > > Cc: Masahiro Yamada <masahiroy@kernel.org>
> > > > Cc: Michal Marek <michal.lkml@markovi.net>
> > > > Cc: Nick Desaulniers <ndesaulniers@google.com>
> > > > Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> > > > Signed-off-by: Jiri Slaby <jslaby@suse.cz>
> > >
> > > Nice catch.
> > > ( I have seen some slowdown in my build-time. Will test and report. )
> > >
> > > Reviewed-by: Sedat Dilek <sedat.dilek@gmail.com>
> > >
> >
> > No measurable difference in build-time.
> >
> > Tested-by: Sedat Dilek <sedat.dilek@gmail.com> # LLVM-14 (x86-64)
> >
>
> [ CC Nathan and Sami ]
>
> I re-checked as my build-time takes approx. 15mins more than usual
> with Linux v5.17.
>
> [ Perf - seconds time elapsed ]
> v5.17: approx. 12500
> v5.19: approx. 13500
>
> +1.000 secs = +16,67mins
>
> I am still investigating.
>
> I use LLVM v14.0.4 and CONFIG_LTO_CLANG_THIN=y.
>
> When I check for...
>
> $ egrep 'Makefile.vmlinux_o|jobserver-exec|link-vmlinux.sh'
> build-log_5.19.0-rc2-2-amd64-clang14-lto.txt
> 61414:  sh scripts/link-vmlinux.sh "ld.lld" "-m elf_x86_64
> --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5"
> "--emit-relocs --discard-none -z max-page-size=0x200000
> --build-id=sha1  -X --orphan-handling=warn";  true
> 61426:+ make -f ./scripts/Makefile.vmlinux_o
> 61427:  python3 ./scripts/jobserver-exec perl
> scripts/generate_initcall_order.pl arch/x86/kernel/head_64.o
> arch/x86/kernel/head64.o arch/x86/kernel/ebda.o
> arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a
> arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a
> fs/built-in.a ipc/built-in.a security/built-i
> n.a crypto/built-in.a block/built-in.a lib/built-in.a
> arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a
> drivers/built-in.a sound/built-in.a net/built-in.a virt/buil
> t-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a
> arch/x86/video/built-in.a > .tmp_initcalls.lds
> 61693:+ echo vmlinux: scripts/link-vmlinux.sh
>
> Plus vmlinux.o
>
> 61428:  ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm
> -import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds
> --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o
> arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o
> init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a
> certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a
> security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a
> arch/x86/lib/built-in.a  lib/lib.a  arch/x86/lib/lib.a
> drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a
> arch/x86/pci/built-in.a arch/x86/power/built-in.a
> arch/x86/video/built-in.a --no-whole-archive --start-group
> --end-group  ; ./tools/objtool/objtool  --hacks=jump_label
> --hacks=noinstr   --mcount  --orc  --retpoline    --static-call
> --uaccess  --link     --
> link vmlinux.o
>
> Pooh, truncated via Gmail? Attached as a file.
>
> Checking via ps...
>
> $ ps -ef | grep l[l]d > lld-jobserver.txt
>
> ...and top util shows me only one lld link-job is running (file attached).
>
> Unsure, if in my setup lld linker benefits from jobserver feature or not.
>
> Am I missing something?
>
> My kernel-config is attached.
>

Found this...

$ ld.lld-14 --help

  --thinlto-jobs=<value>  Number of ThinLTO jobs. Default to --threads=

-Sedat-