Re: [PATCH kernel] powerpc/llvm/lto: Allow LLVM LTO builds

From: Nick Desaulniers <ndesaulniers@google.com>
To: Nathan Chancellor <nathan@kernel.org>
Cc: Alexey Kardashevskiy <aik@ozlabs.ru>,
	llvm@lists.linux.dev, linuxppc-dev@lists.ozlabs.org,
	 Nicholas Piggin <npiggin@gmail.com>,
	Michael Ellerman <mpe@ellerman.id.au>,
	 Sathvika Vasireddy <sv@linux.ibm.com>,
	Russell Currey <ruscur@russell.cc>,
	 "Naveen N . Rao" <naveen.n.rao@linux.vnet.ibm.com>,
	Sami Tolvanen <samitolvanen@google.com>
Subject: Re: [PATCH kernel] powerpc/llvm/lto: Allow LLVM LTO builds
Date: Mon, 9 May 2022 11:19:55 -0700	[thread overview]
Message-ID: <CAKwvOdk_wNfo1GA1MsOehTuTpBQqNi3xhHHyLmkpyTr0wPj1Tg@mail.gmail.com> (raw)
In-Reply-To: <YnlYemdzBDCobK/d@dev-arch.thelio-3990X>

On Mon, May 9, 2022 at 11:08 AM Nathan Chancellor <nathan@kernel.org> wrote:
>
> Hi Alexey,
>
> On Mon, May 09, 2022 at 05:42:59PM +1000, Alexey Kardashevskiy wrote:
> >
> >
> > On 5/9/22 15:18, Alexey Kardashevskiy wrote:
> > >
> > >
> > > On 5/4/22 07:21, Nick Desaulniers wrote:
> > > > On Thu, Apr 28, 2022 at 11:46 PM Alexey Kardashevskiy
> > > > <aik@ozlabs.ru> wrote:
> > > > >
> > > > > This enables LTO_CLANG builds on POWER with the upstream version of
> > > > > LLVM.
> > > > >
> > > > > LTO optimizes the output vmlinux binary and this may affect the FTP
> > > > > alternative section if alt branches use "bc" (Branch Conditional) which
> > > > > is limited by 16 bit offsets. This shows up in errors like:
> > > > >
> > > > > ld.lld: error: InputSection too large for range extension thunk
> > > > > vmlinux.o:(__ftr_alt_97+0xF0)
> > > > >
> > > > > This works around the issue by replacing "bc" in FTR_SECTION_ELSE with
> > > > > "b" which allows 26 bit offsets.
> > > > >
> > > > > This catches the problem instructions in vmlinux.o before it LTO'ed:
> > > > >
> > > > > $ objdump -d -M raw -j __ftr_alt_97 vmlinux.o | egrep '\S+\s*\<bc\>'
> > > > >    30:   00 00 82 40     bc      4,eq,30 <__ftr_alt_97+0x30>
> > > > >    f0:   00 00 82 40     bc      4,eq,f0 <__ftr_alt_97+0xf0>
> > > > >
> > > > > This allows LTO builds for ppc64le_defconfig plus LTO options.
> > > > > Note that DYNAMIC_FTRACE/FUNCTION_TRACER is not supported by LTO builds
> > > > > but this is not POWERPC-specific.
> > > >
> > > > $ ARCH=powerpc make LLVM=1 -j72 ppc64le_defconfig
> > > > $ ARCH=powerpc make LLVM=1 -j72 menuconfig
> > > > <disable FTRACE, enable LTO_CLANG_THIN>
> > > > $ ARCH=powerpc make LLVM=1 -j72
> > > > ...
> > > >    VDSO64L arch/powerpc/kernel/vdso/vdso64.so.dbg
> > > > /usr/bin/powerpc64le-linux-gnu-ld:
> > > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: error
> > > > loading plugin:
> > > > /android0/llvm-project/llvm/build/bin/../lib/LLVMgold.so: cannot open
> > > > shared object file: No such file or directory
> > > > clang-15: error: linker command failed with exit code 1 (use -v to see
> > > > invocation)
> > > > make[1]: *** [arch/powerpc/kernel/vdso/Makefile:67:
> > > > arch/powerpc/kernel/vdso/vdso64.so.dbg] Error 1
> > > >
> > > > Looks like LLD isn't being invoked correctly to link the vdso.
> > > > Probably need to revisit
> > > > https://lore.kernel.org/lkml/20200901222523.1941988-1-ndesaulniers@google.com/
> > > >
> > > > How were you working around this issue? Perhaps you built clang to
> > > > default to LLD? (there's a cmake option for that)
> > >
> > >
> > > What option is that? I only add  -DLLVM_ENABLE_LLD=ON  which (I think)
>
> The option Nick is referring to here is CLANG_DEFAULT_LINKER, which sets
> the default linker when clang is invoked as the linker driver, which the
> PowerPC vDSO Makefile does. We have been trying to move all parts of the
> kernel to compile with $(CC) and link with $(LD) so that the default
> linker invoked by the compiler is taken out of the equation.
>
> For what it's worth, I think that the error Nick is seeing is due to a
> lack of the LLVMgold.so plugin on his system, which is built when
> -DLLVM_BINUTILS_INCDIR=... is included in the list of CMake variables,
> which you have. The LLVMgold.so plugin is needed when doing LTO with
> GNU ld, which is the case with the vDSO currently for the reason I
> mentioned above. We could work around this with Nick's proposed
> '-fuse-ld=lld' patch or just disable LTO for the vDSO.
>
> https://llvm.org/docs/GoldPlugin.html

Ah, and ld.gold is currently banned by
commit 75959d44f9dc ("kbuild: Fail if gold linker is detected")
which landed in v5.4-rc1. So the ppc vdso build isn't quite as
hermetic as we'd like.

>
> My version of the hack would probably be:
>
> ccflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld
> asflags-$(CONFIG_LD_IS_LLD) += -fuse-ld=lld

LGTM; want to send that as a formal patch? That would also address
https://github.com/ClangBuiltLinux/linux/issues/774.

>
> > > tells cmake to use lld to link the LLVM being built but does not seem to
> > > tell what the built clang should do.
>
> Right, -DLLVM_ENABLE_LLD=ON is equivalent to -DLLVM_USE_LINKER=lld,
> except when doing a multi-stage build:
>
> https://llvm.org/docs/CMake.html#llvm-related-variables
>
> > >
> > > Without -DLLVM_ENABLE_LLD=ON, building just fails:
> > >
> > > [fstn1-p1 ~/pbuild/llvm/llvm-lto-latest-cleanbuild]$ ninja -j 100
> > > [619/3501] Linking CXX executable bin/not
> > > FAILED: bin/not
> > > : && /usr/bin/clang++ -fPIC -fvisibility-inlines-hidden
> > > -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra
> > > -Wno-unused-parameter -Wwrite-strings -Wcast-qual
> > > -Wmissing-field-initializers -pedantic -Wno-long-long
> > > -Wc++98-compat-extra-semi -Wimplicit-fallthrough
> > > -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor
> > > -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion
> > > -Wmisleading-indentation -fdiagnostics-color -ffunction-sections
> > > -fdata-sections -flto -O3 -DNDEBUG -flto
> > > -Wl,-rpath-link,/home/aik/pbuild/llvm/llvm-lto-latest-cleanbuild/./lib
> > > -Wl,--gc-sections utils/not/CMakeFiles/not.dir/not.cpp.o -o bin/not
> > > -Wl,-rpath,"\$ORIGIN/../lib"  -lpthread  lib/libLLVMSupport.a  -lrt
> > > -ldl  -lpthread  -lm  /usr/lib/powerpc64le-linux-gnu/libz.so
> > > /usr/lib/powerpc64le-linux-gnu/libtinfo.so  lib/libLLVMDemangle.a && :
> > > /usr/bin/ld: lib/libLLVMSupport.a: error adding symbols: archive has no
> > > index; run ranlib to add one

Sounds like a bug with llvm cmake on ppc hosts.

The other error about

> /usr/bin/ld: lib/libLLVMObject.a: error adding symbols: file format not
recognized

Is likely because GNU ld doesn't recognize LLVM IR, which is what's
passed to the linker when you build LLVM itself with LTO via
`-DLLVM_ENABLE_LTO=ON`.
-- 
Thanks,
~Nick Desaulniers