From: Sami Tolvanen <samitolvanen@google.com> To: Arnd Bergmann <arnd@kernel.org> Cc: Masahiro Yamada <masahiroy@kernel.org>, Steven Rostedt <rostedt@goodmis.org>, Will Deacon <will@kernel.org>, Josh Poimboeuf <jpoimboe@redhat.com>, Peter Zijlstra <peterz@infradead.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, "Paul E. McKenney" <paulmck@kernel.org>, Kees Cook <keescook@chromium.org>, Nick Desaulniers <ndesaulniers@google.com>, clang-built-linux <clang-built-linux@googlegroups.com>, Kernel Hardening <kernel-hardening@lists.openwall.com>, linux-arch <linux-arch@vger.kernel.org>, Linux ARM <linux-arm-kernel@lists.infradead.org>, Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, linux-pci <linux-pci@vger.kernel.org> Subject: Re: [PATCH v8 00/16] Add support for Clang LTO Date: Tue, 8 Dec 2020 08:43:06 -0800 [thread overview] Message-ID: <CABCJKueCHo2RYfx_A21m+=d1gQLR9QsOOxCsHFeicCqyHkb-Kg@mail.gmail.com> (raw) In-Reply-To: <CAK8P3a1WEAo2SEgKUEs3SB7n7QeeHa0=cx_nO==rDK0jjDArow@mail.gmail.com> On Tue, Dec 8, 2020 at 4:15 AM Arnd Bergmann <arnd@kernel.org> wrote: > > On Tue, Dec 1, 2020 at 10:37 PM 'Sami Tolvanen' via Clang Built Linux > <clang-built-linux@googlegroups.com> wrote: > > > > This patch series adds support for building the kernel with Clang's > > Link Time Optimization (LTO). In addition to performance, the primary > > motivation for LTO is to allow Clang's Control-Flow Integrity (CFI) > > to be used in the kernel. Google has shipped millions of Pixel > > devices running three major kernel versions with LTO+CFI since 2018. > > > > Most of the patches are build system changes for handling LLVM > > bitcode, which Clang produces with LTO instead of ELF object files, > > postponing ELF processing until a later stage, and ensuring initcall > > ordering. > > > > Note that arm64 support depends on Will's memory ordering patches > > [1]. I will post x86_64 patches separately after we have fixed the > > remaining objtool warnings [2][3]. > > > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/lto > > [2] https://lore.kernel.org/lkml/20201120040424.a3wctajzft4ufoiw@treble/ > > [3] https://git.kernel.org/pub/scm/linux/kernel/git/jpoimboe/linux.git/log/?h=objtool-vmlinux > > > > You can also pull this series from > > > > https://github.com/samitolvanen/linux.git lto-v8 > > I've tried pull this into my randconfig test tree to give it a spin. Great, thank you for testing this! > So far I have > not managed to get a working build out of it, the main problem so far being > that it is really slow to build because the link stage only uses one CPU. > These are the other issues I've seen so far: You may want to limit your testing only to ThinLTO at first, because full LTO is going to be extremely slow with larger configs, especially when building arm64 kernels. > - one build seems to take even longer to link. It's currently at 35GB RAM > usage and 40 minutes into the final link, but I'm worried it might > not complete > before it runs out of memory. I only have 128GB installed, and google-chrome > uses another 30GB of that, and I'm also doing some other builds in parallel. > Is there a minimum recommended amount of memory for doing LTO builds? When building arm64 defconfig, the maximum memory usage I measured with ThinLTO was 3.5 GB, and with full LTO 20.3 GB. I haven't measured larger configurations, but I believe LLD can easily consume 3-4x that much with full LTO allyesconfig. > - One build failed with > ld.lld -EL -maarch64elf -mllvm -import-instr-limit=5 -r -o vmlinux.o > -T .tmp_initcalls.lds --whole-archive arch/arm64/kernel/head.o > init/built-in.a usr/built-in.a arch/arm64/built-in.a kernel/built-in.a > certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a > security/built-in.a crypto/built-in.a block/built-in.a > arch/arm64/lib/built-in.a lib/built-in.a drivers/built-in.a > sound/built-in.a net/built-in.a virt/built-in.a --no-whole-archive > --start-group arch/arm64/lib/lib.a lib/lib.a > ./drivers/firmware/efi/libstub/lib.a --end-group > "ld.lld: error: arch/arm64/kernel/head.o: invalid symbol index" > after about 30 minutes That's interesting. Did you use LLVM_IAS=1? > - CONFIG_CPU_BIG_ENDIAN doesn't seem to work with lld, and LTO > doesn't work with ld.bfd. > I've added a CPU_LITTLE_ENDIAN dependency to > ARCH_SUPPORTS_LTO_CLANG{,THIN} Ah, good point. I'll fix this in v9. [...] > Not sure if these are all known issues. If there is one you'd like me try > take a closer look at for finding which config options break it, I can try No, none of these are known issues. I would be happy to take a closer look if you can share configs that reproduce these. Sami
WARNING: multiple messages have this Message-ID (diff)
From: Sami Tolvanen <samitolvanen@google.com> To: Arnd Bergmann <arnd@kernel.org> Cc: linux-arch <linux-arch@vger.kernel.org>, Kees Cook <keescook@chromium.org>, "Paul E. McKenney" <paulmck@kernel.org>, Kernel Hardening <kernel-hardening@lists.openwall.com>, Peter Zijlstra <peterz@infradead.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Masahiro Yamada <masahiroy@kernel.org>, Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>, Nick Desaulniers <ndesaulniers@google.com>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, Steven Rostedt <rostedt@goodmis.org>, clang-built-linux <clang-built-linux@googlegroups.com>, linux-pci <linux-pci@vger.kernel.org>, Josh Poimboeuf <jpoimboe@redhat.com>, Will Deacon <will@kernel.org>, Linux ARM <linux-arm-kernel@lists.infradead.org> Subject: Re: [PATCH v8 00/16] Add support for Clang LTO Date: Tue, 8 Dec 2020 08:43:06 -0800 [thread overview] Message-ID: <CABCJKueCHo2RYfx_A21m+=d1gQLR9QsOOxCsHFeicCqyHkb-Kg@mail.gmail.com> (raw) In-Reply-To: <CAK8P3a1WEAo2SEgKUEs3SB7n7QeeHa0=cx_nO==rDK0jjDArow@mail.gmail.com> On Tue, Dec 8, 2020 at 4:15 AM Arnd Bergmann <arnd@kernel.org> wrote: > > On Tue, Dec 1, 2020 at 10:37 PM 'Sami Tolvanen' via Clang Built Linux > <clang-built-linux@googlegroups.com> wrote: > > > > This patch series adds support for building the kernel with Clang's > > Link Time Optimization (LTO). In addition to performance, the primary > > motivation for LTO is to allow Clang's Control-Flow Integrity (CFI) > > to be used in the kernel. Google has shipped millions of Pixel > > devices running three major kernel versions with LTO+CFI since 2018. > > > > Most of the patches are build system changes for handling LLVM > > bitcode, which Clang produces with LTO instead of ELF object files, > > postponing ELF processing until a later stage, and ensuring initcall > > ordering. > > > > Note that arm64 support depends on Will's memory ordering patches > > [1]. I will post x86_64 patches separately after we have fixed the > > remaining objtool warnings [2][3]. > > > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/lto > > [2] https://lore.kernel.org/lkml/20201120040424.a3wctajzft4ufoiw@treble/ > > [3] https://git.kernel.org/pub/scm/linux/kernel/git/jpoimboe/linux.git/log/?h=objtool-vmlinux > > > > You can also pull this series from > > > > https://github.com/samitolvanen/linux.git lto-v8 > > I've tried pull this into my randconfig test tree to give it a spin. Great, thank you for testing this! > So far I have > not managed to get a working build out of it, the main problem so far being > that it is really slow to build because the link stage only uses one CPU. > These are the other issues I've seen so far: You may want to limit your testing only to ThinLTO at first, because full LTO is going to be extremely slow with larger configs, especially when building arm64 kernels. > - one build seems to take even longer to link. It's currently at 35GB RAM > usage and 40 minutes into the final link, but I'm worried it might > not complete > before it runs out of memory. I only have 128GB installed, and google-chrome > uses another 30GB of that, and I'm also doing some other builds in parallel. > Is there a minimum recommended amount of memory for doing LTO builds? When building arm64 defconfig, the maximum memory usage I measured with ThinLTO was 3.5 GB, and with full LTO 20.3 GB. I haven't measured larger configurations, but I believe LLD can easily consume 3-4x that much with full LTO allyesconfig. > - One build failed with > ld.lld -EL -maarch64elf -mllvm -import-instr-limit=5 -r -o vmlinux.o > -T .tmp_initcalls.lds --whole-archive arch/arm64/kernel/head.o > init/built-in.a usr/built-in.a arch/arm64/built-in.a kernel/built-in.a > certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a > security/built-in.a crypto/built-in.a block/built-in.a > arch/arm64/lib/built-in.a lib/built-in.a drivers/built-in.a > sound/built-in.a net/built-in.a virt/built-in.a --no-whole-archive > --start-group arch/arm64/lib/lib.a lib/lib.a > ./drivers/firmware/efi/libstub/lib.a --end-group > "ld.lld: error: arch/arm64/kernel/head.o: invalid symbol index" > after about 30 minutes That's interesting. Did you use LLVM_IAS=1? > - CONFIG_CPU_BIG_ENDIAN doesn't seem to work with lld, and LTO > doesn't work with ld.bfd. > I've added a CPU_LITTLE_ENDIAN dependency to > ARCH_SUPPORTS_LTO_CLANG{,THIN} Ah, good point. I'll fix this in v9. [...] > Not sure if these are all known issues. If there is one you'd like me try > take a closer look at for finding which config options break it, I can try No, none of these are known issues. I would be happy to take a closer look if you can share configs that reproduce these. Sami _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-12-08 16:44 UTC|newest] Thread overview: 141+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-12-01 21:36 [PATCH v8 00/16] Add support for Clang LTO Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` [PATCH v8 01/16] tracing: move function tracer options to Kconfig Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:47 ` Steven Rostedt 2020-12-01 21:47 ` Steven Rostedt 2020-12-01 21:36 ` [PATCH v8 02/16] kbuild: add support for Clang LTO Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-02 2:59 ` Masahiro Yamada 2020-12-02 2:59 ` Masahiro Yamada 2020-12-03 0:07 ` Nick Desaulniers 2020-12-03 0:07 ` Nick Desaulniers 2020-12-03 0:07 ` Nick Desaulniers 2020-12-01 21:36 ` [PATCH v8 03/16] kbuild: lto: fix module versioning Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` [PATCH v8 04/16] kbuild: lto: limit inlining Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` [PATCH v8 05/16] kbuild: lto: merge module sections Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` [PATCH v8 06/16] kbuild: lto: remove duplicate dependencies from .mod files Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` [PATCH v8 07/16] init: lto: ensure initcall ordering Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` [PATCH v8 08/16] init: lto: fix PREL32 relocations Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:36 ` Sami Tolvanen 2020-12-01 21:37 ` [PATCH v8 09/16] PCI: Fix PREL32 relocations for LTO Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` [PATCH v8 10/16] modpost: lto: strip .lto from module names Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` [PATCH v8 11/16] scripts/mod: disable LTO for empty.c Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` [PATCH v8 12/16] efi/libstub: disable LTO Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` [PATCH v8 13/16] drivers/misc/lkdtm: disable LTO for rodata.o Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` [PATCH v8 14/16] arm64: vdso: disable LTO Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` [PATCH v8 15/16] arm64: disable recordmcount with DYNAMIC_FTRACE_WITH_REGS Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` [PATCH v8 16/16] arm64: allow LTO to be selected Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-01 21:37 ` Sami Tolvanen 2020-12-03 0:01 ` [PATCH v8 00/16] Add support for Clang LTO Nick Desaulniers 2020-12-03 0:01 ` Nick Desaulniers 2020-12-03 0:01 ` Nick Desaulniers 2020-12-03 11:26 ` Will Deacon 2020-12-03 11:26 ` Will Deacon 2020-12-03 17:07 ` Sami Tolvanen 2020-12-03 17:07 ` Sami Tolvanen 2020-12-03 17:07 ` Sami Tolvanen 2020-12-03 18:21 ` Nathan Chancellor 2020-12-03 18:21 ` Nathan Chancellor 2020-12-03 18:22 ` Will Deacon 2020-12-03 18:22 ` Will Deacon 2020-12-03 22:32 ` Nick Desaulniers 2020-12-03 22:32 ` Nick Desaulniers 2020-12-03 22:32 ` Nick Desaulniers 2020-12-04 9:35 ` Will Deacon 2020-12-04 9:35 ` Will Deacon 2020-12-04 22:52 ` Sami Tolvanen 2020-12-04 22:52 ` Sami Tolvanen 2020-12-04 22:52 ` Sami Tolvanen 2020-12-06 6:50 ` Nathan Chancellor 2020-12-06 6:50 ` Nathan Chancellor 2020-12-06 20:09 ` Sami Tolvanen 2020-12-06 20:09 ` Sami Tolvanen 2020-12-06 20:09 ` Sami Tolvanen 2020-12-08 0:46 ` Nathan Chancellor 2020-12-08 0:46 ` Nathan Chancellor 2020-12-08 12:15 ` Arnd Bergmann 2020-12-08 12:15 ` Arnd Bergmann 2020-12-08 12:15 ` Arnd Bergmann 2020-12-08 13:54 ` Arnd Bergmann 2020-12-08 13:54 ` Arnd Bergmann 2020-12-08 13:54 ` Arnd Bergmann 2020-12-08 16:53 ` Sami Tolvanen 2020-12-08 16:53 ` Sami Tolvanen 2020-12-08 16:53 ` Sami Tolvanen 2020-12-08 16:56 ` Arnd Bergmann 2020-12-08 16:56 ` Arnd Bergmann 2020-12-08 16:56 ` Arnd Bergmann 2020-12-08 16:43 ` Sami Tolvanen [this message] 2020-12-08 16:43 ` Sami Tolvanen 2020-12-08 16:43 ` Sami Tolvanen 2020-12-08 20:59 ` Arnd Bergmann 2020-12-08 20:59 ` Arnd Bergmann 2020-12-08 21:02 ` Arnd Bergmann 2020-12-08 21:02 ` Arnd Bergmann 2020-12-09 5:23 ` Fāng-ruì Sòng 2020-12-09 5:23 ` Fāng-ruì Sòng 2020-12-09 5:23 ` Fāng-ruì Sòng 2020-12-09 9:07 ` Arnd Bergmann 2020-12-09 9:07 ` Arnd Bergmann 2020-12-09 9:07 ` Arnd Bergmann 2020-12-09 9:59 ` Arnd Bergmann 2020-12-09 9:59 ` Arnd Bergmann 2020-12-09 16:09 ` Sami Tolvanen 2020-12-09 16:09 ` Sami Tolvanen 2020-12-09 16:09 ` Sami Tolvanen 2020-12-09 19:24 ` Arnd Bergmann 2020-12-09 19:24 ` Arnd Bergmann 2020-12-09 19:24 ` Arnd Bergmann 2020-12-08 21:09 ` Nick Desaulniers 2020-12-08 21:09 ` Nick Desaulniers 2020-12-08 21:09 ` Nick Desaulniers 2020-12-08 22:20 ` Arnd Bergmann 2020-12-08 22:20 ` Arnd Bergmann 2020-12-08 22:20 ` Arnd Bergmann 2020-12-09 16:11 ` Sami Tolvanen 2020-12-09 16:11 ` Sami Tolvanen 2020-12-09 16:11 ` Sami Tolvanen 2020-12-09 4:55 ` Fangrui Song 2020-12-09 4:55 ` Fangrui Song 2020-12-09 9:19 ` Arnd Bergmann 2020-12-09 9:19 ` Arnd Bergmann 2020-12-09 9:19 ` Arnd Bergmann 2020-12-09 12:35 ` Arnd Bergmann 2020-12-09 12:35 ` Arnd Bergmann 2020-12-09 12:35 ` Arnd Bergmann 2020-12-09 16:25 ` Sami Tolvanen 2020-12-09 16:25 ` Sami Tolvanen 2020-12-09 16:25 ` Sami Tolvanen 2020-12-09 17:51 ` Arnd Bergmann 2020-12-09 17:51 ` Arnd Bergmann 2020-12-09 17:51 ` Arnd Bergmann
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='CABCJKueCHo2RYfx_A21m+=d1gQLR9QsOOxCsHFeicCqyHkb-Kg@mail.gmail.com' \ --to=samitolvanen@google.com \ --cc=arnd@kernel.org \ --cc=clang-built-linux@googlegroups.com \ --cc=gregkh@linuxfoundation.org \ --cc=jpoimboe@redhat.com \ --cc=keescook@chromium.org \ --cc=kernel-hardening@lists.openwall.com \ --cc=linux-arch@vger.kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kbuild@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-pci@vger.kernel.org \ --cc=masahiroy@kernel.org \ --cc=ndesaulniers@google.com \ --cc=paulmck@kernel.org \ --cc=peterz@infradead.org \ --cc=rostedt@goodmis.org \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.