linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Sami Tolvanen <samitolvanen@google.com>
To: Masahiro Yamada <masahiroy@kernel.org>
Cc: Will Deacon <will@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Steven Rostedt <rostedt@goodmis.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"Paul E. McKenney" <paulmck@kernel.org>,
	Kees Cook <keescook@chromium.org>,
	Nick Desaulniers <ndesaulniers@google.com>,
	clang-built-linux <clang-built-linux@googlegroups.com>,
	Kernel Hardening <kernel-hardening@lists.openwall.com>,
	linux-arch <linux-arch@vger.kernel.org>,
	linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
	Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	linux-pci@vger.kernel.org, X86 ML <x86@kernel.org>
Subject: Re: [PATCH v2 00/28] Add support for Clang LTO
Date: Thu, 10 Sep 2020 08:17:15 -0700	[thread overview]
Message-ID: <20200910151715.GB2041735@google.com> (raw)
In-Reply-To: <CAK7LNAR9zzP0ZU3b__PZv8gRtKrwz6-8GE1zG5UyFx1wDpOBzQ@mail.gmail.com>

On Thu, Sep 10, 2020 at 10:18:05AM +0900, Masahiro Yamada wrote:
> On Wed, Sep 9, 2020 at 8:46 AM Sami Tolvanen <samitolvanen@google.com> wrote:
> >
> > On Sun, Sep 06, 2020 at 09:24:38AM +0900, Masahiro Yamada wrote:
> > > On Fri, Sep 4, 2020 at 5:30 AM Sami Tolvanen <samitolvanen@google.com> wrote:
> > > >
> > > > This patch series adds support for building x86_64 and arm64 kernels
> > > > with Clang's Link Time Optimization (LTO).
> > > >
> > > > In addition to performance, the primary motivation for LTO is
> > > > to allow Clang's Control-Flow Integrity (CFI) to be used in the
> > > > kernel. Google has shipped millions of Pixel devices running three
> > > > major kernel versions with LTO+CFI since 2018.
> > > >
> > > > Most of the patches are build system changes for handling LLVM
> > > > bitcode, which Clang produces with LTO instead of ELF object files,
> > > > postponing ELF processing until a later stage, and ensuring initcall
> > > > ordering.
> > > >
> > > > Note that patches 1-4 are not directly related to LTO, but are
> > > > needed to compile LTO kernels with ToT Clang, so I'm including them
> > > > in the series for your convenience:
> > > >
> > > >  - Patches 1-3 are required for building the kernel with ToT Clang,
> > > >    and IAS, and patch 4 is needed to build allmodconfig with LTO.
> > > >
> > > >  - Patches 3-4 are already in linux-next, but not yet in 5.9-rc.
> > > >
> > >
> > >
> > > I still do not understand how this patch set works.
> > > (only me?)
> > >
> > > Please let me ask fundamental questions.
> > >
> > >
> > >
> > > I applied this series on top of Linus' tree,
> > > and compiled for ARCH=arm64.
> > >
> > > I compared the kernel size with/without LTO.
> > >
> > >
> > >
> > > [1] No LTO  (arm64 defconfig, CONFIG_LTO_NONE)
> > >
> > > $ llvm-size   vmlinux
> > >    text    data     bss     dec     hex filename
> > > 15848692 10099449 493060 26441201 19375f1 vmlinux
> > >
> > >
> > >
> > > [2] Clang LTO  (arm64 defconfig + CONFIG_LTO_CLANG)
> > >
> > > $ llvm-size   vmlinux
> > >    text    data     bss     dec     hex filename
> > > 15906864 10197445 490804 26595113 195cf29 vmlinux
> > >
> > >
> > > I compared the size of raw binary, arch/arm64/boot/Image.
> > > Its size increased too.
> > >
> > >
> > >
> > > So, in my experiment, enabling CONFIG_LTO_CLANG
> > > increases the kernel size.
> > > Is this correct?
> >
> > Yes. LTO does produce larger binaries, mostly due to function
> > inlining between translation units, I believe. The compiler people
> > can probably give you a more detailed answer here. Without -mllvm
> > -import-instr-limit, the binaries would be even larger.
> >
> > > One more thing, could you teach me
> > > how Clang LTO optimizes the code against
> > > relocatable objects?
> > >
> > >
> > >
> > > When I learned Clang LTO first, I read this document:
> > > https://llvm.org/docs/LinkTimeOptimization.html
> > >
> > > It is easy to confirm the final executable
> > > does not contain foo2, foo3...
> > >
> > >
> > >
> > > In contrast to userspace programs,
> > > kernel modules are basically relocatable objects.
> > >
> > > Does Clang drop unused symbols from relocatable objects?
> > > If so, how?
> >
> > I don't think the compiler can legally drop global symbols from
> > relocatable objects, but it can rename and possibly even drop static
> > functions.
> 
> 
> Compilers can drop static functions without LTO.
> Rather, it is a compiler warning
> (-Wunused-function), so the code should be cleaned up.
> 
> 
> 
> > This is why we need global wrappers for initcalls, for
> > example, to have stable symbol names.
> >
> > Sami
> 
> 
> 
> At first, I thought the motivation of LTO
> was to remove unused global symbols, and
> to perform further optimization.
> 
> 
> It is true for userspace programs.
> In fact, the example of
> https://llvm.org/docs/LinkTimeOptimization.html
> produces a smaller binary.
> 
> 
> In contrast, this patch set produces a bigger kernel
> because LTO cannot remove any unused symbol.
> 
> So, I do not understand what the benefit is.
> 
> 
> Is inlining beneficial?
> I am not sure.
> 
> 
> Documentation/process/coding-style.rst
> "15) The inline disease"
> mentions that inlining is not always
> a good thing.
> 
> 
> As a whole, I still do not understand
> the motivation of this patch set.

Clang produces faster code with LTO even if unused functions are not
removed, and I'm not sure how many unused globals there really are in
the kernel that aren't exported for modules. However, as I mentioned in
the cover letter, we also need LTO for Control-Flow Integrity (CFI),
which we have used in Pixel kernels for a couple of years now, and plan
to use in more Android devices in future:

  https://clang.llvm.org/docs/ControlFlowIntegrity.html

Sami

  reply	other threads:[~2020-09-10 19:44 UTC|newest]

Thread overview: 207+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-24 20:31 [PATCH 00/22] add support for Clang LTO Sami Tolvanen
2020-06-24 20:31 ` [PATCH 01/22] objtool: use sh_info to find the base for .rela sections Sami Tolvanen
2020-06-24 20:31 ` [PATCH 02/22] kbuild: add support for Clang LTO Sami Tolvanen
2020-06-24 20:53   ` Nick Desaulniers
2020-06-24 21:29     ` Sami Tolvanen
2020-06-25  2:26   ` Nathan Chancellor
2020-06-25 16:13     ` Sami Tolvanen
2020-06-24 20:31 ` [PATCH 03/22] kbuild: lto: fix module versioning Sami Tolvanen
2020-06-24 20:31 ` [PATCH 04/22] kbuild: lto: fix recordmcount Sami Tolvanen
2020-06-24 21:27   ` Peter Zijlstra
2020-06-24 21:45     ` Sami Tolvanen
2020-06-25  7:45       ` Peter Zijlstra
2020-06-25 16:15         ` Sami Tolvanen
2020-06-25 20:02           ` [RFC][PATCH] objtool,x86_64: Replace recordmcount with objtool Peter Zijlstra
2020-06-25 20:54             ` Nick Desaulniers
2020-06-25 22:40             ` Sami Tolvanen
2020-06-26 11:29               ` Peter Zijlstra
2020-06-26 11:42                 ` Peter Zijlstra
2020-07-17 17:28                 ` Sami Tolvanen
2020-07-17 17:36                   ` Steven Rostedt
2020-07-17 17:47                     ` Sami Tolvanen
2020-07-17 18:05                       ` Steven Rostedt
2020-07-20 16:52                         ` Sami Tolvanen
2020-07-22 17:58                           ` Steven Rostedt
2020-07-22 18:07                             ` Sami Tolvanen
2020-07-22 17:55                 ` Steven Rostedt
2020-07-22 18:41                   ` Peter Zijlstra
2020-07-22 19:09                     ` Steven Rostedt
2020-07-22 20:03                       ` Sami Tolvanen
2020-07-22 23:56                       ` Peter Zijlstra
2020-07-23  0:06                         ` Steven Rostedt
2020-08-06 22:09                           ` Sami Tolvanen
2020-06-24 20:31 ` [PATCH 05/22] kbuild: lto: postpone objtool Sami Tolvanen
2020-06-24 21:19   ` Peter Zijlstra
2020-06-24 21:49     ` Sami Tolvanen
2020-06-25  7:47       ` Peter Zijlstra
2020-06-25 16:22         ` Sami Tolvanen
2020-06-25 18:33           ` Peter Zijlstra
2020-06-25 19:32             ` Sami Tolvanen
2020-06-24 20:31 ` [PATCH 06/22] kbuild: lto: limit inlining Sami Tolvanen
2020-06-24 21:20   ` Peter Zijlstra
2020-06-24 23:37     ` Sami Tolvanen
2020-06-24 20:31 ` [PATCH 07/22] kbuild: lto: merge module sections Sami Tolvanen
2020-06-24 21:01   ` Nick Desaulniers
2020-06-24 21:31     ` Sami Tolvanen
2020-06-24 20:31 ` [PATCH 08/22] kbuild: lto: remove duplicate dependencies from .mod files Sami Tolvanen
2020-06-24 21:13   ` Nick Desaulniers
2020-06-24 20:31 ` [PATCH 09/22] init: lto: ensure initcall ordering Sami Tolvanen
2020-06-24 20:31 ` [PATCH 10/22] init: lto: fix PREL32 relocations Sami Tolvanen
2020-06-24 20:31 ` [PATCH 11/22] pci: " Sami Tolvanen
2020-07-17 20:26   ` Bjorn Helgaas
2020-07-22 18:15     ` Sami Tolvanen
2020-06-24 20:31 ` [PATCH 12/22] modpost: lto: strip .lto from module names Sami Tolvanen
2020-06-24 22:05   ` Nick Desaulniers
2020-06-24 20:31 ` [PATCH 13/22] scripts/mod: disable LTO for empty.c Sami Tolvanen
2020-06-24 20:57   ` Nick Desaulniers
2020-06-24 20:31 ` [PATCH 14/22] efi/libstub: disable LTO Sami Tolvanen
2020-06-24 20:31 ` [PATCH 15/22] drivers/misc/lkdtm: disable LTO for rodata.o Sami Tolvanen
2020-06-24 20:31 ` [PATCH 16/22] arm64: export CC_USING_PATCHABLE_FUNCTION_ENTRY Sami Tolvanen
2020-06-24 20:31 ` [PATCH 17/22] arm64: vdso: disable LTO Sami Tolvanen
2020-06-24 20:58   ` Nick Desaulniers
2020-06-24 21:09     ` Nick Desaulniers
2020-06-24 23:51       ` Andi Kleen
2020-06-24 21:52     ` Sami Tolvanen
2020-06-24 23:05       ` Nick Desaulniers
2020-06-24 23:39         ` Sami Tolvanen
2020-06-24 20:31 ` [PATCH 18/22] arm64: allow LTO_CLANG and THINLTO to be selected Sami Tolvanen
2020-06-24 20:31 ` [PATCH 19/22] x86, vdso: disable LTO only for vDSO Sami Tolvanen
2020-06-24 20:31 ` [PATCH 20/22] x86, ftrace: disable recordmcount for ftrace_make_nop Sami Tolvanen
2020-06-24 20:31 ` [PATCH 21/22] x86, relocs: Ignore L4_PAGE_OFFSET relocations Sami Tolvanen
2020-06-24 20:32 ` [PATCH 22/22] x86, build: allow LTO_CLANG and THINLTO to be selected Sami Tolvanen
2020-06-24 21:15 ` [PATCH 00/22] add support for Clang LTO Peter Zijlstra
2020-06-24 21:30   ` Sami Tolvanen
2020-06-25  8:27     ` Will Deacon
2020-06-24 21:31   ` Nick Desaulniers
2020-06-25  8:03     ` Peter Zijlstra
2020-06-25  8:24       ` Peter Zijlstra
2020-06-25  8:57         ` Peter Zijlstra
2020-06-30 19:19           ` Marco Elver
2020-06-30 20:12             ` Peter Zijlstra
2020-06-30 20:30               ` Paul E. McKenney
2020-07-01  9:10                 ` Peter Zijlstra
2020-07-01 14:20                   ` David Laight
2020-07-01 16:06                     ` Paul E. McKenney
2020-07-02  9:37                       ` David Laight
2020-07-02 18:00                         ` Paul E. McKenney
2020-07-01  9:41                 ` Marco Elver
2020-07-01 10:03                   ` Will Deacon
2020-07-01 11:40                   ` Peter Zijlstra
2020-07-01 14:06                     ` Paul E. McKenney
2020-07-01 15:05                       ` Peter Zijlstra
2020-07-01 16:03                         ` Paul E. McKenney
2020-07-02  8:20                           ` Peter Zijlstra
2020-07-02 17:59                             ` Paul E. McKenney
2020-07-03 13:13                               ` Peter Zijlstra
2020-07-03 13:25                                 ` Peter Zijlstra
2020-07-03 14:51                                   ` Paul E. McKenney
2020-07-03 14:42                                 ` Paul E. McKenney
2020-07-06 16:26                                   ` Paul E. McKenney
2020-07-06 18:29                                     ` Peter Zijlstra
2020-07-06 18:39                                       ` Paul E. McKenney
2020-07-06 19:40                                         ` Peter Zijlstra
2020-07-06 23:41                                           ` Paul E. McKenney
2020-06-28 16:56 ` Masahiro Yamada
2020-06-29 23:20   ` Sami Tolvanen
2020-07-07 15:51     ` Sami Tolvanen
2020-07-07 16:05       ` Sami Tolvanen
2020-07-07 16:56         ` Jakub Kicinski
2020-07-07 17:17           ` Nick Desaulniers
2020-07-07 17:30             ` Jakub Kicinski
2020-07-11 16:32 ` Paul Menzel
2020-07-12  8:59   ` Sedat Dilek
2020-07-12 18:40     ` Nathan Chancellor
2020-07-14  9:44       ` Sedat Dilek
2020-07-14 17:54         ` Nick Desaulniers
2020-07-12 23:34   ` Sami Tolvanen
2020-07-14 12:16     ` Paul Menzel
2020-07-14 12:35       ` Sedat Dilek
2020-09-03 20:30 ` [PATCH v2 00/28] Add " Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 01/28] x86/boot/compressed: Disable relocation relaxation Sami Tolvanen
2020-09-03 21:44     ` Kees Cook
2020-09-03 23:42       ` Arvind Sankar
2020-09-04  7:14         ` Nathan Chancellor
2020-09-03 20:30   ` [PATCH v2 02/28] x86/asm: Replace __force_order with memory clobber Sami Tolvanen
2020-09-03 21:45     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 03/28] lib/string.c: implement stpcpy Sami Tolvanen
2020-09-03 21:47     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 04/28] RAS/CEC: Fix cec_init() prototype Sami Tolvanen
2020-09-03 21:50     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 05/28] objtool: Add a pass for generating __mcount_loc Sami Tolvanen
2020-09-03 21:51     ` Kees Cook
2020-09-03 22:03       ` Sami Tolvanen
2020-09-04  9:31         ` peterz
2020-09-10 18:29           ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 06/28] objtool: Don't autodetect vmlinux.o Sami Tolvanen
2020-09-03 21:52     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 07/28] kbuild: add support for objtool mcount Sami Tolvanen
2020-09-03 21:56     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 08/28] x86, build: use " Sami Tolvanen
2020-09-03 21:58     ` Kees Cook
2020-09-03 22:11       ` Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 09/28] kbuild: add support for Clang LTO Sami Tolvanen
2020-09-03 22:08     ` Kees Cook
2020-09-08 17:02       ` Sami Tolvanen
2020-09-05 19:36     ` Masahiro Yamada
2020-09-08 17:10       ` Sami Tolvanen
2020-09-05 20:17     ` Masahiro Yamada
2020-09-08 17:14       ` Sami Tolvanen
2020-09-07 15:30     ` Masahiro Yamada
2020-09-08 17:30       ` Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 10/28] kbuild: lto: fix module versioning Sami Tolvanen
2020-09-03 22:11     ` Kees Cook
2020-09-08 18:23       ` Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 11/28] kbuild: lto: postpone objtool Sami Tolvanen
2020-09-03 22:19     ` Kees Cook
2020-09-08 20:56       ` Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 12/28] kbuild: lto: limit inlining Sami Tolvanen
2020-09-03 22:20     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 13/28] kbuild: lto: merge module sections Sami Tolvanen
2020-09-03 22:23     ` Kees Cook
2020-09-07 15:25     ` Masahiro Yamada
2020-09-08 21:07       ` Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 14/28] kbuild: lto: remove duplicate dependencies from .mod files Sami Tolvanen
2020-09-03 22:29     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 15/28] init: lto: ensure initcall ordering Sami Tolvanen
2020-09-03 22:40     ` Kees Cook
2020-09-08 21:16       ` Sami Tolvanen
2020-09-10  9:25     ` David Woodhouse
2020-09-10 15:07       ` Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 16/28] init: lto: fix PREL32 relocations Sami Tolvanen
2020-09-03 22:41     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 17/28] PCI: Fix PREL32 relocations for LTO Sami Tolvanen
2020-09-03 22:42     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 18/28] modpost: lto: strip .lto from module names Sami Tolvanen
2020-09-03 22:42     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 19/28] scripts/mod: disable LTO for empty.c Sami Tolvanen
2020-09-03 22:43     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 20/28] efi/libstub: disable LTO Sami Tolvanen
2020-09-03 22:43     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 21/28] drivers/misc/lkdtm: disable LTO for rodata.o Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 22/28] arm64: export CC_USING_PATCHABLE_FUNCTION_ENTRY Sami Tolvanen
2020-09-03 22:44     ` Kees Cook
2020-09-08 21:23       ` Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 23/28] arm64: vdso: disable LTO Sami Tolvanen
2020-09-03 22:45     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 24/28] KVM: arm64: disable LTO for the nVHE directory Sami Tolvanen
2020-09-03 22:45     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 25/28] arm64: allow LTO_CLANG and THINLTO to be selected Sami Tolvanen
2020-09-03 22:45     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 26/28] x86, vdso: disable LTO only for vDSO Sami Tolvanen
2020-09-03 22:46     ` Kees Cook
2020-09-03 20:30   ` [PATCH v2 27/28] x86, relocs: Ignore L4_PAGE_OFFSET relocations Sami Tolvanen
2020-09-03 22:47     ` Kees Cook
2020-09-08 23:28       ` Sami Tolvanen
2020-09-03 20:30   ` [PATCH v2 28/28] x86, build: allow LTO_CLANG and THINLTO to be selected Sami Tolvanen
2020-09-03 22:48     ` Kees Cook
2020-09-03 23:34   ` [PATCH v2 00/28] Add support for Clang LTO Kees Cook
2020-09-04  4:45     ` Nathan Chancellor
2020-09-03 23:38   ` Kees Cook
2020-09-04  7:53   ` Sedat Dilek
2020-09-04  8:55   ` peterz
2020-09-04  9:08     ` Sedat Dilek
2020-09-06  0:24   ` Masahiro Yamada
2020-09-08 23:46     ` Sami Tolvanen
2020-09-10  1:18       ` Masahiro Yamada
2020-09-10 15:17         ` Sami Tolvanen [this message]
2020-09-10 18:18         ` Kees Cook

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200910151715.GB2041735@google.com \
    --to=samitolvanen@google.com \
    --cc=clang-built-linux@googlegroups.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=keescook@chromium.org \
    --cc=kernel-hardening@lists.openwall.com \
    --cc=linux-arch@vger.kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kbuild@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=masahiroy@kernel.org \
    --cc=ndesaulniers@google.com \
    --cc=paulmck@kernel.org \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=will@kernel.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).