From: Sedat Dilek <sedat.dilek@gmail.com>
To: Bill Wendling <morbo@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>,
Masahiro Yamada <masahiroy@kernel.org>,
Linux Doc Mailing List <linux-doc@vger.kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>,
Clang-Built-Linux ML <clang-built-linux@googlegroups.com>,
Andrew Morton <akpm@linux-foundation.org>,
Nathan Chancellor <natechancellor@gmail.com>,
Nick Desaulniers <ndesaulniers@google.com>,
Sami Tolvanen <samitolvanen@google.com>
Subject: Re: [PATCH v5] pgo: add clang's Profile Guided Optimization infrastructure
Date: Sun, 17 Jan 2021 11:44:40 +0100 [thread overview]
Message-ID: <CA+icZUWf05ek+DFsJNyBc-4cg0s6cVrn=rNJDyL4RJ6=fMO5NA@mail.gmail.com> (raw)
In-Reply-To: <CAGG=3QXZTR_f9pKzAR=LrALmMzdDqsvWM_zrTXOb2PpiDGB-+A@mail.gmail.com>
On Sat, Jan 16, 2021 at 9:23 PM Bill Wendling <morbo@google.com> wrote:
>
> On Sat, Jan 16, 2021 at 9:39 AM Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > On Sat, Jan 16, 2021 at 10:44 AM 'Bill Wendling' via Clang Built Linux
> > <clang-built-linux@googlegroups.com> wrote:
> > >
> > > From: Sami Tolvanen <samitolvanen@google.com>
> > >
> > > Enable the use of clang's Profile-Guided Optimization[1]. To generate a
> > > profile, the kernel is instrumented with PGO counters, a representative
> > > workload is run, and the raw profile data is collected from
> > > /sys/kernel/debug/pgo/profraw.
> > >
> > > The raw profile data must be processed by clang's "llvm-profdata" tool
> > > before it can be used during recompilation:
> > >
> > > $ cp /sys/kernel/debug/pgo/profraw vmlinux.profraw
> > > $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > >
> > > Multiple raw profiles may be merged during this step.
> > >
> > > The data can now be used by the compiler:
> > >
> > > $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > >
> > > This initial submission is restricted to x86, as that's the platform we
> > > know works. This restriction can be lifted once other platforms have
> > > been verified to work with PGO.
> > >
> > > Note that this method of profiling the kernel is clang-native, unlike
> > > the clang support in kernel/gcov.
> > >
> > > [1] https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > >
> > > Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
> > > Co-developed-by: Bill Wendling <morbo@google.com>
> > > Signed-off-by: Bill Wendling <morbo@google.com>
> > > ---
> > > v2: - Added "__llvm_profile_instrument_memop" based on Nathan Chancellor's
> > > testing.
> > > - Corrected documentation, re PGO flags when using LTO, based on Fangrui
> > > Song's comments.
> > > v3: - Added change log section based on Sedat Dilek's comments.
> > > v4: - Remove non-x86 Makfile changes and se "hweight64" instead of using our
> > > own popcount implementation, based on Nick Desaulniers's comment.
> > > v5: - Correct padding calculation, discovered by Nathan Chancellor.
> > > ---
> > > Documentation/dev-tools/index.rst | 1 +
> > > Documentation/dev-tools/pgo.rst | 127 +++++++++
> > > MAINTAINERS | 9 +
> > > Makefile | 3 +
> > > arch/Kconfig | 1 +
> > > arch/x86/Kconfig | 1 +
> > > arch/x86/boot/Makefile | 1 +
> > > arch/x86/boot/compressed/Makefile | 1 +
> > > arch/x86/crypto/Makefile | 2 +
> > > arch/x86/entry/vdso/Makefile | 1 +
> > > arch/x86/kernel/vmlinux.lds.S | 2 +
> > > arch/x86/platform/efi/Makefile | 1 +
> > > arch/x86/purgatory/Makefile | 1 +
> > > arch/x86/realmode/rm/Makefile | 1 +
> > > arch/x86/um/vdso/Makefile | 1 +
> > > drivers/firmware/efi/libstub/Makefile | 1 +
> > > include/asm-generic/vmlinux.lds.h | 44 +++
> > > kernel/Makefile | 1 +
> > > kernel/pgo/Kconfig | 35 +++
> > > kernel/pgo/Makefile | 5 +
> > > kernel/pgo/fs.c | 382 ++++++++++++++++++++++++++
> > > kernel/pgo/instrument.c | 185 +++++++++++++
> > > kernel/pgo/pgo.h | 206 ++++++++++++++
> > > scripts/Makefile.lib | 10 +
> > > 24 files changed, 1022 insertions(+)
> > > create mode 100644 Documentation/dev-tools/pgo.rst
> > > create mode 100644 kernel/pgo/Kconfig
> > > create mode 100644 kernel/pgo/Makefile
> > > create mode 100644 kernel/pgo/fs.c
> > > create mode 100644 kernel/pgo/instrument.c
> > > create mode 100644 kernel/pgo/pgo.h
> > >
> > > diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst
> > > index f7809c7b1ba9e..8d6418e858062 100644
> > > --- a/Documentation/dev-tools/index.rst
> > > +++ b/Documentation/dev-tools/index.rst
> > > @@ -26,6 +26,7 @@ whole; patches welcome!
> > > kgdb
> > > kselftest
> > > kunit/index
> > > + pgo
> > >
> > >
> > > .. only:: subproject and html
> > > diff --git a/Documentation/dev-tools/pgo.rst b/Documentation/dev-tools/pgo.rst
> > > new file mode 100644
> > > index 0000000000000..b7f11d8405b73
> > > --- /dev/null
> > > +++ b/Documentation/dev-tools/pgo.rst
> > > @@ -0,0 +1,127 @@
> > > +.. SPDX-License-Identifier: GPL-2.0
> > > +
> > > +===============================
> > > +Using PGO with the Linux kernel
> > > +===============================
> > > +
> > > +Clang's profiling kernel support (PGO_) enables profiling of the Linux kernel
> > > +when building with Clang. The profiling data is exported via the ``pgo``
> > > +debugfs directory.
> > > +
> > > +.. _PGO: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization
> > > +
> > > +
> > > +Preparation
> > > +===========
> > > +
> > > +Configure the kernel with:
> > > +
> > > +.. code-block:: make
> > > +
> > > + CONFIG_DEBUG_FS=y
> > > + CONFIG_PGO_CLANG=y
> > > +
> > > +Note that kernels compiled with profiling flags will be significantly larger
> > > +and run slower.
> > > +
> > > +Profiling data will only become accessible once debugfs has been mounted:
> > > +
> > > +.. code-block:: sh
> > > +
> > > + mount -t debugfs none /sys/kernel/debug
> > > +
> > > +
> > > +Customization
> > > +=============
> > > +
> > > +You can enable or disable profiling for individual file and directories by
> > > +adding a line similar to the following to the respective kernel Makefile:
> > > +
> > > +- For a single file (e.g. main.o)
> > > +
> > > + .. code-block:: make
> > > +
> > > + PGO_PROFILE_main.o := y
> > > +
> > > +- For all files in one directory
> > > +
> > > + .. code-block:: make
> > > +
> > > + PGO_PROFILE := y
> > > +
> > > +To exclude files from being profiled use
> > > +
> > > + .. code-block:: make
> > > +
> > > + PGO_PROFILE_main.o := n
> > > +
> > > +and
> > > +
> > > + .. code-block:: make
> > > +
> > > + PGO_PROFILE := n
> > > +
> > > +Only files which are linked to the main kernel image or are compiled as kernel
> > > +modules are supported by this mechanism.
> > > +
> > > +
> > > +Files
> > > +=====
> > > +
> > > +The PGO kernel support creates the following files in debugfs:
> > > +
> > > +``/sys/kernel/debug/pgo``
> > > + Parent directory for all PGO-related files.
> > > +
> > > +``/sys/kernel/debug/pgo/reset``
> > > + Global reset file: resets all coverage data to zero when written to.
> > > +
> > > +``/sys/kernel/debug/profraw``
> > > + The raw PGO data that must be processed with ``llvm_profdata``.
> > > +
> > > +
> > > +Workflow
> > > +========
> > > +
> > > +The PGO kernel can be run on the host or test machines. The data though should
> > > +be analyzed with Clang's tools from the same Clang version as the kernel was
> > > +compiled. Clang's tolerant of version skew, but it's easier to use the same
> > > +Clang version.
> > > +
> > > +The profiling data is useful for optimizing the kernel, analyzing coverage,
> > > +etc. Clang offers tools to perform these tasks.
> > > +
> > > +Here is an example workflow for profiling an instrumented kernel with PGO and
> > > +using the result to optimize the kernel:
> > > +
> > > +1) Install the kernel on the TEST machine.
> > > +
> > > +2) Reset the data counters right before running the load tests
> > > +
> > > + .. code-block:: sh
> > > +
> > > + $ echo 1 > /sys/kernel/debug/pgo/reset
> > > +
> >
> > I do not get this...
> >
> > # mount | grep debugfs
> > debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
> >
> > After the load-test...?
> >
> > echo 0 > /sys/kernel/debug/pgo/reset
> >
> Writing anything to /sys/kernel/debug/pgo/reset will cause it to reset
> the profiling counters. I picked 1 (one) semi-randomly, but it could
> be any number, letter, your favorite short story, etc. You don't want
> to reset it before collecting the profiling data from your load tests
> though.
>
> > > +3) Run the load tests.
> > > +
> > > +4) Collect the raw profile data
> > > +
> > > + .. code-block:: sh
> > > +
> > > + $ cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > +
> >
> > This is only 4,9M small and seen from the date 5mins before I run the
> > echo-1 line.
> >
> > # ll /sys/kernel/debug/pgo
> > insgesamt 0
> > drwxr-xr-x 2 root root 0 16. Jan 17:29 .
> > drwx------ 41 root root 0 16. Jan 17:29 ..
> > -rw------- 1 root root 0 16. Jan 17:29 profraw
> > --w------- 1 root root 0 16. Jan 18:19 reset
> >
> > # cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> >
> > # ll /tmp/vmlinux.profraw
> > -rw------- 1 root root 4,9M 16. Jan 17:29 /tmp/vmlinux.profraw
> >
> > For me there was no prof-data collected from my defconfig kernel-build.
> >
> The /sys/kernel/debug/pgo/profraw file is read-only. Nothing writes to
> it, not even the kernel. All it does is serialize the profiling
> counters from a memory location in the kernel into a format that
> LLVM's tools can understand.
>
> > > +5) (Optional) Download the raw profile data to the HOST machine.
> > > +
> > > +6) Process the raw profile data
> > > +
> > > + .. code-block:: sh
> > > +
> > > + $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > +
> >
> > Is that executed in /path/to/linux/git?
> >
> The llvm-profdata tool is not in the linux source tree. You need to
> grab it from a clang distribution (or built from clang's git repo).
>
> > > + Note that multiple raw profile data files can be merged during this step.
> > > +
> > > +7) Rebuild the kernel using the profile data (PGO disabled)
> > > +
> > > + .. code-block:: sh
> > > +
> > > + $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> >
> > How big is vmlinux.profdata (make defconfig)?
> >
> I don't have numbers for this, but from what you listed here, it's ~5M
> in size. The size is proportional to the number of counters
> instrumented in the kernel.
>
> > Do I need to do a full defconfig build or can I stop the build after
> > let me say 10mins?
> >
> You should do a full rebuild. Make sure that PGO is disabled during the rebuild.
>
Thanks Bill for all the information.
And sorry if I am so pedantic.
I have installed my Debian system with Legacy-BIOS enabled.
When I rebuild with KCFLAGS=-fprofile-use=vmlinux.profdata (LLVM=1 I
have as a default) my system hangs on reboot.
[ diffconfig ]
$ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo
/boot/config-5.11.0-rc3-9-amd64-clang12-pgo
BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-9-amd64-clang12-pgo"
PGO_CLANG y -> n
[ my make line ]
$ cat ../start-build_5.11.0-rc3-9-amd64-clang12-pgo.txt
dileks 63120 63095 0 06:47 pts/2 00:00:00 /usr/bin/perf_5.10
stat make V=1 -j4 HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang
LD=ld.lld LLVM=1 LLVM_IAS=1 PAHOLE=/opt/pahole/bin/pahole
LOCALVERSION=-9-amd64-clang12-pgo KBUILD_VERBOSE=1
KBUILD_BUILD_HOST=iniza KBUILD_BUILD_USER=sedat.dilek@gmail.com
KBUILD_BUILD_TIMESTAMP=2021-01-17 bindeb-pkg
KDEB_PKGVERSION=5.11.0~rc3-9~bullseye+dileks1
KCFLAGS=-fprofile-use=vmlinux.profdata
( Yes, 06:47 a.m. in the morning :-). )
When I boot with the rebuild Linux-kernel I see:
Wrong EFI loader signature
...
Decompressing
Parsing EFI
Performing Relocations done.
Booting the Kernel.
*** SYSTEM HANGS ***
( I waited for approx 1 min )
I tried to turn UEFI support ON and OFF.
No success.
Does Clang-PGO support Legacy-BIOS or is something different wrong?
Thanks.
- Sedat -
next prev parent reply other threads:[~2021-01-17 10:46 UTC|newest]
Thread overview: 116+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-11 8:18 [PATCH] pgo: add clang's Profile Guided Optimization infrastructure Bill Wendling
2021-01-11 8:39 ` Sedat Dilek
2021-01-11 8:42 ` Sedat Dilek
2021-01-11 9:17 ` Bill Wendling
2021-01-11 9:57 ` Sedat Dilek
2021-01-11 18:28 ` Nathan Chancellor
2021-01-11 20:12 ` Fangrui Song
2021-01-11 20:23 ` Bill Wendling
2021-01-11 20:31 ` Fangrui Song
2021-01-12 0:37 ` Bill Wendling
2021-01-12 0:44 ` Fāng-ruì Sòng
2021-01-11 21:04 ` Nathan Chancellor
2021-01-11 21:17 ` Nick Desaulniers
2021-01-11 21:32 ` Bill Wendling
2021-01-12 5:14 ` [PATCH v2] " Bill Wendling
2021-01-12 5:17 ` Sedat Dilek
2021-01-12 5:31 ` [PATCH v3] " Bill Wendling
[not found] ` <202101121755.pyYoRozB-lkp@intel.com>
2021-01-12 17:22 ` Nathan Chancellor
2021-01-13 6:19 ` [PATCH v4] " Bill Wendling
2021-01-13 20:55 ` Nathan Chancellor
2021-01-13 21:59 ` Bill Wendling
2021-01-14 4:07 ` Nick Desaulniers
2021-01-16 0:01 ` Nick Desaulniers
2021-01-16 0:13 ` Nick Desaulniers
2021-01-16 4:30 ` Sedat Dilek
2021-01-16 5:07 ` Sedat Dilek
2021-01-16 5:18 ` Sedat Dilek
2021-01-18 0:57 ` Sedat Dilek
2021-01-13 23:01 ` Nick Desaulniers
2021-01-16 9:43 ` [PATCH v5] " Bill Wendling
2021-01-16 17:38 ` Sedat Dilek
2021-01-16 18:36 ` Sedat Dilek
2021-01-16 20:23 ` Bill Wendling
2021-01-17 10:44 ` Sedat Dilek [this message]
2021-01-17 10:53 ` Sedat Dilek
2021-01-17 11:23 ` Sedat Dilek
2021-01-17 11:42 ` Sedat Dilek
2021-01-17 11:58 ` Sedat Dilek
[not found] ` <CA+icZUXmn15w=kSq2CZzQD5JggJw_9AEam=Sz13M0KpJ68MWZg@mail.gmail.com>
2021-01-17 17:42 ` Sedat Dilek
2021-01-17 20:34 ` Bill Wendling
[not found] ` <CA+icZUU1HihUFaEHzF69+01+Picg8aq6HAqHupxiRqyDGJ=Mpw@mail.gmail.com>
[not found] ` <CA+icZUUuzA5JEXyVzKbVX+T3xeOdRAU6-mntbo+VwwTxqmN7LA@mail.gmail.com>
2021-01-18 2:32 ` Bill Wendling
2021-01-18 12:39 ` Sedat Dilek
2021-01-18 17:26 ` Sedat Dilek
2021-01-18 21:56 ` Bill Wendling
2021-01-18 23:29 ` Sedat Dilek
2021-01-21 2:03 ` Sedat Dilek
2021-01-21 22:44 ` Sedat Dilek
2021-01-22 1:42 ` Nick Desaulniers
2021-01-22 1:49 ` Sedat Dilek
2021-01-22 1:52 ` Nick Desaulniers
2021-01-20 1:02 ` Nick Desaulniers
2021-01-21 0:51 ` Nick Desaulniers
2021-01-21 8:24 ` Bill Wendling
2021-01-21 8:24 ` [PATCH v6] " Bill Wendling
2021-01-21 10:34 ` Sedat Dilek
2021-01-22 1:44 ` Nick Desaulniers
2021-01-22 1:51 ` Sedat Dilek
2021-01-22 0:14 ` Sedat Dilek
2021-01-22 0:58 ` Sedat Dilek
2021-01-22 1:29 ` Nick Desaulniers
2021-01-22 10:11 ` [PATCH v7] " Bill Wendling
2021-01-22 11:31 ` Sedat Dilek
2021-01-22 18:41 ` Nick Desaulniers
[not found] ` <CA+icZUU=XfwqMcXYonQKcD4QgqTBW-mA+d_84b7cU2R3HYPOSQ@mail.gmail.com>
2021-01-28 21:12 ` Nick Desaulniers
2021-01-28 21:19 ` Sedat Dilek
2021-01-28 21:24 ` Nick Desaulniers
2021-01-28 21:39 ` Sedat Dilek
2021-01-29 7:43 ` Sedat Dilek
2021-01-29 21:48 ` Nick Desaulniers
2021-02-10 23:25 ` Bill Wendling
2021-02-22 21:52 ` Bill Wendling
2021-02-26 22:20 ` [PATCH v8] " Bill Wendling
2021-02-26 22:55 ` Bill Wendling
2021-02-28 18:52 ` Fangrui Song
2021-02-28 21:50 ` Fangrui Song
2021-01-12 17:37 ` [PATCH v2] " Nick Desaulniers
2021-01-12 17:45 ` Fāng-ruì Sòng
2021-01-21 2:21 ` [PATCH] " Sedat Dilek
2021-01-22 1:34 ` Nick Desaulniers
2021-01-22 1:43 ` Sedat Dilek
2021-04-07 21:17 ` [PATCH v9] " Bill Wendling
2021-04-07 21:22 ` Kees Cook
2021-04-07 21:44 ` Fāng-ruì Sòng
2021-04-07 21:47 ` Nathan Chancellor
2021-04-07 21:58 ` Bill Wendling
2021-05-19 21:37 ` Kees Cook
2021-05-22 23:51 ` Bill Wendling
2021-05-31 21:12 ` Nathan Chancellor
2021-06-01 17:31 ` Nick Desaulniers
2021-06-12 16:59 ` Peter Zijlstra
2021-06-12 17:25 ` Bill Wendling
2021-06-12 18:15 ` Peter Zijlstra
2021-06-12 19:10 ` Bill Wendling
2021-06-12 19:28 ` Bill Wendling
2021-06-12 20:25 ` Peter Zijlstra
2021-06-12 20:56 ` Bill Wendling
2021-06-12 22:47 ` Bill Wendling
2021-06-13 18:07 ` Bill Wendling
2021-06-14 9:43 ` Peter Zijlstra
2021-06-14 10:18 ` Peter Zijlstra
2021-06-14 7:51 ` Peter Zijlstra
2021-06-14 9:01 ` Peter Zijlstra
2021-06-14 9:39 ` Bill Wendling
2021-06-14 10:44 ` Peter Zijlstra
2021-06-14 11:41 ` Bill Wendling
2021-06-14 11:43 ` Bill Wendling
2021-06-14 14:16 ` Marco Elver
2021-06-14 15:26 ` Kees Cook
2021-06-14 15:35 ` Peter Zijlstra
2021-06-14 16:22 ` Kees Cook
2021-06-14 18:07 ` Nick Desaulniers
2021-06-14 20:49 ` Nick Desaulniers
2021-06-14 15:46 ` Peter Zijlstra
2021-06-14 16:03 ` Nick Desaulniers
2021-06-12 20:20 ` Fangrui Song
2021-06-12 20:31 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CA+icZUWf05ek+DFsJNyBc-4cg0s6cVrn=rNJDyL4RJ6=fMO5NA@mail.gmail.com' \
--to=sedat.dilek@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=clang-built-linux@googlegroups.com \
--cc=corbet@lwn.net \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kbuild@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=masahiroy@kernel.org \
--cc=morbo@google.com \
--cc=natechancellor@gmail.com \
--cc=ndesaulniers@google.com \
--cc=samitolvanen@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).