From: Fangrui Song <maskray@google.com>
To: Nick Desaulniers <ndesaulniers@google.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
Tom Stellard <tstellar@redhat.com>,
felixonmars@archlinux.org, Masahiro Yamada <masahiroy@kernel.org>,
Nathan Chancellor <nathan@kernel.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
clang-built-linux <clang-built-linux@googlegroups.com>,
Serge Guelton <sguelton@redhat.com>,
Sylvestre Ledru <sylvestre@mozilla.com>
Subject: Re: Very slow clang kernel config ..
Date: Fri, 30 Apr 2021 19:16:42 -0700 [thread overview]
Message-ID: <20210501021642.yertoclugdbqq6m4@google.com> (raw)
In-Reply-To: <CAKwvOdk0nxxUATg2jEKgx4HutXCMXcW92SX3DT+uCTgqBwQHBg@mail.gmail.com>
On 2021-04-30, Nick Desaulniers wrote:
>On Fri, Apr 30, 2021 at 6:22 PM Linus Torvalds
><torvalds@linux-foundation.org> wrote:
>>
>> On Fri, Apr 30, 2021 at 5:25 PM Nick Desaulniers
>> <ndesaulniers@google.com> wrote:
>> >
>> > Ah, no, sorry, these are the runtime link editor/loader. So probably
>> > spending quite some time resolving symbols in large binaries.
>>
>> Yeah. Appended is the profile I see when I profile that "make
>> oldconfig", so about 45% of all time seems to be spent in just symbol
>> lookup and relocation.
>>
>> And a fair amount of time just creating and tearing down that huge
>> executable (with a lot of copy-on-write overhead too), with the kernel
>> side of that being another 15%. The cost of that is likely also fairly
>> directly linked to all the dynamic linking costs, which brings in all
>> that data.
>>
>> Just to compare, btw, this is the symbol lookup overhead for the gcc case:
>>
>> 1.43% ld-2.33.so do_lookup_x
>> 0.96% ld-2.33.so _dl_relocate_object
>> 0.69% ld-2.33.so _dl_lookup_symbol_x
>>
>> so it really does seem to be something very odd going on with the clang binary.
>>
>> Maybe the Fedora binary is built some odd way, but it's likely just
>> the default clang build.
>>
>> Linus
>>
>> ----
>> 23.59% ld-2.33.so _dl_lookup_symbol_x
>> 11.41% ld-2.33.so _dl_relocate_object
>> 9.95% ld-2.33.so do_lookup_x
>> 4.00% [kernel.vmlinux] copy_page
>> 3.98% [kernel.vmlinux] next_uptodate_page
>> 3.05% [kernel.vmlinux] zap_pte_range
>> 1.81% [kernel.vmlinux] clear_page_rep
>> 1.68% [kernel.vmlinux] asm_exc_page_fault
>> 1.33% ld-2.33.so strcmp
>> 1.33% ld-2.33.so check_match
>
>47.61% spent in symbol table lookup. Nice. (Not counting probably a
>fair amount of the libc calls below).
>
>> 0.92% libLLVM-12.so llvm::StringMapImpl::LookupBucketFor
>
>^ wait a minute; notice how in your profile the `Shared Object` is
>attributed to `libLLVM-12.so` while mine is `clang-13`? Clang can be
>built as either having libllvm statically linked or dynamically; see
>the cmake variables
>LLVM_BUILD_LLVM_DYLIB:BOOL
>LLVM_LINK_LLVM_DYLIB:BOOL
>BUILD_SHARED_LIBS:BOOL
>https://llvm.org/docs/CMake.html
>
>I think those are frowned upon; useful for cutting down on developers
>iteration speed due to not having to relink llvm when developing
>clang. But shipping that in production? I just checked and it doesn't
>look like we do that for AOSP's build of LLVM.
>
>Tom, is one of the above intentionally set for clang builds on Fedora?
>I'm guessing it's intentional that there are packages for
>libLLVM-12.so and libclang-cpp.so.12, perhaps they have other
>dependents?
LLVM_LINK_LLVM_DYLIB (linking against libLLVM.so instead of libLLVM*.a)
has been around for a while.
Tom added CLANG_LINK_CLANG_DYLIB in 2019
(https://reviews.llvm.org/D63503 link against libclang-cpp.so instead of
libclang*.a or libclang*.so) :) So I'd guess this is a concious decision
for Fedora.
Arch Linux has switched to -DCLANG_LINK_CLANG_DYLIB=on as well
https://github.com/archlinux/svntogit-packages/blob/packages/clang/trunk/PKGBUILD
This is useful to make the total size of LLVM/clang dependent packages
(ccls, zig, etc) small.
If we don't let distributions use libLLVM.so libclang-cpp.so, hmmmm, I guess
their only choice will be crunchgen[1]-style
clang+lld+llvm-objcopy+llvm-objdump+llvm-ar+llvm-nm+llvm-strings+llvm-readelf+...+clang-format+clang-offload-bundler+...
(executables from packages which are usually named llvm, clang, and clang-tools)
[1]: https://www.freebsd.org/cgi/man.cgi?query=crunchgen&sektion=1
>> 0.83% [kernel.vmlinux] rmqueue_bulk
>> 0.77% conf yylex
>> 0.75% libc-2.33.so __gconv_transform_utf8_internal
>> 0.74% libc-2.33.so _int_malloc
>> 0.69% libc-2.33.so __strlen_avx2
>> 0.62% [kernel.vmlinux] pagecache_get_page
>> 0.58% [kernel.vmlinux] page_remove_rmap
>> 0.56% [kernel.vmlinux] __handle_mm_fault
>> 0.54% [kernel.vmlinux] filemap_map_pages
>> 0.54% libc-2.33.so __strcmp_avx2
>> 0.54% [kernel.vmlinux] __free_one_page
>> 0.52% [kernel.vmlinux] release_pages
>--
>Thanks,
>~Nick Desaulniers
next prev parent reply other threads:[~2021-05-01 2:16 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-29 21:53 Very slow clang kernel config Linus Torvalds
2021-04-30 0:19 ` Nick Desaulniers
2021-04-30 2:22 ` Nick Desaulniers
2021-05-01 0:19 ` Nick Desaulniers
2021-05-01 0:23 ` Nick Desaulniers
2021-05-01 0:25 ` Nick Desaulniers
2021-05-01 0:40 ` Nick Desaulniers
2021-05-01 1:22 ` Linus Torvalds
2021-05-01 1:48 ` Nick Desaulniers
2021-05-01 2:16 ` Fangrui Song [this message]
2021-05-01 3:32 ` Tom Stellard
2021-05-01 16:32 ` Linus Torvalds
2021-05-01 19:57 ` Serge Guelton
2021-05-01 22:39 ` Linus Torvalds
2021-05-01 23:55 ` Fangrui Song
2021-05-01 21:58 ` David Laight
2021-05-02 9:31 ` Adrian Bunk
2021-05-02 11:35 ` David Laight
2021-05-02 16:12 ` Linus Torvalds
2021-05-02 16:45 ` Adrian Bunk
2021-05-02 16:49 ` Linus Torvalds
2021-05-02 17:55 ` Adrian Bunk
2021-05-02 17:59 ` Linus Torvalds
2021-05-02 21:48 ` Adrian Bunk
2021-05-04 22:02 ` Miguel Ojeda
2021-05-05 0:58 ` Theodore Ts'o
2021-05-05 17:21 ` Miguel Ojeda
2021-05-04 21:32 ` Miguel Ojeda
2021-05-05 11:05 ` David Laight
2021-05-05 13:53 ` Miguel Ojeda
2021-05-05 14:13 ` David Laight
2021-05-05 16:06 ` Miguel Ojeda
2021-05-05 16:25 ` David Laight
2021-05-05 17:55 ` Miguel Ojeda
2021-05-03 1:03 ` Maciej W. Rozycki
2021-05-03 14:38 ` Theodore Ts'o
2021-05-03 14:54 ` Theodore Ts'o
2021-05-03 17:14 ` Maciej W. Rozycki
2021-05-03 16:09 ` David Laight
2021-05-04 23:04 ` Greg Stark
2021-05-05 0:55 ` Theodore Ts'o
2021-05-01 23:37 ` Mike Hommey
2021-05-02 5:19 ` Dan Aloni
2021-05-03 16:48 ` Tom Stellard
2021-05-03 19:00 ` Fangrui Song
2021-04-30 0:52 ` Nathan Chancellor
2021-04-30 2:21 ` Nick Desaulniers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210501021642.yertoclugdbqq6m4@google.com \
--to=maskray@google.com \
--cc=clang-built-linux@googlegroups.com \
--cc=felixonmars@archlinux.org \
--cc=linux-kernel@vger.kernel.org \
--cc=masahiroy@kernel.org \
--cc=nathan@kernel.org \
--cc=ndesaulniers@google.com \
--cc=sguelton@redhat.com \
--cc=sylvestre@mozilla.com \
--cc=torvalds@linux-foundation.org \
--cc=tstellar@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).