From: Nick Desaulniers <ndesaulniers@google.com> To: Linus Torvalds <torvalds@linux-foundation.org>, Tom Stellard <tstellar@redhat.com> Cc: Masahiro Yamada <masahiroy@kernel.org>, Nathan Chancellor <nathan@kernel.org>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, clang-built-linux <clang-built-linux@googlegroups.com>, Fangrui Song <maskray@google.com>, Serge Guelton <sguelton@redhat.com>, Sylvestre Ledru <sylvestre@mozilla.com> Subject: Re: Very slow clang kernel config .. Date: Fri, 30 Apr 2021 18:48:11 -0700 [thread overview] Message-ID: <CAKwvOdk0nxxUATg2jEKgx4HutXCMXcW92SX3DT+uCTgqBwQHBg@mail.gmail.com> (raw) In-Reply-To: <CAHk-=wi1yiBBr3b3RbCEte6-yzAApsZN5zRdr3xoW8Av9jOX=Q@mail.gmail.com> On Fri, Apr 30, 2021 at 6:22 PM Linus Torvalds <torvalds@linux-foundation.org> wrote: > > On Fri, Apr 30, 2021 at 5:25 PM Nick Desaulniers > <ndesaulniers@google.com> wrote: > > > > Ah, no, sorry, these are the runtime link editor/loader. So probably > > spending quite some time resolving symbols in large binaries. > > Yeah. Appended is the profile I see when I profile that "make > oldconfig", so about 45% of all time seems to be spent in just symbol > lookup and relocation. > > And a fair amount of time just creating and tearing down that huge > executable (with a lot of copy-on-write overhead too), with the kernel > side of that being another 15%. The cost of that is likely also fairly > directly linked to all the dynamic linking costs, which brings in all > that data. > > Just to compare, btw, this is the symbol lookup overhead for the gcc case: > > 1.43% ld-2.33.so do_lookup_x > 0.96% ld-2.33.so _dl_relocate_object > 0.69% ld-2.33.so _dl_lookup_symbol_x > > so it really does seem to be something very odd going on with the clang binary. > > Maybe the Fedora binary is built some odd way, but it's likely just > the default clang build. > > Linus > > ---- > 23.59% ld-2.33.so _dl_lookup_symbol_x > 11.41% ld-2.33.so _dl_relocate_object > 9.95% ld-2.33.so do_lookup_x > 4.00% [kernel.vmlinux] copy_page > 3.98% [kernel.vmlinux] next_uptodate_page > 3.05% [kernel.vmlinux] zap_pte_range > 1.81% [kernel.vmlinux] clear_page_rep > 1.68% [kernel.vmlinux] asm_exc_page_fault > 1.33% ld-2.33.so strcmp > 1.33% ld-2.33.so check_match 47.61% spent in symbol table lookup. Nice. (Not counting probably a fair amount of the libc calls below). > 0.92% libLLVM-12.so llvm::StringMapImpl::LookupBucketFor ^ wait a minute; notice how in your profile the `Shared Object` is attributed to `libLLVM-12.so` while mine is `clang-13`? Clang can be built as either having libllvm statically linked or dynamically; see the cmake variables LLVM_BUILD_LLVM_DYLIB:BOOL LLVM_LINK_LLVM_DYLIB:BOOL BUILD_SHARED_LIBS:BOOL https://llvm.org/docs/CMake.html I think those are frowned upon; useful for cutting down on developers iteration speed due to not having to relink llvm when developing clang. But shipping that in production? I just checked and it doesn't look like we do that for AOSP's build of LLVM. Tom, is one of the above intentionally set for clang builds on Fedora? I'm guessing it's intentional that there are packages for libLLVM-12.so and libclang-cpp.so.12, perhaps they have other dependents? > 0.83% [kernel.vmlinux] rmqueue_bulk > 0.77% conf yylex > 0.75% libc-2.33.so __gconv_transform_utf8_internal > 0.74% libc-2.33.so _int_malloc > 0.69% libc-2.33.so __strlen_avx2 > 0.62% [kernel.vmlinux] pagecache_get_page > 0.58% [kernel.vmlinux] page_remove_rmap > 0.56% [kernel.vmlinux] __handle_mm_fault > 0.54% [kernel.vmlinux] filemap_map_pages > 0.54% libc-2.33.so __strcmp_avx2 > 0.54% [kernel.vmlinux] __free_one_page > 0.52% [kernel.vmlinux] release_pages -- Thanks, ~Nick Desaulniers
next prev parent reply other threads:[~2021-05-01 1:49 UTC|newest] Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-04-29 21:53 Linus Torvalds 2021-04-30 0:19 ` Nick Desaulniers 2021-04-30 2:22 ` Nick Desaulniers 2021-05-01 0:19 ` Nick Desaulniers 2021-05-01 0:23 ` Nick Desaulniers 2021-05-01 0:25 ` Nick Desaulniers 2021-05-01 0:40 ` Nick Desaulniers 2021-05-01 1:22 ` Linus Torvalds 2021-05-01 1:48 ` Nick Desaulniers [this message] 2021-05-01 2:16 ` Fangrui Song 2021-05-01 3:32 ` Tom Stellard 2021-05-01 16:32 ` Linus Torvalds 2021-05-01 19:57 ` Serge Guelton 2021-05-01 22:39 ` Linus Torvalds 2021-05-01 23:55 ` Fangrui Song 2021-05-01 21:58 ` David Laight 2021-05-02 9:31 ` Adrian Bunk 2021-05-02 11:35 ` David Laight 2021-05-02 16:12 ` Linus Torvalds 2021-05-02 16:45 ` Adrian Bunk 2021-05-02 16:49 ` Linus Torvalds 2021-05-02 17:55 ` Adrian Bunk 2021-05-02 17:59 ` Linus Torvalds 2021-05-02 21:48 ` Adrian Bunk 2021-05-04 22:02 ` Miguel Ojeda 2021-05-05 0:58 ` Theodore Ts'o 2021-05-05 17:21 ` Miguel Ojeda 2021-05-04 21:32 ` Miguel Ojeda 2021-05-05 11:05 ` David Laight 2021-05-05 13:53 ` Miguel Ojeda 2021-05-05 14:13 ` David Laight 2021-05-05 16:06 ` Miguel Ojeda 2021-05-05 16:25 ` David Laight 2021-05-05 17:55 ` Miguel Ojeda 2021-05-03 1:03 ` Maciej W. Rozycki 2021-05-03 14:38 ` Theodore Ts'o 2021-05-03 14:54 ` Theodore Ts'o 2021-05-03 17:14 ` Maciej W. Rozycki 2021-05-03 16:09 ` David Laight 2021-05-04 23:04 ` Greg Stark 2021-05-05 0:55 ` Theodore Ts'o 2021-05-01 23:37 ` Mike Hommey 2021-05-02 5:19 ` Dan Aloni 2021-05-03 16:48 ` Tom Stellard 2021-05-03 19:00 ` Fangrui Song 2021-04-30 0:52 ` Nathan Chancellor 2021-04-30 2:21 ` Nick Desaulniers
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=CAKwvOdk0nxxUATg2jEKgx4HutXCMXcW92SX3DT+uCTgqBwQHBg@mail.gmail.com \ --to=ndesaulniers@google.com \ --cc=clang-built-linux@googlegroups.com \ --cc=linux-kernel@vger.kernel.org \ --cc=masahiroy@kernel.org \ --cc=maskray@google.com \ --cc=nathan@kernel.org \ --cc=sguelton@redhat.com \ --cc=sylvestre@mozilla.com \ --cc=torvalds@linux-foundation.org \ --cc=tstellar@redhat.com \ --subject='Re: Very slow clang kernel config ..' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).