Re: Very slow clang kernel config ..

From: Tom Stellard <tstellar@redhat.com>
To: Nick Desaulniers <ndesaulniers@google.com>,
	Linus Torvalds <torvalds@linux-foundation.org>
Cc: Masahiro Yamada <masahiroy@kernel.org>,
	Nathan Chancellor <nathan@kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	clang-built-linux <clang-built-linux@googlegroups.com>,
	Fangrui Song <maskray@google.com>,
	Serge Guelton <sguelton@redhat.com>,
	Sylvestre Ledru <sylvestre@mozilla.com>
Subject: Re: Very slow clang kernel config ..
Date: Fri, 30 Apr 2021 20:32:29 -0700	[thread overview]
Message-ID: <1c5e05fa-a246-9456-ff4e-287960acb18c@redhat.com> (raw)
In-Reply-To: <CAKwvOdk0nxxUATg2jEKgx4HutXCMXcW92SX3DT+uCTgqBwQHBg@mail.gmail.com>

On 4/30/21 6:48 PM, Nick Desaulniers wrote:
> On Fri, Apr 30, 2021 at 6:22 PM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>>
>> On Fri, Apr 30, 2021 at 5:25 PM Nick Desaulniers
>> <ndesaulniers@google.com> wrote:
>>>
>>> Ah, no, sorry, these are the runtime link editor/loader. So probably
>>> spending quite some time resolving symbols in large binaries.
>>
>> Yeah. Appended is the profile I see when I profile that "make
>> oldconfig", so about 45% of all time seems to be spent in just symbol
>> lookup and relocation.
>>
>> And a fair amount of time just creating and tearing down that huge
>> executable (with a lot of copy-on-write overhead too), with the kernel
>> side of that being another 15%. The cost of that is likely also fairly
>> directly linked to all the dynamic linking costs, which brings in all
>> that data.
>>
>> Just to compare, btw, this is the symbol lookup overhead for the gcc case:
>>
>>     1.43%  ld-2.33.so             do_lookup_x
>>     0.96%  ld-2.33.so             _dl_relocate_object
>>     0.69%  ld-2.33.so             _dl_lookup_symbol_x
>>
>> so it really does seem to be something very odd going on with the clang binary.
>>
>> Maybe the Fedora binary is built some odd way, but it's likely just
>> the default clang build.
>>
>>               Linus
>>
>> ----
>>    23.59%  ld-2.33.so          _dl_lookup_symbol_x
>>    11.41%  ld-2.33.so          _dl_relocate_object
>>     9.95%  ld-2.33.so          do_lookup_x
>>     4.00%  [kernel.vmlinux]    copy_page
>>     3.98%  [kernel.vmlinux]    next_uptodate_page
>>     3.05%  [kernel.vmlinux]    zap_pte_range
>>     1.81%  [kernel.vmlinux]    clear_page_rep
>>     1.68%  [kernel.vmlinux]    asm_exc_page_fault
>>     1.33%  ld-2.33.so          strcmp
>>     1.33%  ld-2.33.so          check_match
> 
> 47.61% spent in symbol table lookup. Nice. (Not counting probably a
> fair amount of the libc calls below).
> 
>>     0.92%  libLLVM-12.so       llvm::StringMapImpl::LookupBucketFor
> 
> ^ wait a minute; notice how in your profile the `Shared Object` is
> attributed to `libLLVM-12.so` while mine is `clang-13`?  Clang can be
> built as either having libllvm statically linked or dynamically; see
> the cmake variables
> LLVM_BUILD_LLVM_DYLIB:BOOL
> LLVM_LINK_LLVM_DYLIB:BOOL
> BUILD_SHARED_LIBS:BOOL
> https://llvm.org/docs/CMake.html
> 
> I think those are frowned upon; useful for cutting down on developers
> iteration speed due to not having to relink llvm when developing
> clang. But shipping that in production? I just checked and it doesn't
> look like we do that for AOSP's build of LLVM.
> 

BUILD_SHARED_LIBS is the only one that is discouraged and we don't use
that in Fedora any more.  We just use LLVM_LINK_LLVM_DYLIB and the
clang equivalent.

> Tom, is one of the above intentionally set for clang builds on Fedora?
> I'm guessing it's intentional that there are packages for
> libLLVM-12.so and libclang-cpp.so.12, perhaps they have other
> dependents?
> 

Yes, it's intentional.  Dynamic linking libraries from other packages is
the Fedora policy[1], and clang and llvm are separate packages (in Fedora).

- Tom

[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/#_statically_linking_executables

>>     0.83%  [kernel.vmlinux]    rmqueue_bulk
>>     0.77%  conf                yylex
>>     0.75%  libc-2.33.so        __gconv_transform_utf8_internal
>>     0.74%  libc-2.33.so        _int_malloc
>>     0.69%  libc-2.33.so        __strlen_avx2
>>     0.62%  [kernel.vmlinux]    pagecache_get_page
>>     0.58%  [kernel.vmlinux]    page_remove_rmap
>>     0.56%  [kernel.vmlinux]    __handle_mm_fault
>>     0.54%  [kernel.vmlinux]    filemap_map_pages
>>     0.54%  libc-2.33.so        __strcmp_avx2
>>     0.54%  [kernel.vmlinux]    __free_one_page
>>     0.52%  [kernel.vmlinux]    release_pages