Re: [PATCH dwarves 3/3] dwarf_loader: add option to merge more dwarf cu's into one pahole cu

[Yonghong Song <yhs@fb.com>]

On 3/26/21 7:41 AM, Arnaldo Carvalho de Melo wrote:
> Em Wed, Mar 24, 2021 at 11:53:32PM -0700, Yonghong Song escreveu:
>> This patch added an option "merge_cus", which will permit
>> to merge all debug info cu's into one pahole cu.
>> For vmlinux built with clang thin-lto or lto, there exist
>> cross cu type references. For example, you could have
>>    compile unit 1:
>>       tag 10:  type A
>>    compile unit 2:
>>       ...
>>         refer to type A (tag 10 in compile unit 1)
>> I only checked a few but have seen type A may be a simple type
>> like "unsigned char" or a complex type like an array of base types.
>>
>> There are two different ways to resolve this issue:
>> (1). merge all compile units as one pahole cu so tags/types
>>       can be resolved easily, or
>> (2). try to do on-demand type traversal in other debuginfo cu's
>>       when we do die_process().
>> The method (2) is much more complicated so I picked method (1).
>> An option "merge_cus" is added to permit such an operation.
>>
>> Merging cu's will create a single cu with lots of types, tags
>> and functions. For example with clang thin-lto built vmlinux,
>> I saw 9M entries in types table, 5.2M in tags table. The
>> below are pahole wallclock time for different hashbits:
>> command line: time pahole -J --merge_cus vmlinux
>>        # of hashbits            wallclock time in seconds
>>            15                       460
>>            16                       255
>>            17                       131
>>            18                       97
>>            19                       75
>>            20                       69
>>            21                       64
>>            22                       62
>>            23                       58
>>            24                       64
>>
>> Note that the number of hashbits 24 makes performance worse
>> than 23. The reason could be that 23 hashbits can cover 8M
>> buckets (close to 9M for the number of entries in types table).
>> Higher number of hash bits allocates more memory and becomes
>> less cache efficient compared to 23 hashbits.
>>
>> This patch picks # of hashbits 21 as the starting value
>> and will try to allocate memory based on that, if memory
>> allocation fails, we will go with less hashbits until
>> we reach hashbits 15 which is the default for
>> non merge-cu case.
> 
> I'll probably add a way to specify the starting max_hashbits to be able
> to use 'perf stat' to show what causes the performance difference.

The problem is with hashtags__find(), esp. the loop

         uint32_t bucket = hashtags__fn(id);
         const struct hlist_head *head = hashtable + bucket;

         hlist_for_each_entry(tpos, pos, head, hash_node) {
                 if (tpos->id == id)
                         return tpos;
         }

Say we have 8M types and (1 << 15) buckets, that means
each bucket will 64 elements. So each lookup will traverse
the loop 32 iterations on average.

If we have 1 << 21 buckets, then each buckets will have 4 elements,
and the average number of loop iterations for hashtags__find()
will be 2.

If the patch needs respin, I can add the above descriptions
in the commit message.

> 
> I'm also adding the man page patch below, now to build the kernel with
> your bpf-next patch to test it.

Thanks for adding man page and testing, let me know if you
need any help!

> 
> - Arnaldo
> 
> [acme@five pahole]$ git diff
> diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
> index cbbefbf22556412c..1be2a293ad4bcc50 100644
> --- a/man-pages/pahole.1
> +++ b/man-pages/pahole.1
> @@ -208,6 +208,12 @@ information has float types.
>   .B \-\-btf_gen_all
>   Allow using all the BTF features supported by pahole.
> 
> +.TP
> +.B \-\-merge_cus
> +Merge all cus (except possible types_cu) when loading DWARF, this is needed
> +when processing files that have inter-CU references, this happens, for instance
> +when building the Linux kernel with clang using thin-LTO or LTO.
> +
>   .TP
>   .B \-l, \-\-show_first_biggest_size_base_type_member
>   Show first biggest size base_type member.
> [acme@five pahole]$
>