[FIXED] Re: 5:11: in-kernel BTF is malformed

* [FIXED] Re: 5:11: in-kernel BTF is malformed
       [not found]               ` <20210204132625.GB910119@kernel.org>
@ 2021-02-04 16:33                 ` Arnaldo Carvalho de Melo
  2021-02-05  3:10                   ` Chris Murphy
  0 siblings, 1 reply; 3+ messages in thread
From: Arnaldo Carvalho de Melo @ 2021-02-04 16:33 UTC (permalink / raw)
  To: Chris Murphy
  Cc: bpf, Jiri Olsa, dwarves, Nick Desaulniers, Yonghong Song,
	Andrii Nakryiko, Masahiro Yamada, Nathan Chancellor,
	Andrew Morton, LKML, Clang-Built-Linux ML,
	Linux Kbuild mailing list, linux-arch, Jakub Jelinek,
	Fangrui Song, Caroline Tice, Nick Clifton

Em Thu, Feb 04, 2021 at 10:26:25AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Wed, Feb 03, 2021 at 07:10:52PM -0700, Chris Murphy escreveu:
> > On Wed, Feb 3, 2021 at 7:05 PM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> > > So it's a bitfield offset breakage that should be fixed in pahole 1.20.
 
> > Rawhide is currently still on
> > dwarves-1.19-2.fc34.x86_64
 
> > This might also be related:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1922707#c9
 
> With about to be released pahole v1.20 it all seems to work, tldr;
 
> [root@seventh ~]# ~acme/git/pahole/btfdiff /lib/modules/`uname -r`/build/vmlinux
> [root@seventh ~]#
 
> Same thing. I'll do more tests after some errands, but I've put the pre-release
> rpm packages for 1.20 at:
 
> http://vger.kernel.org/~acme/pahole/rpms/1.20-0/RPMS/x86_64/Packages/
 
> Can you please try it with gcc 11?

Looking at the original source code in the kernel:

        /* Used for emulating ABI behavior of previous Linux versions: */
        unsigned int                    personality;

        /* Scheduler bits, serialized by scheduler locks: */
        unsigned                        sched_reset_on_fork:1;
        unsigned                        sched_contributes_to_load:1;
        unsigned                        sched_migrated:1;
#ifdef CONFIG_PSI
        unsigned                        sched_psi_wake_requeue:1;
#endif

        /* Force alignment to the next boundary: */
        unsigned                        :0;

        /* Unserialized, strictly 'current' */

        /*
         * This field must not be in the scheduler word above due to wakelist
         * queueing no longer being serialized by p->on_cpu. However:
         *
         * p->XXX = X;                  ttwu()
         * schedule()                     if (p->on_rq && ..) // false
         *   smp_mb__after_spinlock();    if (smp_load_acquire(&p->on_cpu) && //true
         *   deactivate_task()                ttwu_queue_wakelist())
         *     p->on_rq = 0;                    p->sched_remote_wakeup = Y;
         *
         * guarantees all stores of 'current' are visible before
         * ->sched_remote_wakeup gets used, so it can be in this word.
         */
        unsigned                        sched_remote_wakeup:1;

        /* Bit to tell LSMs we're in execve(): */
        unsigned                        in_execve:1;
        unsigned                        in_iowait:1;

Then the 'struct task_struct' as reconstructed by pahole from:
  
  $ ls -la /sys/kernel/btf/vmlinux
  -r--r--r--. 1 root root 4560322 Feb  4 10:16 /sys/kernel/btf/vmlinux
  $ ls -lah /sys/kernel/btf/vmlinux
  -r--r--r--. 1 root root 4.4M Feb  4 10:16 /sys/kernel/btf/vmlinux
  $

That was generated by 'pahole -J' from the default DWARF version
generated by gcc 11, we see that 'unsigned :0;' preserved (and that
isn't directly encoded in DWARF, so, to produce source code from it the
tool needs to add that "manually") and the byte offset (not
DW_AT_data_member_location isn't provided in such case) calculated from
the DW_AT_bit_offset (DWARF4/5 only stuff):

[acme@seventh perf]$ readelf -wi ../build/v5.11.0-rc6+/vmlinux | grep -m1 sched_reset_on_fork -A40 | grep -v DW_AT_decl_
    <1149>   DW_AT_name        : (indirect string, offset: 0x53da): sched_reset_on_fork
    <1151>   DW_AT_type        : <0x69>
    <1155>   DW_AT_bit_size    : 1
    <1155>   DW_AT_data_bit_offset: 18208
 <2><1157>: Abbrev Number: 29 (DW_TAG_member)
    <1158>   DW_AT_name        : (indirect string, offset: 0x3940): sched_contributes_to_load
    <1160>   DW_AT_type        : <0x69>
    <1164>   DW_AT_bit_size    : 1
    <1164>   DW_AT_data_bit_offset: 18209
 <2><1166>: Abbrev Number: 29 (DW_TAG_member)
    <1167>   DW_AT_name        : (indirect string, offset: 0x4883): sched_migrated
    <116f>   DW_AT_type        : <0x69>
    <1173>   DW_AT_bit_size    : 1
    <1173>   DW_AT_data_bit_offset: 18210
 <2><1175>: Abbrev Number: 29 (DW_TAG_member)
    <1176>   DW_AT_name        : (indirect string, offset: 0x3245): sched_psi_wake_requeue
    <117e>   DW_AT_type        : <0x69>
    <1182>   DW_AT_bit_size    : 1
    <1182>   DW_AT_data_bit_offset: 18211
 <2><1184>: Abbrev Number: 29 (DW_TAG_member)
    <1185>   DW_AT_name        : (indirect string, offset: 0xe4b): sched_remote_wakeup
    <118d>   DW_AT_type        : <0x69>
    <1191>   DW_AT_bit_size    : 1
    <1191>   DW_AT_data_bit_offset: 18240
 <2><1193>: Abbrev Number: 29 (DW_TAG_member)
    <1194>   DW_AT_name        : (indirect string, offset: 0x3ad4): in_execve
[acme@seventh perf]$ 

Looking at the options passed to gcc:

[acme@seventh perf]$ readelf -wi ../build/v5.11.0-rc6+/vmlinux | grep DW_AT_producer -m2
    <1c>   DW_AT_producer    : (indirect string, offset: 0x51): GNU AS 2.35.1
    <2f>   DW_AT_producer    : (indirect string, offset: 0x124a): GNU C89 11.0.0 20210123 (Red Hat 11.0.0-0) -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -mindirect-branch=thunk-extern -mindirect-branch-register -mrecord-mcount -mfentry -march=x86-64 -g -O2 -std=gnu90 -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -falign-jumps=1 -falign-loops=1 -fno-asynchronous-unwind-tables -fno-jump-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fno-strict-overflow -fstack-check=no -fconserve-stack -fcf-protection=none -fno-stack-protector -fno-builtin
[acme@seventh perf]$

its just plain '-g'

Also using pahole's friend, fullcircle, we can see that it uses pfunct
to generate a compileable .c file from the DWARF thing, building it with
the same set of options used to build a .o file and then compares the
degugging info in the .c generated from the debugging info and checks
it matches, i.e. full circle.

[acme@seventh tmp]$ cp ~/git/build/v5.11.0-rc6+/net/ipv4/tcp.o .
[acme@seventh tmp]$ pahole -C task_struct tcp.o | tail

	/* --- cacheline 194 boundary (12416 bytes) --- */
	struct thread_struct       thread __attribute__((__aligned__(64))); /* 12416  4352 */

	/* size: 16768, cachelines: 262, members: 252 */
	/* sum members: 16579, holes: 22, sum holes: 173 */
	/* sum bitfield members: 77 bits, bit holes: 2, sum bit holes: 51 bits */
	/* paddings: 4, sum paddings: 21 */
	/* forced alignments: 8, forced holes: 2, sum forced holes: 96 */
} __attribute__((__aligned__(64)));
[acme@seventh tmp]$
[acme@seventh tmp]$ ~acme/git/pahole/fullcircle tcp.o
[acme@seventh tmp]$

For reference, here is the script:

  https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/fullcircle

Then the next test is to build and use tools/bpf/runqslower/ that uses
CO-RE, i.e. it touches task_struct fields, etc, after building it:

[root@seventh linux]# tools/bpf/runqslower/.output/runqslower
Tracing run queue latency higher than 10000 us
TIME     COMM             PID           LAT(us)

And on the other terminal:

[root@seventh ~]# bpftool prog | grep handle__sched -A3
93: type 26  name handle__sched_w  tag 77e6e21acbdbb0f6  gpl
	loaded_at 2021-02-04T13:22:32-0300  uid 0
	xlated 152B  jited 93B  memlock 4096B  map_ids 23,21
	btf_id 194
95: type 26  name handle__sched_w  tag 77e6e21acbdbb0f6  gpl
	loaded_at 2021-02-04T13:22:32-0300  uid 0
	xlated 152B  jited 93B  memlock 4096B  map_ids 23,21
	btf_id 194
96: type 26  name handle__sched_s  tag 1c25b248d7358175  gpl
	loaded_at 2021-02-04T13:22:32-0300  uid 0
	xlated 560B  jited 332B  memlock 4096B  map_ids 23,21,22
	btf_id 194
[root@seventh ~]#

Ok, system is running smoothly, lets try with a lower threshold:

[root@seventh linux]# tools/bpf/runqslower/.output/runqslower 1000
Tracing run queue latency higher than 1000 us
TIME     COMM             PID           LAT(us)
13:24:31 swapper/3        5070             1405
13:24:31 swapper/2        2037             1369
13:24:31 swapper/3        5070             1263
13:24:32 swapper/0        645              1384
13:24:32 swapper/1        881              1384
13:24:32 abrt-dump-journ  5070             2064
13:24:32 systemd-journal  882              2135
13:24:32 abrt-dump-journ  645              2240
13:24:32 runqslower       2947             1317
13:24:32 iceccd           3839             1451
13:24:32 runqslower       241              1500
13:24:32 rcu_sched        5070             1794
13:24:32 swapper/2        176              1369
^C
[root@seventh linux]#

And furthermore:

[root@seventh linux]# file tools/bpf/runqslower/.output/runqslower.bpf.o
tools/bpf/runqslower/.output/runqslower.bpf.o: ELF 64-bit LSB relocatable, eBPF, version 1 (SYSV), not stripped
[root@seventh linux]# eu-readelf -winfo tools/bpf/runqslower/.output/runqslower.bpf.o
eu-readelf: cannot get debug context descriptor: No DWARF information found
[root@seventh linux]#
[root@seventh linux]# pahole -C task_struct tools/bpf/runqslower/.output/runqslower.bpf.o > BTF.generated-by-clang-for-a-BPF.target
[root@seventh linux]# pahole task_struct > BTF.generated-by-pahole-J-from-default-DWWARF-generated-by-gcc11-for-a-x86_64.target
[root@seventh linux]# diff -u BTF.generated-by-clang-for-a-BPF.target BTF.generated-by-pahole-J-from-default-DWWARF-generated-by-gcc11-for-a-x86_64.target

--- BTF.generated-by-clang-for-a-BPF.target	2021-02-04 13:26:16.215883852 -0300
+++ BTF.generated-by-pahole-J-from-default-DWWARF-generated-by-gcc11-for-a-x86_64.target	2021-02-04 13:27:07.483184784 -0300
@@ -49,8 +49,8 @@
 	/* --- cacheline 30 boundary (1920 bytes) was 56 bytes ago --- */
 	void *                     migration_pending;    /*  1976     8 */
 	/* --- cacheline 31 boundary (1984 bytes) --- */
-	unsigned short             migration_disabled;   /*  1984     2 */
-	unsigned short             migration_flags;      /*  1986     2 */
+	short unsigned int         migration_disabled;   /*  1984     2 */
+	short unsigned int         migration_flags;      /*  1986     2 */

 	/* XXX 4 bytes hole, try to pack */

@@ -284,8 +284,8 @@
 	struct list_head           perf_event_list;      /*  7016    16 */
 	struct mempolicy *         mempolicy;            /*  7032     8 */
 	/* --- cacheline 110 boundary (7040 bytes) --- */
-	short                      il_prev;              /*  7040     2 */
-	short                      pref_node_fork;       /*  7042     2 */
+	short int                  il_prev;              /*  7040     2 */
+	short int                  pref_node_fork;       /*  7042     2 */
 	int                        numa_scan_seq;        /*  7044     4 */
 	unsigned int               numa_scan_period;     /*  7048     4 */
 	unsigned int               numa_scan_period_max; /*  7052     4 */
[root@seventh linux]#

Not really a difference, but I'll check if I can remove this annoyance
in > 1.20 :-)

[root@seventh linux]# clang -v |& head -1
clang version 12.0.0 (https://github.com/llvm/llvm-project 87369c626114ae17f4c637635c119e6de0856a9a)
[root@seventh linux]#

So I think that for the problems related to building the kernel with gcc
11 in Fedora Rawhide using the default that is now DWARF5, pahole 1.20
is good to go and I'll tag it now.

- Arnaldo

^ permalink raw reply	[flat|nested] 3+ messages in thread