From: Jiri Olsa <jolsa@redhat.com>
To: Brendan Jackman <jackmanb@google.com>
Cc: Sandipan Das <sandipan@linux.ibm.com>,
"Naveen N. Rao" <naveen.n.rao@linux.ibm.com>,
bpf <bpf@vger.kernel.org>, Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii.nakryiko@gmail.com>,
KP Singh <kpsingh@chromium.org>,
Florent Revest <revest@chromium.org>,
John Fastabend <john.fastabend@gmail.com>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [BUG soft lockup] Re: [PATCH bpf-next v3] bpf: Propagate stack bounds to registers in atomics w/ BPF_FETCH
Date: Thu, 1 Jul 2021 12:16:36 +0200 [thread overview]
Message-ID: <YN2WBP9kjGQxHrKS@krava> (raw)
In-Reply-To: <CA+i-1C2-MGe0BziQc8t4ry3mj45W0ULVrGsU+uQw9952tFZ1nA@mail.gmail.com>
On Thu, Jul 01, 2021 at 10:18:39AM +0200, Brendan Jackman wrote:
> On Wed, 30 Jun 2021 at 14:42, Jiri Olsa <jolsa@redhat.com> wrote:
> >
> > On Wed, Jun 30, 2021 at 12:34:58PM +0200, Brendan Jackman wrote:
> > > On Tue, 29 Jun 2021 at 23:09, Jiri Olsa <jolsa@redhat.com> wrote:
> > > >
> > > > On Tue, Jun 29, 2021 at 06:41:24PM +0200, Jiri Olsa wrote:
> > > > > On Tue, Jun 29, 2021 at 06:25:33PM +0200, Brendan Jackman wrote:
> > > > > > On Tue, 29 Jun 2021 at 18:04, Jiri Olsa <jolsa@redhat.com> wrote:
> > > > > > > On Tue, Jun 29, 2021 at 04:10:12PM +0200, Jiri Olsa wrote:
> > > > > > > > On Mon, Jun 28, 2021 at 11:21:42AM +0200, Brendan Jackman wrote:
> > >
> > > > > > > > > atomics in .imm). Any idea if this test was ever passing on PowerPC?
> > > > > > > > >
> > > > > > > >
> > > > > > > > hum, I guess not.. will check
> > > > > > >
> > > > > > > nope, it locks up the same:
> > > > > >
> > > > > > Do you mean it locks up at commit 91c960b0056 too?
> > >
> > > Sorry I was being stupid here - the test didn't exist at this commit
> > >
> > > > > I tried this one:
> > > > > 37086bfdc737 bpf: Propagate stack bounds to registers in atomics w/ BPF_FETCH
> > > > >
> > > > > I will check also 91c960b0056, but I think it's the new test issue
> > >
> > > So yeah hard to say whether this was broken on PowerPC all along. How
> > > hard is it for me to get set up to reproduce the failure? Is there a
> > > rootfs I can download, and some instructions for running a PowerPC
> > > QEMU VM? If so if you can also share your config and I'll take a look.
> > >
> > > If it's not as simple as that, I'll stare at the code for a while and
> > > see if anything jumps out.
> > >
> >
> > I have latest fedora ppc server and compile/install latest bpf-next tree
> > I think it will be reproduced also on vm, I attached my config
>
> OK, getting set up to boot a PowerPC QEMU isn't practical here unless
> someone's got commands I can copy-paste (suspect it will need .config
> hacking too). Looks like you need to build a proper bootloader, and
> boot an installer disk.
>
> Looked at the code for a bit but nothing jumped out. It seems like the
> verifier is seeing a BPF_ADD | BPF_FETCH, which means it doesn't
> detect an infinite loop, but then we lose the BPF_FETCH flag somewhere
> between do_check in verifier.c and bpf_jit_build_body in
> bpf_jit_comp64.c. That would explain why we don't get the "eBPF filter
> atomic op code %02x (@%d) unsupported", and would also explain the
> lockup because a normal atomic add without fetch would leave BPF R1
> unchanged.
>
> We should be able to confirm that theory by disassembling the JITted
> code that gets hexdumped by bpf_jit_dump when bpf_jit_enable is set to
> 2... at least for PowerPC 32-bit... maybe you could paste those lines
> into the 64-bit version too? Here's some notes I made for
> disassembling the hexdump on x86, I guess you'd just need to change
> the objdump flags:
>
> --
>
> - Enable console JIT output:
> ```shell
> echo 2 > /proc/sys/net/core/bpf_jit_enable
> ```
> - Load & run the program of interest.
> - Copy the hex code from the kernel console to `/tmp/jit.txt`. Here's what a
> short program looks like. This includes a line of context - don't paste the
> `flen=` line.
> ```
> [ 79.381020] flen=8 proglen=54 pass=4 image=000000001af6f390
> from=test_verifier pid=258
> [ 79.389568] JIT code: 00000000: 0f 1f 44 00 00 66 90 55 48 89 e5 48 81 ec 08 00
> [ 79.397411] JIT code: 00000010: 00 00 48 c7 45 f8 64 00 00 00 bf 04 00 00 00 48
> [ 79.405965] JIT code: 00000020: f7 df f0 48 29 7d f8 8b 45 f8 48 83 f8 60 74 02
> [ 79.414719] JIT code: 00000030: c9 c3 31 c0 eb fa
> ```
> - This incantation will split out and decode the hex, then disassemble the
> result:
> ```shell
> cat /tmp/jit.txt | cut -d: -f2- | xxd -r >/tmp/obj && objdump -D -b
> binary -m i386:x86-64 /tmp/obj
> ```
that's where I decided to write to list and ask for help before
googling ppc assembly ;-)
I changed the test_verifier to stop before executing the test
so I can dump the program via bpftool:
[root@ibm-p9z-07-lp1 bpf-next]# bpftool prog dump xlated id 48
0: (b7) r0 = 0
1: (7b) *(u64 *)(r10 -8) = r0
2: (b7) r1 = 1
3: (db) r1 = atomic64_fetch_add((u64 *)(r10 -8), r1)
4: (55) if r1 != 0x0 goto pc-1
5: (95) exit
[root@ibm-p9z-07-lp1 bpf-next]# bpftool prog dump jited id 48
bpf_prog_a2eb9104e5e8a5bf:
0: nop
4: nop
8: stdu r1,-112(r1)
c: std r31,104(r1)
10: addi r31,r1,48
14: li r8,0
18: std r8,-8(r31)
1c: li r3,1
20: addi r9,r31,-8
24: ldarx r10,0,r9
28: add r10,r10,r3
2c: stdcx. r10,0,r9
30: bne 0x0000000000000024
34: cmpldi r3,0
38: bne 0x0000000000000034
3c: nop
40: ld r31,104(r1)
44: addi r1,r1,112
48: mr r3,r8
4c: blr
I wanted to also do it through bpf_jit_enable and bpf_jit_dump, but I need to check
the setup, because I can't set bpf_jit_enable to 2 at the moment.. might take some time
[root@ibm-p9z-07-lp1 bpf-next]# echo 2 > /proc/sys/net/core/bpf_jit_enable
-bash: echo: write error: Invalid argument
jirka
>
> --
>
> Sandipan, Naveen, do you know of anything in the PowerPC code that
> might be leading us to drop the BPF_FETCH flag from the atomic
> instruction in tools/testing/selftests/bpf/verifier/atomic_bounds.c?
>
next prev parent reply other threads:[~2021-07-01 10:16 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-02 13:50 [PATCH bpf-next v3] bpf: Propagate stack bounds to registers in atomics w/ BPF_FETCH Brendan Jackman
2021-02-03 17:07 ` Yonghong Song
2021-02-03 17:37 ` Alexei Starovoitov
2021-06-27 15:34 ` [BUG soft lockup] " Jiri Olsa
2021-06-28 9:21 ` Brendan Jackman
2021-06-29 14:10 ` Jiri Olsa
2021-06-29 16:04 ` Jiri Olsa
2021-06-29 16:25 ` Brendan Jackman
2021-06-29 16:41 ` Jiri Olsa
2021-06-29 21:08 ` Jiri Olsa
2021-06-30 10:34 ` Brendan Jackman
[not found] ` <YNxmwZGtnqiXGnF0@krava>
2021-07-01 8:18 ` Brendan Jackman
2021-07-01 10:16 ` Jiri Olsa [this message]
2021-07-01 11:02 ` Naveen N. Rao
2021-07-01 14:15 ` Jiri Olsa
2021-07-02 9:44 ` Brendan Jackman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YN2WBP9kjGQxHrKS@krava \
--to=jolsa@redhat.com \
--cc=andrii.nakryiko@gmail.com \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=jackmanb@google.com \
--cc=john.fastabend@gmail.com \
--cc=kpsingh@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=naveen.n.rao@linux.ibm.com \
--cc=revest@chromium.org \
--cc=sandipan@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).