From: Arnaldo Carvalho de Melo <firstname.lastname@example.org>
To: "Naveen N. Rao" <email@example.com>
Cc: firstname.lastname@example.org, Jiri Olsa <email@example.com>,
Linux Kernel Mailing List <firstname.lastname@example.org>,
Namhyung Kim <email@example.com>
Subject: Re: [PATCH] perf trace: Fix SIGSEGV when processing augmented args
Date: Tue, 15 Mar 2022 14:52:05 -0300 [thread overview]
Message-ID: <YjDSRb1wwswKpJNJ@kernel.org> (raw)
Em Tue, Mar 15, 2022 at 10:57:57PM +0530, Naveen N. Rao escreveu:
> Arnaldo Carvalho de Melo wrote:
> > Em Thu, Mar 10, 2022 at 04:17:41PM +0530, Naveen N. Rao escreveu:
> > > On powerpc, 'perf trace' is crashing with a SIGSEGV when trying to
> > > process a perf data file created with 'perf trace record -p':
> > > #0 0x00000001225b8988 in syscall_arg__scnprintf_augmented_string <snip> at builtin-trace.c:1492
> > > #1 syscall_arg__scnprintf_filename <snip> at builtin-trace.c:1492
> > > #2 syscall_arg__scnprintf_filename <snip> at builtin-trace.c:1486
> > > #3 0x00000001225bdd9c in syscall_arg_fmt__scnprintf_val <snip> at builtin-trace.c:1973
> > > #4 syscall__scnprintf_args <snip> at builtin-trace.c:2041
> > > #5 0x00000001225bff04 in trace__sys_enter <snip> at builtin-trace.c:2319
> > > The size captured in the augmented arg looks corrupt, resulting in the
> > > augmented arg pointer being adjusted incorrectly. Fix this by checking
> > > that the size is reasonable.
> > > Reported-by: Claudio Carvalho <firstname.lastname@example.org>
> > > Signed-off-by: Naveen N. Rao <email@example.com>
> > > ---
> > > While this resolves the 'perf trace' crash, I'm not yet sure why the
> > > size for the augmented arg is corrupt. This looks to be happening when
> > > processing the sample for 'read' syscall. Any pointers?
> > Strange indeed, the augmented args code should kick in when the payload
> > for some tracepoint is bigger than what is expected, i.e. more than the
> > sum of its arguments, in which case it assumes that what is coming after
> > the expected payload comes with, say, the pathname for an open, openat,
> > etc syscall, that otherwise would be just a pointer.
> > This augmentation is done using something like
> > tools/perf/examples/bpf/augmented_raw_syscalls.c, i.e.:
> > [root@quaco ~]# perf trace -e openat sleep 1
> > 0.000 openat(dfd: CWD, filename: 0x1bbb0b6, flags: RDONLY|CLOEXEC) = 3
> > 0.021 openat(dfd: CWD, filename: 0x1bc8e80, flags: RDONLY|CLOEXEC) = 3
> > 0.201 openat(dfd: CWD, filename: 0x1b34f20, flags: RDONLY|CLOEXEC) = 3
> > [root@quaco ~]# perf trace -e ~acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c,openat sleep 1
> > 0.000 openat(dfd: CWD, filename: "/etc/ld.so.cache", flags: RDONLY|CLOEXEC) = 3
> > 0.023 openat(dfd: CWD, filename: "/lib64/libc.so.6", flags: RDONLY|CLOEXEC) = 3
> > 0.225 openat(dfd: CWD, filename: "/usr/lib/locale/locale-archive", flags: RDONLY|CLOEXEC) = 3
> > [root@quaco ~]#
> Thanks, that clarifies things a bit. I was under the impression that
> syscalls are augmented through the bpf program by default. But, it looks
> like that isn't the case (at least on the distro where this problem was
Its not by default at the moment.
This was an experiment in getting perf to get assistance from eBPF, but
this predates eBPF skeletons, sleepable tracepoints, tons of stuff that
it should instead be using by now.
We now have:
⬢[acme@toolbox perf]$ ls -la tools/perf/util/bpf_skel/
drwxr-xr-x. 1 acme acme 232 Mar 14 17:56 .
drwxr-xr-x. 1 acme acme 6978 Mar 14 19:15 ..
-rw-r--r--. 1 acme acme 4592 Mar 14 17:54 bperf_cgroup.bpf.c
-rw-r--r--. 1 acme acme 1764 Mar 14 17:55 bperf_follower.bpf.c
-rw-r--r--. 1 acme acme 1438 Mar 14 17:55 bperf_leader.bpf.c
-rw-r--r--. 1 acme acme 285 Mar 14 17:54 bperf_u.h
-rw-r--r--. 1 acme acme 2290 Mar 14 17:55 bpf_prog_profiler.bpf.c
-rw-r--r--. 1 acme acme 2115 Mar 14 17:56 func_latency.bpf.c
-rw-r--r--. 1 acme acme 53 Nov 6 17:43 .gitignore
'perf trace' needs to switch to using that method as well.
> > So it seems the perf.data file being processed is corrupted somehow...
> > Perhaps we should check if the syscall has pointer args and if not don't
> > kick in the augmented code and instead emit some warning about having
> > more payload than expected.
> Yes, it looks like the current check in 'perf' isn't working. The below
> patch also resolves the crash we are seeing:
> diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
> index 2f1d20553a0aa3..86b459f4ebdd61 100644
> --- a/tools/perf/builtin-trace.c
> +++ b/tools/perf/builtin-trace.c
> @@ -2326,7 +2326,7 @@ static int trace__sys_enter(struct trace *trace, struct evsel *evsel,
> * thinking that the extra 2 u64 args are the augmented filename, so just check
> * here and avoid using augmented syscalls when the evsel is the raw_syscalls one.
> - if (evsel != trace->syscalls.events.sys_enter)
> + if (strcmp(evsel__name(evsel), "raw_syscalls:sys_enter"))
> augmented_args = syscall__augmented_args(sc, sample, &augmented_args_size, trace->raw_augmented_syscalls_args_size);
> ttrace->entry_time = sample->time;
> msg = ttrace->entry_str;
Interesting, that should be equivalent :-\ humm, not really, understood,
when processing perf.data files we don't setup
switching from strcmp() to something cheaper but equivalent should be
the fix for now.
next prev parent reply other threads:[~2022-03-15 17:52 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-10 10:47 [PATCH] perf trace: Fix SIGSEGV when processing augmented args Naveen N. Rao
2022-03-14 22:09 ` Arnaldo Carvalho de Melo
2022-03-15 17:27 ` Naveen N. Rao
2022-03-15 17:52 ` Arnaldo Carvalho de Melo [this message]
2022-03-16 20:36 ` Arnaldo Carvalho de Melo
2022-03-17 13:24 ` Naveen N. Rao
2022-07-06 12:31 ` Naveen N. Rao
2022-07-06 15:50 ` Arnaldo Carvalho de Melo
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.