* [BUG] perf: intel_pt won't display kernel function @ 2019-04-03 14:37 Jiri Olsa 2019-04-03 14:53 ` Arnaldo Carvalho de Melo 2019-04-03 17:05 ` Song Liu 0 siblings, 2 replies; 14+ messages in thread From: Jiri Olsa @ 2019-04-03 14:37 UTC (permalink / raw) To: Adrian Hunter Cc: Andi Kleen, acme, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen, Song Liu hi, perf script --call-trace stop working for me recently, and displays only user space functions I bisected that to: 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs data from following comands will display user space functions only: # perf-with-kcore record pt -e intel_pt// -- ls # perf-with-kcore script pt --call-trace when I disable the bpf synthesizing (patch below), kernel functions are back I guess the new events mess up with intel_pt decoder somehow thanks, jirka --- diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c index 4e2d953d4bc5..3daa78bc6549 100644 --- a/tools/perf/builtin-record.c +++ b/tools/perf/builtin-record.c @@ -1114,10 +1114,12 @@ static int record__synthesize(struct record *rec, bool tail) return err; } +#if 0 err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts); if (err < 0) pr_warning("Couldn't synthesize bpf events.\n"); +#endif err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, process_synthesized_event, opts->sample_address, ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 14:37 [BUG] perf: intel_pt won't display kernel function Jiri Olsa @ 2019-04-03 14:53 ` Arnaldo Carvalho de Melo 2019-04-03 15:15 ` Arnaldo Carvalho de Melo 2019-04-03 17:05 ` Song Liu 1 sibling, 1 reply; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2019-04-03 14:53 UTC (permalink / raw) To: Jiri Olsa Cc: Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen, Song Liu Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu: > hi, > perf script --call-trace stop working for me recently, > and displays only user space functions > > I bisected that to: > 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs > > data from following comands will display user space functions only: > # perf-with-kcore record pt -e intel_pt// -- ls > # perf-with-kcore script pt --call-trace > > when I disable the bpf synthesizing (patch below), kernel functions are back > > I guess the new events mess up with intel_pt decoder somehow And also that synthesizing should't take place when we use perf record --no-bpf-event I.e. I'm adding the patch below to my perf/urgent branch. The --no-bpf-event option was put in place exactly for that for explicitely disabling it if one thinks it is messing with something, like seems to be the case here. - Arnaldo diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c index 4e2d953d4bc5..17d772f192ad 100644 --- a/tools/perf/builtin-record.c +++ b/tools/perf/builtin-record.c @@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail) return err; } - err = perf_event__synthesize_bpf_events(session, process_synthesized_event, - machine, opts); - if (err < 0) - pr_warning("Couldn't synthesize bpf events.\n"); + if (!opts->no_bpf_event) { + err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts); + if (err < 0) + pr_warning("Couldn't synthesize bpf events.\n"); + } err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, process_synthesized_event, opts->sample_address, > > thanks, > jirka > > > --- > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c > index 4e2d953d4bc5..3daa78bc6549 100644 > --- a/tools/perf/builtin-record.c > +++ b/tools/perf/builtin-record.c > @@ -1114,10 +1114,12 @@ static int record__synthesize(struct record *rec, bool tail) > return err; > } > > +#if 0 > err = perf_event__synthesize_bpf_events(session, process_synthesized_event, > machine, opts); > if (err < 0) > pr_warning("Couldn't synthesize bpf events.\n"); > +#endif > > err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, > process_synthesized_event, opts->sample_address, -- - Arnaldo ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 14:53 ` Arnaldo Carvalho de Melo @ 2019-04-03 15:15 ` Arnaldo Carvalho de Melo 2019-04-03 16:27 ` Song Liu 0 siblings, 1 reply; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2019-04-03 15:15 UTC (permalink / raw) To: Song Liu, Jiri Olsa Cc: Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu: > Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu: > > hi, > > perf script --call-trace stop working for me recently, > > and displays only user space functions > > > > I bisected that to: > > 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs > > > > data from following comands will display user space functions only: > > # perf-with-kcore record pt -e intel_pt// -- ls > > # perf-with-kcore script pt --call-trace > > > > when I disable the bpf synthesizing (patch below), kernel functions are back > > > > I guess the new events mess up with intel_pt decoder somehow > > I.e. I'm adding the patch below to my perf/urgent branch. Song, that is what I have, can I have your Acked-by and please consider taking a look at the bug Jiri reported, Thanks, - Arnaldo commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503 Author: Arnaldo Carvalho de Melo <acme@redhat.com> Date: Wed Apr 3 12:05:15 2019 -0300 perf record: Do not synthesize BPF records when --no-bpf-event is used By default we synthesize and ask the kernel for BPF events, having a --no-bpf-event option to disable that, which can be useful, for instance, if there are still bugs in that code, which seems to be the case as reported by Jiri Olsa in: "[BUG] perf: intel_pt won't display kernel function" https://lore.kernel.org/lkml/20190403143738.GB32001@krava So add the check for record_opts.no_bpf_event when considering synthesizing BPF events for pre-existing BPF programs in 'perf record'. The reported bug needs further analysis and is a separate matter. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Jiri Olsa <jolsa@kernel.org> Cc: Martin KaFai Lau <kafai@fb.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <songliubraving@fb.com> Cc: Yonghong Song <yhs@fb.com> Fixes: 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs") Link: https://lore.kernel.org/lkml/20190403145353.GE32553@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c index 4e2d953d4bc5..17d772f192ad 100644 --- a/tools/perf/builtin-record.c +++ b/tools/perf/builtin-record.c @@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail) return err; } - err = perf_event__synthesize_bpf_events(session, process_synthesized_event, - machine, opts); - if (err < 0) - pr_warning("Couldn't synthesize bpf events.\n"); + if (!opts->no_bpf_event) { + err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts); + if (err < 0) + pr_warning("Couldn't synthesize bpf events.\n"); + } err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, process_synthesized_event, opts->sample_address, ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 15:15 ` Arnaldo Carvalho de Melo @ 2019-04-03 16:27 ` Song Liu 2019-04-03 18:50 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 14+ messages in thread From: Song Liu @ 2019-04-03 16:27 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Jiri Olsa, Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen > On Apr 3, 2019, at 8:15 AM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: > > Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu: >> Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu: >>> hi, >>> perf script --call-trace stop working for me recently, >>> and displays only user space functions >>> >>> I bisected that to: >>> 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs >>> >>> data from following comands will display user space functions only: >>> # perf-with-kcore record pt -e intel_pt// -- ls >>> # perf-with-kcore script pt --call-trace >>> >>> when I disable the bpf synthesizing (patch below), kernel functions are back >>> >>> I guess the new events mess up with intel_pt decoder somehow >> >> I.e. I'm adding the patch below to my perf/urgent branch. > > Song, that is what I have, can I have your Acked-by and please consider > taking a look at the bug Jiri reported, > > Thanks, > > - Arnaldo Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not PERF_RECORD_BPF_EVENT: https://elixir.bootlin.com/linux/v5.1-rc3/source/tools/perf/util/bpf-event.c#L254 I will look into the intel_pt problem. In the meanwhile, let's fix it for now. Acked-by: Song Liu <songliubraving@fb.com> Thanks, Song > commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503 > Author: Arnaldo Carvalho de Melo <acme@redhat.com> > Date: Wed Apr 3 12:05:15 2019 -0300 > > perf record: Do not synthesize BPF records when --no-bpf-event is used > > By default we synthesize and ask the kernel for BPF events, having a > --no-bpf-event option to disable that, which can be useful, for > instance, if there are still bugs in that code, which seems to be the > case as reported by Jiri Olsa in: > > "[BUG] perf: intel_pt won't display kernel function" > https://lore.kernel.org/lkml/20190403143738.GB32001@krava > > So add the check for record_opts.no_bpf_event when considering > synthesizing BPF events for pre-existing BPF programs in 'perf record'. > > The reported bug needs further analysis and is a separate matter. > > Cc: Adrian Hunter <adrian.hunter@intel.com> > Cc: Alexei Starovoitov <ast@kernel.org> > Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> > Cc: Daniel Borkmann <daniel@iogearbox.net> > Cc: Jiri Olsa <jolsa@kernel.org> > Cc: Martin KaFai Lau <kafai@fb.com> > Cc: Namhyung Kim <namhyung@kernel.org> > Cc: Peter Zijlstra <peterz@infradead.org> > Cc: Song Liu <songliubraving@fb.com> > Cc: Yonghong Song <yhs@fb.com> > Fixes: 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs") > Link: https://lore.kernel.org/lkml/20190403145353.GE32553@kernel.org > Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> > > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c > index 4e2d953d4bc5..17d772f192ad 100644 > --- a/tools/perf/builtin-record.c > +++ b/tools/perf/builtin-record.c > @@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail) > return err; > } > > - err = perf_event__synthesize_bpf_events(session, process_synthesized_event, > - machine, opts); > - if (err < 0) > - pr_warning("Couldn't synthesize bpf events.\n"); > + if (!opts->no_bpf_event) { > + err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts); > + if (err < 0) > + pr_warning("Couldn't synthesize bpf events.\n"); > + } > > err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, > process_synthesized_event, opts->sample_address, ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 16:27 ` Song Liu @ 2019-04-03 18:50 ` Arnaldo Carvalho de Melo 2019-04-03 18:55 ` Song Liu 0 siblings, 1 reply; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2019-04-03 18:50 UTC (permalink / raw) To: Song Liu Cc: Arnaldo Carvalho de Melo, Jiri Olsa, Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen Em Wed, Apr 03, 2019 at 04:27:56PM +0000, Song Liu escreveu: > > > > On Apr 3, 2019, at 8:15 AM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: > > > > Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu: > >> Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu: > >>> hi, > >>> perf script --call-trace stop working for me recently, > >>> and displays only user space functions > >>> > >>> I bisected that to: > >>> 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs > >>> > >>> data from following comands will display user space functions only: > >>> # perf-with-kcore record pt -e intel_pt// -- ls > >>> # perf-with-kcore script pt --call-trace > >>> > >>> when I disable the bpf synthesizing (patch below), kernel functions are back > >>> > >>> I guess the new events mess up with intel_pt decoder somehow > >> > >> I.e. I'm adding the patch below to my perf/urgent branch. > > > > Song, that is what I have, can I have your Acked-by and please consider > > taking a look at the bug Jiri reported, > > > > Thanks, > > > > - Arnaldo > > Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not > PERF_RECORD_BPF_EVENT: I see... The opts is done later, after querying the kernel for existing BPF programs so that at least the KSYMBOL ones can be generated. So I'll keep it as is, no need for this patch. > https://elixir.bootlin.com/linux/v5.1-rc3/source/tools/perf/util/bpf-event.c#L254 > > I will look into the intel_pt problem. Ok. > In the meanwhile, let's fix it for now. I'll wait a bit. > Acked-by: Song Liu <songliubraving@fb.com> > > Thanks, > Song > > > > commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503 > > Author: Arnaldo Carvalho de Melo <acme@redhat.com> > > Date: Wed Apr 3 12:05:15 2019 -0300 > > > > perf record: Do not synthesize BPF records when --no-bpf-event is used > > > > By default we synthesize and ask the kernel for BPF events, having a > > --no-bpf-event option to disable that, which can be useful, for > > instance, if there are still bugs in that code, which seems to be the > > case as reported by Jiri Olsa in: > > > > "[BUG] perf: intel_pt won't display kernel function" > > https://lore.kernel.org/lkml/20190403143738.GB32001@krava > > > > So add the check for record_opts.no_bpf_event when considering > > synthesizing BPF events for pre-existing BPF programs in 'perf record'. > > > > The reported bug needs further analysis and is a separate matter. > > > > Cc: Adrian Hunter <adrian.hunter@intel.com> > > Cc: Alexei Starovoitov <ast@kernel.org> > > Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> > > Cc: Daniel Borkmann <daniel@iogearbox.net> > > Cc: Jiri Olsa <jolsa@kernel.org> > > Cc: Martin KaFai Lau <kafai@fb.com> > > Cc: Namhyung Kim <namhyung@kernel.org> > > Cc: Peter Zijlstra <peterz@infradead.org> > > Cc: Song Liu <songliubraving@fb.com> > > Cc: Yonghong Song <yhs@fb.com> > > Fixes: 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs") > > Link: https://lore.kernel.org/lkml/20190403145353.GE32553@kernel.org > > Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> > > > > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c > > index 4e2d953d4bc5..17d772f192ad 100644 > > --- a/tools/perf/builtin-record.c > > +++ b/tools/perf/builtin-record.c > > @@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail) > > return err; > > } > > > > - err = perf_event__synthesize_bpf_events(session, process_synthesized_event, > > - machine, opts); > > - if (err < 0) > > - pr_warning("Couldn't synthesize bpf events.\n"); > > + if (!opts->no_bpf_event) { > > + err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts); > > + if (err < 0) > > + pr_warning("Couldn't synthesize bpf events.\n"); > > + } > > > > err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, > > process_synthesized_event, opts->sample_address, -- - Arnaldo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 18:50 ` Arnaldo Carvalho de Melo @ 2019-04-03 18:55 ` Song Liu 2019-04-03 18:59 ` Song Liu 0 siblings, 1 reply; 14+ messages in thread From: Song Liu @ 2019-04-03 18:55 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Jiri Olsa, Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen > On Apr 3, 2019, at 11:50 AM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: > > Em Wed, Apr 03, 2019 at 04:27:56PM +0000, Song Liu escreveu: >> >> >>> On Apr 3, 2019, at 8:15 AM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: >>> >>> Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu: >>>> Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu: >>>>> hi, >>>>> perf script --call-trace stop working for me recently, >>>>> and displays only user space functions >>>>> >>>>> I bisected that to: >>>>> 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs >>>>> >>>>> data from following comands will display user space functions only: >>>>> # perf-with-kcore record pt -e intel_pt// -- ls >>>>> # perf-with-kcore script pt --call-trace >>>>> >>>>> when I disable the bpf synthesizing (patch below), kernel functions are back >>>>> >>>>> I guess the new events mess up with intel_pt decoder somehow >>>> >>>> I.e. I'm adding the patch below to my perf/urgent branch. >>> >>> Song, that is what I have, can I have your Acked-by and please consider >>> taking a look at the bug Jiri reported, >>> >>> Thanks, >>> >>> - Arnaldo >> >> Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not >> PERF_RECORD_BPF_EVENT: > > I see... The opts is done later, after querying the kernel for existing > BPF programs so that at least the KSYMBOL ones can be generated. So I'll > keep it as is, no need for this patch. > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__elixir.bootlin.com_linux_v5.1-2Drc3_source_tools_perf_util_bpf-2Devent.c-23L254&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=dR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=8ITYgIrqFBIy1uQFQPHrsX5kBX5RlmhwcknKYJa2eGs&s=MdB2zA9JstwC93qKEonsfTQeQYiw-yeOLbPxm3QMRo8&e= >> >> I will look into the intel_pt problem. > > Ok. > >> In the meanwhile, let's fix it for now. > > I'll wait a bit. I guess something like this might be a better fix? (Sorry for missing commit message): diff --git i/tools/perf/util/map.c w/tools/perf/util/map.c index e32628cd20a7..741430a35dca 100644 --- i/tools/perf/util/map.c +++ w/tools/perf/util/map.c @@ -261,6 +261,12 @@ bool __map__is_extra_kernel_map(const struct map *map) return kmap && kmap->name[0]; } +bool __map__is_bpf_prog(const struct map *map) +{ + const char *name = map->dso->short_name; + return name && (strstr(name, "bpf_prog_") == name); +} + bool map__has_symbols(const struct map *map) { return dso__has_symbols(map->dso); diff --git i/tools/perf/util/map.h w/tools/perf/util/map.h index 0e20749f2c55..01079f1f4375 100644 --- i/tools/perf/util/map.h +++ w/tools/perf/util/map.h @@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name, bool __map__is_kernel(const struct map *map); bool __map__is_extra_kernel_map(const struct map *map); +bool __map__is_bpf_prog(const struct map *map); static inline bool __map__is_kmodule(const struct map *map) { - return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map); + return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) && + !__map__is_bpf_prog(map); } bool map__has_symbols(const struct map *map); Thanks, Song > >> Acked-by: Song Liu <songliubraving@fb.com> >> >> Thanks, >> Song >> >> >>> commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503 >>> Author: Arnaldo Carvalho de Melo <acme@redhat.com> >>> Date: Wed Apr 3 12:05:15 2019 -0300 >>> >>> perf record: Do not synthesize BPF records when --no-bpf-event is used >>> >>> By default we synthesize and ask the kernel for BPF events, having a >>> --no-bpf-event option to disable that, which can be useful, for >>> instance, if there are still bugs in that code, which seems to be the >>> case as reported by Jiri Olsa in: >>> >>> "[BUG] perf: intel_pt won't display kernel function" >>> https://lore.kernel.org/lkml/20190403143738.GB32001@krava >>> >>> So add the check for record_opts.no_bpf_event when considering >>> synthesizing BPF events for pre-existing BPF programs in 'perf record'. >>> >>> The reported bug needs further analysis and is a separate matter. >>> >>> Cc: Adrian Hunter <adrian.hunter@intel.com> >>> Cc: Alexei Starovoitov <ast@kernel.org> >>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> >>> Cc: Daniel Borkmann <daniel@iogearbox.net> >>> Cc: Jiri Olsa <jolsa@kernel.org> >>> Cc: Martin KaFai Lau <kafai@fb.com> >>> Cc: Namhyung Kim <namhyung@kernel.org> >>> Cc: Peter Zijlstra <peterz@infradead.org> >>> Cc: Song Liu <songliubraving@fb.com> >>> Cc: Yonghong Song <yhs@fb.com> >>> Fixes: 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs") >>> Link: https://lore.kernel.org/lkml/20190403145353.GE32553@kernel.org >>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> >>> >>> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c >>> index 4e2d953d4bc5..17d772f192ad 100644 >>> --- a/tools/perf/builtin-record.c >>> +++ b/tools/perf/builtin-record.c >>> @@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail) >>> return err; >>> } >>> >>> - err = perf_event__synthesize_bpf_events(session, process_synthesized_event, >>> - machine, opts); >>> - if (err < 0) >>> - pr_warning("Couldn't synthesize bpf events.\n"); >>> + if (!opts->no_bpf_event) { >>> + err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts); >>> + if (err < 0) >>> + pr_warning("Couldn't synthesize bpf events.\n"); >>> + } >>> >>> err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, >>> process_synthesized_event, opts->sample_address, > > -- > > - Arnaldo ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 18:55 ` Song Liu @ 2019-04-03 18:59 ` Song Liu 2019-04-03 21:48 ` Song Liu 0 siblings, 1 reply; 14+ messages in thread From: Song Liu @ 2019-04-03 18:59 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Jiri Olsa, Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen > On Apr 3, 2019, at 11:55 AM, Song Liu <songliubraving@fb.com> wrote: > > > >> On Apr 3, 2019, at 11:50 AM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: >> >> Em Wed, Apr 03, 2019 at 04:27:56PM +0000, Song Liu escreveu: >>> >>> >>>> On Apr 3, 2019, at 8:15 AM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: >>>> >>>> Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu: >>>>> Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu: >>>>>> hi, >>>>>> perf script --call-trace stop working for me recently, >>>>>> and displays only user space functions >>>>>> >>>>>> I bisected that to: >>>>>> 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs >>>>>> >>>>>> data from following comands will display user space functions only: >>>>>> # perf-with-kcore record pt -e intel_pt// -- ls >>>>>> # perf-with-kcore script pt --call-trace >>>>>> >>>>>> when I disable the bpf synthesizing (patch below), kernel functions are back >>>>>> >>>>>> I guess the new events mess up with intel_pt decoder somehow >>>>> >>>>> I.e. I'm adding the patch below to my perf/urgent branch. >>>> >>>> Song, that is what I have, can I have your Acked-by and please consider >>>> taking a look at the bug Jiri reported, >>>> >>>> Thanks, >>>> >>>> - Arnaldo >>> >>> Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not >>> PERF_RECORD_BPF_EVENT: >> >> I see... The opts is done later, after querying the kernel for existing >> BPF programs so that at least the KSYMBOL ones can be generated. So I'll >> keep it as is, no need for this patch. >> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__elixir.bootlin.com_linux_v5.1-2Drc3_source_tools_perf_util_bpf-2Devent.c-23L254&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=dR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=8ITYgIrqFBIy1uQFQPHrsX5kBX5RlmhwcknKYJa2eGs&s=MdB2zA9JstwC93qKEonsfTQeQYiw-yeOLbPxm3QMRo8&e= >>> >>> I will look into the intel_pt problem. >> >> Ok. >> >>> In the meanwhile, let's fix it for now. >> >> I'll wait a bit. > > I guess something like this might be a better fix? (Sorry for missing > commit message): > > diff --git i/tools/perf/util/map.c w/tools/perf/util/map.c > index e32628cd20a7..741430a35dca 100644 > --- i/tools/perf/util/map.c > +++ w/tools/perf/util/map.c > @@ -261,6 +261,12 @@ bool __map__is_extra_kernel_map(const struct map *map) > return kmap && kmap->name[0]; > } > > +bool __map__is_bpf_prog(const struct map *map) > +{ > + const char *name = map->dso->short_name; > + return name && (strstr(name, "bpf_prog_") == name); > +} > + > bool map__has_symbols(const struct map *map) > { > return dso__has_symbols(map->dso); > diff --git i/tools/perf/util/map.h w/tools/perf/util/map.h > index 0e20749f2c55..01079f1f4375 100644 > --- i/tools/perf/util/map.h > +++ w/tools/perf/util/map.h > @@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name, > > bool __map__is_kernel(const struct map *map); > bool __map__is_extra_kernel_map(const struct map *map); > +bool __map__is_bpf_prog(const struct map *map); > > static inline bool __map__is_kmodule(const struct map *map) > { > - return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map); > + return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) && > + !__map__is_bpf_prog(map); > } > > bool map__has_symbols(const struct map *map); > > Thanks, > Song Actually, it should be something like this: diff --git i/tools/perf/util/map.c w/tools/perf/util/map.c index e32628cd20a7..31cd23612529 100644 --- i/tools/perf/util/map.c +++ w/tools/perf/util/map.c @@ -261,6 +261,11 @@ bool __map__is_extra_kernel_map(const struct map *map) return kmap && kmap->name[0]; } +bool __map__is_bpf_prog(const struct map *map) +{ + return map->dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO; +} + bool map__has_symbols(const struct map *map) { return dso__has_symbols(map->dso); diff --git i/tools/perf/util/map.h w/tools/perf/util/map.h index 0e20749f2c55..01079f1f4375 100644 --- i/tools/perf/util/map.h +++ w/tools/perf/util/map.h @@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name, bool __map__is_kernel(const struct map *map); bool __map__is_extra_kernel_map(const struct map *map); +bool __map__is_bpf_prog(const struct map *map); static inline bool __map__is_kmodule(const struct map *map) { - return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map); + return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) && + !__map__is_bpf_prog(map); } bool map__has_symbols(const struct map *map); > >> >>> Acked-by: Song Liu <songliubraving@fb.com> >>> >>> Thanks, >>> Song >>> >>> >>>> commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503 >>>> Author: Arnaldo Carvalho de Melo <acme@redhat.com> >>>> Date: Wed Apr 3 12:05:15 2019 -0300 >>>> >>>> perf record: Do not synthesize BPF records when --no-bpf-event is used >>>> >>>> By default we synthesize and ask the kernel for BPF events, having a >>>> --no-bpf-event option to disable that, which can be useful, for >>>> instance, if there are still bugs in that code, which seems to be the >>>> case as reported by Jiri Olsa in: >>>> >>>> "[BUG] perf: intel_pt won't display kernel function" >>>> https://lore.kernel.org/lkml/20190403143738.GB32001@krava >>>> >>>> So add the check for record_opts.no_bpf_event when considering >>>> synthesizing BPF events for pre-existing BPF programs in 'perf record'. >>>> >>>> The reported bug needs further analysis and is a separate matter. >>>> >>>> Cc: Adrian Hunter <adrian.hunter@intel.com> >>>> Cc: Alexei Starovoitov <ast@kernel.org> >>>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> >>>> Cc: Daniel Borkmann <daniel@iogearbox.net> >>>> Cc: Jiri Olsa <jolsa@kernel.org> >>>> Cc: Martin KaFai Lau <kafai@fb.com> >>>> Cc: Namhyung Kim <namhyung@kernel.org> >>>> Cc: Peter Zijlstra <peterz@infradead.org> >>>> Cc: Song Liu <songliubraving@fb.com> >>>> Cc: Yonghong Song <yhs@fb.com> >>>> Fixes: 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs") >>>> Link: https://lore.kernel.org/lkml/20190403145353.GE32553@kernel.org >>>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> >>>> >>>> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c >>>> index 4e2d953d4bc5..17d772f192ad 100644 >>>> --- a/tools/perf/builtin-record.c >>>> +++ b/tools/perf/builtin-record.c >>>> @@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail) >>>> return err; >>>> } >>>> >>>> - err = perf_event__synthesize_bpf_events(session, process_synthesized_event, >>>> - machine, opts); >>>> - if (err < 0) >>>> - pr_warning("Couldn't synthesize bpf events.\n"); >>>> + if (!opts->no_bpf_event) { >>>> + err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts); >>>> + if (err < 0) >>>> + pr_warning("Couldn't synthesize bpf events.\n"); >>>> + } >>>> >>>> err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, >>>> process_synthesized_event, opts->sample_address, >> >> -- >> >> - Arnaldo ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 18:59 ` Song Liu @ 2019-04-03 21:48 ` Song Liu 2019-04-04 9:14 ` Jiri Olsa 0 siblings, 1 reply; 14+ messages in thread From: Song Liu @ 2019-04-03 21:48 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Jiri Olsa, Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen > On Apr 3, 2019, at 11:59 AM, Song Liu <songliubraving@fb.com> wrote: > > > >> On Apr 3, 2019, at 11:55 AM, Song Liu <songliubraving@fb.com> wrote: >> >> >> >>> On Apr 3, 2019, at 11:50 AM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: >>> >>> Em Wed, Apr 03, 2019 at 04:27:56PM +0000, Song Liu escreveu: >>>> >>>> >>>>> On Apr 3, 2019, at 8:15 AM, Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com> wrote: >>>>> >>>>> Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu: >>>>>> Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu: >>>>>>> hi, >>>>>>> perf script --call-trace stop working for me recently, >>>>>>> and displays only user space functions >>>>>>> >>>>>>> I bisected that to: >>>>>>> 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs >>>>>>> >>>>>>> data from following comands will display user space functions only: >>>>>>> # perf-with-kcore record pt -e intel_pt// -- ls >>>>>>> # perf-with-kcore script pt --call-trace >>>>>>> >>>>>>> when I disable the bpf synthesizing (patch below), kernel functions are back >>>>>>> >>>>>>> I guess the new events mess up with intel_pt decoder somehow >>>>>> >>>>>> I.e. I'm adding the patch below to my perf/urgent branch. >>>>> >>>>> Song, that is what I have, can I have your Acked-by and please consider >>>>> taking a look at the bug Jiri reported, >>>>> >>>>> Thanks, >>>>> >>>>> - Arnaldo >>>> >>>> Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not >>>> PERF_RECORD_BPF_EVENT: >>> >>> I see... The opts is done later, after querying the kernel for existing >>> BPF programs so that at least the KSYMBOL ones can be generated. So I'll >>> keep it as is, no need for this patch. >>> >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__elixir.bootlin.com_linux_v5.1-2Drc3_source_tools_perf_util_bpf-2Devent.c-23L254&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=dR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=8ITYgIrqFBIy1uQFQPHrsX5kBX5RlmhwcknKYJa2eGs&s=MdB2zA9JstwC93qKEonsfTQeQYiw-yeOLbPxm3QMRo8&e= >>>> >>>> I will look into the intel_pt problem. >>> >>> Ok. >>> >>>> In the meanwhile, let's fix it for now. >>> >>> I'll wait a bit. >> >> I guess something like this might be a better fix? (Sorry for missing >> commit message): >> >> diff --git i/tools/perf/util/map.c w/tools/perf/util/map.c >> index e32628cd20a7..741430a35dca 100644 >> --- i/tools/perf/util/map.c >> +++ w/tools/perf/util/map.c >> @@ -261,6 +261,12 @@ bool __map__is_extra_kernel_map(const struct map *map) >> return kmap && kmap->name[0]; >> } >> >> +bool __map__is_bpf_prog(const struct map *map) >> +{ >> + const char *name = map->dso->short_name; >> + return name && (strstr(name, "bpf_prog_") == name); >> +} >> + >> bool map__has_symbols(const struct map *map) >> { >> return dso__has_symbols(map->dso); >> diff --git i/tools/perf/util/map.h w/tools/perf/util/map.h >> index 0e20749f2c55..01079f1f4375 100644 >> --- i/tools/perf/util/map.h >> +++ w/tools/perf/util/map.h >> @@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name, >> >> bool __map__is_kernel(const struct map *map); >> bool __map__is_extra_kernel_map(const struct map *map); >> +bool __map__is_bpf_prog(const struct map *map); >> >> static inline bool __map__is_kmodule(const struct map *map) >> { >> - return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map); >> + return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) && >> + !__map__is_bpf_prog(map); >> } >> >> bool map__has_symbols(const struct map *map); >> >> Thanks, >> Song > > Actually, it should be something like this: > > diff --git i/tools/perf/util/map.c w/tools/perf/util/map.c > index e32628cd20a7..31cd23612529 100644 > --- i/tools/perf/util/map.c > +++ w/tools/perf/util/map.c > @@ -261,6 +261,11 @@ bool __map__is_extra_kernel_map(const struct map *map) > return kmap && kmap->name[0]; > } > > +bool __map__is_bpf_prog(const struct map *map) > +{ > + return map->dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO; > +} > + > bool map__has_symbols(const struct map *map) > { > return dso__has_symbols(map->dso); > diff --git i/tools/perf/util/map.h w/tools/perf/util/map.h > index 0e20749f2c55..01079f1f4375 100644 > --- i/tools/perf/util/map.h > +++ w/tools/perf/util/map.h > @@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name, > > bool __map__is_kernel(const struct map *map); > bool __map__is_extra_kernel_map(const struct map *map); > +bool __map__is_bpf_prog(const struct map *map); > > static inline bool __map__is_kmodule(const struct map *map) > { > - return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map); > + return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) && > + !__map__is_bpf_prog(map); > } > > bool map__has_symbols(const struct map *map); > > >> >>> >>>> Acked-by: Song Liu <songliubraving@fb.com> >>>> >>>> Thanks, >>>> Song >>>> >>>> >>>>> commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503 >>>>> Author: Arnaldo Carvalho de Melo <acme@redhat.com> >>>>> Date: Wed Apr 3 12:05:15 2019 -0300 >>>>> >>>>> perf record: Do not synthesize BPF records when --no-bpf-event is used >>>>> >>>>> By default we synthesize and ask the kernel for BPF events, having a >>>>> --no-bpf-event option to disable that, which can be useful, for >>>>> instance, if there are still bugs in that code, which seems to be the >>>>> case as reported by Jiri Olsa in: >>>>> >>>>> "[BUG] perf: intel_pt won't display kernel function" >>>>> https://lore.kernel.org/lkml/20190403143738.GB32001@krava >>>>> >>>>> So add the check for record_opts.no_bpf_event when considering >>>>> synthesizing BPF events for pre-existing BPF programs in 'perf record'. >>>>> >>>>> The reported bug needs further analysis and is a separate matter. >>>>> >>>>> Cc: Adrian Hunter <adrian.hunter@intel.com> >>>>> Cc: Alexei Starovoitov <ast@kernel.org> >>>>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> >>>>> Cc: Daniel Borkmann <daniel@iogearbox.net> >>>>> Cc: Jiri Olsa <jolsa@kernel.org> >>>>> Cc: Martin KaFai Lau <kafai@fb.com> >>>>> Cc: Namhyung Kim <namhyung@kernel.org> >>>>> Cc: Peter Zijlstra <peterz@infradead.org> >>>>> Cc: Song Liu <songliubraving@fb.com> >>>>> Cc: Yonghong Song <yhs@fb.com> >>>>> Fixes: 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs") >>>>> Link: https://lore.kernel.org/lkml/20190403145353.GE32553@kernel.org >>>>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> >>>>> >>>>> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c >>>>> index 4e2d953d4bc5..17d772f192ad 100644 >>>>> --- a/tools/perf/builtin-record.c >>>>> +++ b/tools/perf/builtin-record.c >>>>> @@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail) >>>>> return err; >>>>> } >>>>> >>>>> - err = perf_event__synthesize_bpf_events(session, process_synthesized_event, >>>>> - machine, opts); >>>>> - if (err < 0) >>>>> - pr_warning("Couldn't synthesize bpf events.\n"); >>>>> + if (!opts->no_bpf_event) { >>>>> + err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts); >>>>> + if (err < 0) >>>>> + pr_warning("Couldn't synthesize bpf events.\n"); >>>>> + } >>>>> >>>>> err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, >>>>> process_synthesized_event, opts->sample_address, >>> >>> -- >>> >>> - Arnaldo Yet another version, with commit message and a little more logic: From 8c842b52905e747487447a014cef8c98d0db94ef Mon Sep 17 00:00:00 2001 From: Song Liu <songliubraving@fb.com> Date: Wed, 3 Apr 2019 14:24:02 -0700 Subject: [PATCH] perf util: check maps for bpf programs As reported by Jiri Olsa in: "[BUG] perf: intel_pt won't display kernel function" https://lore.kernel.org/lkml/20190403143738.GB32001@krava Recent changes to support PERF_RECORD_KSYMBOL and PERF_RECORD_BPF_EVENT broke --kallsyms option. This is because it broke test __map__is_kmodule. This patch fixes this by adding check for bpf program, so that these maps are not mistaken as kernel modules. Reported-by: Jiri Olsa <jolsa@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Martin KaFai Lau <kafai@fb.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Song Liu <songliubraving@fb.com> Cc: Yonghong Song <yhs@fb.com> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Fixes: 76193a94522f ("perf, bpf: Introduce PERF_RECORD_KSYMBOL") Link: https://lore.kernel.org/lkml/20190403145353.GE32553@kernel.org Signed-off-by: Song Liu <songliubraving@fb.com> --- tools/perf/util/map.c | 16 ++++++++++++++++ tools/perf/util/map.h | 4 +++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c index e32628cd20a7..28d484ef74ae 100644 --- a/tools/perf/util/map.c +++ b/tools/perf/util/map.c @@ -261,6 +261,22 @@ bool __map__is_extra_kernel_map(const struct map *map) return kmap && kmap->name[0]; } +bool __map__is_bpf_prog(const struct map *map) +{ + const char *name; + + if (map->dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO) + return true; + + /* + * If PERF_RECORD_BPF_EVENT is not included, the dso will not have + * type of DSO_BINARY_TYPE__BPF_PROG_INFO. In such cases, we can + * guess the type based on name. + */ + name = map->dso->short_name; + return name && (strstr(name, "bpf_prog_") == name); +} + bool map__has_symbols(const struct map *map) { return dso__has_symbols(map->dso); diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h index 0e20749f2c55..01079f1f4375 100644 --- a/tools/perf/util/map.h +++ b/tools/perf/util/map.h @@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name, bool __map__is_kernel(const struct map *map); bool __map__is_extra_kernel_map(const struct map *map); +bool __map__is_bpf_prog(const struct map *map); static inline bool __map__is_kmodule(const struct map *map) { - return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map); + return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) && + !__map__is_bpf_prog(map); } bool map__has_symbols(const struct map *map); -- 2.17.1 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 21:48 ` Song Liu @ 2019-04-04 9:14 ` Jiri Olsa 2019-04-04 12:25 ` Jiri Olsa 0 siblings, 1 reply; 14+ messages in thread From: Jiri Olsa @ 2019-04-04 9:14 UTC (permalink / raw) To: Song Liu Cc: Arnaldo Carvalho de Melo, Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen On Wed, Apr 03, 2019 at 09:48:23PM +0000, Song Liu wrote: SNIP > >>>>> commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503 > >>>>> Author: Arnaldo Carvalho de Melo <acme@redhat.com> > >>>>> Date: Wed Apr 3 12:05:15 2019 -0300 > >>>>> > >>>>> perf record: Do not synthesize BPF records when --no-bpf-event is used > >>>>> > >>>>> By default we synthesize and ask the kernel for BPF events, having a > >>>>> --no-bpf-event option to disable that, which can be useful, for > >>>>> instance, if there are still bugs in that code, which seems to be the > >>>>> case as reported by Jiri Olsa in: > >>>>> > >>>>> "[BUG] perf: intel_pt won't display kernel function" > >>>>> https://lore.kernel.org/lkml/20190403143738.GB32001@krava > >>>>> > >>>>> So add the check for record_opts.no_bpf_event when considering > >>>>> synthesizing BPF events for pre-existing BPF programs in 'perf record'. > >>>>> > >>>>> The reported bug needs further analysis and is a separate matter. > >>>>> > >>>>> Cc: Adrian Hunter <adrian.hunter@intel.com> > >>>>> Cc: Alexei Starovoitov <ast@kernel.org> > >>>>> Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> > >>>>> Cc: Daniel Borkmann <daniel@iogearbox.net> > >>>>> Cc: Jiri Olsa <jolsa@kernel.org> > >>>>> Cc: Martin KaFai Lau <kafai@fb.com> > >>>>> Cc: Namhyung Kim <namhyung@kernel.org> > >>>>> Cc: Peter Zijlstra <peterz@infradead.org> > >>>>> Cc: Song Liu <songliubraving@fb.com> > >>>>> Cc: Yonghong Song <yhs@fb.com> > >>>>> Fixes: 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs") > >>>>> Link: https://lore.kernel.org/lkml/20190403145353.GE32553@kernel.org > >>>>> Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> > >>>>> > >>>>> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c > >>>>> index 4e2d953d4bc5..17d772f192ad 100644 > >>>>> --- a/tools/perf/builtin-record.c > >>>>> +++ b/tools/perf/builtin-record.c > >>>>> @@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail) > >>>>> return err; > >>>>> } > >>>>> > >>>>> - err = perf_event__synthesize_bpf_events(session, process_synthesized_event, > >>>>> - machine, opts); > >>>>> - if (err < 0) > >>>>> - pr_warning("Couldn't synthesize bpf events.\n"); > >>>>> + if (!opts->no_bpf_event) { > >>>>> + err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts); > >>>>> + if (err < 0) > >>>>> + pr_warning("Couldn't synthesize bpf events.\n"); > >>>>> + } > >>>>> > >>>>> err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, > >>>>> process_synthesized_event, opts->sample_address, > >>> > >>> -- > >>> > >>> - Arnaldo > > Yet another version, with commit message and a little more logic: > > > From 8c842b52905e747487447a014cef8c98d0db94ef Mon Sep 17 00:00:00 2001 > From: Song Liu <songliubraving@fb.com> > Date: Wed, 3 Apr 2019 14:24:02 -0700 > Subject: [PATCH] perf util: check maps for bpf programs > > As reported by Jiri Olsa in: > > "[BUG] perf: intel_pt won't display kernel function" > https://lore.kernel.org/lkml/20190403143738.GB32001@krava > > Recent changes to support PERF_RECORD_KSYMBOL and PERF_RECORD_BPF_EVENT > broke --kallsyms option. This is because it broke test __map__is_kmodule. > > This patch fixes this by adding check for bpf program, so that these maps > are not mistaken as kernel modules. > > Reported-by: Jiri Olsa <jolsa@kernel.org> > Cc: Adrian Hunter <adrian.hunter@intel.com> > Cc: Alexei Starovoitov <ast@kernel.org> > Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com> > Cc: Daniel Borkmann <daniel@iogearbox.net> > Cc: Martin KaFai Lau <kafai@fb.com> > Cc: Namhyung Kim <namhyung@kernel.org> > Cc: Peter Zijlstra <peterz@infradead.org> > Cc: Song Liu <songliubraving@fb.com> > Cc: Yonghong Song <yhs@fb.com> > Cc: Arnaldo Carvalho de Melo <acme@redhat.com> > Fixes: 76193a94522f ("perf, bpf: Introduce PERF_RECORD_KSYMBOL") > Link: https://lore.kernel.org/lkml/20190403145353.GE32553@kernel.org > > Signed-off-by: Song Liu <songliubraving@fb.com> > --- > tools/perf/util/map.c | 16 ++++++++++++++++ > tools/perf/util/map.h | 4 +++- > 2 files changed, 19 insertions(+), 1 deletion(-) > > diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c > index e32628cd20a7..28d484ef74ae 100644 > --- a/tools/perf/util/map.c > +++ b/tools/perf/util/map.c > @@ -261,6 +261,22 @@ bool __map__is_extra_kernel_map(const struct map *map) > return kmap && kmap->name[0]; > } > > +bool __map__is_bpf_prog(const struct map *map) > +{ > + const char *name; > + > + if (map->dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO) > + return true; > + > + /* > + * If PERF_RECORD_BPF_EVENT is not included, the dso will not have > + * type of DSO_BINARY_TYPE__BPF_PROG_INFO. In such cases, we can > + * guess the type based on name. > + */ > + name = map->dso->short_name; > + return name && (strstr(name, "bpf_prog_") == name); > +} > + > bool map__has_symbols(const struct map *map) > { > return dso__has_symbols(map->dso); > diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h > index 0e20749f2c55..01079f1f4375 100644 > --- a/tools/perf/util/map.h > +++ b/tools/perf/util/map.h > @@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name, > > bool __map__is_kernel(const struct map *map); > bool __map__is_extra_kernel_map(const struct map *map); > +bool __map__is_bpf_prog(const struct map *map); > > static inline bool __map__is_kmodule(const struct map *map) > { > - return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map); > + return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) && > + !__map__is_bpf_prog(map); > } > > bool map__has_symbols(const struct map *map); it shows the kernel portion now, but it segfaults on me in ksymbol map processing recorded data with: # perf-with-kcore record pt -e intel_pt// -- sleep 1 gdb session: (gdb) r script -i pt/perf.data --kallsyms=./pt/kcore_dir/kallsyms Starting program: /home/jolsa/bin/perf script -i pt/perf.data --kallsyms=./pt/kcore_dir/kallsyms ... Program received signal SIGABRT, Aborted. 0x00007ffff75e60f5 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-29.fc30.x86_64 elfutils-libelf-0.176-1.fc30.x86_64 elfutils-libs-0.176-1.fc30.x86_64 libgcc-9.0.1-0.10.fc30.x86_64 libunwind-1.3.1-2.fc30.x86_64 libxcrypt-4.4.4-2.fc30.x86_64 numactl-libs-2.0.12-2.fc30.x86_64 perl-libs-5.28.1-434.fc30.x86_64 python2-libs-2.7.16-1.fc30.x86_64 slang-2.3.2-5.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64 (gdb) bt #0 0x00007ffff75e60f5 in raise () from /lib64/libc.so.6 #1 0x00007ffff75d0895 in abort () from /lib64/libc.so.6 #2 0x00007ffff75d0769 in __assert_fail_base.cold () from /lib64/libc.so.6 #3 0x00007ffff75de596 in __assert_fail () from /lib64/libc.so.6 #4 0x00000000004fc006 in refcount_sub_and_test (i=1, r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:131 #5 refcount_dec_and_test (r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:148 #6 map__put (map=0x1224df0) at util/map.c:299 #7 0x00000000004fdb95 in __maps__remove (map=0x1224df0, maps=0xb17d80) at util/map.c:953 #8 maps__remove (maps=0xb17d80, map=0x1224df0) at util/map.c:959 #9 0x00000000004f7d8a in map_groups__remove (map=<optimized out>, mg=<optimized out>) at util/map_groups.h:65 #10 machine__process_ksymbol_unregister (sample=<optimized out>, event=0x7ffff7279670, machine=<optimized out>) at util/machine.c:728 #11 machine__process_ksymbol (machine=<optimized out>, event=0x7ffff7279670, sample=<optimized out>) at util/machine.c:741 #12 0x00000000004fffbb in perf_session__deliver_event (session=0xb11390, event=0x7ffff7279670, tool=0x7fffffffc7b0, file_offset=13936) at util/session.c:1362 #13 0x00000000005039bb in do_flush (show_progress=false, oe=0xb17e80) at util/ordered-events.c:243 #14 __ordered_events__flush (oe=0xb17e80, how=OE_FLUSH__ROUND, timestamp=<optimized out>) at util/ordered-events.c:322 #15 0x00000000005005e4 in perf_session__process_user_event (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1402 #16 0x000000000050091e in perf_session__process_event (file_offset=191224, event=0x7ffff72a4af8, session=0xb11390) at util/session.c:1529 #17 process_simple (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1962 #18 0x0000000000501f44 in reader__process_events (prog=0x7fffffffc680, session=0xb11390, rd=<synthetic pointer>) at util/session.c:1931 #19 __perf_session__process_events (session=0xb11390) at util/session.c:1985 #20 perf_session__process_events (session=0xb11390) at util/session.c:2018 #21 0x000000000045bb42 in __cmd_script (script=0x7fffffffc7b0) at builtin-script.c:2429 #22 cmd_script (argc=<optimized out>, argv=<optimized out>) at builtin-script.c:3770 #23 0x00000000004a65cb in run_builtin (p=p@entry=0x994918 <commands+408>, argc=argc@entry=4, argv=argv@entry=0x7fffffffe160) at perf.c:303 #24 0x000000000042da1e in handle_internal_command (argv=0x7fffffffe160, argc=4) at perf.c:355 #25 run_argv (argcp=<synthetic pointer>, argv=<synthetic pointer>) at perf.c:399 #26 main (argc=4, argv=0x7fffffffe160) at perf.c:521 jirka ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-04 9:14 ` Jiri Olsa @ 2019-04-04 12:25 ` Jiri Olsa 2019-04-04 17:08 ` Song Liu 0 siblings, 1 reply; 14+ messages in thread From: Jiri Olsa @ 2019-04-04 12:25 UTC (permalink / raw) To: Song Liu Cc: Arnaldo Carvalho de Melo, Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen On Thu, Apr 04, 2019 at 11:14:38AM +0200, Jiri Olsa wrote: SNIP > Program received signal SIGABRT, Aborted. > 0x00007ffff75e60f5 in raise () from /lib64/libc.so.6 > Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-29.fc30.x86_64 elfutils-libelf-0.176-1.fc30.x86_64 elfutils-libs-0.176-1.fc30.x86_64 libgcc-9.0.1-0.10.fc30.x86_64 libunwind-1.3.1-2.fc30.x86_64 libxcrypt-4.4.4-2.fc30.x86_64 numactl-libs-2.0.12-2.fc30.x86_64 perl-libs-5.28.1-434.fc30.x86_64 python2-libs-2.7.16-1.fc30.x86_64 slang-2.3.2-5.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64 > (gdb) bt > #0 0x00007ffff75e60f5 in raise () from /lib64/libc.so.6 > #1 0x00007ffff75d0895 in abort () from /lib64/libc.so.6 > #2 0x00007ffff75d0769 in __assert_fail_base.cold () from /lib64/libc.so.6 > #3 0x00007ffff75de596 in __assert_fail () from /lib64/libc.so.6 > #4 0x00000000004fc006 in refcount_sub_and_test (i=1, r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:131 > #5 refcount_dec_and_test (r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:148 > #6 map__put (map=0x1224df0) at util/map.c:299 > #7 0x00000000004fdb95 in __maps__remove (map=0x1224df0, maps=0xb17d80) at util/map.c:953 > #8 maps__remove (maps=0xb17d80, map=0x1224df0) at util/map.c:959 > #9 0x00000000004f7d8a in map_groups__remove (map=<optimized out>, mg=<optimized out>) at util/map_groups.h:65 > #10 machine__process_ksymbol_unregister (sample=<optimized out>, event=0x7ffff7279670, machine=<optimized out>) at util/machine.c:728 > #11 machine__process_ksymbol (machine=<optimized out>, event=0x7ffff7279670, sample=<optimized out>) at util/machine.c:741 > #12 0x00000000004fffbb in perf_session__deliver_event (session=0xb11390, event=0x7ffff7279670, tool=0x7fffffffc7b0, file_offset=13936) at util/session.c:1362 > #13 0x00000000005039bb in do_flush (show_progress=false, oe=0xb17e80) at util/ordered-events.c:243 > #14 __ordered_events__flush (oe=0xb17e80, how=OE_FLUSH__ROUND, timestamp=<optimized out>) at util/ordered-events.c:322 > #15 0x00000000005005e4 in perf_session__process_user_event (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, > file_offset=file_offset@entry=191224) at util/session.c:1402 > #16 0x000000000050091e in perf_session__process_event (file_offset=191224, event=0x7ffff72a4af8, session=0xb11390) at util/session.c:1529 > #17 process_simple (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1962 > #18 0x0000000000501f44 in reader__process_events (prog=0x7fffffffc680, session=0xb11390, rd=<synthetic pointer>) at util/session.c:1931 > #19 __perf_session__process_events (session=0xb11390) at util/session.c:1985 > #20 perf_session__process_events (session=0xb11390) at util/session.c:2018 > #21 0x000000000045bb42 in __cmd_script (script=0x7fffffffc7b0) at builtin-script.c:2429 > #22 cmd_script (argc=<optimized out>, argv=<optimized out>) at builtin-script.c:3770 > #23 0x00000000004a65cb in run_builtin (p=p@entry=0x994918 <commands+408>, argc=argc@entry=4, argv=argv@entry=0x7fffffffe160) at perf.c:303 > #24 0x000000000042da1e in handle_internal_command (argv=0x7fffffffe160, argc=4) at perf.c:355 > #25 run_argv (argcp=<synthetic pointer>, argv=<synthetic pointer>) at perf.c:399 > #26 main (argc=4, argv=0x7fffffffe160) at perf.c:521 we dont increase the map's reference for same name, which we need to.. I assume we we re able to hit this path with bpf maps/dso with same name jirka --- diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c index 58ae5fc1f0e9..cdc49113ba38 100644 --- a/tools/perf/util/map.c +++ b/tools/perf/util/map.c @@ -929,10 +929,11 @@ static void __maps__insert_name(struct maps *maps, struct map *map) else if (rc > 0) p = &(*p)->rb_right; else - return; + goto out; } rb_link_node(&map->rb_node_name, parent, p); rb_insert_color(&map->rb_node_name, &maps->names); +out: map__get(map); } ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-04 12:25 ` Jiri Olsa @ 2019-04-04 17:08 ` Song Liu 2019-04-04 17:38 ` Jiri Olsa 0 siblings, 1 reply; 14+ messages in thread From: Song Liu @ 2019-04-04 17:08 UTC (permalink / raw) To: Jiri Olsa Cc: Arnaldo Carvalho de Melo, Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen > On Apr 4, 2019, at 5:25 AM, Jiri Olsa <jolsa@redhat.com> wrote: > > On Thu, Apr 04, 2019 at 11:14:38AM +0200, Jiri Olsa wrote: > > SNIP > >> Program received signal SIGABRT, Aborted. >> 0x00007ffff75e60f5 in raise () from /lib64/libc.so.6 >> Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-29.fc30.x86_64 elfutils-libelf-0.176-1.fc30.x86_64 elfutils-libs-0.176-1.fc30.x86_64 libgcc-9.0.1-0.10.fc30.x86_64 libunwind-1.3.1-2.fc30.x86_64 libxcrypt-4.4.4-2.fc30.x86_64 numactl-libs-2.0.12-2.fc30.x86_64 perl-libs-5.28.1-434.fc30.x86_64 python2-libs-2.7.16-1.fc30.x86_64 slang-2.3.2-5.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64 >> (gdb) bt >> #0 0x00007ffff75e60f5 in raise () from /lib64/libc.so.6 >> #1 0x00007ffff75d0895 in abort () from /lib64/libc.so.6 >> #2 0x00007ffff75d0769 in __assert_fail_base.cold () from /lib64/libc.so.6 >> #3 0x00007ffff75de596 in __assert_fail () from /lib64/libc.so.6 >> #4 0x00000000004fc006 in refcount_sub_and_test (i=1, r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:131 >> #5 refcount_dec_and_test (r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:148 >> #6 map__put (map=0x1224df0) at util/map.c:299 >> #7 0x00000000004fdb95 in __maps__remove (map=0x1224df0, maps=0xb17d80) at util/map.c:953 >> #8 maps__remove (maps=0xb17d80, map=0x1224df0) at util/map.c:959 >> #9 0x00000000004f7d8a in map_groups__remove (map=<optimized out>, mg=<optimized out>) at util/map_groups.h:65 >> #10 machine__process_ksymbol_unregister (sample=<optimized out>, event=0x7ffff7279670, machine=<optimized out>) at util/machine.c:728 >> #11 machine__process_ksymbol (machine=<optimized out>, event=0x7ffff7279670, sample=<optimized out>) at util/machine.c:741 >> #12 0x00000000004fffbb in perf_session__deliver_event (session=0xb11390, event=0x7ffff7279670, tool=0x7fffffffc7b0, file_offset=13936) at util/session.c:1362 >> #13 0x00000000005039bb in do_flush (show_progress=false, oe=0xb17e80) at util/ordered-events.c:243 >> #14 __ordered_events__flush (oe=0xb17e80, how=OE_FLUSH__ROUND, timestamp=<optimized out>) at util/ordered-events.c:322 >> #15 0x00000000005005e4 in perf_session__process_user_event (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, >> file_offset=file_offset@entry=191224) at util/session.c:1402 >> #16 0x000000000050091e in perf_session__process_event (file_offset=191224, event=0x7ffff72a4af8, session=0xb11390) at util/session.c:1529 >> #17 process_simple (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1962 >> #18 0x0000000000501f44 in reader__process_events (prog=0x7fffffffc680, session=0xb11390, rd=<synthetic pointer>) at util/session.c:1931 >> #19 __perf_session__process_events (session=0xb11390) at util/session.c:1985 >> #20 perf_session__process_events (session=0xb11390) at util/session.c:2018 >> #21 0x000000000045bb42 in __cmd_script (script=0x7fffffffc7b0) at builtin-script.c:2429 >> #22 cmd_script (argc=<optimized out>, argv=<optimized out>) at builtin-script.c:3770 >> #23 0x00000000004a65cb in run_builtin (p=p@entry=0x994918 <commands+408>, argc=argc@entry=4, argv=argv@entry=0x7fffffffe160) at perf.c:303 >> #24 0x000000000042da1e in handle_internal_command (argv=0x7fffffffe160, argc=4) at perf.c:355 >> #25 run_argv (argcp=<synthetic pointer>, argv=<synthetic pointer>) at perf.c:399 >> #26 main (argc=4, argv=0x7fffffffe160) at perf.c:521 > For some reason, I cannot repro this issue. I tried to load two bpf programs with same name, but that doesn't trigger it either. > > we dont increase the map's reference for same name, > which we need to.. I assume we we re able to hit this > path with bpf maps/dso with same name > > jirka Does the following change fix the issue on your side? Thanks, Song > > > --- > diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c > index 58ae5fc1f0e9..cdc49113ba38 100644 > --- a/tools/perf/util/map.c > +++ b/tools/perf/util/map.c > @@ -929,10 +929,11 @@ static void __maps__insert_name(struct maps *maps, struct map *map) > else if (rc > 0) > p = &(*p)->rb_right; > else > - return; > + goto out; > } > rb_link_node(&map->rb_node_name, parent, p); > rb_insert_color(&map->rb_node_name, &maps->names); > +out: > map__get(map); > } > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-04 17:08 ` Song Liu @ 2019-04-04 17:38 ` Jiri Olsa 0 siblings, 0 replies; 14+ messages in thread From: Jiri Olsa @ 2019-04-04 17:38 UTC (permalink / raw) To: Song Liu Cc: Arnaldo Carvalho de Melo, Adrian Hunter, Andi Kleen, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen On Thu, Apr 04, 2019 at 05:08:55PM +0000, Song Liu wrote: > > > > On Apr 4, 2019, at 5:25 AM, Jiri Olsa <jolsa@redhat.com> wrote: > > > > On Thu, Apr 04, 2019 at 11:14:38AM +0200, Jiri Olsa wrote: > > > > SNIP > > > >> Program received signal SIGABRT, Aborted. > >> 0x00007ffff75e60f5 in raise () from /lib64/libc.so.6 > >> Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-29.fc30.x86_64 elfutils-libelf-0.176-1.fc30.x86_64 elfutils-libs-0.176-1.fc30.x86_64 libgcc-9.0.1-0.10.fc30.x86_64 libunwind-1.3.1-2.fc30.x86_64 libxcrypt-4.4.4-2.fc30.x86_64 numactl-libs-2.0.12-2.fc30.x86_64 perl-libs-5.28.1-434.fc30.x86_64 python2-libs-2.7.16-1.fc30.x86_64 slang-2.3.2-5.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64 > >> (gdb) bt > >> #0 0x00007ffff75e60f5 in raise () from /lib64/libc.so.6 > >> #1 0x00007ffff75d0895 in abort () from /lib64/libc.so.6 > >> #2 0x00007ffff75d0769 in __assert_fail_base.cold () from /lib64/libc.so.6 > >> #3 0x00007ffff75de596 in __assert_fail () from /lib64/libc.so.6 > >> #4 0x00000000004fc006 in refcount_sub_and_test (i=1, r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:131 > >> #5 refcount_dec_and_test (r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:148 > >> #6 map__put (map=0x1224df0) at util/map.c:299 > >> #7 0x00000000004fdb95 in __maps__remove (map=0x1224df0, maps=0xb17d80) at util/map.c:953 > >> #8 maps__remove (maps=0xb17d80, map=0x1224df0) at util/map.c:959 > >> #9 0x00000000004f7d8a in map_groups__remove (map=<optimized out>, mg=<optimized out>) at util/map_groups.h:65 > >> #10 machine__process_ksymbol_unregister (sample=<optimized out>, event=0x7ffff7279670, machine=<optimized out>) at util/machine.c:728 > >> #11 machine__process_ksymbol (machine=<optimized out>, event=0x7ffff7279670, sample=<optimized out>) at util/machine.c:741 > >> #12 0x00000000004fffbb in perf_session__deliver_event (session=0xb11390, event=0x7ffff7279670, tool=0x7fffffffc7b0, file_offset=13936) at util/session.c:1362 > >> #13 0x00000000005039bb in do_flush (show_progress=false, oe=0xb17e80) at util/ordered-events.c:243 > >> #14 __ordered_events__flush (oe=0xb17e80, how=OE_FLUSH__ROUND, timestamp=<optimized out>) at util/ordered-events.c:322 > >> #15 0x00000000005005e4 in perf_session__process_user_event (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, > >> file_offset=file_offset@entry=191224) at util/session.c:1402 > >> #16 0x000000000050091e in perf_session__process_event (file_offset=191224, event=0x7ffff72a4af8, session=0xb11390) at util/session.c:1529 > >> #17 process_simple (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1962 > >> #18 0x0000000000501f44 in reader__process_events (prog=0x7fffffffc680, session=0xb11390, rd=<synthetic pointer>) at util/session.c:1931 > >> #19 __perf_session__process_events (session=0xb11390) at util/session.c:1985 > >> #20 perf_session__process_events (session=0xb11390) at util/session.c:2018 > >> #21 0x000000000045bb42 in __cmd_script (script=0x7fffffffc7b0) at builtin-script.c:2429 > >> #22 cmd_script (argc=<optimized out>, argv=<optimized out>) at builtin-script.c:3770 > >> #23 0x00000000004a65cb in run_builtin (p=p@entry=0x994918 <commands+408>, argc=argc@entry=4, argv=argv@entry=0x7fffffffe160) at perf.c:303 > >> #24 0x000000000042da1e in handle_internal_command (argv=0x7fffffffe160, argc=4) at perf.c:355 > >> #25 run_argv (argcp=<synthetic pointer>, argv=<synthetic pointer>) at perf.c:399 > >> #26 main (argc=4, argv=0x7fffffffe160) at perf.c:521 > > > > For some reason, I cannot repro this issue. I tried to load two > bpf programs with same name, but that doesn't trigger it either. for some reason I see multiple same name of bpf programs on my setup.. need to check for the reason > > > > > > we dont increase the map's reference for same name, > > which we need to.. I assume we we re able to hit this > > path with bpf maps/dso with same name > > > > jirka > > Does the following change fix the issue on your side? yes, I'll post full patch later jirka ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 14:37 [BUG] perf: intel_pt won't display kernel function Jiri Olsa 2019-04-03 14:53 ` Arnaldo Carvalho de Melo @ 2019-04-03 17:05 ` Song Liu 2019-04-03 18:10 ` Jiri Olsa 1 sibling, 1 reply; 14+ messages in thread From: Song Liu @ 2019-04-03 17:05 UTC (permalink / raw) To: Jiri Olsa Cc: Adrian Hunter, Andi Kleen, acme, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen > On Apr 3, 2019, at 7:37 AM, Jiri Olsa <jolsa@redhat.com> wrote: > > hi, > perf script --call-trace stop working for me recently, > and displays only user space functions > > I bisected that to: > 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs > > data from following comands will display user space functions only: > # perf-with-kcore record pt -e intel_pt// -- ls > # perf-with-kcore script pt --call-trace Seems the following are ok (upstream as-is)? ./perf record -e intel_pt// -- ls ./perf script --call-trace Jiri, could you please verify this? (Sorry I am new to perf-with-kcore). Thanks, Song > > when I disable the bpf synthesizing (patch below), kernel functions are back > > I guess the new events mess up with intel_pt decoder somehow > > > thanks, > jirka > > > --- > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c > index 4e2d953d4bc5..3daa78bc6549 100644 > --- a/tools/perf/builtin-record.c > +++ b/tools/perf/builtin-record.c > @@ -1114,10 +1114,12 @@ static int record__synthesize(struct record *rec, bool tail) > return err; > } > > +#if 0 > err = perf_event__synthesize_bpf_events(session, process_synthesized_event, > machine, opts); > if (err < 0) > pr_warning("Couldn't synthesize bpf events.\n"); > +#endif > > err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, > process_synthesized_event, opts->sample_address, ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [BUG] perf: intel_pt won't display kernel function 2019-04-03 17:05 ` Song Liu @ 2019-04-03 18:10 ` Jiri Olsa 0 siblings, 0 replies; 14+ messages in thread From: Jiri Olsa @ 2019-04-03 18:10 UTC (permalink / raw) To: Song Liu Cc: Adrian Hunter, Andi Kleen, acme, jolsa, namhyung, linux-kernel, linux-perf-users, Andi Kleen On Wed, Apr 03, 2019 at 05:05:02PM +0000, Song Liu wrote: > > > > On Apr 3, 2019, at 7:37 AM, Jiri Olsa <jolsa@redhat.com> wrote: > > > > hi, > > perf script --call-trace stop working for me recently, > > and displays only user space functions > > > > I bisected that to: > > 7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs > > > > data from following comands will display user space functions only: > > # perf-with-kcore record pt -e intel_pt// -- ls > > # perf-with-kcore script pt --call-trace > > Seems the following are ok (upstream as-is)? > > ./perf record -e intel_pt// -- ls > ./perf script --call-trace yes, I think it's the --kallsyms otion which is added by perf-with-kcore that's broken jirka > > Jiri, could you please verify this? (Sorry I am new to perf-with-kcore). > > Thanks, > Song > > > > > when I disable the bpf synthesizing (patch below), kernel functions are back > > > > I guess the new events mess up with intel_pt decoder somehow > > > > > > thanks, > > jirka > > > > > > --- > > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c > > index 4e2d953d4bc5..3daa78bc6549 100644 > > --- a/tools/perf/builtin-record.c > > +++ b/tools/perf/builtin-record.c > > @@ -1114,10 +1114,12 @@ static int record__synthesize(struct record *rec, bool tail) > > return err; > > } > > > > +#if 0 > > err = perf_event__synthesize_bpf_events(session, process_synthesized_event, > > machine, opts); > > if (err < 0) > > pr_warning("Couldn't synthesize bpf events.\n"); > > +#endif > > > > err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads, > > process_synthesized_event, opts->sample_address, > ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2019-04-04 17:38 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-04-03 14:37 [BUG] perf: intel_pt won't display kernel function Jiri Olsa 2019-04-03 14:53 ` Arnaldo Carvalho de Melo 2019-04-03 15:15 ` Arnaldo Carvalho de Melo 2019-04-03 16:27 ` Song Liu 2019-04-03 18:50 ` Arnaldo Carvalho de Melo 2019-04-03 18:55 ` Song Liu 2019-04-03 18:59 ` Song Liu 2019-04-03 21:48 ` Song Liu 2019-04-04 9:14 ` Jiri Olsa 2019-04-04 12:25 ` Jiri Olsa 2019-04-04 17:08 ` Song Liu 2019-04-04 17:38 ` Jiri Olsa 2019-04-03 17:05 ` Song Liu 2019-04-03 18:10 ` Jiri Olsa
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.