[PATCH 0/3] Add PERF_EVENT_IOC_QUERY_KPROBE ioctl

[PATCH 0/3] Add PERF_EVENT_IOC_QUERY_KPROBE ioctl

[Daniel Xu]

It's useful to know kprobe's nmissed and nhit stats. For example with
tracing tools, it's important to know when events may have been lost.
There is currently no way to get that information from the perf API.
This patch adds a new ioctl that lets users query this information.

Daniel Xu (3):
  tracing/kprobe: Add PERF_EVENT_IOC_QUERY_KPROBE ioctl
  libbpf: Add helper to extract perf fd from bpf_link
  tracing/kprobe: Add self test for PERF_EVENT_IOC_QUERY_KPROBE

 include/linux/trace_events.h                  |  6 +++
 include/uapi/linux/perf_event.h               | 23 ++++++++++
 kernel/events/core.c                          | 11 +++++
 kernel/trace/trace_kprobe.c                   | 25 +++++++++++
 tools/include/uapi/linux/perf_event.h         | 23 ++++++++++
 tools/lib/bpf/libbpf.c                        | 13 ++++++
 tools/lib/bpf/libbpf.h                        |  1 +
 tools/lib/bpf/libbpf.map                      |  5 +++
 .../selftests/bpf/prog_tests/attach_probe.c   | 43 +++++++++++++++++++
 9 files changed, 150 insertions(+)

-- 
2.20.1

Re: [PATCH 0/3] Add PERF_EVENT_IOC_QUERY_KPROBE ioctl

[Yonghong Song]

On 8/6/19 4:38 PM, Daniel Xu wrote:
> It's useful to know kprobe's nmissed and nhit stats. For example with
> tracing tools, it's important to know when events may have been lost.
> There is currently no way to get that information from the perf API.
> This patch adds a new ioctl that lets users query this information.

Probably worthwhile to further elaborate the use case.
The debugfs does have a way to count kprobe nmissed and nhit
through /sys/kernel/debug/tracing/kprobe_profile.
But there is no uapi to count the nmissed and nhit for
FD-based kprobe interface. bpf program itself might be able
to count nhit, but still no uapi to get the nmissed counters.

> 
> Daniel Xu (3):
>    tracing/kprobe: Add PERF_EVENT_IOC_QUERY_KPROBE ioctl
>    libbpf: Add helper to extract perf fd from bpf_link
>    tracing/kprobe: Add self test for PERF_EVENT_IOC_QUERY_KPROBE
> 
>   include/linux/trace_events.h                  |  6 +++
>   include/uapi/linux/perf_event.h               | 23 ++++++++++
>   kernel/events/core.c                          | 11 +++++
>   kernel/trace/trace_kprobe.c                   | 25 +++++++++++
>   tools/include/uapi/linux/perf_event.h         | 23 ++++++++++
>   tools/lib/bpf/libbpf.c                        | 13 ++++++
>   tools/lib/bpf/libbpf.h                        |  1 +
>   tools/lib/bpf/libbpf.map                      |  5 +++
>   .../selftests/bpf/prog_tests/attach_probe.c   | 43 +++++++++++++++++++
>   9 files changed, 150 insertions(+)
> 

Re: [PATCH 0/3] Add PERF_EVENT_IOC_QUERY_KPROBE ioctl

[Song Liu]

On Tue, Aug 6, 2019 at 4:39 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
>
> It's useful to know kprobe's nmissed and nhit stats. For example with
> tracing tools, it's important to know when events may have been lost.
> There is currently no way to get that information from the perf API.
> This patch adds a new ioctl that lets users query this information.
>

Part of the change is on perf side, so please include linux-kernel and
perf mantainers and reviewers:

M:      Peter Zijlstra <peterz@infradead.org>
M:      Ingo Molnar <mingo@redhat.com>
M:      Arnaldo Carvalho de Melo <acme@kernel.org>
R:      Alexander Shishkin <alexander.shishkin@linux.intel.com>
R:      Jiri Olsa <jolsa@redhat.com>
R:      Namhyung Kim <namhyung@kernel.org>
L:      linux-kernel@vger.kernel.org

I guess this is based on bpf-next tree. So please use subject perfix
[PATCH bpf-next] ([PATCH v2 bpf-next] when resend).

Thanks,
Song

Re: [PATCH 0/3] Add PERF_EVENT_IOC_QUERY_KPROBE ioctl

[Andrii Nakryiko]

On Tue, Aug 6, 2019 at 4:39 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
>
> It's useful to know kprobe's nmissed and nhit stats. For example with

Is nmissed/nhit kprobe-specific? What about tracepoints and raw
tracepoints, do they have something similar or they can never be
missed? At least nhit still seems useful, so would be nice to have
ability to get that with the same API, is it possible?

> tracing tools, it's important to know when events may have been lost.
> There is currently no way to get that information from the perf API.
> This patch adds a new ioctl that lets users query this information.
>
> Daniel Xu (3):
>   tracing/kprobe: Add PERF_EVENT_IOC_QUERY_KPROBE ioctl
>   libbpf: Add helper to extract perf fd from bpf_link
>   tracing/kprobe: Add self test for PERF_EVENT_IOC_QUERY_KPROBE
>
>  include/linux/trace_events.h                  |  6 +++
>  include/uapi/linux/perf_event.h               | 23 ++++++++++
>  kernel/events/core.c                          | 11 +++++
>  kernel/trace/trace_kprobe.c                   | 25 +++++++++++
>  tools/include/uapi/linux/perf_event.h         | 23 ++++++++++
>  tools/lib/bpf/libbpf.c                        | 13 ++++++
>  tools/lib/bpf/libbpf.h                        |  1 +
>  tools/lib/bpf/libbpf.map                      |  5 +++
>  .../selftests/bpf/prog_tests/attach_probe.c   | 43 +++++++++++++++++++
>  9 files changed, 150 insertions(+)
>
> --
> 2.20.1
>

Re: [PATCH 0/3] Add PERF_EVENT_IOC_QUERY_KPROBE ioctl

[Daniel Xu]

On Wed, Aug 7, 2019, at 12:03 PM, Andrii Nakryiko wrote:
> On Tue, Aug 6, 2019 at 4:39 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
> >
> > It's useful to know kprobe's nmissed and nhit stats. For example with
> 
> Is nmissed/nhit kprobe-specific? What about tracepoints and raw
> tracepoints, do they have something similar or they can never be
> missed? At least nhit still seems useful, so would be nice to have
> ability to get that with the same API, is it possible?
> 

I'm still trying to grok all the tracepoint/ftrace machinery, but it appears
to me like it is kprobe/uprobe specific. My guess is that b/c tracepoints are
inline (and don't require trapping interrupts), it cannot really "miss".

This brings up a good point, though. I think we want the same querying
functionality for uprobes so it might be worthwhile to make this API generic.
Something like PERF_EVENT_IOC_QUERY_PROBE so we can later add in
uprobe stats. And maybe tracepoint if it makes sense.

Thoughts?

Re: [PATCH 0/3] Add PERF_EVENT_IOC_QUERY_KPROBE ioctl

[Andrii Nakryiko]

On Thu, Aug 8, 2019 at 3:32 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
>
> On Wed, Aug 7, 2019, at 12:03 PM, Andrii Nakryiko wrote:
> > On Tue, Aug 6, 2019 at 4:39 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
> > >
> > > It's useful to know kprobe's nmissed and nhit stats. For example with
> >
> > Is nmissed/nhit kprobe-specific? What about tracepoints and raw
> > tracepoints, do they have something similar or they can never be
> > missed? At least nhit still seems useful, so would be nice to have
> > ability to get that with the same API, is it possible?
> >
>
> I'm still trying to grok all the tracepoint/ftrace machinery, but it appears
> to me like it is kprobe/uprobe specific. My guess is that b/c tracepoints are
> inline (and don't require trapping interrupts), it cannot really "miss".
>
> This brings up a good point, though. I think we want the same querying
> functionality for uprobes so it might be worthwhile to make this API generic.
> Something like PERF_EVENT_IOC_QUERY_PROBE so we can later add in
> uprobe stats. And maybe tracepoint if it makes sense.
>
> Thoughts?

Yeah, we should definitely unify uprobes, if possible. Having nhit
would be great for tracepoints/raw_tracepoints, but we need to look
whether it's possible and whether it hurts performance. With bpf_stats
we trigger them only from time to time, so counting overhead might be
noticeable for some use cases.