Re: [PATCH v3 0/2] kstats: kernel metric collector

From: Luigi Rizzo <lrizzo@google.com>
To: "Toke Høiland-Jørgensen" <toke@redhat.com>
Cc: linux-kernel@vger.kernel.org,
	Masami Hiramatsu <mhiramat@kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Greg KH <gregkh@linuxfoundation.org>,
	naveen.n.rao@linux.ibm.com, ardb@kernel.org,
	Luigi Rizzo <rizzo@iet.unipi.it>, Paolo Abeni <pabeni@redhat.com>,
	giuseppe.lettieri@unipi.it,
	Jesper Dangaard Brouer <hawk@kernel.org>,
	mingo@redhat.com, acme@kernel.org,
	Steven Rostedt <rostedt@goodmis.org>,
	Peter Zijlstra <peterz@infradead.org>
Subject: Re: [PATCH v3 0/2] kstats: kernel metric collector
Date: Thu, 27 Feb 2020 02:31:03 -0800	[thread overview]
Message-ID: <CAMOZA0+C4SyGLVhFAa10WPFMarBVVnT+Cysfat-bcJS9mBySmg@mail.gmail.com> (raw)
In-Reply-To: <878skpx7th.fsf@toke.dk>

On Wed, Feb 26, 2020 at 3:11 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Luigi Rizzo <lrizzo@google.com> writes:
>
> > - the runtime cost and complexity of hooking bpf code is still a bit
> > unclear to me. kretprobe or tracepoints are expensive, I suppose that
> > some lean hook replace register_kretprobe() may exist and the
> > difference from inline annotations would be marginal (we'd still need
> > to put in the hooks around the code we want to time, though, so it
> > wouldn't be a pure bpf solution). Any pointers to this are welcome;
> > Alexei mentioned fentry/fexit and bpf trampolines, but I haven't found
> > an example that lets me do something equivalent to kretprobe (take a
> > timestamp before and one after a function without explicit
> > instrumentation)
>
> As Alexei said, with fentry/fexit the overhead should be on par with
> your example. This functionality is pretty new, though, so I can
> understand why it's not obvious how to do things with it yet :)
>
> I think the best place to look is currently in selftests/bpf in the
> kernel sources. Grep for 'fexit' and 'fentry' in the progs/ subdir.
> test_overhead.c and kfree_skb.c seem to have some examples you may be
> able to work from.

Thank you for the precise reference, Toke.
I tweaked test_overhead.c to measure (using kstats) the cost of the various
hooks and I can confirm that fentry and fexit are pretty fast. The
following table
shows the p90 runtime of __set_task_comm() at low (100/s) and high (1M/s) rates:

                      90 percentile of __set_task_comm() runtime
(accuracy: 30ns)
call rate          base     kprobe   kretprobe  tracepoint   fentry   fexit
100/sec          270       870        1220         500             400       450
 >1M/s            60        120         210          90
70          80

For high rate operation, the overhead of fentry and fexit is quite good,
even better than tracepoints, and well below the clock's accuracy
(more detailed measurements indicate ~5ns for fentry, ~10ns for fexit).
At very low call rates there is an extra 150-200ns
but that is expected due to the out of line code.

cheers
luigi