Re: [PATCH perf,bpf 0/5] reveal invisible bpf programs

From: Song Liu <songliubraving@fb.com>
To: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Alexei Starovoitov <ast@kernel.org>,
	"Daniel Borkmann" <daniel@iogearbox.net>,
	Kernel Team <Kernel-team@fb.com>
Subject: Re: [PATCH perf,bpf 0/5] reveal invisible bpf programs
Date: Tue, 27 Nov 2018 19:21:41 +0000	[thread overview]
Message-ID: <E268FE5F-E2CE-4266-8B1F-A134BFAC6462@fb.com> (raw)
In-Reply-To: <20181126152726.GA5518@kernel.org>

> On Nov 26, 2018, at 7:27 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> 
> Em Mon, Nov 26, 2018 at 03:50:04PM +0100, Peter Zijlstra escreveu:
>> Now, I'm not saying this patch set is useless; but I'm saying most
>> people should not need this, and it is massive overkill for the needs of
>> most people.
> 
> So, the comparision is sort of with kernel modules, that can come and go
> while you're profiling/tracing, if that happens, then samples, in post
> processing, are not resolvable, and that is the case for kernel modules
> right now. Sure, you're right, that doesn't happen so frequently, so
> nobody hollered (thankfully that is now verbotten ;-)) at us so far.
> 
> You need to have the load-kernel-bin/unload-kernel-bin events recorded,
> and you need to somehow match those addresses to some symtab/src(for
> people that want to have src mixed up with assembly) and you need that
> jitted code, with timestamps of when it was loaded and it was unloaded.
> 
> People doing post processing analysis of weird problems need all those
> details.
> 
> Now I don't know how frequently those binary blobs gets loaded/unloaded
> in the brave new world of eBPF, but for completeness sake, we need those
> load/unload events and we need to grab a copy of the raw jitted binary,
> etc.

BPF programs get loaded/unloaded more often than kernel modules. This is 
because BPF verifier makes sure BPF program will never crash the kernel, 
thus BPF programs could often be debugged as user space apps. 

Typical debug/tune process is like:

    perf record -- ./test_work
    perf report

    # make changes to bpf program and repeat

or even:

    perf record -o perf_1.data -- ./test_v1
    perf record -o perf_2.data -- ./test_v2
    perf record -o perf_3.data -- ./test_v3

    # perf report in 3 terminals, and compare the output

In such use cases, including all details in the perf.data is very helpful, 
as /proc/kcore doesn't have details of the program when perf-report runs. 

Thanks,
Song