Re: [PATCH] libbpf: kprobe.multi: Filter with blacklist and available_filter_functions

[Jiri Olsa <olsajiri@gmail.com>]

On Wed, May 24, 2023 at 09:19:48AM +0800, Jackie Liu wrote:
> Hi Jiri.
> 
> 在 2023/5/24 09:03, Jackie Liu 写道:
> > Hi Jiri.
> > 
> > 在 2023/5/24 00:17, Jiri Olsa 写道:
> > > On Tue, May 23, 2023 at 09:25:47PM +0800, Jackie Liu wrote:
> > > > From: Jackie Liu <liuyun01@kylinos.cn>
> > > > 
> > > > When using regular expression matching with "kprobe multi", it scans all
> > > > the functions under "/proc/kallsyms" that can be matched.
> > > > However, not all
> > > > of them can be traced by kprobe.multi. If any one of the functions fails
> > > > to be traced, it will result in the failure of all functions. The best
> > > > approach is to filter out the functions that cannot be traced to ensure
> > > > proper tracking of the functions.
> > > > 
> > > > But, the addition of these checks will frequently probe whether
> > > > a function
> > > > complies with "available_filter_functions" and ensure that it
> > > > has not been
> > > > filtered by kprobe's blacklist. As a result, it may take a longer time
> > > > during startup. The function implementation is referenced from BCC's
> > > > "kprobe_exists()"
> > > > 
> > > > Here is the test eBPF program [1].
> > > > [1] https://github.com/JackieLiu1/ketones/commit/a9e76d1ba57390e533b8b3eadde97f7a4535e867
> > > > 
> > > > Signed-off-by: Jackie Liu <liuyun01@kylinos.cn>
> > > > ---
> > > >   tools/lib/bpf/libbpf.c | 47 ++++++++++++++++++++++++++++++++++++++++++
> > > >   1 file changed, 47 insertions(+)
> > > > 
> > > > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > > > index ad1ec893b41b..6a201267fa08 100644
> > > > --- a/tools/lib/bpf/libbpf.c
> > > > +++ b/tools/lib/bpf/libbpf.c
> > > > @@ -10421,6 +10421,50 @@ struct kprobe_multi_resolve {
> > > >       size_t cnt;
> > > >   };
> > > > +static bool filter_available_function(const char *name)
> > > > +{
> > > > +    char addr_range[256];
> > > > +    char sym_name[256];
> > > > +    FILE *f;
> > > > +    int ret;
> > > > +
> > > > +    f = fopen("/sys/kernel/debug/kprobes/blacklist", "r");
> > > > +    if (!f)
> > > > +        goto avail_filter;
> > > > +
> > > > +    while (true) {
> > > > +        ret = fscanf(f, "%s %s%*[^\n]\n", addr_range, sym_name);
> > > > +        if (ret == EOF && feof(f))
> > > > +            break;
> > > > +        if (ret != 2)
> > > > +            break;
> > > > +        if (!strcmp(name, sym_name)) {
> > > > +            fclose(f);
> > > > +            return false;
> > > > +        }
> > > > +    }
> > > > +    fclose(f);
> > > 
> > > so available_filter_functions already contains all traceable symbols
> > > for kprobe_multi/fprobe
> > > 
> > > kprobes/blacklist is kprobe specific and does not apply to fprobe,
> > > is there a crash when attaching function from kprobes/blacklist ?
> > 
> > No, I haven't got crash before, Simply because BCC's kprobe_exists has
> > implemented it so I added this, Yes, I also don't think
> > kprobes/blacklist will affect FPROBE, so I will remove it.
> > 
> > > 
> > > > +
> > > > +avail_filter:
> > > > +    f =
> > > > fopen("/sys/kernel/debug/tracing/available_filter_functions",
> > > > "r");
> > > > +    if (!f)
> > > > +        return true;
> > > > +
> > > > +    while (true) {
> > > > +        ret = fscanf(f, "%s%*[^\n]\n", sym_name);
> > > > +        if (ret == EOF && feof(f))
> > > > +            break;
> > > > +        if (ret != 1)
> > > > +            break;
> > > > +        if (!strcmp(name, sym_name)) {
> > > > +            fclose(f);
> > > > +            return true;
> > > > +        }
> > > > +    }
> > > > +    fclose(f);
> > > > +    return false;
> > > > +}
> > > > +
> > > >   static int
> > > >   resolve_kprobe_multi_cb(unsigned long long sym_addr, char sym_type,
> > > >               const char *sym_name, void *ctx)
> > > > @@ -10431,6 +10475,9 @@ resolve_kprobe_multi_cb(unsigned long
> > > > long sym_addr, char sym_type,
> > > >       if (!glob_match(sym_name, res->pattern))
> > > >           return 0;
> > > > +    if (!filter_available_function(sym_name))
> > > > +        return 0;
> > > 
> > > I think it'd be better to parse available_filter_functions directly
> > > for kprobe_multi instead of filtering out kallsyms entries
> > > 
> > > we could add libbpf_available_filter_functions_parse function with
> > > similar callback to go over available_filter_functions file
> > > 
> > 
> > Sure, if available_filter_functions not found, fallback to /proc/kallsyms.
> > 
> 
> Um.
> 
> It is difficult to judge available_filter_functions directly, because we
> not only need the function name, but also obtain its address and other
> information, but we can indeed obtain the function set from
> available_filter_functions first, and then obtain the function address
> from /proc/kallsyms. which will be slightly faster than reading
> available_filter_functions later, because if this function does not
> exist in available_filter_functions, it will take a long time to read
> the entire file.
> 
> Of course, it would be better if the kernel directly provided an
> available_filter_functions -like file containing function address
> information.

you don't need to resolve symbols, you can pass just array of symbols
to create kprobe_multi link and they will get resolved in kernel:

	struct bpf_link_create_opts {

			struct {
				__u32 flags;
				__u32 cnt;
		--->		const char **syms;
				const unsigned long *addrs;
				const __u64 *cookies;
			} kprobe_multi;
	}

I resolved the symbols in bpf_program__attach_kprobe_multi_opts mostly
because the address was available right away when parsing kallsyms,
but passing just symbols for pattern is fine

jirka