Re: [PATCH] libbpf: kprobe.multi: Filter with blacklist and available_filter_functions

[Jackie Liu <liu.yun@linux.dev>]

Hi Jiri.

在 2023/5/24 14:47, Jiri Olsa 写道:
> On Wed, May 24, 2023 at 09:19:48AM +0800, Jackie Liu wrote:
>> Hi Jiri.
>>
>> 在 2023/5/24 09:03, Jackie Liu 写道:
>>> Hi Jiri.
>>>
>>> 在 2023/5/24 00:17, Jiri Olsa 写道:
>>>> On Tue, May 23, 2023 at 09:25:47PM +0800, Jackie Liu wrote:
>>>>> From: Jackie Liu <liuyun01@kylinos.cn>
>>>>>
>>>>> When using regular expression matching with "kprobe multi", it scans all
>>>>> the functions under "/proc/kallsyms" that can be matched.
>>>>> However, not all
>>>>> of them can be traced by kprobe.multi. If any one of the functions fails
>>>>> to be traced, it will result in the failure of all functions. The best
>>>>> approach is to filter out the functions that cannot be traced to ensure
>>>>> proper tracking of the functions.
>>>>>
>>>>> But, the addition of these checks will frequently probe whether
>>>>> a function
>>>>> complies with "available_filter_functions" and ensure that it
>>>>> has not been
>>>>> filtered by kprobe's blacklist. As a result, it may take a longer time
>>>>> during startup. The function implementation is referenced from BCC's
>>>>> "kprobe_exists()"
>>>>>
>>>>> Here is the test eBPF program [1].
>>>>> [1] https://github.com/JackieLiu1/ketones/commit/a9e76d1ba57390e533b8b3eadde97f7a4535e867
>>>>>
>>>>> Signed-off-by: Jackie Liu <liuyun01@kylinos.cn>
>>>>> ---
>>>>>    tools/lib/bpf/libbpf.c | 47 ++++++++++++++++++++++++++++++++++++++++++
>>>>>    1 file changed, 47 insertions(+)
>>>>>
>>>>> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
>>>>> index ad1ec893b41b..6a201267fa08 100644
>>>>> --- a/tools/lib/bpf/libbpf.c
>>>>> +++ b/tools/lib/bpf/libbpf.c
>>>>> @@ -10421,6 +10421,50 @@ struct kprobe_multi_resolve {
>>>>>        size_t cnt;
>>>>>    };
>>>>> +static bool filter_available_function(const char *name)
>>>>> +{
>>>>> +    char addr_range[256];
>>>>> +    char sym_name[256];
>>>>> +    FILE *f;
>>>>> +    int ret;
>>>>> +
>>>>> +    f = fopen("/sys/kernel/debug/kprobes/blacklist", "r");
>>>>> +    if (!f)
>>>>> +        goto avail_filter;
>>>>> +
>>>>> +    while (true) {
>>>>> +        ret = fscanf(f, "%s %s%*[^\n]\n", addr_range, sym_name);
>>>>> +        if (ret == EOF && feof(f))
>>>>> +            break;
>>>>> +        if (ret != 2)
>>>>> +            break;
>>>>> +        if (!strcmp(name, sym_name)) {
>>>>> +            fclose(f);
>>>>> +            return false;
>>>>> +        }
>>>>> +    }
>>>>> +    fclose(f);
>>>>
>>>> so available_filter_functions already contains all traceable symbols
>>>> for kprobe_multi/fprobe
>>>>
>>>> kprobes/blacklist is kprobe specific and does not apply to fprobe,
>>>> is there a crash when attaching function from kprobes/blacklist ?
>>>
>>> No, I haven't got crash before, Simply because BCC's kprobe_exists has
>>> implemented it so I added this, Yes, I also don't think
>>> kprobes/blacklist will affect FPROBE, so I will remove it.
>>>
>>>>
>>>>> +
>>>>> +avail_filter:
>>>>> +    f =
>>>>> fopen("/sys/kernel/debug/tracing/available_filter_functions",
>>>>> "r");
>>>>> +    if (!f)
>>>>> +        return true;
>>>>> +
>>>>> +    while (true) {
>>>>> +        ret = fscanf(f, "%s%*[^\n]\n", sym_name);
>>>>> +        if (ret == EOF && feof(f))
>>>>> +            break;
>>>>> +        if (ret != 1)
>>>>> +            break;
>>>>> +        if (!strcmp(name, sym_name)) {
>>>>> +            fclose(f);
>>>>> +            return true;
>>>>> +        }
>>>>> +    }
>>>>> +    fclose(f);
>>>>> +    return false;
>>>>> +}
>>>>> +
>>>>>    static int
>>>>>    resolve_kprobe_multi_cb(unsigned long long sym_addr, char sym_type,
>>>>>                const char *sym_name, void *ctx)
>>>>> @@ -10431,6 +10475,9 @@ resolve_kprobe_multi_cb(unsigned long
>>>>> long sym_addr, char sym_type,
>>>>>        if (!glob_match(sym_name, res->pattern))
>>>>>            return 0;
>>>>> +    if (!filter_available_function(sym_name))
>>>>> +        return 0;
>>>>
>>>> I think it'd be better to parse available_filter_functions directly
>>>> for kprobe_multi instead of filtering out kallsyms entries
>>>>
>>>> we could add libbpf_available_filter_functions_parse function with
>>>> similar callback to go over available_filter_functions file
>>>>
>>>
>>> Sure, if available_filter_functions not found, fallback to /proc/kallsyms.
>>>
>>
>> Um.
>>
>> It is difficult to judge available_filter_functions directly, because we
>> not only need the function name, but also obtain its address and other
>> information, but we can indeed obtain the function set from
>> available_filter_functions first, and then obtain the function address
>> from /proc/kallsyms. which will be slightly faster than reading
>> available_filter_functions later, because if this function does not
>> exist in available_filter_functions, it will take a long time to read
>> the entire file.
>>
>> Of course, it would be better if the kernel directly provided an
>> available_filter_functions -like file containing function address
>> information.
> 
> you don't need to resolve symbols, you can pass just array of symbols
> to create kprobe_multi link and they will get resolved in kernel:
> 
> 	struct bpf_link_create_opts {
> 
> 			struct {
> 				__u32 flags;
> 				__u32 cnt;
> 		--->		const char **syms;
> 				const unsigned long *addrs;
> 				const __u64 *cookies;
> 			} kprobe_multi;
> 	}
> 
> I resolved the symbols in bpf_program__attach_kprobe_multi_opts mostly
> because the address was available right away when parsing kallsyms,
> but passing just symbols for pattern is fine

I see, let me try. Thanks.

-- 
Jackie Liu

> 
> jirka