Headers for whitelisted kernel functions available to BPF programs

Headers for whitelisted kernel functions available to BPF programs

[Kenny Ho]

Hi,

I understand that helper functions available to bpf programs are
listed in include/uapi/linux/bpf.h and kernel headers can be made
available at /sys/kernel/kheaders.tar.xz with CONFIG_IKHEADERS.  But
with the support of calling kernel functions from bpf programs, how
would one know which functions are whitelisted?  Are the headers for
these whitelisted functions available via something like "bpftool btf
dump file /sys/kernel/btf/vmlinux format c"?

Regards,
Kenny

Re: Headers for whitelisted kernel functions available to BPF programs

[Martin KaFai Lau]

On Thu, Jun 03, 2021 at 02:01:13PM -0400, Kenny Ho wrote:
> Hi,
> 
> I understand that helper functions available to bpf programs are
> listed in include/uapi/linux/bpf.h and kernel headers can be made
> available at /sys/kernel/kheaders.tar.xz with CONFIG_IKHEADERS.  But
> with the support of calling kernel functions from bpf programs, how
> would one know which functions are whitelisted?  Are the headers for
> these whitelisted functions available via something like "bpftool btf
> dump file /sys/kernel/btf/vmlinux format c"?
Like other whitelisted functions in BPF, the list is not in the vmlinux btf
now but could be a BTF extension in the future (Cc: Yonghong).

Making the kfunc call whitelist more accessible is useful in general.
The bpf tcp-cc struct_ops is the only prog type supporting kfunc call.
What is your use case to introspect this whitelist?

Re: Headers for whitelisted kernel functions available to BPF programs

[Yonghong Song]

On 6/3/21 11:13 PM, Martin KaFai Lau wrote:
> On Thu, Jun 03, 2021 at 02:01:13PM -0400, Kenny Ho wrote:
>> Hi,
>>
>> I understand that helper functions available to bpf programs are
>> listed in include/uapi/linux/bpf.h and kernel headers can be made
>> available at /sys/kernel/kheaders.tar.xz with CONFIG_IKHEADERS.  But
>> with the support of calling kernel functions from bpf programs, how
>> would one know which functions are whitelisted?  Are the headers for
>> these whitelisted functions available via something like "bpftool btf
>> dump file /sys/kernel/btf/vmlinux format c"?
> Like other whitelisted functions in BPF, the list is not in the vmlinux btf
> now but could be a BTF extension in the future (Cc: Yonghong).

Currently I am working to add a custom dwarf attribute in llvm so we can
annotate global/static variables, global/static functions, function
arguments, struct/unions, struct/union fields, with arbitrary strings
and these strings will be preserved in dwarf so we can eventually
encode in BTF. The WIP patch is here https://reviews.llvm.org/D103667.

> 
> Making the kfunc call whitelist more accessible is useful in general.
> The bpf tcp-cc struct_ops is the only prog type supporting kfunc call.
> What is your use case to introspect this whitelist?

Agree. It would be good if you can share your use case.

Re: Headers for whitelisted kernel functions available to BPF programs

[Kenny Ho]

On Fri, Jun 4, 2021 at 12:11 PM Yonghong Song <yhs@fb.com> wrote:
> On 6/3/21 11:13 PM, Martin KaFai Lau wrote:
> >
> > Making the kfunc call whitelist more accessible is useful in general.
> > The bpf tcp-cc struct_ops is the only prog type supporting kfunc call.
> > What is your use case to introspect this whitelist?
>
> Agree. It would be good if you can share your use case.

At the high level, I am trying to see if we can use bpf in the drm
subsystem and gpu drivers which are kernel modules.  My initial
motivation was to use bpf for dynamic/run-time reconfiguration of the
drm/gpu driver (for experimentation.)  But now that I learned more
about bpf, I think there are quite a few more things I can do with it.
(Debugging during GPU hw bring-ups, profiling driver performance in
live system, etc.)  I have been looking into bpf with kprobe and
struct_ops.

In terms of kernel module support for bpf/btf, Andrii told me about it
last year and I see that his feature is in (I was able to do a btf
dump file for /sys/kernel/btf/amdgpu, /sys/kernel/btf/drm_ttm_helper,
for example.)  The next thing I thought about was having helper
functions from kernel modules and Toke pointed me to Martin's patch
around "unstable helpers"/calling whitelisted kernel functions and
this is where we are at.

Re: Headers for whitelisted kernel functions available to BPF programs

[Kenny Ho]

On Fri, Jun 4, 2021 at 4:47 PM Kenny Ho <y2kenny@gmail.com> wrote:
>
> On Fri, Jun 4, 2021 at 12:11 PM Yonghong Song <yhs@fb.com> wrote:
> > On 6/3/21 11:13 PM, Martin KaFai Lau wrote:
> > >
> > > Making the kfunc call whitelist more accessible is useful in general.
> > > The bpf tcp-cc struct_ops is the only prog type supporting kfunc call.
> > > What is your use case to introspect this whitelist?
> >
> > Agree. It would be good if you can share your use case.
>
> At the high level, I am trying to see if we can use bpf in the drm
> subsystem and gpu drivers which are kernel modules.  My initial
> motivation was to use bpf for dynamic/run-time reconfiguration of the
> drm/gpu driver (for experimentation.)  But now that I learned more
> about bpf, I think there are quite a few more things I can do with it.
> (Debugging during GPU hw bring-ups, profiling driver performance in
> live system, etc.)  I have been looking into bpf with kprobe and
> struct_ops.
>
> In terms of kernel module support for bpf/btf, Andrii told me about it
> last year and I see that his feature is in (I was able to do a btf
> dump file for /sys/kernel/btf/amdgpu, /sys/kernel/btf/drm_ttm_helper,
> for example.)  The next thing I thought about was having helper
> functions from kernel modules and Toke pointed me to Martin's patch
> around "unstable helpers"/calling whitelisted kernel functions and
> this is where we are at.

Yonghong and Martin, does the use case make sense?  Or am I trying to
do something stupid?

Kenny

Re: Headers for whitelisted kernel functions available to BPF programs

[Yonghong Song]

On 6/8/21 12:51 PM, Kenny Ho wrote:
> On Fri, Jun 4, 2021 at 4:47 PM Kenny Ho <y2kenny@gmail.com> wrote:
>>
>> On Fri, Jun 4, 2021 at 12:11 PM Yonghong Song <yhs@fb.com> wrote:
>>> On 6/3/21 11:13 PM, Martin KaFai Lau wrote:
>>>>
>>>> Making the kfunc call whitelist more accessible is useful in general.
>>>> The bpf tcp-cc struct_ops is the only prog type supporting kfunc call.
>>>> What is your use case to introspect this whitelist?
>>>
>>> Agree. It would be good if you can share your use case.
>>
>> At the high level, I am trying to see if we can use bpf in the drm
>> subsystem and gpu drivers which are kernel modules.  My initial
>> motivation was to use bpf for dynamic/run-time reconfiguration of the
>> drm/gpu driver (for experimentation.)  But now that I learned more
>> about bpf, I think there are quite a few more things I can do with it.
>> (Debugging during GPU hw bring-ups, profiling driver performance in
>> live system, etc.)  I have been looking into bpf with kprobe and
>> struct_ops.
>>
>> In terms of kernel module support for bpf/btf, Andrii told me about it
>> last year and I see that his feature is in (I was able to do a btf
>> dump file for /sys/kernel/btf/amdgpu, /sys/kernel/btf/drm_ttm_helper,
>> for example.)  The next thing I thought about was having helper
>> functions from kernel modules and Toke pointed me to Martin's patch
>> around "unstable helpers"/calling whitelisted kernel functions and
>> this is where we are at.
> 
> Yonghong and Martin, does the use case make sense?  Or am I trying to
> do something stupid?

So your intention is to call functions in 
drivers/gpu/drm/drm_gem_ttm_helper.c, right? How do you get function
parameters? What kinds of programs you intend to call
this functions?

kprobe probably won't work as kernel does not capture
traced function types. fentry program might be a good
choice.

if you can *craft* a bpf program to show how
kernel function is used that will be great. We will
then see how verifier can help enforce safety.

> 
> Kenny
> 

Re: Headers for whitelisted kernel functions available to BPF programs

[Kenny Ho]

On Wed, Jun 9, 2021 at 2:42 AM Yonghong Song <yhs@fb.com> wrote:
> So your intention is to call functions in
> drivers/gpu/drm/drm_gem_ttm_helper.c, right? How do you get function
> parameters? What kinds of programs you intend to call
> this functions?
ok... sounds like my use case was not concret enough.  Perhaps I can
elaborate further with the following examples:

In the GPU scheduler, there's a trace point
"trace_drm_run_job(sched_job, entity)":

https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/scheduler/sched_main.c#L813

If I want to analyze the jobs being scheduled, I can potentially
attach a bpf prog with this tracepoint.  Each driver has its own
run_job and sched_job implementation so I was thinking the drivers can
provide a bpf helper function to resolve this.  Alternatively, there
could be tracepoint in the driver implementation that one can attach
bpf to, but tracepoints are not universally put in place (have trace:
https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c#L221;
not have trace:
https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/etnaviv/etnaviv_sched.c#L72
.)  So in cases without tracepoint, I guess I would be using kprobe or
fentry?

Note that all of these are in kernel modules.  My understanding is
that BTF will work but having helper functions from the kernel modules
are not yet available?  So let say I want to whitelist and call "
amdgpu_device_gpu_recover"
(https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c#L4521)
from inside a bpf prog, or whitelist and call
"drm_sched_increase_karma"
https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/scheduler/sched_main.c#L362,
I wouldn't be able to do so?  Are there any criteria in terms of what
kernel function should or should not be whitelisted (separate from the
kernel module support question)?  For example, would
amdgpu_device_gpu_recover be not "whitelist-able" because of the hw
interaction or complexity or likelihood to crash the kernel while
drm_sched_increase_karma is ok because it's more or less manipulation
of some data structure?

A quick side question, does container_of work inside a bpf prog?

> kprobe probably won't work as kernel does not capture
> traced function types. fentry program might be a good
> choice.
Thanks for the pointer.  I am not familiar with fentry but I will look
into it.  By function type, what do you mean by that?

Re: Headers for whitelisted kernel functions available to BPF programs

[Yonghong Song]

On 6/15/21 12:34 PM, Kenny Ho wrote:
> On Wed, Jun 9, 2021 at 2:42 AM Yonghong Song <yhs@fb.com> wrote:
>> So your intention is to call functions in
>> drivers/gpu/drm/drm_gem_ttm_helper.c, right? How do you get function
>> parameters? What kinds of programs you intend to call
>> this functions?
> ok... sounds like my use case was not concret enough.  Perhaps I can
> elaborate further with the following examples:
> 
> In the GPU scheduler, there's a trace point
> "trace_drm_run_job(sched_job, entity)":
> 
> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/scheduler/sched_main.c#L813
> 
> If I want to analyze the jobs being scheduled, I can potentially
> attach a bpf prog with this tracepoint.  Each driver has its own
> run_job and sched_job implementation so I was thinking the drivers can
> provide a bpf helper function to resolve this.  Alternatively, there
> could be tracepoint in the driver implementation that one can attach
> bpf to, but tracepoints are not universally put in place (have trace:
> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c#L221 ;
> not have trace:
> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/etnaviv/etnaviv_sched.c#L72
> .)  So in cases without tracepoint, I guess I would be using kprobe or
> fentry?

Yes, either kprobe or fentry should work.

> 
> Note that all of these are in kernel modules.  My understanding is
> that BTF will work but having helper functions from the kernel modules
> are not yet available?  So let say I want to whitelist and call "

We could you have some kernel helpers interacts with kernel subsystems.
For example, ipv6 could be a module, some kernel helpers actually works
fine with ipv6 module, you need have some configure time checking as 
well as some runtime checking.


> amdgpu_device_gpu_recover"
> (https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c#L4521 )
> from inside a bpf prog, or whitelist and call
> "drm_sched_increase_karma"
> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/scheduler/sched_main.c#L362 ,
> I wouldn't be able to do so?  Are there any criteria in terms of what
> kernel function should or should not be whitelisted (separate from the
> kernel module support question)?  For example, would
> amdgpu_device_gpu_recover be not "whitelist-able" because of the hw
> interaction or complexity or likelihood to crash the kernel while
> drm_sched_increase_karma is ok because it's more or less manipulation
> of some data structure?

To determine whether a function can be used in what context indeed a 
very tricky question. The function amdgpu_device_gpu_recover is very
complex and has a lot of side effect. Maybe it can still be used,
but need to tag the function with some conditions and unless these
conditions are verified by verifier, the function cannot be used
in bpf program.

> A quick side question, does container_of work inside a bpf prog?

Yes, the macro is defined in bpf_helpers.h.

> 
>> kprobe probably won't work as kernel does not capture
>> traced function types. fentry program might be a good
>> choice.
> Thanks for the pointer.  I am not familiar with fentry but I will look
> into it.  By function type, what do you mean by that?

Sorry, probably typing too fast. "function type" here I actually
mean "btf type". To call kernel function inside the bpf program,
we need btf type to verify kernel function signature, precondition
etc. kprobe uses pt_regs and all types are lost.

>