Re: [RFC PATCH bpf-next 0/2] unprivileged BPF_PROG_TEST_RUN

Re: [RFC PATCH bpf-next 0/2] unprivileged BPF_PROG_TEST_RUN

[Y Song]

Added cc to bpf@vger.kernel.org.
For bpf related patches, please cc bpf@vger.kernel.org which is major
place for bpf discussions.

On Wed, Dec 18, 2019 at 6:16 PM Edwin Peer <epeer@juniper.net> wrote:
>
> Being able to load, verify and test BPF programs in unprivileged
> build environments is desirable. The two phase load and then
> test API makes this goal difficult to achieve, since relaxing
> permissions for BPF_PROG_TEST_RUN alone would be insufficient.
>
> The approach taken in this proposal defers CAP_SYS_ADMIN checks
> until program attach time in order to unencumber BPF_PROG_LOAD.

I like the idea to *test* bpf program in unprivileged mode as sudo always
has some risk to break the development server.

Have you tried your patch with some bpf programs? verifier and jit  put some
restrictions on unpriv programs. To truely test the program, most if
not all these
restrictions should be lifted, so the same tested program should be able to
run on production server and vice verse.

>
> Edwin Peer (2):
>   bpf: defer capability checks until program attach
>   bpf: relax CAP_SYS_ADMIN requirement for BPF_PROG_TEST_RUN
>
>  include/linux/filter.h |  3 ++-
>  kernel/bpf/syscall.c   | 27 +++++++++++++++++----------
>  2 files changed, 19 insertions(+), 11 deletions(-)
>
> --
> 2.24.1

Re: [RFC PATCH bpf-next 0/2] unprivileged BPF_PROG_TEST_RUN

[Edwin Peer]

On 12/18/19, 23:19, "Y Song" <ys114321@gmail.com> wrote:

>  Added cc to bpf@vger.kernel.org.
    
Thank you, I will remember to do this next time.

> Have you tried your patch with some bpf programs? verifier and jit  put some
> restrictions on unpriv programs. To truely test the program, most if not all these
> restrictions should be lifted, so the same tested program should be able to
> run on production server and vice verse.

Agreed, I am aware of some of these differences in the load/verifier behavior with and without
CAP_SYS_ADMIN. In particular, without CAP_SYS_ADMIN programs are still restricted to 4k, some helpers are not available (spin locks, trace printk) and there are some differences in context access checks.

I think these can be addressed incrementally, assuming folk are on board with this approach in general?

Regards,
Edwin Peer
 

Re: [RFC PATCH bpf-next 0/2] unprivileged BPF_PROG_TEST_RUN

[Daniel Borkmann]

On Thu, Dec 19, 2019 at 02:50:42PM +0000, Edwin Peer wrote:
> On 12/18/19, 23:19, "Y Song" <ys114321@gmail.com> wrote:
> 
> >  Added cc to bpf@vger.kernel.org.
>
> Thank you, I will remember to do this next time.
>
> > Have you tried your patch with some bpf programs? verifier and jit  put some
> > restrictions on unpriv programs. To truely test the program, most if not all these
> > restrictions should be lifted, so the same tested program should be able to
> > run on production server and vice verse.
> 
> Agreed, I am aware of some of these differences in the load/verifier behavior with and without
> CAP_SYS_ADMIN. In particular, without CAP_SYS_ADMIN programs are still restricted to 4k, some helpers are not available (spin locks, trace printk) and there are some differences in context access checks.
> 
> I think these can be addressed incrementally, assuming folk are on board with this approach in general?

What about CAP_BPF? IIRC, there are also other issues e.g. you could abuse
the test interface as a packet generator (bpf_clone_redirect) which is not
something fully unpriv should be doing.

Thanks,
Daniel

Re: [RFC PATCH bpf-next 0/2] unprivileged BPF_PROG_TEST_RUN

[Edwin Peer]

On 12/19/19, 07:47, "Daniel Borkmann" <daniel@iogearbox.net> wrote:

>  What about CAP_BPF?

What is the status of this? It might solve some of the problems, but it is still puts testing
BPF outside reach of normal users.

> IIRC, there are also other issues e.g. you could abuse the test interface as a packet
>  generator (bpf_clone_redirect) which is not something fully unpriv should be doing.

Good point. I suspect solutions exist - I'm trying to ascertain if they are worth pursuing
or if the idea of unprivileged testing is a complete non-starter to begin with.

Are there other helpers of concern that come immediately to mind? A first stab might
add these to the list in the verifier that require privilege. This has the drawback that
programs that actually need this kind of functionality are beyond the test framework.

Another idea might be to have some kind of test classification stored in the program
context. That way, dangerous helpers could be replaced by versions that mock the
functionality in some safe way. Perhaps having test programs only have access to a
subset of virtual netdevs for clone redirect, for example. 

Regards,
Edwin Peer
  

Re: [RFC PATCH bpf-next 0/2] unprivileged BPF_PROG_TEST_RUN

[Alexei Starovoitov]

On Thu, Dec 19, 2019 at 05:05:42PM +0000, Edwin Peer wrote:
> On 12/19/19, 07:47, "Daniel Borkmann" <daniel@iogearbox.net> wrote:
> 
> >  What about CAP_BPF?
> 
> What is the status of this? It might solve some of the problems, but it is still puts testing
> BPF outside reach of normal users.

why?
I think CAP_BPF is solving exactly what you're trying to achieve.
Use CAP_BPF to load _any_ program type and use prog_test_run to run it.
While discussing CAP_BPF during plumbers conf we realized that the kernel doesn't need
to check CAP_BPF for prog_test_run. It's user supplied data. No security risk. Though
the kernel needs to make sure that dangerous helpers are not used for prog_test_run.
Whether bpf_clone_redirect() is such helper is still tbd. Unpriv user can flood netdevs
without any bpf.

> > IIRC, there are also other issues e.g. you could abuse the test interface as a packet
> >  generator (bpf_clone_redirect) which is not something fully unpriv should be doing.
> 
> Good point. I suspect solutions exist - I'm trying to ascertain if they are worth pursuing
> or if the idea of unprivileged testing is a complete non-starter to begin with.
> 
> Are there other helpers of concern that come immediately to mind? A first stab might
> add these to the list in the verifier that require privilege. This has the drawback that
> programs that actually need this kind of functionality are beyond the test framework.

So far majority of programs require root-only verifier features. The programs are
getting more complex and benefit the most from testing. Relaxing test_run for unpriv
progs is imo very narrow use case. I'd rather use CAP_BPF.

Re: [RFC PATCH bpf-next 0/2] unprivileged BPF_PROG_TEST_RUN

[Edwin Peer]

On 12/19/19, 11:26, "Alexei Starovoitov" <alexei.starovoitov@gmail.com> wrote:

> On Thu, Dec 19, 2019 at 05:05:42PM +0000, Edwin Peer wrote:
>> On 12/19/19, 07:47, "Daniel Borkmann" <daniel@iogearbox.net> wrote:
>> 
>> >  What about CAP_BPF?
>> 
>> What is the status of this? It might solve some of the problems, but it is still puts testing
>> BPF outside reach of normal users.
>    
> why?
> I think CAP_BPF is solving exactly what you're trying to achieve.

I'm trying to provide access to BPF testing infrastructure for unprivileged
users (assuming it can be done in a safe way, which I'm as yet unsure of).
CAP_BPF is not the same thing, because at least some kind of root
intervention is required to attain CAP_BPF in the first place.

> Whether bpf_clone_redirect() is such helper is still tbd. Unpriv user can flood netdevs
> without any bpf.
   
True, but presumably such would still be subject to administrator
controlled QoS and firewall policy? Also unprivileged users presumably
can't create arbitrary packets coming from spoofed IPs / MACs, which I
believe requires CAP_NET_RAW?
 
>> Are there other helpers of concern that come immediately to mind? A first stab might
>> add these to the list in the verifier that require privilege. This has the drawback that
>> programs that actually need this kind of functionality are beyond the test framework.
>    
>   So far majority of programs require root-only verifier features. The programs are
>   getting more complex and benefit the most from testing. Relaxing test_run for unpriv
>   progs is imo very narrow use case. I'd rather use CAP_BPF.
    
The more elaborate proposal called for mocking these aspects for
testing, which could conceivably resolve this? That said, I see an
incremental path to this, adding such as needed. The narrowness
of the use case really depends on exactly what you're trying to do.
Something in XDP, for example, has very little kernel dependencies
(possibly none that would be affected here) and represents an entire
class of use cases that could have unprivileged testing be supported.

Regards,
Edwin Peer

Re: [RFC PATCH bpf-next 0/2] unprivileged BPF_PROG_TEST_RUN

[Alexei Starovoitov]

On Thu, Dec 19, 2019 at 08:06:21PM +0000, Edwin Peer wrote:
> On 12/19/19, 11:26, "Alexei Starovoitov" <alexei.starovoitov@gmail.com> wrote:
> 
> > On Thu, Dec 19, 2019 at 05:05:42PM +0000, Edwin Peer wrote:
> >> On 12/19/19, 07:47, "Daniel Borkmann" <daniel@iogearbox.net> wrote:
> >> 
> >> >  What about CAP_BPF?
> >> 
> >> What is the status of this? It might solve some of the problems, but it is still puts testing
> >> BPF outside reach of normal users.
> >    
> > why?
> > I think CAP_BPF is solving exactly what you're trying to achieve.
> 
> I'm trying to provide access to BPF testing infrastructure for unprivileged
> users (assuming it can be done in a safe way, which I'm as yet unsure of).
> CAP_BPF is not the same thing, because at least some kind of root
> intervention is required to attain CAP_BPF in the first place.

yes and test infra can bet setup with CAP_BPF.
The desire of testing frameworks to work without root was one of the main
motivations for us to work on CAP_BPF.

> > Whether bpf_clone_redirect() is such helper is still tbd. Unpriv user can flood netdevs
> > without any bpf.
>    
> True, but presumably such would still be subject to administrator
> controlled QoS and firewall policy? Also unprivileged users presumably
> can't create arbitrary packets coming from spoofed IPs / MACs, which I
> believe requires CAP_NET_RAW?
>  
> >> Are there other helpers of concern that come immediately to mind? A first stab might
> >> add these to the list in the verifier that require privilege. This has the drawback that
> >> programs that actually need this kind of functionality are beyond the test framework.
> >    
> >   So far majority of programs require root-only verifier features. The programs are
> >   getting more complex and benefit the most from testing. Relaxing test_run for unpriv
> >   progs is imo very narrow use case. I'd rather use CAP_BPF.
>     
> The more elaborate proposal called for mocking these aspects for
> testing, which could conceivably resolve this? That said, I see an
> incremental path to this, adding such as needed. The narrowness
> of the use case really depends on exactly what you're trying to do.
> Something in XDP, for example, has very little kernel dependencies
> (possibly none that would be affected here) and represents an entire
> class of use cases that could have unprivileged testing be supported.

I'm looking at public and non-public XDP progs and none of them are verifiable
as unpriv. I don't think it's a good idea to build infra for toy programs.