Re: [PATCH bpf-next v3 1/5] bpf: Support chain calling multiple BPF programs after each other

["Toke Høiland-Jørgensen" <toke@redhat.com>]

Edward Cree <ecree@solarflare.com> writes:

> On 17/10/2019 13:11, Toke Høiland-Jørgensen wrote:
>> I think there's a conceptual disconnect here in how we view what an XDP
>> program is. In my mind, an XDP program is a stand-alone entity tied to a
>> particular application; not a library function that can just be inserted
>> into another program.

> To me, an XDP (or any other eBPF) program is a function that is already
>  being 'inserted into another program', namely, the kernel.  It's a
>  function that's being wired up to a hook in the kernel.  Which isn't
>  so different to wiring it up to a hook in a function that's wired up to
>  a hook in the kernel (which is what my proposal effectively does).

Yes, see: Different mental models leading to different notions of what
is the natural way to do things :)

>> Setting aside that for a moment; the reason I don't think this belongs
>> in userspace is that putting it there would carry a complexity cost that
>> is higher than having it in the kernel.
> Complexity in the kernel is more expensive than in userland.  There are
>  several reasons for this, such as:
> * The kernel's reliability requirements are stricter — a daemon that
>   crashes can be restarted, a kernel that crashes ruins your day.
> * Userland has libraries available for many common tasks that can't be
>   used in the kernel.
> * Anything ABI-visible (which this would be) has to be kept forever even
>   if it turns out to be a Bad Idea™, because We Do Not Break
> Userspace™.

To me, the first and last of those are actually arguments for putting
this into the kernel (from the consuming application's PoV). Surely we
want something that's reliable and well-supported? ;)

> The last of these is the big one, and means that wherever possible the
> proper course is to prototype functionality in userspace, and then
> once the ABI is solid and known-useful, it can move to the kernel if
> there's an advantage to doing so

To me, the prototyping was the tail call-based stuff Facebook and
Cloudflare has been doing, and this is an attempt to synthesise that
into something that we can actually agree to support as part of the XDP
feature set.

>> - Add a new dependency to using XDP (now you not only need the kernel
>>   and libraries, you'll also need the daemon).
> You'll need *a* daemon. You won't be tied to a specific
> implementation.

But the point is that I *want* this to be a specific implementation; or
rather, a specific interface. I.e., I want this to be an interface that
people can rely on being available, rather than have a proliferation of
slightly different ways to achieve this that are subtly incompatible.

>>     - Keeping track of which XDP programs are loaded and attached to
>>       each interface
> There's already an API to query this.

There's an API, but the daemon still has to deal with it.

> You would probably want an atomic cmpxchg operation, so that you can
> detect if someone else is fiddling with XDP and scream noisy warnings.

Yup, probably going to do some sort of atomic program replace API in any
case :)

>>     - Some kind of interface with the verifier; if an app does
>>       xdpd_rpc_load(prog), how is the verifier result going to get back
>>       to the caller?
> The daemon will get the verifier log back when it tries to update the
> program; it might want to do a bit of translation before passing it
> on, but an RPC call can definitely return errors to the caller.

I wasn't implying that it was impossible to report errors over RPC.
I was saying that you "a bit of translation" is not as trivial as you
make it out to be...


> In the Ideal World of kernel dynamic linking, of course, each app prog
> gets submitted to the verifier by the app to create a floating function
> in the kernel that's not bound to any XDP hook (app gets its verifier
> responses at this point)

I believe this is what Alexei means by "indirect calls". That is
different, though, because it implies that each program lives as a
separate object in the kernel - and so it might actually work. What you
were talking about (until this paragraph) was something that was
entirely in userspace, and all the kernel sees is a blob of the eBPF
equivalent of `cat *.so > my_composite_prog.so`.

-Toke