Re: [PATCH bpf-next 1/4] xdp: Support specifying expected existing program when attaching XDP

[Daniel Borkmann <daniel@iogearbox.net>]

On 3/31/20 12:13 PM, Toke Høiland-Jørgensen wrote:
> Andrii Nakryiko <andrii.nakryiko@gmail.com> writes:
> 
>>>> So you install your libxdp-based firewalls and are happy. Then you
>>>> decide to install this awesome packet analyzer, which doesn't know
>>>> about libxdp yet. Suddenly, you get all packets analyzer, but no more
>>>> firewall, until users somehow notices that it's gone. Or firewall
>>>> periodically checks that it's still runinng. Both not great, IMO, but
>>>> might be acceptable for some users, I guess. But imagine all the
>>>> confusion for user, especially if he doesn't give a damn about XDP and
>>>> other buzzwords, but only needs a reliable firewall :)
>>>
>>> Yes, whereas if the firewall is using bpf_link, then the packet analyser
>>> will be locked out and can't do its thing. Either way you end up with a
>>> broken application; it's just moving the breakage. In the case of
>>
>> Hm... In one case firewall installation reported success and stopped
>> working afterwards with no notification and user having no clue. In
>> another, packet analyzer refused to start and reported error to user.
>> Let's agree to disagree that those are not at all equivalent. To me
>> silent failure is so much worse, than application failing to start in
>> the first place.

I sort of agree with both of you that either case is not great. The silent
override we currently have is not great since it can be evicted at any time
but also bpf_link to lock-out other programs at XDP layer is not great either
since there is also huge potential to break existing programs. It's probably
best to discuss on an actual proposal to see the concrete semantics, but my
concerns, assuming I didn't misunderstand or got confused on something along
the way (if so, please let me know), currently are:

  - System service XYZ starts to use XDP with bpf_link one day. Somehow this
    application gets shipped by default on mainstream distros and starts up
    during init, then effectively locking out everyone else that used to use
    the hook today "just fine" given they owned / orchestrated the underlying
    networking on the host namespace for the nodes they manage (and for that
    it worked before). Now such latter app somehow needs to work around this
    breakage by undoing the damage that XYZ did in order to be able to operate
    again. There was mentioned 'human override'. I presume whatever it will be,
    it will also be done by applications when they don't have another choice.
    Otherwise we need to go and tell users that XDP is now only _entirely_
    reserved for system service XYZ if you run distro ABC, but not for everyone
    else anymore; what answer is there to this? From a PoV where one owns the
    entire distro and ecosystem, this is fine, but where this is not the case
    as in the rest of the world having to rely on mainstream distros, what is
    the answer to users (and especially "those that don't give a damn about XDP,
    but just want to get stuff to work" that used to work before, even if we
    think silent override is broken)? If the answer is to just 'shrug' and tell
    'sorry that's the new way it is right now', then apps will try to use whatever
    'human override' there is, and we're back to square one. To provide a
    concrete example: if Cilium was configured to load some of its programs on
    XDP hook, it currently replaces whatever it was there before. The assumption
    is, that in the scenario we're in, we can orchestrate the hostns networking
    just fine on K8s nodes since there is just one CNI plugin taking care of that
    (and that generally also holds true for the other hooks we're using today).
    Now, while we could switch to bpf_link as well and implement it in iproute2
    for this specific case, what if someone else starting up earlier and locks
    our stuff out? How would we work around it?

  - Assuming we have XDP with bpf_link in place with the above, now applications
    are forced to start using bpf_link in order to not be locked out by others
    using bpf_link as otherwise their application would break. So they need to
    support the "old" way of attaching programs as we have today for older
    kernels and need to support the bpf_link attachment for newer kernels since
    they cannot rely on the old / existing API anymore. There is also a world
    outside of C/C++ and thus libbpf / lib{xdp,dispatcher} or whatever, so the
    whole rest of the ecosystem is forced to implement it as well due to breakage
    concerns, understandable, but quite a burden.

  - Equally, in case of Toke's implementation for the cmpxchg-like mechanism in
    XDP itself, what happens if an application uses this API and assuming the
    library would return the error to the application using it if the expected
    program is not attached? Then the application would go for a forceful override
    with the existing API today or would it voluntarily bail out and refusing to
    work if some other non-cooperating application was loaded in the meantime?
    What is the cmpxchg-like mechanism then solving realistically? (And again,
    please keep all in mind we cannot force the entire world to use one single
    library to rule 'em all, there are plenty of other language runtimes out in
    the wild that cannot just import C/C++.)

Thoughts?

Thanks,
Daniel