Re: [PATCHv3 iproute2-next 0/5] iproute2: add libbpf support

[Jamal Hadi Salim <jhs@mojatatu.com>]

On 2020-11-05 4:01 p.m., Andrii Nakryiko wrote:
> On Thu, Nov 5, 2020 at 6:05 AM Jamal Hadi Salim <jhs@mojatatu.com> wrote:
>>
>> On 2020-11-04 10:19 p.m., David Ahern wrote:
>>
>> [..]

[..]

>> 2cents feedback from a dabbler in ebpf on user experience:
>>
>> What David described above *has held me back*.
>> Over time it seems things have gotten better with libbpf
>> (although a few times i find myself copying includes from the
>> latest iproute into libbpf). I ended up just doing static links.
>> The idea of upgrading clang/llvm every 2 months i revisit ebpf is
>> the most painful. At times code that used to compile just fine
>> earlier doesnt anymore. There's a minor issue of requiring i install
> 
> Do you have a specific example of something that stopped compiling?
> I'm not saying that can't happen, but we definitely try hard to avoid
> any regressions. I might be forgetting something, but I don't recall
> the situation when something would stop compiling just due to newer
> libbpf.
> 

Unfortunately the ecosystem is more than libbpf; sometimes it is
the kernel code that is being exercised by libbpf that is problematic.
This may sound unfair to libbpf but it is hard to separate the two for
someone who is dabbling like me.

The last issue iirc correctly had to do with one of the tcp notifier
variants either in samples or selftests(both user space and kernel).
I can go back and look at the details.
The fix always more than half the time was need to upgrade
clang/llvm. At one point i think it required that i had to grab
the latest and greatest git version. I think the machine i have
right now has version 11. The first time i found out about these
clang upgrades was trying to go from 8->9 or maybe it was 9->10.
Somewhere along there also was discovery that something that
compiled under earlier version wasnt compiling under newer version.

>> kernel headers every time i want to run something in samples, etc
>> but i am probably lacking knowledge on how to ease the pain in that
>> regard.
>>
>> I find the loader and associated tooling in iproute2/tc to be quiet
>> stable (not shiny but works everytime).
>> And for that reason i often find myself sticking to just tc instead
>> of toying with other areas.
> 
> That's the part that others on this thread mentioned is bit rotting?

Yes. Reason is i dont have to deal with new discoveries of things
that require some upgrade or copying etc.
I should be clear on the "it is the ecosystem": this is not just because
of user space code but also the simplicity of writing the tc kernel code
and loading it with tc tooling and then have a separate user tool for
control.
Lately i started linking the control tool with static libbpf instead.

Bpftool seems improved last time i tried to load something in XDP. I 
like the load-map-then-attach-program approach that bpftool gets
out of libbpf. I dont think that feature is possible with tc tooling.

However, I am still loading with tc and xdp with ip because of old
habits and what i consider to be a very simple workflow.

> Doesn't seem like everyone is happy about that, though. Stopping any
> development definitely makes things stable by definition. BPF and
> libbpf try to be stable while not stagnating, which is harder than
> just stopping any development, unfortunately.
> 

I am for moving to libbpf. I think it is a bad idea to have multiple
loaders for example. Note: I am not a demanding user, but there
are a few useful features that i feel i need that are missing in
iproute2 version. e.g, one thing i was playing with about a month
ago was some TOCTOU issue in the kernel code and getting
the bpf_lock integrated into the tc code proved challenging.
I ended rewriting the code to work around the tooling.

The challenge - when making changes in the name of progress - is to
not burden a user like myself with a complex workflow but still give
me the features i need.

>> Slight tangent:
>> One thing that would help libbpf adoption is to include an examples/
>> directory. Put a bunch of sample apps for tc, probes, xdp etc.
>> And have them compile outside of the kernel. Maybe useful Makefiles
>> that people can cutnpaste from. Every time you add a new feature
>> put some sample code in the examples.
> 
> That's what tools/testing/selftests/bpf in kernel source are for. It's
> not the greatest showcase of examples, but all the new features have a
> test demonstrating its usage. I do agree about having simple Makefiles
> and we do have that at [0]. I'm also about to do another sample repo
> with a lot of things pre-setup, for tinkering and using that as a
> bootstrap for BPF development with libbpf.
> 
>    [0] https://github.com/iovisor/bcc/tree/master/libbpf-tools


I pull that tree regularly.
selftests is good for aggregating things developers submit and
then have the robots test.
For better usability, it has to be something that is standalone that 
would work out of the box with libbf.
selftests and samples are not what i would consider for the
faint-hearted.
It may look easy to you because you eat this stuff for
breakfast but consider all those masses you want to be part of this.
They dont have the skills and people with average skills dont
have the patience.

This again comes back to "the ecosystem" - just getting libbpf to get
things stable for userland is not enough. Maybe have part of the libbpf
testing also to copy things from selftests.

cheers,
jamal