[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Jiri Kosina]

As everyone is pretty much aware, eBPF adoption is quickly expanding for 
various usecases in the kernel. For example, there has recently been 
effort invested into adding eBPF support to HID subsystem [1], in order to 
make adding quirks for specific devices easier, not requiring a "full" 
proper driver for devices that just need a tiny bit of special handling 
here and there but apart from that can be handled by the generic driver 
just fine.

While there are many proponents of "eBPF is good for everything and your 
grandma" aproach, this opinion is not universally shared. One big risk is 
that this will eventually lead to possibility of having whole drivers / 
core code written in eBPF, which could potentially lead to decreased 
maintainability and supportability, also due to big fragmentation of the 
code (eBPF programs might not necessarily be shipped together with the 
kernel codebase).

This could potentially be a big risk for distros as well, as we (as a 
distro vendor) might be very quickly losing control over what is actually 
running in the context of the kernel they are bound to be supporting.

In the specific case of drivers, there also is also some similarity in 
principle with UDI, which is a concept that hasn't proved viable quite 
some time ago already :)

I feel like we are currently lacking a clear borderline, defining what is 
still acceptable by the community to be implemented in terms of eBPF, and 
what is over the line as it'd be causing big supportability and 
maintainability concerns (see e.g. Christoph's concern to the HID eBPF 
implementation implications [2]).

So I'd like to propose a session where we'd ideally converge closer to 
defining a borderline between acceptable and non-acceptable usecases for 
eBPF in the kernel.

[1] https://lore.kernel.org/all/20220518205924.399291-1-benjamin.tissoires@redhat.com/
[2] https://lore.kernel.org/all/YoX7iHddAd4FkQRQ@infradead.org/

Thanks,

-- 
Jiri Kosina
SUSE Labs

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Linus Walleij]

On Wed, Jun 15, 2022 at 8:55 AM Jiri Kosina <jkosina@suse.cz> wrote:

> As everyone is pretty much aware, eBPF adoption is quickly expanding for
> various usecases in the kernel. For example, there has recently been
> effort invested into adding eBPF support to HID subsystem [1], in order to
> make adding quirks for specific devices easier, not requiring a "full"
> proper driver for devices that just need a tiny bit of special handling
> here and there but apart from that can be handled by the generic driver
> just fine.

I see your concern as subsystem maintainer not wanting HID to turn
into a dumping ground for various vendor extensions. I submitted
a Razer keyboard bling driver that was nixed with "put it in userspace",
i.e. using hidraw, and I was directed to the project called "razertest"
https://github.com/z3ntu/razer_test
https://patchwork.kernel.org/project/linux-input/patch/20181128222443.13062-1-linus.walleij@linaro.org/

One of the concerns raised was (Luca) "I wouldn't put too much functionality
into the kernel driver as new devices are being released all the time and it
would probably take very long for those patches to arrive at users of
non-rolling distros."

The problem with this is that while the kernel has a highway into *all*
distros and how many distros even carry "razer_test"? Not Fedora
for example. That surely has not reached users of any distro AFAICT.

So what we have here is three prominent HID contributors: Benjamin,
Luca and Jiri. One of you work for RedHat and one of you work for SuSE.
And yet none of your distributions have packaged Luca's userspace project?
Can't you see that this isn't working?

I feel HID is maybe missing the point that creating new userspace
projects is really hard and resource-craving, involving a lot of social
problem, process and lobbying compared to maintaining
a few more driver files in the kernel.

Referring people to misc half-adopted userspace projects for full support
of the hardware on their desk isn't really sustainable, no matter if
they use eBPF or not. (Asking them to go and create their own userspace
driver extension libraries is even less helpful.)

I once created libmtp for MTP communication with storage devices
such as all Android phones. In a way it is a success since it is
deployed in all distros and even used on Mac by Google for
accessing Android devices. Would I implement it today I would
seriously discuss with Greg to make this a kernel driver instead.
Simply because the kernel has more momentum, community
and attention. More bugs get fixed there, maintenance actually
happens. I'm really suspicious of other microkernel-like projects
such as libsane and cups as well, but they can talk for
themselves.

To me the big question is rather: does HID have a working userspace
project community outside of the kernel, which is adopted by all major
distributions? I know it is a chicken and egg problem, and that you
need to create something to get accepted but is there some
momentum in these projects?

At a minimum, do what Bartosz did for the libgpiod library that we
(reluctantly) developed for userspace GPIO access, and put the git
for the userspace on git.kernel.org so people know where to find it
and get the cuddly feeling that it has something to do with the official
Linux kernel.

> While there are many proponents of "eBPF is good for everything and your
> grandma" aproach, this opinion is not universally shared. One big risk is
> that this will eventually lead to possibility of having whole drivers /
> core code written in eBPF, which could potentially lead to decreased
> maintainability and supportability, also due to big fragmentation of the
> code (eBPF programs might not necessarily be shipped together with the
> kernel codebase).
(...)
> I feel like we are currently lacking a clear borderline, defining what is
> still acceptable by the community to be implemented in terms of eBPF, and
> what is over the line as it'd be causing big supportability and
> maintainability concerns (see e.g. Christoph's concern to the HID eBPF
> implementation implications [2]).

I must say from my own experience I side with Christoph here.

I just want to add some code to the kernel so my hardware works
(better) out of the box, is that really so voluminous for the HID maintainers
to maintain that it need to be referred to userspace?

I kind of feel like rebasing and resubmitting my razer driver from 2018
to raise a discussion here. Will I be asked to rewrite it in eBPF
if I do?

Yours,
Linus Walleij

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Jiri Kosina]

On Wed, 15 Jun 2022, Linus Walleij wrote:

> > As everyone is pretty much aware, eBPF adoption is quickly expanding for
> > various usecases in the kernel. For example, there has recently been
> > effort invested into adding eBPF support to HID subsystem [1], in order to
> > make adding quirks for specific devices easier, not requiring a "full"
> > proper driver for devices that just need a tiny bit of special handling
> > here and there but apart from that can be handled by the generic driver
> > just fine.
> 
> I see your concern as subsystem maintainer not wanting HID to turn into 
> a dumping ground for various vendor extensions. 

Just to clarify the point I was trying to make here -- I am not saying 
that I do not like this particular case (i.e. the HID-BFP patchset that 
Benjamin is developing).

My point was that unless we properly define what are the reasonable 
usecases for eBPF and what is the borderline beyond which we shouldn't go 
if we want to maintain sanity of the ecosystem (and people having to 
support the kernels :) ), we will be getting this discussion popping up 
over and over again.

> I submitted a Razer keyboard bling driver that was nixed with "put it in 
> userspace", i.e. using hidraw, and I was directed to the project called 
> "razertest" https://github.com/z3ntu/razer_test 
> https://patchwork.kernel.org/project/linux-input/patch/20181128222443.13062-1-linus.walleij@linaro.org/
> 
> One of the concerns raised was (Luca) "I wouldn't put too much functionality
> into the kernel driver as new devices are being released all the time and it
> would probably take very long for those patches to arrive at users of
> non-rolling distros."

I actually don't buy that as an argument. Distros are quite happily 
backporting to their kernels whatever the users / customers request from 
them, so I don't see why this would be an exception.

If there is a good use for it in a particular distro, it's up to that 
distro (or embedded vendor kernel) to have it ported.

> The problem with this is that while the kernel has a highway into *all*
> distros and how many distros even carry "razer_test"? Not Fedora
> for example. That surely has not reached users of any distro AFAICT.
> 
> So what we have here is three prominent HID contributors: Benjamin,
> Luca and Jiri. One of you work for RedHat and one of you work for SuSE.
> And yet none of your distributions have packaged Luca's userspace project?
> Can't you see that this isn't working?

I am probably much more inclined into taking the drivers into kernel than 
some of the others. I have always had issues with drivers in userspace (no 
matter whether it's libusb, UIO, hidraw, or whatever else there is), 
exactly because of the issues it brings when it comes to distributing it.

With perhaps an exception of things that can be made to work with an udev 
rule / script.

> To me the big question is rather: does HID have a working userspace 
> project community outside of the kernel, which is adopted by all major 
> distributions? I know it is a chicken and egg problem, and that you need 
> to create something to get accepted but is there some momentum in these 
> projects?

Even though I brought HID as an example on which I wanted to just 
demonstrate the point, I'd actually like to propose this discussion to 
stay completely abstracted away from any particular subsystem.

I didn't bring it up because of HID specifically, but as a general issue 
that spans across subsystems.

> > While there are many proponents of "eBPF is good for everything and your
> > grandma" aproach, this opinion is not universally shared. One big risk is
> > that this will eventually lead to possibility of having whole drivers /
> > core code written in eBPF, which could potentially lead to decreased
> > maintainability and supportability, also due to big fragmentation of the
> > code (eBPF programs might not necessarily be shipped together with the
> > kernel codebase).
> (...)
> > I feel like we are currently lacking a clear borderline, defining what is
> > still acceptable by the community to be implemented in terms of eBPF, and
> > what is over the line as it'd be causing big supportability and
> > maintainability concerns (see e.g. Christoph's concern to the HID eBPF
> > implementation implications [2]).
> 
> I must say from my own experience I side with Christoph here.
> 
> I just want to add some code to the kernel so my hardware works (better) 
> out of the box, is that really so voluminous for the HID maintainers to 
> maintain that it need to be referred to userspace?
> 
> I kind of feel like rebasing and resubmitting my razer driver from 2018 
> to raise a discussion here. 

I actually don't remember that submission from back 2018, sorry, but I 
personally wouldn't see a big problem accepting the driver as-is. But 
that's a discussion to have on a regular thread after you (re)submit it, 
but is probably not so relevant for the point I wanted to make here.

> Will I be asked to rewrite it in eBPF if I do?

Definitely not by me :)

Thanks,

-- 
Jiri Kosina
SUSE Labs

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Jan Kara]

On Wed 15-06-22 10:36:41, Jiri Kosina wrote:
> On Wed, 15 Jun 2022, Linus Walleij wrote:
> 
> > > As everyone is pretty much aware, eBPF adoption is quickly expanding for
> > > various usecases in the kernel. For example, there has recently been
> > > effort invested into adding eBPF support to HID subsystem [1], in order to
> > > make adding quirks for specific devices easier, not requiring a "full"
> > > proper driver for devices that just need a tiny bit of special handling
> > > here and there but apart from that can be handled by the generic driver
> > > just fine.
> > 
> > I see your concern as subsystem maintainer not wanting HID to turn into 
> > a dumping ground for various vendor extensions. 
> 
> Just to clarify the point I was trying to make here -- I am not saying 
> that I do not like this particular case (i.e. the HID-BFP patchset that 
> Benjamin is developing).
> 
> My point was that unless we properly define what are the reasonable 
> usecases for eBPF and what is the borderline beyond which we shouldn't go 
> if we want to maintain sanity of the ecosystem (and people having to 
> support the kernels :) ), we will be getting this discussion popping up 
> over and over again.

Yeah. Related to this is the fact that this way eBPF hooks become de-facto
API of the kernel with all the implications for required stability. It is
one thing to use eBPF for kernel introspection / monitoring (there
occasional breakage is kind of expected by userspace) and another to enable
hardware with it where people just expect things to work...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Jonathan Corbet]

Jan Kara <jack@suse.cz> writes:

> Yeah. Related to this is the fact that this way eBPF hooks become de-facto
> API of the kernel with all the implications for required stability. It is
> one thing to use eBPF for kernel introspection / monitoring (there
> occasional breakage is kind of expected by userspace) and another to enable
> hardware with it where people just expect things to work...

Relevant to this whole thing, I think, is the multigenerational LRU
discussion at LSFMM: https://lwn.net/Articles/894859/

One aspect that was only dropped in as a response to a question is that
the plan is to add a BPF hook to manage the movement of pages between
LRU generations; this will be done because nobody really knows how to
come up with an optimal solution to that problem for the general case.
That's about as core as it gets and, if users are tweaking memory
management with BPF, that will surely achieve ABI status in short order.

Beyond that, though, is the temptation to say "we don't have to figure
out this hard problem, we'll just add a BPF hook and let the users find
a solution for themselves".  The line between providing a useful means
for expert users to optimize their systems and skipping out on our own
duty to optimize things is not always clear, at least to me.

jon

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Jiri Kosina]

On Wed, 15 Jun 2022, Paul E. McKenney wrote:

> > Relevant to this whole thing, I think, is the multigenerational LRU
> > discussion at LSFMM: https://lwn.net/Articles/894859/
> > 
> > One aspect that was only dropped in as a response to a question is that
> > the plan is to add a BPF hook to manage the movement of pages between
> > LRU generations; this will be done because nobody really knows how to
> > come up with an optimal solution to that problem for the general case.
> > That's about as core as it gets and, if users are tweaking memory
> > management with BPF, that will surely achieve ABI status in short order.
> > 
> > Beyond that, though, is the temptation to say "we don't have to figure
> > out this hard problem, we'll just add a BPF hook and let the users find
> > a solution for themselves".  The line between providing a useful means
> > for expert users to optimize their systems and skipping out on our own
> > duty to optimize things is not always clear, at least to me.
> 
> I might as well ask the naive question:  Should subsystems document
> which hooks they intend to treat as ABI?  ;-)

Unfortunately, this "just select a subset" aproach has been proven not to 
work with tracepoints (which is exactly why some subsytems systematically 
refused to add tracepoints in the first place, because they explicitly did 
want to avoid being constrained by tracepoints having to be stable), which 
in this particular aspect is a similar problem.

-- 
Jiri Kosina
SUSE Labs

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Luis Chamberlain]

On Wed, Jun 15, 2022 at 08:55:47AM +0200, Jiri Kosina wrote:
> I feel like we are currently lacking a clear borderline, defining what is 
> still acceptable by the community to be implemented in terms of eBPF, and 
> what is over the line as it'd be causing big supportability and 
> maintainability concerns (see e.g. Christoph's concern to the HID eBPF 
> implementation implications [2]).
> 
> So I'd like to propose a session where we'd ideally converge closer to 
> defining a borderline between acceptable and non-acceptable usecases for 
> eBPF in the kernel.

Can't the equivalent of an LSM be used for these considerations?

  Luis

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Jiri Kosina]

On Wed, 15 Jun 2022, Benjamin Tissoires wrote:

> One point that was also raised in the various HID-BPF patch series is 
> that for "hardware enablement" like support, the eBPF programs would be 
> in-tree, and automatically loaded by the kernel itself.
> 
> Alexei has some ideas on how to implement that, I had others, but the 
> hallway track discussions showed that everybody has a different idea on 
> the automatic mechanism, but it is a requirement and worth discussing :)
> 
> Which means that in that case, eBPF would be a more convenient way for 
> users to fix their device, without having to rely on a full or partial 
> kernel recompilation.

That definitely does solve one of the issues. It's basically following the 
model of perf, where the ABI must not be kept intact, because the user(s) 
of it are in-tree and released in lockstep with the ABI changes.

> While working on eBPF, I came to realize how hidraw is bad in terms of 
> development model for userspace "drivers". 

As the original author of hidraw, I have to agree :) I didn't expect such 
an explosion of userspace drivers being dependent on it, it was primarily 
meant for debugging and tweaking.

libusb is a similar example, I believe.

-- 
Jiri Kosina
SUSE Labs

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Linus Walleij]

On Thu, Jun 16, 2022 at 10:02 AM Jiri Kosina <jikos@kernel.org> wrote:
> On Wed, 15 Jun 2022, Benjamin Tissoires wrote:
>
> > One point that was also raised in the various HID-BPF patch series is
> > that for "hardware enablement" like support, the eBPF programs would be
> > in-tree, and automatically loaded by the kernel itself.
> >
> > Alexei has some ideas on how to implement that, I had others, but the
> > hallway track discussions showed that everybody has a different idea on
> > the automatic mechanism, but it is a requirement and worth discussing :)
> >
> > Which means that in that case, eBPF would be a more convenient way for
> > users to fix their device, without having to rely on a full or partial
> > kernel recompilation.

Convenient for some definition of convenient. I might be biased in asking
how much harder it is to set up a kernel compile, rebuild a module
and run a few insmod/rmmod to find a quirk compared to setting up
the eBPF compiler and figure out how to compile and push in such
programs.

I guess I could be convinced.

> That definitely does solve one of the issues. It's basically following the
> model of perf, where the ABI must not be kept intact, because the user(s)
> of it are in-tree and released in lockstep with the ABI changes.

I agree, I would actually go so far as to taint the kernel if programs that are
not in-tree are used. That is fine for the goal here: users can create new
eBPF snippets and debug them, but they can't ship them because then
the kernel gets tainted, so they MUST be submitted upstream.

Do you think we could do this? Pushing taint in the face of people who
don't follow our established contribution process is essentially the big
hammer we have to stop fragmentation.

Yours,
Linus Walleij

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Benjamin Tissoires]

On Thu, Jun 16, 2022 at 1:38 PM Linus Walleij <linus.walleij@linaro.org> wrote:
>
> On Thu, Jun 16, 2022 at 10:02 AM Jiri Kosina <jikos@kernel.org> wrote:
> > On Wed, 15 Jun 2022, Benjamin Tissoires wrote:
> >
> > > One point that was also raised in the various HID-BPF patch series is
> > > that for "hardware enablement" like support, the eBPF programs would be
> > > in-tree, and automatically loaded by the kernel itself.
> > >
> > > Alexei has some ideas on how to implement that, I had others, but the
> > > hallway track discussions showed that everybody has a different idea on
> > > the automatic mechanism, but it is a requirement and worth discussing :)
> > >
> > > Which means that in that case, eBPF would be a more convenient way for
> > > users to fix their device, without having to rely on a full or partial
> > > kernel recompilation.
>
> Convenient for some definition of convenient. I might be biased in asking
> how much harder it is to set up a kernel compile, rebuild a module
> and run a few insmod/rmmod to find a quirk compared to setting up
> the eBPF compiler and figure out how to compile and push in such
> programs.
>
> I guess I could be convinced.

The plan is to forward to the end user the sources, of course, but
also the binary blob. The binary can be checked that it is doing the
correct thing because it's not so much of a black box and we can
easily check that it is the one built from the source. For extra check
the maintainer can also sign the binary.

And for the end user, it is thus: 1. trust your kernel maintainer (see
above if you don't), and 2. drop the binary in the file system and
forget. So no compilation involved :)

>
> > That definitely does solve one of the issues. It's basically following the
> > model of perf, where the ABI must not be kept intact, because the user(s)
> > of it are in-tree and released in lockstep with the ABI changes.
>
> I agree, I would actually go so far as to taint the kernel if programs that are
> not in-tree are used. That is fine for the goal here: users can create new
> eBPF snippets and debug them, but they can't ship them because then
> the kernel gets tainted, so they MUST be submitted upstream.
>
> Do you think we could do this? Pushing taint in the face of people who
> don't follow our established contribution process is essentially the big
> hammer we have to stop fragmentation.

I would like to have that too. The in-kernel eBPF binaries are using a
different path to be loaded so I would assume this is OK to
differentiate, but I'll leave it to Alexei to give a more definitive
answer.

Cheers,
Benjamin

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Benjamin Tissoires]

On Thu, Jun 16, 2022 at 1:49 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Wed, Jun 15, 2022 at 08:34:19PM +0200, Benjamin Tissoires wrote:
> > Also, one of the things I'd like to have in kfuncs is to be able to
> > restrict them to a set of tracepoints, not just "all tracing
> > programs".
>
> Yes.
>
> > Because the thing I do not want is users hijacking the kfunc API I
> > define in other use cases I haven't thought of, and this would prevent
> > changes.
>
> Yes.  And I also think the BTF_IDs for kfuncs need to become something
> like EXPORT_SYMBOL_EPBF to be more explicit and greppable and need to
> see the same process.  That is in-tree users and no guarantee of stability
> for out of tree users.

Sounds interesting :)

>
> > And AFAICT, I consider everything eBPF I do in HID as ABI, and treat
> > it as such, and document it from day one.
>
> Yikes.  So you're adding a UDI-like stable driver ABI?
>

No, I mean I am carefully defining the eBPF api I am willing to export
to user space, and will restrain myself from shuffling arguments every
single release.

The last change in the HID core protocol was 20 years old, so we
roughly know what it does. It is not so much of a task to define what
will be unlikely to change.

Maybe a better way of putting it: "I consider *almost* everything eBPF
I do in HID as *UAPI*, and treat it as such, and document it from day
one."

Cheers,
Benjamin

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Steven Rostedt]

On Wed, 15 Jun 2022 20:25:41 +0200 (CEST)
Jiri Kosina <jikos@kernel.org> wrote:

> > I might as well ask the naive question:  Should subsystems document
> > which hooks they intend to treat as ABI?  ;-)  

I would like there to be a decree that eBPF is denoted as the same as a
fancy module. And be treated the same as modules. That is, there is NO ABI!

A eBPF program that works on one kernel should have no guarantee that it
will work on another version of the kernel. Because eBPF is basically just
that, a module. It is compiled into native code that runs in kernel space.
Exactly like a module, with the caveat that it must first go through a
verifier.

> 
> Unfortunately, this "just select a subset" aproach has been proven not to 
> work with tracepoints (which is exactly why some subsytems systematically 
> refused to add tracepoints in the first place, because they explicitly did 
> want to avoid being constrained by tracepoints having to be stable), which 
> in this particular aspect is a similar problem.

The difference between eBPF and tracepoints (actually only trace events),
is that trace events are exported visibly to user space and attached via
the perf system call. The trace event's format is shown in the tracefs
events directory. Just like any file in /proc, the trace events can easily
be read by a privileged user space application.

Now tracepoints are not exported to user space.

  The difference between a tracepoint and trace event is that a tracepoint
  is the "trace_foo()" in the kernel, where as the trace event is the data
  extracted from the tracepoint via the TRACE_EVENT() macro and listed in
  the format files in the tracefs events directory.

The tracepoint interface is just like any other C function in the kernel,
and should never be considered an ABI.

I wanted to bring this up at MS as well, so I'd like to extend this topic,
and say eBPF programs *are* modules.

We had an issue [1] that we added a parameter to the sched_switch
tracepooint (not trace event), and that broke some eBPF programs. Since all
they wanted was for use to reorder the parameters of the tracepoint call,
we obliged. But we made it a point that this must not set a precedent.
The mere fact that we had to do this has brought up major concerns that
eBPF is starting to become too invasive and may limit the ability of kernel
development.

[1] https://lore.kernel.org/all/c8a6930dfdd58a4a5755fc01732675472979732b.camel@fb.com/T/#mc2ec6eded478552fc01d10e32dc4a892f95a9900

-- Steve

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[James Bottomley]

On Thu, 2022-06-16 at 12:26 -0400, Steven Rostedt wrote:
> On Wed, 15 Jun 2022 20:25:41 +0200 (CEST)
> Jiri Kosina <jikos@kernel.org> wrote:
> 
> > > I might as well ask the naive question:  Should subsystems
> > > document
> > > which hooks they intend to treat as ABI?  ;-)  
> 
> I would like there to be a decree that eBPF is denoted as the same as
> a fancy module. And be treated the same as modules. That is, there is
> NO ABI!

Well, you can wish for a Pony, but there's no guarantee you'll get one
...

> 
> A eBPF program that works on one kernel should have no guarantee that
> it will work on another version of the kernel. Because eBPF is
> basically just that, a module. It is compiled into native code that
> runs in kernel space. Exactly like a module, with the caveat that it
> must first go through a verifier.

Based on the encouragement we gave as kernel developers, certain
tracing as a service companies that previously had propritary modules
(Sysdig for instance) are now moving over to using tracing with eBPF. 
At the time we thought this was good for the kernel; if we now try to
tell them "actually you can't use the interface because it's completely
unstable" that's going to undermine our whole argument to them for
dumping proprietary modules.  So I don't think no eBPF at all is stable
is a tenable position for us.

Equally well, I don't think it's as hard as a userspace ABI meaning it
can never change.  I think we can get away with changes that force
tracing and other value added service providers to change their eBPF, I
just don't think we can do it very often without damaging the value of
eBPF over proprietary modules.

> > Unfortunately, this "just select a subset" aproach has been proven
> > not to work with tracepoints (which is exactly why some subsytems
> > systematically refused to add tracepoints in the first place,
> > because they explicitly did want to avoid being constrained by
> > tracepoints having to be stable), which in this particular aspect
> > is a similar problem.

As I said above, I think we have to provide *some* stability, but for a
sophisticated consumer it doesn't have to be the absolute stability
guarantee of an ABI.  I think we should debate what kinds of slightly
unstable stability we could provide

James

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Steven Rostedt]

On Thu, 16 Jun 2022 12:38:24 -0400
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> > 
> > A eBPF program that works on one kernel should have no guarantee that
> > it will work on another version of the kernel. Because eBPF is
> > basically just that, a module. It is compiled into native code that
> > runs in kernel space. Exactly like a module, with the caveat that it
> > must first go through a verifier.  
> 
> Based on the encouragement we gave as kernel developers, certain
> tracing as a service companies that previously had propritary modules
> (Sysdig for instance) are now moving over to using tracing with eBPF. 

I wasn't in this conversation, so I'm unaware of the advantages of it.

> At the time we thought this was good for the kernel; if we now try to
> tell them "actually you can't use the interface because it's completely
> unstable" that's going to undermine our whole argument to them for
> dumping proprietary modules.  So I don't think no eBPF at all is stable
> is a tenable position for us.
> 
> Equally well, I don't think it's as hard as a userspace ABI meaning it
> can never change.  I think we can get away with changes that force
> tracing and other value added service providers to change their eBPF, I
> just don't think we can do it very often without damaging the value of
> eBPF over proprietary modules.

Perhaps we should add a place in tools/bpf for "upstream bpf programs". Or
is there already something there? Again, I would like to treat bpf no
different than modules. Sorry, I don't see a "convince proprietary
module folks to create proprietary bpf programs" as progress. If anything,
it's going backwards.

If you want a "stable ebpf program" then you submit it upstream and we can
make sure that it works with any internal API changes, the same way we do
for modules. Those with out-of-tree modules will have the technical debt of
changing every time a new kernel release is out, and so should out-of-tree
bpf programs.

> 
> > > Unfortunately, this "just select a subset" aproach has been proven
> > > not to work with tracepoints (which is exactly why some subsytems
> > > systematically refused to add tracepoints in the first place,
> > > because they explicitly did want to avoid being constrained by
> > > tracepoints having to be stable), which in this particular aspect
> > > is a similar problem.  
> 
> As I said above, I think we have to provide *some* stability, but for a
> sophisticated consumer it doesn't have to be the absolute stability
> guarantee of an ABI.  I think we should debate what kinds of slightly
> unstable stability we could provide

What you define as "somewhat stable" others may define as "not stable at
all". There is no happy medium (as you said, "you can wish for a Pony, but
there's no guarantee you'll get one").

Once you say you will have some stability, that will become an albatross
around your neck forever.

-- Steve

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[James Bottomley]

On Thu, 2022-06-16 at 12:51 -0400, Steven Rostedt wrote:
> On Thu, 16 Jun 2022 12:38:24 -0400
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> > > A eBPF program that works on one kernel should have no guarantee
> > > that it will work on another version of the kernel. Because eBPF
> > > is basically just that, a module. It is compiled into native code
> > > that runs in kernel space. Exactly like a module, with the caveat
> > > that it must first go through a verifier.  
> > 
> > Based on the encouragement we gave as kernel developers, certain
> > tracing as a service companies that previously had propritary
> > modules (Sysdig for instance) are now moving over to using tracing
> > with eBPF. 
> 
> I wasn't in this conversation, so I'm unaware of the advantages of
> it.

Unaware of the advantages of using eBPF over proprietary modules? 
We've had a long standing policy of discouraging proprietary modules. 
We use the carrot and stick methods.  The stick is definitely the API
is unstable and changes often and will break you (and even some kernel
developers will go out of their way to change the API your using).  The
carrot is showing them there are better and easier ways of doing the
things they're trying to achieve without using proprietary modules.

> > At the time we thought this was good for the kernel; if we now try
> > to tell them "actually you can't use the interface because it's
> > completely unstable" that's going to undermine our whole argument
> > to them for dumping proprietary modules.  So I don't think no eBPF
> > at all is stable is a tenable position for us.
> > 
> > Equally well, I don't think it's as hard as a userspace ABI meaning
> > it can never change.  I think we can get away with changes that
> > force tracing and other value added service providers to change
> > their eBPF, I just don't think we can do it very often without
> > damaging the value of eBPF over proprietary modules.
> 
> Perhaps we should add a place in tools/bpf for "upstream bpf
> programs". Or is there already something there? Again, I would like
> to treat bpf no different than modules. Sorry, I don't see a
> "convince proprietary module folks to create proprietary bpf
> programs" as progress. If anything, it's going backwards.

That's a false dichotomy: it's not true the only way of forcing
openness is to pull something into the kernel tree.  I gave the
proprietary module as one example, but not everything that uses eBPF is
proprietary ... although it is also true that not everything that's
open is under a licence compatible with the kernel.

> If you want a "stable ebpf program" then you submit it upstream and
> we can make sure that it works with any internal API changes, the
> same way we do for modules. Those with out-of-tree modules will have
> the technical debt of changing every time a new kernel release is
> out, and so should out-of-tree bpf programs.

Assuming eBPF takes off, that would have some poor maintainer managing
the whole of the compatibility changes for the entire eBPF ecosystem
... I really don't think that's scalable.

> > > > Unfortunately, this "just select a subset" aproach has been
> > > > proven not to work with tracepoints (which is exactly why some
> > > > subsytems systematically refused to add tracepoints in the
> > > > first place, because they explicitly did want to avoid being
> > > > constrained by tracepoints having to be stable), which in this
> > > > particular aspect is a similar problem.  
> > 
> > As I said above, I think we have to provide *some* stability, but
> > for a sophisticated consumer it doesn't have to be the absolute
> > stability guarantee of an ABI.  I think we should debate what kinds
> > of slightly unstable stability we could provide
> 
> What you define as "somewhat stable" others may define as "not stable
> at all". There is no happy medium (as you said, "you can wish for a
> Pony, but there's no guarantee you'll get one").
> 
> Once you say you will have some stability, that will become an
> albatross around your neck forever.

Not if it's managed correctly.  One method could simply be to not care
but be careful.  After all, lots of companies still manage to produce
proprietary modules even with us actively trying to break the API. 
Either it's not as difficult as we think or API changes are easier to
cope with than we assume.

James

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Steven Rostedt]

On Thu, 16 Jun 2022 14:25:44 -0400
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> Not if it's managed correctly.  One method could simply be to not care
> but be careful.  After all, lots of companies still manage to produce
> proprietary modules even with us actively trying to break the API. 
> Either it's not as difficult as we think or API changes are easier to
> cope with than we assume.

All I read from this is that eBPF is the new way that proprietary modules
can have their stable API.

The problem I have here is that eBPF is becoming extremely invasive. It is
now hooking to any function in the kernel that isn't marked notrace. If
there's not a clear border on what is considered non-stable now, we will be
fighting with those that say "hey this broke my bpf program, revert it
please" for a long time coming.

-- Steve

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Jiri Kosina]

On Thu, 16 Jun 2022, James Bottomley wrote:

> > If you want a "stable ebpf program" then you submit it upstream and
> > we can make sure that it works with any internal API changes, the
> > same way we do for modules. Those with out-of-tree modules will have
> > the technical debt of changing every time a new kernel release is
> > out, and so should out-of-tree bpf programs.
> 
> Assuming eBPF takes off, that would have some poor maintainer managing
> the whole of the compatibility changes for the entire eBPF ecosystem
> ... I really don't think that's scalable.

I nevertheless still see this as the best and only option we have; that 
is, have an infrastructure in the kernel tree for maintaining eBPF 
programs, somehow sorted per subsystem so that it mirrors the standard 
maintainership / subsystem structure proper, and have the maintainers 
responsible for keeping the eBPF programs related to their subsystem in 
sync with the internal changes happening in the subsystem.

At the end of the day, it will be the subsystem maintainers themsleves 
accepting the program into the tree in the first place, so it's not like 
they are receiving responsibility for something they never wanted in the 
first place. So we'll probably end up with subsystems with many eBPF 
programs, and also subsystems with zero. Similarly to tracepoints.

I.e. pretty much the 'perf' model, but on much wider scale (as eBPF can 
hook to just about anything).

Any other option seems to lead to having eBPF programs sprinkled all over 
the internet that depend on particular kernel version / API, leading to 
nothing else than unhappy users, because "I downloaded it, it didn't work, 
Linux sucks".

-- 
Jiri Kosina
SUSE Labs

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Linus Walleij]

On Fri, Jun 17, 2022 at 9:54 AM Jiri Kosina <jikos@kernel.org> wrote:
> On Thu, 16 Jun 2022, James Bottomley wrote:
>
> > > If you want a "stable ebpf program" then you submit it upstream and
> > > we can make sure that it works with any internal API changes, the
> > > same way we do for modules. Those with out-of-tree modules will have
> > > the technical debt of changing every time a new kernel release is
> > > out, and so should out-of-tree bpf programs.
> >
> > Assuming eBPF takes off, that would have some poor maintainer managing
> > the whole of the compatibility changes for the entire eBPF ecosystem
> > ... I really don't think that's scalable.
>
> I nevertheless still see this as the best and only option we have; that
> is, have an infrastructure in the kernel tree for maintaining eBPF
> programs, somehow sorted per subsystem so that it mirrors the standard
> maintainership / subsystem structure proper, and have the maintainers
> responsible for keeping the eBPF programs related to their subsystem in
> sync with the internal changes happening in the subsystem.

It's similar to what we have for Device Tree bindings already,
and those are in many cases handled by the relevant subsystem
maintainer, both review and merge.

> At the end of the day, it will be the subsystem maintainers themsleves
> accepting the program into the tree in the first place, so it's not like
> they are receiving responsibility for something they never wanted in the
> first place. So we'll probably end up with subsystems with many eBPF
> programs, and also subsystems with zero. Similarly to tracepoints.

The concern is that as subsystem maintainers had to learn YAML
to deal with device tree bindings, they now have to learn eBPF
syntax, and there is also Rust in the kernel around the corner to
learn.

Plenty of languages to learn. Oh well, they keep us on the move.

Yours,
Linus Walleij

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Jan Kara]

On Fri 17-06-22 09:53:52, Jiri Kosina wrote:
> On Thu, 16 Jun 2022, James Bottomley wrote:
> 
> > > If you want a "stable ebpf program" then you submit it upstream and
> > > we can make sure that it works with any internal API changes, the
> > > same way we do for modules. Those with out-of-tree modules will have
> > > the technical debt of changing every time a new kernel release is
> > > out, and so should out-of-tree bpf programs.
> > 
> > Assuming eBPF takes off, that would have some poor maintainer managing
> > the whole of the compatibility changes for the entire eBPF ecosystem
> > ... I really don't think that's scalable.
> 
> I nevertheless still see this as the best and only option we have; that 
> is, have an infrastructure in the kernel tree for maintaining eBPF 
> programs, somehow sorted per subsystem so that it mirrors the standard 
> maintainership / subsystem structure proper, and have the maintainers 
> responsible for keeping the eBPF programs related to their subsystem in 
> sync with the internal changes happening in the subsystem.
> 
> At the end of the day, it will be the subsystem maintainers themsleves 
> accepting the program into the tree in the first place, so it's not like 
> they are receiving responsibility for something they never wanted in the 
> first place. So we'll probably end up with subsystems with many eBPF 
> programs, and also subsystems with zero. Similarly to tracepoints.
> 
> I.e. pretty much the 'perf' model, but on much wider scale (as eBPF can 
> hook to just about anything).
> 
> Any other option seems to lead to having eBPF programs sprinkled all over 
> the internet that depend on particular kernel version / API, leading to 
> nothing else than unhappy users, because "I downloaded it, it didn't work, 
> Linux sucks".

OK, but if we keep eBPF programs this closely coupled to the kernel, then
what is the advantage of using eBPF, say for HID as was discussed earlier
in this thread, compared to just making sure HID has appropriate hooks and
the handling of the device quirks is done in normal C code (kernel module)
attached to these hooks?

Because frankly for me the main value of eBPF over patching and recompiling
the kernel is that I can tweak the eBPF code exactly to my needs and load
it to the kernel without needing to reboot, over time it is less work than
maintaining a source code patch out of tree, and usually it is a hacky
tweak I use for some debugging so merging it upstream does not make sense.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Benjamin Tissoires]

On Fri, Jun 17, 2022 at 12:31 PM Jan Kara <jack@suse.cz> wrote:
>
> On Fri 17-06-22 09:53:52, Jiri Kosina wrote:
> > On Thu, 16 Jun 2022, James Bottomley wrote:
> >
> > > > If you want a "stable ebpf program" then you submit it upstream and
> > > > we can make sure that it works with any internal API changes, the
> > > > same way we do for modules. Those with out-of-tree modules will have
> > > > the technical debt of changing every time a new kernel release is
> > > > out, and so should out-of-tree bpf programs.
> > >
> > > Assuming eBPF takes off, that would have some poor maintainer managing
> > > the whole of the compatibility changes for the entire eBPF ecosystem
> > > ... I really don't think that's scalable.
> >
> > I nevertheless still see this as the best and only option we have; that
> > is, have an infrastructure in the kernel tree for maintaining eBPF
> > programs, somehow sorted per subsystem so that it mirrors the standard
> > maintainership / subsystem structure proper, and have the maintainers
> > responsible for keeping the eBPF programs related to their subsystem in
> > sync with the internal changes happening in the subsystem.
> >
> > At the end of the day, it will be the subsystem maintainers themsleves
> > accepting the program into the tree in the first place, so it's not like
> > they are receiving responsibility for something they never wanted in the
> > first place. So we'll probably end up with subsystems with many eBPF
> > programs, and also subsystems with zero. Similarly to tracepoints.
> >
> > I.e. pretty much the 'perf' model, but on much wider scale (as eBPF can
> > hook to just about anything).
> >
> > Any other option seems to lead to having eBPF programs sprinkled all over
> > the internet that depend on particular kernel version / API, leading to
> > nothing else than unhappy users, because "I downloaded it, it didn't work,
> > Linux sucks".
>
> OK, but if we keep eBPF programs this closely coupled to the kernel, then
> what is the advantage of using eBPF, say for HID as was discussed earlier
> in this thread, compared to just making sure HID has appropriate hooks and
> the handling of the device quirks is done in normal C code (kernel module)
> attached to these hooks?

[sorry some of my messages are kept in the moderation queue]

This is something I tried to explain in my talk at Kernel Recipes 2
weeks ago [1].

Basically, for regular users, it will be simpler for them to test patches:
A developer patches the device through a bpf program, compiles it and
sends to the user the source and the bytecode. The user just has to
drop the bytecode in the file system and a udev rule automatically
loads it without having to reboot the current kernel, making testing
way faster and simpler for users.

Then developers take the burden of upstreaming the bpf program through
the usual email submission.

This way users are testing the actual code that is upstreamed without
too much hurdle.

IMO those kinds of fixes should be in-tree because they are actual fixes.


>
> Because frankly for me the main value of eBPF over patching and recompiling
> the kernel is that I can tweak the eBPF code exactly to my needs and load
> it to the kernel without needing to reboot, over time it is less work than
> maintaining a source code patch out of tree, and usually it is a hacky
> tweak I use for some debugging so merging it upstream does not make sense.
>

And that's also why I want to give *some* guarantees that the eBPF
hooks I am putting in HID are somewhat stable. Or at least I won't
break everything at each release and look carefully when there is a
change in the API.

But these guarantees are just *my* constraints I put on *my*
subsystem. I have reasons to believe I can ensure that, so I will.

But I am not saying any internal HID function will have ABI
guarantees. Only the few hooks I am explicitly documenting as such.

So the idea is that your debugging program can live in your own tree,
without being submitted, but this is just a contract between myself as
a subsystem maintainer and you, not the entire eBPF or ftrace
community.

I made this so I can put any fancy new kernel API out in eBPF
programs, that would be handled by the actual user of that kernel API.
If a user wants to set some fancy feature on a particular device,
instead of having a kernel parameter that no one will ever use besides
that program that may be short-lived, we can then load that
functionality with the eBPF program.
Or if a user is only interested in a particular event in a report for
a device (with the custom protocols some gaming devices are enjoying),
then that user can load a BPF program and keep it outside of the tree
because the program doesn't make sense without the userspace
component.

I hope it clarifies a little bit now. For HID at least.

Cheers,
Benjamin

[1] https://kernel-recipes.org/en/2022/talks/hid-bpf/ and
https://www.youtube.com/watch?v=nhJqaZT94z0&t=14923s

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Benjamin Tissoires]

On Fri, Jun 17, 2022 at 1:16 PM Hans de Goede <hdegoede@redhat.com> wrote:
>
> Hi Benjamin,
>
> Deliberate offlist reply.

[re-adding the list as your last comment]

>
> On 6/17/22 13:04, Benjamin Tissoires wrote:
> > On Fri, Jun 17, 2022 at 12:31 PM Jan Kara <jack@suse.cz> wrote:
> >>
> >> On Fri 17-06-22 09:53:52, Jiri Kosina wrote:
> >>> On Thu, 16 Jun 2022, James Bottomley wrote:
> >>>
> >>>>> If you want a "stable ebpf program" then you submit it upstream and
> >>>>> we can make sure that it works with any internal API changes, the
> >>>>> same way we do for modules. Those with out-of-tree modules will have
> >>>>> the technical debt of changing every time a new kernel release is
> >>>>> out, and so should out-of-tree bpf programs.
> >>>>
> >>>> Assuming eBPF takes off, that would have some poor maintainer managing
> >>>> the whole of the compatibility changes for the entire eBPF ecosystem
> >>>> ... I really don't think that's scalable.
> >>>
> >>> I nevertheless still see this as the best and only option we have; that
> >>> is, have an infrastructure in the kernel tree for maintaining eBPF
> >>> programs, somehow sorted per subsystem so that it mirrors the standard
> >>> maintainership / subsystem structure proper, and have the maintainers
> >>> responsible for keeping the eBPF programs related to their subsystem in
> >>> sync with the internal changes happening in the subsystem.
> >>>
> >>> At the end of the day, it will be the subsystem maintainers themsleves
> >>> accepting the program into the tree in the first place, so it's not like
> >>> they are receiving responsibility for something they never wanted in the
> >>> first place. So we'll probably end up with subsystems with many eBPF
> >>> programs, and also subsystems with zero. Similarly to tracepoints.
> >>>
> >>> I.e. pretty much the 'perf' model, but on much wider scale (as eBPF can
> >>> hook to just about anything).
> >>>
> >>> Any other option seems to lead to having eBPF programs sprinkled all over
> >>> the internet that depend on particular kernel version / API, leading to
> >>> nothing else than unhappy users, because "I downloaded it, it didn't work,
> >>> Linux sucks".
> >>
> >> OK, but if we keep eBPF programs this closely coupled to the kernel, then
> >> what is the advantage of using eBPF, say for HID as was discussed earlier
> >> in this thread, compared to just making sure HID has appropriate hooks and
> >> the handling of the device quirks is done in normal C code (kernel module)
> >> attached to these hooks?
> >
> > [sorry some of my messages are kept in the moderation queue]
> >
> > This is something I tried to explain in my talk at Kernel Recipes 2
> > weeks ago [1].
> >
> > Basically, for regular users, it will be simpler for them to test patches:
> > A developer patches the device through a bpf program, compiles it and
> > sends to the user the source and the bytecode. The user just has to
> > drop the bytecode in the file system and a udev rule automatically
> > loads it without having to reboot the current kernel, making testing
> > way faster and simpler for users.
> >
> > Then developers take the burden of upstreaming the bpf program through
> > the usual email submission.
> >
> > This way users are testing the actual code that is upstreamed without
> > too much hurdle.
> >
> > IMO those kinds of fixes should be in-tree because they are actual fixes.
> >
> >
> >>
> >> Because frankly for me the main value of eBPF over patching and recompiling
> >> the kernel is that I can tweak the eBPF code exactly to my needs and load
> >> it to the kernel without needing to reboot, over time it is less work than
> >> maintaining a source code patch out of tree, and usually it is a hacky
> >> tweak I use for some debugging so merging it upstream does not make sense.
> >>
> >
> > And that's also why I want to give *some* guarantees that the eBPF
> > hooks I am putting in HID are somewhat stable. Or at least I won't
> > break everything at each release and look carefully when there is a
> > change in the API.
> >
> > But these guarantees are just *my* constraints I put on *my*
> > subsystem. I have reasons to believe I can ensure that, so I will.
> >
> > But I am not saying any internal HID function will have ABI
> > guarantees. Only the few hooks I am explicitly documenting as such.
> >
> > So the idea is that your debugging program can live in your own tree,
> > without being submitted, but this is just a contract between myself as
> > a subsystem maintainer and you, not the entire eBPF or ftrace
> > community.
> >
> > I made this so I can put any fancy new kernel API out in eBPF
> > programs, that would be handled by the actual user of that kernel API.
>
> I think you need to clarify this, to me it now reads that you want
> to use eBPF to implement new APIs, where as what you mean is that
> you want to avoid the need to add new APIs like e.g. sysfs knobs
> or kernel-module-parameters, right ?

Yes. Exactly. It's not about regular and standard API. It's a device
specific property/feature that is usually handled through module
parameters or custom sysfs entries that have only one single user.

>
> > If a user wants to set some fancy feature on a particular device,
> > instead of having a kernel parameter that no one will ever use besides
> > that program that may be short-lived, we can then load that
> > functionality with the eBPF program.
>
> I think you need to clarify here that you mean changing some settings
> on the device through e.g. a HID feature report which would require
> a sysfs-attribute or kernel-module-parameters without ePBF and you want
> to avoid adding more and more sysfs-attributes / kernel-module-parameters?

Yep yep :)

>
> At least I think this is what you are trying to say, and I think that
> without some clarification this is not what most kernel-devs will
> understand here.

Thanks for saying it better than I do :)

>
> Feel free to copy/paste from this reply to use my attemps at clarifying
> things, or just reply to this with the list added back to the Cc :)

And done through the lazy option, just forward to the list :)

Cheers,
Benjamin

>
> Regards,
>
> Hans
>
>
>
> > Or if a user is only interested in a particular event in a report for
> > a device (with the custom protocols some gaming devices are enjoying),
> > then that user can load a BPF program and keep it outside of the tree
> > because the program doesn't make sense without the userspace
> > component.
> >
> > I hope it clarifies a little bit now. For HID at least.
> >
> > Cheers,
> > Benjamin
> >
> > [1] https://kernel-recipes.org/en/2022/talks/hid-bpf/ and
> > https://www.youtube.com/watch?v=nhJqaZT94z0&t=14923s
> >
> > _______________________________________________
> > Ksummit-discuss mailing list
> > Ksummit-discuss@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
> >
>

[Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Hans de Goede]

Hi,

On 6/17/22 13:25, Benjamin Tissoires wrote:
> On Fri, Jun 17, 2022 at 1:16 PM Hans de Goede <hdegoede@redhat.com> wrote:
>>
>> Hi Benjamin,
>>
>> Deliberate offlist reply.
> 
> [re-adding the list as your last comment]
> 
>>
>> On 6/17/22 13:04, Benjamin Tissoires wrote:
>>> On Fri, Jun 17, 2022 at 12:31 PM Jan Kara <jack@suse.cz> wrote:
>>>>
>>>> On Fri 17-06-22 09:53:52, Jiri Kosina wrote:
>>>>> On Thu, 16 Jun 2022, James Bottomley wrote:
>>>>>
>>>>>>> If you want a "stable ebpf program" then you submit it upstream and
>>>>>>> we can make sure that it works with any internal API changes, the
>>>>>>> same way we do for modules. Those with out-of-tree modules will have
>>>>>>> the technical debt of changing every time a new kernel release is
>>>>>>> out, and so should out-of-tree bpf programs.
>>>>>>
>>>>>> Assuming eBPF takes off, that would have some poor maintainer managing
>>>>>> the whole of the compatibility changes for the entire eBPF ecosystem
>>>>>> ... I really don't think that's scalable.
>>>>>
>>>>> I nevertheless still see this as the best and only option we have; that
>>>>> is, have an infrastructure in the kernel tree for maintaining eBPF
>>>>> programs, somehow sorted per subsystem so that it mirrors the standard
>>>>> maintainership / subsystem structure proper, and have the maintainers
>>>>> responsible for keeping the eBPF programs related to their subsystem in
>>>>> sync with the internal changes happening in the subsystem.
>>>>>
>>>>> At the end of the day, it will be the subsystem maintainers themsleves
>>>>> accepting the program into the tree in the first place, so it's not like
>>>>> they are receiving responsibility for something they never wanted in the
>>>>> first place. So we'll probably end up with subsystems with many eBPF
>>>>> programs, and also subsystems with zero. Similarly to tracepoints.
>>>>>
>>>>> I.e. pretty much the 'perf' model, but on much wider scale (as eBPF can
>>>>> hook to just about anything).
>>>>>
>>>>> Any other option seems to lead to having eBPF programs sprinkled all over
>>>>> the internet that depend on particular kernel version / API, leading to
>>>>> nothing else than unhappy users, because "I downloaded it, it didn't work,
>>>>> Linux sucks".
>>>>
>>>> OK, but if we keep eBPF programs this closely coupled to the kernel, then
>>>> what is the advantage of using eBPF, say for HID as was discussed earlier
>>>> in this thread, compared to just making sure HID has appropriate hooks and
>>>> the handling of the device quirks is done in normal C code (kernel module)
>>>> attached to these hooks?
>>>
>>> [sorry some of my messages are kept in the moderation queue]
>>>
>>> This is something I tried to explain in my talk at Kernel Recipes 2
>>> weeks ago [1].
>>>
>>> Basically, for regular users, it will be simpler for them to test patches:
>>> A developer patches the device through a bpf program, compiles it and
>>> sends to the user the source and the bytecode. The user just has to
>>> drop the bytecode in the file system and a udev rule automatically
>>> loads it without having to reboot the current kernel, making testing
>>> way faster and simpler for users.
>>>
>>> Then developers take the burden of upstreaming the bpf program through
>>> the usual email submission.
>>>
>>> This way users are testing the actual code that is upstreamed without
>>> too much hurdle.
>>>
>>> IMO those kinds of fixes should be in-tree because they are actual fixes.
>>>
>>>
>>>>
>>>> Because frankly for me the main value of eBPF over patching and recompiling
>>>> the kernel is that I can tweak the eBPF code exactly to my needs and load
>>>> it to the kernel without needing to reboot, over time it is less work than
>>>> maintaining a source code patch out of tree, and usually it is a hacky
>>>> tweak I use for some debugging so merging it upstream does not make sense.
>>>>
>>>
>>> And that's also why I want to give *some* guarantees that the eBPF
>>> hooks I am putting in HID are somewhat stable. Or at least I won't
>>> break everything at each release and look carefully when there is a
>>> change in the API.
>>>
>>> But these guarantees are just *my* constraints I put on *my*
>>> subsystem. I have reasons to believe I can ensure that, so I will.
>>>
>>> But I am not saying any internal HID function will have ABI
>>> guarantees. Only the few hooks I am explicitly documenting as such.
>>>
>>> So the idea is that your debugging program can live in your own tree,
>>> without being submitted, but this is just a contract between myself as
>>> a subsystem maintainer and you, not the entire eBPF or ftrace
>>> community.
>>>
>>> I made this so I can put any fancy new kernel API out in eBPF
>>> programs, that would be handled by the actual user of that kernel API.
>>
>> I think you need to clarify this, to me it now reads that you want
>> to use eBPF to implement new APIs, where as what you mean is that
>> you want to avoid the need to add new APIs like e.g. sysfs knobs
>> or kernel-module-parameters, right ?
> 
> Yes. Exactly. It's not about regular and standard API. It's a device
> specific property/feature that is usually handled through module
> parameters or custom sysfs entries that have only one single user.
> 
>>
>>> If a user wants to set some fancy feature on a particular device,
>>> instead of having a kernel parameter that no one will ever use besides
>>> that program that may be short-lived, we can then load that
>>> functionality with the eBPF program.
>>
>> I think you need to clarify here that you mean changing some settings
>> on the device through e.g. a HID feature report which would require
>> a sysfs-attribute or kernel-module-parameters without ePBF and you want
>> to avoid adding more and more sysfs-attributes / kernel-module-parameters?
> 
> Yep yep :)
> 
>>
>> At least I think this is what you are trying to say, and I think that
>> without some clarification this is not what most kernel-devs will
>> understand here.
> 
> Thanks for saying it better than I do :)

You're welcome. Actually thinking more about this I think this
is somewhat akin to Windows HID filter drivers and given BPF-s
history in packet-filtering I think that that actually is a helpful
analogy to make.

Regards,

Hans

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Steven Rostedt]

[ Switched over to ksummit@lists.linux.dev ]

On Fri, 17 Jun 2022 13:25:04 +0200
Benjamin Tissoires <benjamin.tissoires@redhat.com> wrote:

> > I think you need to clarify here that you mean changing some settings
> > on the device through e.g. a HID feature report which would require
> > a sysfs-attribute or kernel-module-parameters without ePBF and you want
> > to avoid adding more and more sysfs-attributes / kernel-module-parameters?  
> 
> Yep yep :)

And that is perfectly acceptable, and something I would support.

> 
> >
> > At least I think this is what you are trying to say, and I think that
> > without some clarification this is not what most kernel-devs will
> > understand here.  

This statement is the issue. There's not a clear line of what
constitutes what eBPF is being used for, and worse, what is guaranteed
to work across kernel versions.

My fear is that we will start having an expectation that pretty much
any eBPF program will continue to work, and there's going to be a lot
of disgruntled users when it doesn't.

Having configuration uses is one thing, enabling full device
functionality is another. And anything that touches core interfaces
need to be scrutinized.

If we look at where BPF came from (iptables et.al.) and having it do
nothing more that that but for other parts of the system, it may be a
good start. But even that could introduce dependencies of internal
kernel implementations that could break over time. Basically, we can
try to not break BPF programs but we really need the decree that it's
not user API, and does not follow the "don't break user space" agenda,
as anything that uses eBPF, is *not* user space. It's just another kind
of kernel module.

And if an eBPF program does break, if the source of that program is not
available, then the answer to fixing it should be "tough luck". This
should not be a way for proprietary code to have their kernel API.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Steven Rostedt]

On Mon, 20 Jun 2022 09:13:44 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> This statement is the issue. There's not a clear line of what
> constitutes what eBPF is being used for, and worse, what is guaranteed
> to work across kernel versions.
> 
> My fear is that we will start having an expectation that pretty much
> any eBPF program will continue to work, and there's going to be a lot
> of disgruntled users when it doesn't.
> 
> Having configuration uses is one thing, enabling full device
> functionality is another. And anything that touches core interfaces
> need to be scrutinized.
> 
> If we look at where BPF came from (iptables et.al.) and having it do
> nothing more that that but for other parts of the system, it may be a
> good start. But even that could introduce dependencies of internal
> kernel implementations that could break over time. Basically, we can
> try to not break BPF programs but we really need the decree that it's
> not user API, and does not follow the "don't break user space" agenda,
> as anything that uses eBPF, is *not* user space. It's just another kind
> of kernel module.
> 
> And if an eBPF program does break, if the source of that program is not
> available, then the answer to fixing it should be "tough luck". This
> should not be a way for proprietary code to have their kernel API.

I'm currently at the Open Source Summit, and Dirk basically asked my
question almost verbatim to Linus (during the "Dirk and Linus show").

And Linus stated that there is no defined "API". It's just "do not
break user space tools", and this includes eBPF. That is, if something
that is useful in user space that breaks with an upgrade, what breaks
it either gets reverted, or a "fix" is done to make it work again.

Linus specified that when it gets to eBPF programs that do debugging or
tracing, or other really low level interactions with the kernel, then
if it breaks a user space tool that is doing this low level work, then
that's just part of the work flow (fix user space to match, it's
already dependent on the kernel implementations). He specified if you
break "real user space tools, that normal (non kernel developers) users
use" then you need to fix it, regardless if it is using eBPF or not.

This still does not address the problem. First, where's the line where a
tool becomes something for normal users?

Next, eBPF can now attach to pretty much *any* function, and modify how
that function operates (changing parameters, etc) Basically, eBPF can
do live kernel patching like changes.

If a new feature is implemented with eBPF and that eBPF feature depends
on some internal kernel implementation, where the maintainer of that
code is unaware of this new eBPF implemented feature, if they change
their code and break this, then the burden is on the maintainer to fix
that breakage, not those that implemented the eBPF feature.

This is the worry I have. Maintainers now have no control over what is
exposed to users that can become 'user API", aka break normal user
tooling, without having a clue that something now depends on the
implementation of their code.

We really need to take a step back before we let eBPF become fully
controlling of anything in the kernel. Because that's going to add a
huge burden on maintainers that do not even use eBPF.

Exposing information for consumption only is one thing, and what I
would like to have more of, but once you allow non-passive interactions
with the tracing infrastructure, it changes things where I can see a
lot of maintainers will have more push back against the former (reading
only tracing, as there's no way to keep it from making changes).

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Benjamin Tissoires]

On Tue, Jun 21, 2022 at 5:12 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 20 Jun 2022 09:13:44 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
>
> > This statement is the issue. There's not a clear line of what
> > constitutes what eBPF is being used for, and worse, what is guaranteed
> > to work across kernel versions.
> >
> > My fear is that we will start having an expectation that pretty much
> > any eBPF program will continue to work, and there's going to be a lot
> > of disgruntled users when it doesn't.
> >
> > Having configuration uses is one thing, enabling full device
> > functionality is another. And anything that touches core interfaces
> > need to be scrutinized.
> >
> > If we look at where BPF came from (iptables et.al.) and having it do
> > nothing more that that but for other parts of the system, it may be a
> > good start. But even that could introduce dependencies of internal
> > kernel implementations that could break over time. Basically, we can
> > try to not break BPF programs but we really need the decree that it's
> > not user API, and does not follow the "don't break user space" agenda,
> > as anything that uses eBPF, is *not* user space. It's just another kind
> > of kernel module.
> >
> > And if an eBPF program does break, if the source of that program is not
> > available, then the answer to fixing it should be "tough luck". This
> > should not be a way for proprietary code to have their kernel API.
>
> I'm currently at the Open Source Summit, and Dirk basically asked my
> question almost verbatim to Linus (during the "Dirk and Linus show").

Heh, interesting :)

>
> And Linus stated that there is no defined "API". It's just "do not
> break user space tools", and this includes eBPF. That is, if something
> that is useful in user space that breaks with an upgrade, what breaks
> it either gets reverted, or a "fix" is done to make it work again.
>
> Linus specified that when it gets to eBPF programs that do debugging or
> tracing, or other really low level interactions with the kernel, then
> if it breaks a user space tool that is doing this low level work, then
> that's just part of the work flow (fix user space to match, it's
> already dependent on the kernel implementations). He specified if you
> break "real user space tools, that normal (non kernel developers) users
> use" then you need to fix it, regardless if it is using eBPF or not.

Hmm, OK, but I am not sure we are all talking about the same thing here.

For real user space applications using eBPF, AFAICT, today we have
cgroups and network filtering.
And for those 2 applications, given that there is a well defined eBPF
API, it wouldn't surprise me that maintainers should take care of the
user space breakage.

However, if I decide to attach a random BPF program to random
functions in the kernel without any involvement in the kernel, and use
it to get some metrics, how can we consider this to be plain debugging
or an actual user space application (assuming I share it with the rest
of the world)?

So IMO, we can not assume that any user space application relying on
eBPF should never break if that application is hooking to random
functions in the kernel. If that user space application is using a
well defined (not-an-)API, then yes, this is something maintainers
should be aware of.

I might not win that argument with Linus, but I still prefered your
suggestion to make eBPF for drivers similar to kernel modules.

>
> This still does not address the problem. First, where's the line where a
> tool becomes something for normal users?

I thought this was the initial topic that Jiri raised, and why we need
to have this discussion :)

>
> Next, eBPF can now attach to pretty much *any* function, and modify how
> that function operates (changing parameters, etc) Basically, eBPF can
> do live kernel patching like changes.

AFAICT, technically, when you attach an TRACING bpf program, you only
have read-only capabilities on the arguments.
To be able to change the return value, the kernel needs to explicitly
export the function as such, and to change the incoming data, well,
you need a special kfunc.


>
> If a new feature is implemented with eBPF and that eBPF feature depends
> on some internal kernel implementation, where the maintainer of that
> code is unaware of this new eBPF implemented feature, if they change
> their code and break this, then the burden is on the maintainer to fix
> that breakage, not those that implemented the eBPF feature.

I think I would need a more precise example here.
kfuncs are by explicitly defined in the kernel, meaning that the eBPF
program willing to use that feature needs to go through the kfunc. And
then, I would hope that the writer of the kfunc writes selftests for
it, and we can detect the change early enough before it gets pushed.
So the only other part a maintainer would not be aware of is if the
eBPF program attached a tracing program to one of the internal
functions, and I don't have a solution for this given that it's
already in the wild, and you already had the case once.

>
> This is the worry I have. Maintainers now have no control over what is
> exposed to users that can become 'user API", aka break normal user
> tooling, without having a clue that something now depends on the
> implementation of their code.
>
> We really need to take a step back before we let eBPF become fully
> controlling of anything in the kernel. Because that's going to add a
> huge burden on maintainers that do not even use eBPF.

We already have the painful part available to anybody.

I have a simple example for the gamepads with a touchpad:
I can easily attach a eBPF program that counts how many times a hidraw
devnode is opened, and if the result is positive, disable another
devnode through the sysfs (the input node of the touchpad) that should
not be used at the same time. If I push that code to a userspace tool,
let's say Steam or systemd, this will be considered as "application"
and I am now preventing the maintainer of hidraw or maybe devnode to
not do further changes regarding those functions?

So I would rather have a clear definition of what is an eBPF program
that is an extension of a driver, and how we can deal with internal
kernel changes.

>
> Exposing information for consumption only is one thing, and what I
> would like to have more of, but once you allow non-passive interactions
> with the tracing infrastructure, it changes things where I can see a
> lot of maintainers will have more push back against the former (reading
> only tracing, as there's no way to keep it from making changes).
>

Again, AFAICT, non-passive interaction is opt-in only. It doesn't mean
this won't have any side effect on other maintainers, but if one
maintainer wants non-passive interactions, then that maintainer
probably needs to define what is supposed to be used with (or just
embed the source in the tree and provide selftests).

Cheers,
Benjamin

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Jiri Kosina]

On Tue, 21 Jun 2022, Benjamin Tissoires wrote:

> > I'm currently at the Open Source Summit, and Dirk basically asked my
> > question almost verbatim to Linus (during the "Dirk and Linus show").
> 
> Heh, interesting :)

I guess that also sort of underlines the point that this discussion is 
relevant :)

> Hmm, OK, but I am not sure we are all talking about the same thing here.
> 
> For real user space applications using eBPF, AFAICT, today we have 
> cgroups and network filtering. And for those 2 applications, given that 
> there is a well defined eBPF API, it wouldn't surprise me that 
> maintainers should take care of the user space breakage.
> 
> However, if I decide to attach a random BPF program to random functions 
> in the kernel without any involvement in the kernel, and use it to get 
> some metrics, how can we consider this to be plain debugging or an 
> actual user space application (assuming I share it with the rest of the 
> world)?
> 
> So IMO, we can not assume that any user space application relying on 
> eBPF should never break if that application is hooking to random 
> functions in the kernel. If that user space application is using a well 
> defined (not-an-)API, then yes, this is something maintainers should be 
> aware of.

And we have been exactly here with traceevents/tracepoints, as far as I 
can tell.

It was introduced mostly for debugging and profiling purposes. Then some 
random Joe Developer wrote his 'ubertrace' tracing tool that had hard 
dependency on format of some of those, and they they changed, this 
publicly available tool broke. I don't see how eBPF makes this in 
principle different, and that's exactly why I brought this up. Because 
every maintainer is potentially involved/affected, even those who have no 
idea whatsoever yet what eBPF is.

> > This still does not address the problem. First, where's the line where a
> > tool becomes something for normal users?
> 
> I thought this was the initial topic that Jiri raised, and why we need
> to have this discussion :)

Absolutely, yes.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Alexei Starovoitov]

On Thu, Jun 23, 2022 at 1:15 PM Jiri Kosina <jikos@kernel.org> wrote:
>
> On Tue, 21 Jun 2022, Benjamin Tissoires wrote:
>
> > > I'm currently at the Open Source Summit, and Dirk basically asked my
> > > question almost verbatim to Linus (during the "Dirk and Linus show").
> >
> > Heh, interesting :)
>
> I guess that also sort of underlines the point that this discussion is
> relevant :)
>
> > Hmm, OK, but I am not sure we are all talking about the same thing here.
> >
> > For real user space applications using eBPF, AFAICT, today we have
> > cgroups and network filtering. And for those 2 applications, given that
> > there is a well defined eBPF API, it wouldn't surprise me that
> > maintainers should take care of the user space breakage.
> >
> > However, if I decide to attach a random BPF program to random functions
> > in the kernel without any involvement in the kernel, and use it to get
> > some metrics, how can we consider this to be plain debugging or an
> > actual user space application (assuming I share it with the rest of the
> > world)?
> >
> > So IMO, we can not assume that any user space application relying on
> > eBPF should never break if that application is hooking to random
> > functions in the kernel. If that user space application is using a well
> > defined (not-an-)API, then yes, this is something maintainers should be
> > aware of.
>
> And we have been exactly here with traceevents/tracepoints, as far as I
> can tell.

Apples are similar to oranges in many ways.

> It was introduced mostly for debugging and profiling purposes. Then some
> random Joe Developer wrote his 'ubertrace' tracing tool that had hard
> dependency on format of some of those, and they they changed, this
> publicly available tool broke. I don't see how eBPF makes this in
> principle different, and that's exactly why I brought this up. Because
> every maintainer is potentially involved/affected, even those who have no
> idea whatsoever yet what eBPF is.

tracepoints exposed an api through cat-able and echo-able files
that 'ubertrace' can use without any knowledge of the underlying kernel,
without access to kernel sources, etc.
Now please explain how bpf based tracing tools are similar?
What exposed api do they use?

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] How far to go with eBPF

[Steven Rostedt]

On Thu, 23 Jun 2022 14:23:51 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> tracepoints exposed an api through cat-able and echo-able files
> that 'ubertrace' can use without any knowledge of the underlying kernel,
> without access to kernel sources, etc.
> Now please explain how bpf based tracing tools are similar?
> What exposed api do they use?

Question to you. Can eBPF provide information that normally is not
available without the eBPF program? Could there be someone that creates
a eBPF program based on internal information where that developer had
to read the kernel source to implement. But then provide this program
to other users that can now write programs based on that new
information?

Years and versions can go by, and the code that the eBPF program hasn't
changed so this works for years, without the maintainer of the attached
code even being aware of this use. Then one day that maintainer changes
the code and this eBPF program breaks for the first time in years, and
with that, it breaks all the user space applications that relied on
this.

I can see that change falling under the "broke user space" clause of
the Linux kernel and reverted.