Re: How to orchestrate multiple XDP programs

Re: How to orchestrate multiple XDP programs

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> Hi all,
>
> tl;dr: What does the future look like for Go programs orchestrating
> multiple, chained eBPF network functions? Is it worth considering doing
> the orchestration from C++ (or Rust) instead of Go? I'm hoping the
> Cilium and/or Katran folks (and any other interested people) can weigh
> in!

Thank you for bringing this up! Adding a few people (and bpf@vger) to Cc
to widen the discussion a bit.

> --
>
> I just joined a team working with the orchestration of multiple XDP (and
> TC) programs. By "orchestration" I mean that various eBPF programs can
> be written by multiple teams for different systems and we have logic to
> decide how to run, update, and chain them together. This seems to be a
> fast-moving topic right now and I'm trying to get my bearings as well as
> prepare for the future. Feel free to just point me to relevant docs or
> code
>
> This is essentially what we do today: We have a top level Golang
> orchestration program that has access to a database that contains all
> the business logic (e.g., variable values for the bpf programs depending
> on datacenter), the latest build of the C BPF userspace, and kernel
> programs. Basically, like this:
>
>                   +--> [userspace prog 1] --> [kern prog 1]
>                   |
> [Go orchestrator] +--> [userspace prog 2] --> [kern prog 2]
>                   |
>                   +--> [userspace prog 3] --> [kern prog 3]
>
> The Go program simply executes (fork+exec) the userspace programs with
> the appropriate command-line arguments for their environment. The
> userspace program loads the kernel programs, which need to do a
> bpf_tail_call to the next program, which is exposed with some bpf map
> mojo[1].
>
> I think it's not too dissimilar from what the Cilium and Katran folks
> have been doing. I think our current approach is actually pretty cool
> considering that the project started a couple of years ago, but I'm
> trying to plot a course for the future.
>
> I have a couple of concerns about the current design:
>
> 1. The kernel programs need to make the bpf_tail_call. I'd prefer our
>    internal teams can write network functions without needing to be
>    aware of other network functions.
> 2. The Go orchestrator simply doing a fork+exec feels naughty. I'm
>    assuming there's potentially important error state information that
>    we might be missing out on by not working with an library API.
>
> Regarding #1, Toke Høiland-Jørgensen was kind enough to point me to his
> recent work for the Linux 5.10 kernel and xdp-loader (backed by xdplib)
> that I think addresses the underlying concern. However, I'm not so sure
> how to handle my concern #2.
>
> I think ideally, our new flow would look something like this:
>
>                                +--> [kern prog 1]
>                                |
> [Go orchestrator] --> [xdplib] +--> [kern prog 2]
>                                |
>                                +--> [kern prog 3]
>
> Assuming that make sense, I have a few questions:
>     * is there any work being done for a Go interface for xdplib?
>     * interface for xdplib? Any ideas on the level of effort to do that? 
>
> Alternatively, we could probably just execute the xdp-loader binary from
> Go, but that that goes back to my concern #2 above.

As I see it there are basically four paths to widen the language support
for libxdp/multiprog:

1. Regular language bindings for the C libxdp. I gather this is somewhat
   cumbersome in Go, though, as evidenced by the existence of a
   native-go BPF library.

2. Reimplement the libxdp functionality in each language. Libxdp really
   implements a "protocol" for how to cooperatively build a multiprog
   dispatcher from component programs, including atomic replace etc.
   This could be re-implemented by other libraries / in other languages
   as well, and if people want to go this route I'm happy to write up a
   formal specification if that's helpful. I'm not aware of any efforts
   in this direction thus far, though.

3. Make xdp-loader explicitly support the fork/exec use case you're
   describing above. Nothing says this has to lose any information
   compared to the library, we just have to design for it (JSON output
   and the ability to pass a BPF object file as an fd on exit would be
   the main things missing here, I think). I certainly wouldn't object
   to including this in xdp-loader.

4. Implement an "xdpd" daemon that exposes an IPC interface (say, a
   socket) and does all the stuff libxdp currently does in application
   context. Each language then just has to talk to the daemon which is
   less complex than the full libxdp re-implementation.


I've been resisting the daemon approach because I don't like introducing
another dependency for running XDP. But my main concern is that we end
up with something that is compatible across environments and
implementations, so applications that use XDP don't have to deal with
multiple incompatible ways of loading their programs.

So any feedback anyone has as to which approach(es) would/would not work
for you would be welcome (speaking to everyone here)!

-Toke

Re: How to orchestrate multiple XDP programs

[Brian G. Merrell]

On 20/12/01 01:08PM, Toke Høiland-Jørgensen wrote:
> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
> 
> > Hi all,
> >
> > tl;dr: What does the future look like for Go programs orchestrating
> > multiple, chained eBPF network functions? Is it worth considering doing
> > the orchestration from C++ (or Rust) instead of Go? I'm hoping the
> > Cilium and/or Katran folks (and any other interested people) can weigh
> > in!
> 
> Thank you for bringing this up! Adding a few people (and bpf@vger) to Cc
> to widen the discussion a bit.
> 

[snip]

> 
> As I see it there are basically four paths to widen the language support
> for libxdp/multiprog:

Thanks for enumerating these paths; I think they are all feasible. It's
just a matter of weighing trade-offs, naturally.

> 
> 1. Regular language bindings for the C libxdp. I gather this is somewhat
>    cumbersome in Go, though, as evidenced by the existence of a
>    native-go BPF library.

"Cumbersome" is a nice, succinct way to put it. While we could hack
something together for POC code (if we get to that point before a better
option is available), I don't think this is the right answer for Go
applications in general (based on personal experience, documented
experiences from other projects, and apparently lack of interest in cgo
from the Go team--anyone curious about details need only search the
web).

> 
> 2. Reimplement the libxdp functionality in each language. Libxdp really
>    implements a "protocol" for how to cooperatively build a multiprog
>    dispatcher from component programs, including atomic replace etc.
>    This could be re-implemented by other libraries / in other languages
>    as well, and if people want to go this route I'm happy to write up a
>    formal specification if that's helpful. I'm not aware of any efforts
>    in this direction thus far, though.

With a small edge over option #4, I think this is the best of the
options. The downside is that it is not part of the xdp project, so the
developers of the Go implementation will obviously be responsible for
all of the initial and continuing overhead of a full reimplementation.
Toke, you probably see that as an upside :). Regardless, I think that
libxdp is not so large to make this overly burdensome.

Could you please write up that format specification and we will start
running with it?

> 
> 3. Make xdp-loader explicitly support the fork/exec use case you're
>    describing above. Nothing says this has to lose any information
>    compared to the library, we just have to design for it (JSON output
>    and the ability to pass a BPF object file as an fd on exit would be
>    the main things missing here, I think). I certainly wouldn't object
>    to including this in xdp-loader.

There is some temptation in asking for this approach because it becomes
part of the xdp project. I just can't help but think it will be a
constant struggle to keep the same level of flexibility and control with
this approach versus access to a full library. I will admit that I am
moderately ignorant about exactly what challenges we might face because
I'm still learning more about bpf, xdp, and libxdp every day; I am
definitely open to being convinced that my concerns are wrong.

> 
> 4. Implement an "xdpd" daemon that exposes an IPC interface (say, a
>    socket) and does all the stuff libxdp currently does in application
>    context. Each language then just has to talk to the daemon which is
>    less complex than the full libxdp re-implementation.

This option very tempting. I understand your hesitancy to add another
dependency for running XDP--if it wasn't for that, I may have pushed
harder for this approach. I do like the idea of the xdpd daemon being
part of the xdp project and providing a language agnostic interface to
the library.

There will still need to be a new Go project written to interface with the
daemon. This seems like quite a bit less work than a reimplementation,
but combined with the downside of adding another dependency, I think we
can pass on this option--unless you really have a change of heart :)

> 
> I've been resisting the daemon approach because I don't like introducing
> another dependency for running XDP. But my main concern is that we end
> up with something that is compatible across environments and
> implementations, so applications that use XDP don't have to deal with
> multiple incompatible ways of loading their programs.
> 
> So any feedback anyone has as to which approach(es) would/would not work
> for you would be welcome (speaking to everyone here)!

Yes, please, if anyone has additional thoughts please weigh in, but I
think my team is ready to commit to option #2.

Any concerns about my assessment and request?

Thanks,
Brian

Re: How to orchestrate multiple XDP programs

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> Yes, please, if anyone has additional thoughts please weigh in, but I
> think my team is ready to commit to option #2.
>
> Any concerns about my assessment and request?

Just a quick note - since the holidays are fast approaching I won't have
time to do anything more about it before January anyway. But until then,
thank you for volunteering to do the Go implementation work!

I'll write up the "spec" once I'm back from the holidays, and we can
continue the discussion (and you can start prototyping an
implementation). And if you run away screaming after seeing the gory
details we can of course reconsider ;)

Sounds good?

And if someone else chimes in with opinions before or during that, great!

-Toke

Re: How to orchestrate multiple XDP programs

[Brian G. Merrell]

On 21/01/29 01:02PM, Toke Høiland-Jørgensen wrote:
> Hi Brian
> 
> I've posted a first draft of this protocol description here:
> https://github.com/xdp-project/xdp-tools/blob/master/lib/libxdp/protocol.org
> 
> Please take a look and let me know what you think. And do feel free to
> point out any places that are unclear, as I said this is a first draft,
> and I'm expecting it to evolve as I get feedback from you and others :)
> 
> -Toke
> 

Thanks so much for doing this Toke. There's a lot of great information.
I did one read-through, and didn't notice any surprises compared to the
code that I've read so far.

One thing I have been a little concerned about is the XDP_RUN_CONFIG in
the xdp program function. For our case--with multiple teams writing
independent, composable xdp programs--we don't want the XDP_RUN_CONFIG
policy to be in the xdp program. Instead, we want the Go orchestration
tool to have that policy as part of its configuration data (e.g., what
order to run the xdp program functions in). From what I can tell, it's
possible to omit the XDP_RUN_CONFIG from the xdp program function, and
instead set the values when loading the xdp dispatcher. That's great, and
thanks for the foresight there. I just want to confirm that I'm
understanding that correctly, because it's very important for us.

Also, I do hope that the existing Go BTF libraries are good enough to do
what's needed here, because if I'm understand correctly, that's how I'll
need to approach setting the XDP_RUN_CONFIG values for our use case.

I've been tasked to work on a Go libxdp implementation this quarter, so
I'll be starting on that soon and let you know if I have questions as I
go. I'm also happy to coordinate with anyone else that's interested.

Thanks again,
Brian

Re: How to orchestrate multiple XDP programs

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> On 21/01/29 01:02PM, Toke Høiland-Jørgensen wrote:
>> Hi Brian
>> 
>> I've posted a first draft of this protocol description here:
>> https://github.com/xdp-project/xdp-tools/blob/master/lib/libxdp/protocol.org
>> 
>> Please take a look and let me know what you think. And do feel free to
>> point out any places that are unclear, as I said this is a first draft,
>> and I'm expecting it to evolve as I get feedback from you and others :)
>> 
>> -Toke
>> 
>
> Thanks so much for doing this Toke. There's a lot of great information.
> I did one read-through, and didn't notice any surprises compared to the
> code that I've read so far.

Awesome! :)

> One thing I have been a little concerned about is the XDP_RUN_CONFIG in
> the xdp program function. For our case--with multiple teams writing
> independent, composable xdp programs--we don't want the XDP_RUN_CONFIG
> policy to be in the xdp program. Instead, we want the Go orchestration
> tool to have that policy as part of its configuration data (e.g., what
> order to run the xdp program functions in). From what I can tell, it's
> possible to omit the XDP_RUN_CONFIG from the xdp program function, and
> instead set the values when loading the xdp dispatcher. That's great, and
> thanks for the foresight there. I just want to confirm that I'm
> understanding that correctly, because it's very important for us.

Yes. The values embedded into the program BTF are defaults, and can be
overridden on load. The idea is that an application will set a default
value (e.g., "I'm a firewall, so I want to run early" or "I want to
monitor traffic to the stack so I'll run late"), but if the sysadmin
wants to do things differently they can override the order. The
important bit being that ultimate control of run order is up to the
*user*, not the application developer.

The policy override stuff is not implemented yet, but I am planning to
implement it by having libxdp read a config file with priority overrides
(similar to how libc will read /etc/nsswitch.conf or /etc/hosts which
makes them work in all applications).

And of course, if you're writing an orchestration tool, then you *are*
the user, so having the tool override priorities is definitely in scope
(it'll just be an alternative way to set policy instead of a config
file). How are you planning to specify the effective run order? I am
also quite open to working on a compatible way that can work for both
your tool and libxdp :)

> Also, I do hope that the existing Go BTF libraries are good enough to do
> what's needed here, because if I'm understand correctly, that's how I'll
> need to approach setting the XDP_RUN_CONFIG values for our use case.

You'll need to *parse* BTF to *read* the XDP_RUN_CONFIG. Which is pretty
basic, really, you just need to walk the BTF reference tree. Feel free
to reuse the parsing code in libxdp; that is, in turn, adapted from the
.maps section parsing code in libbpf :)

> I've been tasked to work on a Go libxdp implementation this quarter, so
> I'll be starting on that soon and let you know if I have questions as I
> go. I'm also happy to coordinate with anyone else that's interested.

Sounds great! Will be interesting to see a second implementation;
independent implementations are the ultimate test of any specification :)

Please do keep me in the loop, and don't hesitate to ping me if there
are things that are unclear or that you feel are less-than-ideal in the
way things work. I'm also quite open to evolving the spec to meet
everyone's needs!

-Toke

Re: How to orchestrate multiple XDP programs

[Brian G. Merrell]

On 21/02/11 12:18PM, Toke Høiland-Jørgensen wrote:
> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
> 
> > One thing I have been a little concerned about is the XDP_RUN_CONFIG in
> > the xdp program function. For our case--with multiple teams writing
> > independent, composable xdp programs--we don't want the XDP_RUN_CONFIG
> > policy to be in the xdp program. Instead, we want the Go orchestration
> > tool to have that policy as part of its configuration data (e.g., what
> > order to run the xdp program functions in). From what I can tell, it's
> > possible to omit the XDP_RUN_CONFIG from the xdp program function, and
> > instead set the values when loading the xdp dispatcher. That's great, and
> > thanks for the foresight there. I just want to confirm that I'm
> > understanding that correctly, because it's very important for us.
> 
> Yes. The values embedded into the program BTF are defaults, and can be
> overridden on load. The idea is that an application will set a default
> value (e.g., "I'm a firewall, so I want to run early" or "I want to
> monitor traffic to the stack so I'll run late"), but if the sysadmin
> wants to do things differently they can override the order. The
> important bit being that ultimate control of run order is up to the
> *user*, not the application developer.

Great. In our case, it would be ideal if the application developer
doesn't even need to be aware of the XDP_RUN_CONFIG and can just omit
it. I guess for our implementation that would mean that we don't error
out if the BTF section doesn't exist, and instead we look to our
configuration data (more on that below) for the relevant information.

> The policy override stuff is not implemented yet, but I am planning to
> implement it by having libxdp read a config file with priority overrides
> (similar to how libc will read /etc/nsswitch.conf or /etc/hosts which
> makes them work in all applications).
> 
> And of course, if you're writing an orchestration tool, then you *are*
> the user, so having the tool override priorities is definitely in scope
> (it'll just be an alternative way to set policy instead of a config
> file). How are you planning to specify the effective run order? I am
> also quite open to working on a compatible way that can work for both
> your tool and libxdp :)

As part of our control plane we have a whole process for a sysadmin to
get config data to to our BPF orchestration tool, which is running on
multiple nodes. It very abstractly looks like this:


                                     +---- Node 1
                                     |
UI -> API -> DATABASE -> CONFIG DATA +---- Node 2
                                     |
		                     +---- Node N

So, the sysadmin using the UI or API would dictate which xdp programs
run *and* what their priority is (plus anything else that would
otherwise go into XDP_RUN_CONFIG, plus a bunch of other config data for
various other needs). Then--and hopefully I'm getting this right--when
our (Go) orchestration tool uses (Go) libxdp, the tool needs a way to
set the run order for the XDP programs before the dispatcher loads. I
was planning to set the run order programatically on the XDP program
objects via libxdp calls. It looks like your libxdp implementation
already has ways to do this in the form of xdp_program__set_run_prio()
and xdp_program__chain_call_enabled().

Does that make sense? This is still all very theoretical for me at this
point!

> 
> > Also, I do hope that the existing Go BTF libraries are good enough to do
> > what's needed here, because if I'm understand correctly, that's how I'll
> > need to approach setting the XDP_RUN_CONFIG values for our use case.
> 
> You'll need to *parse* BTF to *read* the XDP_RUN_CONFIG. Which is pretty
> basic, really, you just need to walk the BTF reference tree. Feel free
> to reuse the parsing code in libxdp; that is, in turn, adapted from the
> .maps section parsing code in libbpf :)

OK, that makes sense. Since I want to keep our implementation purely in
Go (if possible), what I trying to say what that I hope there's an
existing Go library that can parse and read BTF (Cillium's Go eBPF
library looks promising). After thinking more about our orchestration
config data use case I was describing above, though, I don't think
reading XDP_RUN_CONFIG from BTF is strictly necessary for our use case.
That said, it obviously would be preferable to conform to the
specification, plus it does look necessary to read the program IDs from
BTF anyway :)

8< snip

> Please do keep me in the loop, and don't hesitate to ping me if there
> are things that are unclear or that you feel are less-than-ideal in the
> way things work. I'm also quite open to evolving the spec to meet
> everyone's needs!

Will do. Thanks for your help!

-Brian

Re: How to orchestrate multiple XDP programs

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> On 21/02/11 12:18PM, Toke Høiland-Jørgensen wrote:
>> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
>> 
>> > One thing I have been a little concerned about is the XDP_RUN_CONFIG in
>> > the xdp program function. For our case--with multiple teams writing
>> > independent, composable xdp programs--we don't want the XDP_RUN_CONFIG
>> > policy to be in the xdp program. Instead, we want the Go orchestration
>> > tool to have that policy as part of its configuration data (e.g., what
>> > order to run the xdp program functions in). From what I can tell, it's
>> > possible to omit the XDP_RUN_CONFIG from the xdp program function, and
>> > instead set the values when loading the xdp dispatcher. That's great, and
>> > thanks for the foresight there. I just want to confirm that I'm
>> > understanding that correctly, because it's very important for us.
>> 
>> Yes. The values embedded into the program BTF are defaults, and can be
>> overridden on load. The idea is that an application will set a default
>> value (e.g., "I'm a firewall, so I want to run early" or "I want to
>> monitor traffic to the stack so I'll run late"), but if the sysadmin
>> wants to do things differently they can override the order. The
>> important bit being that ultimate control of run order is up to the
>> *user*, not the application developer.
>
> Great. In our case, it would be ideal if the application developer
> doesn't even need to be aware of the XDP_RUN_CONFIG and can just omit
> it. I guess for our implementation that would mean that we don't error
> out if the BTF section doesn't exist, and instead we look to our
> configuration data (more on that below) for the relevant information.

That's also what libxdp does; if no XDP_RUN_CONFIG is found, it just
uses the defaults (run prio 50 + chain call on XDP_PASS).

>> The policy override stuff is not implemented yet, but I am planning to
>> implement it by having libxdp read a config file with priority overrides
>> (similar to how libc will read /etc/nsswitch.conf or /etc/hosts which
>> makes them work in all applications).
>> 
>> And of course, if you're writing an orchestration tool, then you *are*
>> the user, so having the tool override priorities is definitely in scope
>> (it'll just be an alternative way to set policy instead of a config
>> file). How are you planning to specify the effective run order? I am
>> also quite open to working on a compatible way that can work for both
>> your tool and libxdp :)
>
> As part of our control plane we have a whole process for a sysadmin to
> get config data to to our BPF orchestration tool, which is running on
> multiple nodes. It very abstractly looks like this:
>
>
>                                      +---- Node 1
>                                      |
> UI -> API -> DATABASE -> CONFIG DATA +---- Node 2
>                                      |
> 		                     +---- Node N
>
> So, the sysadmin using the UI or API would dictate which xdp programs
> run *and* what their priority is (plus anything else that would
> otherwise go into XDP_RUN_CONFIG, plus a bunch of other config data for
> various other needs). Then--and hopefully I'm getting this right--when
> our (Go) orchestration tool uses (Go) libxdp, the tool needs a way to
> set the run order for the XDP programs before the dispatcher loads.

Yeah, and what I was interested in was how the orchestration tool gets
this data (and the BPF programs themselves)? Is there a daemon running
on the nodes that exposes an API? Are you pushing this via SSH/Ansible?
Infinite monkeys with typewriters inputting data? Something else? :)

> I was planning to set the run order programatically on the XDP program
> objects via libxdp calls. It looks like your libxdp implementation
> already has ways to do this in the form of xdp_program__set_run_prio()
> and xdp_program__chain_call_enabled().
>
> Does that make sense? This is still all very theoretical for me at this
> point!

Yup, totally possible to set this programmatically with libxdp as well
today. However, before doing so you still need to communicate the list
of BPF programs and their run configuration to each node. And I'm
thinking it may be worthwhile to specify how to do this as part of the
"protocol" and also teach libxdp about the format, so others won't have
to reinvent the same thing later.

The reason I went with the embedded BTF is that this gets compiled into
the ELF file, and so we can be pretty sure that it doesn't get lost,
without having to keep track of separate configuration files. So this
makes it a good fit for BPF program authors specifying a default: they
can be pretty sure that this will stay with the object code no matter
how it's moved around.

The downside of using BTF is of course the same: it's tightly coupled to
the compiled binary, and it's a bit awkward to parse (and modify). So I
always anticipated that a secondary format that was *decoupled* from the
binary byte code format would be needed, just as you're describing. So
I'm just looking for input on what such a format might reasonably look
like :)

>> > Also, I do hope that the existing Go BTF libraries are good enough to do
>> > what's needed here, because if I'm understand correctly, that's how I'll
>> > need to approach setting the XDP_RUN_CONFIG values for our use case.
>> 
>> You'll need to *parse* BTF to *read* the XDP_RUN_CONFIG. Which is pretty
>> basic, really, you just need to walk the BTF reference tree. Feel free
>> to reuse the parsing code in libxdp; that is, in turn, adapted from the
>> .maps section parsing code in libbpf :)
>
> OK, that makes sense. Since I want to keep our implementation purely
> in Go (if possible), what I trying to say what that I hope there's an
> existing Go library that can parse and read BTF (Cillium's Go eBPF
> library looks promising). After thinking more about our orchestration
> config data use case I was describing above, though, I don't think
> reading XDP_RUN_CONFIG from BTF is strictly necessary for our use
> case.

See above re: my reasons for picking the BTF format. Not sure how you're
developing the BPF programs, but it may turn out to be useful to have
program authors specify defaults as well. E.g., you could have whatever
process *inserts* programs into your database (assuming that's where you
store the available programs) read default values from the BTF and
pre-populate the admin UI with those when someone wants to load
programs?

> That said, it obviously would be preferable to conform to the
> specification, plus it does look necessary to read the program IDs
> from BTF anyway :)

The program IDs are allocated by the kernel on load, so those have
nothing to do with BTF. But you'll likely want to support BTF-defined
maps, and the freplace functionality itself relies on BTF being present;
so you'll need to handle it somehow... :)

-Toke

Re: How to orchestrate multiple XDP programs

[Brian G. Merrell]

On 21/02/15 01:47PM, Toke Høiland-Jørgensen wrote:
> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
> 
> > On 21/02/11 12:18PM, Toke Høiland-Jørgensen wrote:
> >> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
> 
> >> The policy override stuff is not implemented yet, but I am planning to
> >> implement it by having libxdp read a config file with priority overrides
> >> (similar to how libc will read /etc/nsswitch.conf or /etc/hosts which
> >> makes them work in all applications).
> >> 
> >> And of course, if you're writing an orchestration tool, then you *are*
> >> the user, so having the tool override priorities is definitely in scope
> >> (it'll just be an alternative way to set policy instead of a config
> >> file). How are you planning to specify the effective run order? I am
> >> also quite open to working on a compatible way that can work for both
> >> your tool and libxdp :)
> >
> > As part of our control plane we have a whole process for a sysadmin to
> > get config data to to our BPF orchestration tool, which is running on
> > multiple nodes. It very abstractly looks like this:
> >
> >
> >                                      +---- Node 1
> >                                      |
> > UI -> API -> DATABASE -> CONFIG DATA +---- Node 2
> >                                      |
> >                                      +---- Node N
> >
> > So, the sysadmin using the UI or API would dictate which xdp programs
> > run *and* what their priority is (plus anything else that would
> > otherwise go into XDP_RUN_CONFIG, plus a bunch of other config data for
> > various other needs). Then--and hopefully I'm getting this right--when
> > our (Go) orchestration tool uses (Go) libxdp, the tool needs a way to
> > set the run order for the XDP programs before the dispatcher loads.
> 
> Yeah, and what I was interested in was how the orchestration tool gets
> this data (and the BPF programs themselves)? Is there a daemon running
> on the nodes that exposes an API? Are you pushing this via SSH/Ansible?
> Infinite monkeys with typewriters inputting data? Something else? :)

OK, what currently happens is we have a separate, centralized Go web
service exposing an HTTP based API. When the sysadmin calls that API it
stores the config data in a database. Then, we have another service that
periodically queries the database and writes the config data to a
constant database (cdb) and stores that in blob storage. Then, there is
a service running on each node that periodically pulls down the latest
cdb. Our orchestration tool running on each node is watching for new
cdbs using inotify; when the tool sees a new CDB it loads the new
configuration data--which, for us, literally ends up just being JSON
data--and does anything that needs done.

I had omitted those details for a couple of reasons: First, it's kind of a
lot and I didn't know it would be helpful. Second, this is the way it
works currently because, for expediency, we leveraged the internal
ecosystem that was already setup. We will likely move away from it, at
least partially.

So, I think the important part is that our orchestration tool will
periodically get the config data in JSON format. A path to each BPF
program is in the config data and the orchestration tool downloads them
as needed. We may move to just including the BPF program binary in the
config data--TBD. Obviously, we aren't using libxdp yet, so our config
data doesn't have "run priority", instead the config data has the order
the BPF programs need to run, and BPF programs themselves have to do the
bpf_tail_call (and the orchestration tool does a bunch of complicated
orchestration to get the chain in the right order). The config data also
contains a bunch of other information to do the orchestration, e.g.,
interface, ingress or egress, tc or xdp, what userspace code to run and
any config values for that, etc.

Hopefully that answers your question, and sorry if it was too much
information :)

> 
> > I was planning to set the run order programatically on the XDP program
> > objects via libxdp calls. It looks like your libxdp implementation
> > already has ways to do this in the form of xdp_program__set_run_prio()
> > and xdp_program__chain_call_enabled().
> >
> > Does that make sense? This is still all very theoretical for me at this
> > point!
> 
> Yup, totally possible to set this programmatically with libxdp as well
> today. However, before doing so you still need to communicate the list
> of BPF programs and their run configuration to each node. And I'm
> thinking it may be worthwhile to specify how to do this as part of the
> "protocol" and also teach libxdp about the format, so others won't have
> to reinvent the same thing later.

It seems like I must be missing something here, but my plan was to do
this all programmatically by calling libxdp functions from the
orchestration tool by 1) calling something like xdp_program__open_file()
to load the XDP program, and then 2) setting the run configuration by
calling something like xdp_program__set_run_prio() and
xdp_program__chain_call_enabled(), and 3) adding the programs to the
dispatcher and loading it.

Correct me if I'm wrong, but I guess you're saying that it might be
worth creating an abstraction in libxdp where a user can pass in the
necessary config data and libxdp does the work, that I just summarized
in the previous paragraph, on the user's behalf. I can see how that
could be a useful abstraction.

> The reason I went with the embedded BTF is that this gets compiled into
> the ELF file, and so we can be pretty sure that it doesn't get lost,
> without having to keep track of separate configuration files. So this
> makes it a good fit for BPF program authors specifying a default: they
> can be pretty sure that this will stay with the object code no matter
> how it's moved around.
> 
> The downside of using BTF is of course the same: it's tightly coupled to
> the compiled binary, and it's a bit awkward to parse (and modify). So I
> always anticipated that a secondary format that was *decoupled* from the
> binary byte code format would be needed, just as you're describing. So
> I'm just looking for input on what such a format might reasonably look
> like :)

I don't have super strong feelings about this, and there may be use
cases that I'm not thinking about, but my first thought would be to make
the format just be code in libxdp, and have the libxdp "abstraction"
function take an array of objects that contain the necessary data.

I know in a previous e-mail you mentioned having a config file with
priority overrides. That's just not a use case that our team would want
to use. And, my opinion would be that the program using libxdp should be
the one to implement that sort of policy; it keeps libxdp more simple
without needing to worry about parsing config files (and handling config
version changes in the code and the spec). For example, xdp-loader could
have a config file with priority overrides and people could use that
code if they wanted to do something similar.

Hopefully I'm even making sense, but like I said, I don't have strong
feelings about the format, as long as we are able to achieve our
required use case of programmatically setting the run configuration
values from a libxdp user program.

> 
> >> > Also, I do hope that the existing Go BTF libraries are good enough to do
> >> > what's needed here, because if I'm understand correctly, that's how I'll
> >> > need to approach setting the XDP_RUN_CONFIG values for our use case.
> >> 
> >> You'll need to *parse* BTF to *read* the XDP_RUN_CONFIG. Which is pretty
> >> basic, really, you just need to walk the BTF reference tree. Feel free
> >> to reuse the parsing code in libxdp; that is, in turn, adapted from the
> >> .maps section parsing code in libbpf :)
> >
> > OK, that makes sense. Since I want to keep our implementation purely
> > in Go (if possible), what I trying to say what that I hope there's an
> > existing Go library that can parse and read BTF (Cillium's Go eBPF
> > library looks promising). After thinking more about our orchestration
> > config data use case I was describing above, though, I don't think
> > reading XDP_RUN_CONFIG from BTF is strictly necessary for our use
> > case.
> 
> See above re: my reasons for picking the BTF format. Not sure how you're
> developing the BPF programs, but it may turn out to be useful to have
> program authors specify defaults as well. E.g., you could have whatever
> process *inserts* programs into your database (assuming that's where you
> store the available programs) read default values from the BTF and
> pre-populate the admin UI with those when someone wants to load
> programs?

We explicitly do not want defaults set by program authors. We want that
policy to be completely in the hands of the orchestration environment.

> 
> > That said, it obviously would be preferable to conform to the
> > specification, plus it does look necessary to read the program IDs
> > from BTF anyway :)
> 
> The program IDs are allocated by the kernel on load, so those have
> nothing to do with BTF. But you'll likely want to support BTF-defined
> maps, and the freplace functionality itself relies on BTF being present;
> so you'll need to handle it somehow... :)
> 
> -Toke

Thanks again,
Brian

Re: How to orchestrate multiple XDP programs

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> On 21/02/15 01:47PM, Toke Høiland-Jørgensen wrote:
>> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
>> 
>> > On 21/02/11 12:18PM, Toke Høiland-Jørgensen wrote:
>> >> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
>> 
>> >> The policy override stuff is not implemented yet, but I am planning to
>> >> implement it by having libxdp read a config file with priority overrides
>> >> (similar to how libc will read /etc/nsswitch.conf or /etc/hosts which
>> >> makes them work in all applications).
>> >> 
>> >> And of course, if you're writing an orchestration tool, then you *are*
>> >> the user, so having the tool override priorities is definitely in scope
>> >> (it'll just be an alternative way to set policy instead of a config
>> >> file). How are you planning to specify the effective run order? I am
>> >> also quite open to working on a compatible way that can work for both
>> >> your tool and libxdp :)
>> >
>> > As part of our control plane we have a whole process for a sysadmin to
>> > get config data to to our BPF orchestration tool, which is running on
>> > multiple nodes. It very abstractly looks like this:
>> >
>> >
>> >                                      +---- Node 1
>> >                                      |
>> > UI -> API -> DATABASE -> CONFIG DATA +---- Node 2
>> >                                      |
>> >                                      +---- Node N
>> >
>> > So, the sysadmin using the UI or API would dictate which xdp programs
>> > run *and* what their priority is (plus anything else that would
>> > otherwise go into XDP_RUN_CONFIG, plus a bunch of other config data for
>> > various other needs). Then--and hopefully I'm getting this right--when
>> > our (Go) orchestration tool uses (Go) libxdp, the tool needs a way to
>> > set the run order for the XDP programs before the dispatcher loads.
>> 
>> Yeah, and what I was interested in was how the orchestration tool gets
>> this data (and the BPF programs themselves)? Is there a daemon running
>> on the nodes that exposes an API? Are you pushing this via SSH/Ansible?
>> Infinite monkeys with typewriters inputting data? Something else? :)
>
> OK, what currently happens is we have a separate, centralized Go web
> service exposing an HTTP based API. When the sysadmin calls that API it
> stores the config data in a database. Then, we have another service that
> periodically queries the database and writes the config data to a
> constant database (cdb) and stores that in blob storage. Then, there is
> a service running on each node that periodically pulls down the latest
> cdb. Our orchestration tool running on each node is watching for new
> cdbs using inotify; when the tool sees a new CDB it loads the new
> configuration data--which, for us, literally ends up just being JSON
> data--and does anything that needs done.
>
> I had omitted those details for a couple of reasons: First, it's kind of a
> lot and I didn't know it would be helpful. Second, this is the way it
> works currently because, for expediency, we leveraged the internal
> ecosystem that was already setup. We will likely move away from it, at
> least partially.
>
> So, I think the important part is that our orchestration tool will
> periodically get the config data in JSON format. A path to each BPF
> program is in the config data and the orchestration tool downloads them
> as needed. We may move to just including the BPF program binary in the
> config data--TBD. Obviously, we aren't using libxdp yet, so our config
> data doesn't have "run priority", instead the config data has the order
> the BPF programs need to run, and BPF programs themselves have to do the
> bpf_tail_call (and the orchestration tool does a bunch of complicated
> orchestration to get the chain in the right order). The config data also
> contains a bunch of other information to do the orchestration, e.g.,
> interface, ingress or egress, tc or xdp, what userspace code to run and
> any config values for that, etc.
>
> Hopefully that answers your question, and sorry if it was too much
> information :)

No, that was very helpful, thanks! Just the kind of detail I was after
to understand your deployment scenario :)

>> > I was planning to set the run order programatically on the XDP program
>> > objects via libxdp calls. It looks like your libxdp implementation
>> > already has ways to do this in the form of xdp_program__set_run_prio()
>> > and xdp_program__chain_call_enabled().
>> >
>> > Does that make sense? This is still all very theoretical for me at this
>> > point!
>> 
>> Yup, totally possible to set this programmatically with libxdp as well
>> today. However, before doing so you still need to communicate the list
>> of BPF programs and their run configuration to each node. And I'm
>> thinking it may be worthwhile to specify how to do this as part of the
>> "protocol" and also teach libxdp about the format, so others won't have
>> to reinvent the same thing later.
>
> It seems like I must be missing something here, but my plan was to do
> this all programmatically by calling libxdp functions from the
> orchestration tool by 1) calling something like xdp_program__open_file()
> to load the XDP program, and then 2) setting the run configuration by
> calling something like xdp_program__set_run_prio() and
> xdp_program__chain_call_enabled(), and 3) adding the programs to the
> dispatcher and loading it.
>
> Correct me if I'm wrong, but I guess you're saying that it might be
> worth creating an abstraction in libxdp where a user can pass in the
> necessary config data and libxdp does the work, that I just summarized
> in the previous paragraph, on the user's behalf. I can see how that
> could be a useful abstraction.

Yeah, so what I was thinking was whether it would be useful to, for
instance, define a "bundle" format that contains the config data that
libxdp will understand. Could just be a JSON schema containing keys for
priority, chain call actions and a filename, so you can just point
xdp_program__open_file() at that and it will do the rest.

However, I'm still not quite sure I'm convinced that this will be
generally useful. As you say, you can just as well just set the values
programmatically after loading the file, and I suspect that different
deployments will end up having too much custom stuff around this that
they'll bother using such a facility anyway. WDYT?

>> The reason I went with the embedded BTF is that this gets compiled into
>> the ELF file, and so we can be pretty sure that it doesn't get lost,
>> without having to keep track of separate configuration files. So this
>> makes it a good fit for BPF program authors specifying a default: they
>> can be pretty sure that this will stay with the object code no matter
>> how it's moved around.
>> 
>> The downside of using BTF is of course the same: it's tightly coupled to
>> the compiled binary, and it's a bit awkward to parse (and modify). So I
>> always anticipated that a secondary format that was *decoupled* from the
>> binary byte code format would be needed, just as you're describing. So
>> I'm just looking for input on what such a format might reasonably look
>> like :)
>
> I don't have super strong feelings about this, and there may be use
> cases that I'm not thinking about, but my first thought would be to make
> the format just be code in libxdp, and have the libxdp "abstraction"
> function take an array of objects that contain the necessary data.

Well that's basically what 'struct xdp_program' in libxdp is :) You can
create an array of those (setting their properties with the setter
methods as you mentioned above) and pass them to
xdp_program__attach_multi() which will build a dispatcher for all of
them and attach that (incorporating any programs that may already be
attached by their priority).

> I know in a previous e-mail you mentioned having a config file with
> priority overrides. That's just not a use case that our team would want
> to use. And, my opinion would be that the program using libxdp should be
> the one to implement that sort of policy; it keeps libxdp more simple
> without needing to worry about parsing config files (and handling config
> version changes in the code and the spec). For example, xdp-loader could
> have a config file with priority overrides and people could use that
> code if they wanted to do something similar.

Yeah, that's totally what would make sense for your deployment case. The
design where libxdp reads a config file comes from my distro
perspective: We want to build a system whereby different applications
can each incorporate XDP functionality and co-exist; and the goal is to
make libxdp the synchronisation point between them. I.e., we can say to
application authors "just use libxdp when writing your application and
it'll work", while at the same time empowering sysadmins to change the
default application ordering.

By having that configuration be part of the library, applications can be
free to use either the command-line loader or include the loading into
their own user-space binary.

But since you are (notionally) both the application developer and system
owner, that is less of a concern for you as you control the whole stack.

> Hopefully I'm even making sense, but like I said, I don't have strong
> feelings about the format, as long as we are able to achieve our
> required use case of programmatically setting the run configuration
> values from a libxdp user program.

Sure, that you can certainly achieve with implementing what libxdp
includes today. I'm just trying to make sure we explore any
opportunities for standardising something useful so others can benefit
from it as well; so I hope you'll forgive my probing :)

>> >> > Also, I do hope that the existing Go BTF libraries are good enough to do
>> >> > what's needed here, because if I'm understand correctly, that's how I'll
>> >> > need to approach setting the XDP_RUN_CONFIG values for our use case.
>> >> 
>> >> You'll need to *parse* BTF to *read* the XDP_RUN_CONFIG. Which is pretty
>> >> basic, really, you just need to walk the BTF reference tree. Feel free
>> >> to reuse the parsing code in libxdp; that is, in turn, adapted from the
>> >> .maps section parsing code in libbpf :)
>> >
>> > OK, that makes sense. Since I want to keep our implementation purely
>> > in Go (if possible), what I trying to say what that I hope there's an
>> > existing Go library that can parse and read BTF (Cillium's Go eBPF
>> > library looks promising). After thinking more about our orchestration
>> > config data use case I was describing above, though, I don't think
>> > reading XDP_RUN_CONFIG from BTF is strictly necessary for our use
>> > case.
>> 
>> See above re: my reasons for picking the BTF format. Not sure how you're
>> developing the BPF programs, but it may turn out to be useful to have
>> program authors specify defaults as well. E.g., you could have whatever
>> process *inserts* programs into your database (assuming that's where you
>> store the available programs) read default values from the BTF and
>> pre-populate the admin UI with those when someone wants to load
>> programs?
>
> We explicitly do not want defaults set by program authors. We want that
> policy to be completely in the hands of the orchestration environment.

Right, OK. How does the admin configuring the orchestration system
figure out which order to run programs in, BTW? Is this obvious from the
nature of the programs, or do you document it out of band somewhere, or
something like that?

-Toke

Re: How to orchestrate multiple XDP programs

[Brian G. Merrell]

On 21/02/17 04:53PM, Toke Høiland-Jørgensen wrote:
> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
> 
> > On 21/02/15 01:47PM, Toke Høiland-Jørgensen wrote:
> >> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
> >> 
> >> > On 21/02/11 12:18PM, Toke Høiland-Jørgensen wrote:
> >> >> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

8< snip

> > OK, what currently happens is we have a separate, centralized Go web
> > service exposing an HTTP based API. When the sysadmin calls that API it
> > stores the config data in a database. Then, we have another service that
> > periodically queries the database and writes the config data to a
> > constant database (cdb) and stores that in blob storage. Then, there is
> > a service running on each node that periodically pulls down the latest
> > cdb. Our orchestration tool running on each node is watching for new
> > cdbs using inotify; when the tool sees a new CDB it loads the new
> > configuration data--which, for us, literally ends up just being JSON
> > data--and does anything that needs done.
> >
> > I had omitted those details for a couple of reasons: First, it's kind of a
> > lot and I didn't know it would be helpful. Second, this is the way it
> > works currently because, for expediency, we leveraged the internal
> > ecosystem that was already setup. We will likely move away from it, at
> > least partially.
> >
> > So, I think the important part is that our orchestration tool will
> > periodically get the config data in JSON format. A path to each BPF
> > program is in the config data and the orchestration tool downloads them
> > as needed. We may move to just including the BPF program binary in the
> > config data--TBD. Obviously, we aren't using libxdp yet, so our config
> > data doesn't have "run priority", instead the config data has the order
> > the BPF programs need to run, and BPF programs themselves have to do the
> > bpf_tail_call (and the orchestration tool does a bunch of complicated
> > orchestration to get the chain in the right order). The config data also
> > contains a bunch of other information to do the orchestration, e.g.,
> > interface, ingress or egress, tc or xdp, what userspace code to run and
> > any config values for that, etc.
> >
> > Hopefully that answers your question, and sorry if it was too much
> > information :)
> 
> No, that was very helpful, thanks! Just the kind of detail I was after
> to understand your deployment scenario :)

OK, phew :)

> >> > I was planning to set the run order programatically on the XDP program
> >> > objects via libxdp calls. It looks like your libxdp implementation
> >> > already has ways to do this in the form of xdp_program__set_run_prio()
> >> > and xdp_program__chain_call_enabled().
> >> >
> >> > Does that make sense? This is still all very theoretical for me at this
> >> > point!
> >> 
> >> Yup, totally possible to set this programmatically with libxdp as well
> >> today. However, before doing so you still need to communicate the list
> >> of BPF programs and their run configuration to each node. And I'm
> >> thinking it may be worthwhile to specify how to do this as part of the
> >> "protocol" and also teach libxdp about the format, so others won't have
> >> to reinvent the same thing later.
> >
> > It seems like I must be missing something here, but my plan was to do
> > this all programmatically by calling libxdp functions from the
> > orchestration tool by 1) calling something like xdp_program__open_file()
> > to load the XDP program, and then 2) setting the run configuration by
> > calling something like xdp_program__set_run_prio() and
> > xdp_program__chain_call_enabled(), and 3) adding the programs to the
> > dispatcher and loading it.
> >
> > Correct me if I'm wrong, but I guess you're saying that it might be
> > worth creating an abstraction in libxdp where a user can pass in the
> > necessary config data and libxdp does the work, that I just summarized
> > in the previous paragraph, on the user's behalf. I can see how that
> > could be a useful abstraction.
> 
> Yeah, so what I was thinking was whether it would be useful to, for
> instance, define a "bundle" format that contains the config data that
> libxdp will understand. Could just be a JSON schema containing keys for
> priority, chain call actions and a filename, so you can just point
> xdp_program__open_file() at that and it will do the rest.
> 
> However, I'm still not quite sure I'm convinced that this will be
> generally useful. As you say, you can just as well just set the values
> programmatically after loading the file, and I suspect that different
> deployments will end up having too much custom stuff around this that
> they'll bother using such a facility anyway. WDYT?

Yeah, my hunch is that different deployments will have custom config
data and then getting the data into libxdp format will just be another
data conversion step. I could be wrong.

If it is decided to create a data format, then the JSON schema you
described is pretty much exactly what I had in mind, too. For whatever
that is worth!

> > I know in a previous e-mail you mentioned having a config file with
> > priority overrides. That's just not a use case that our team would want
> > to use. And, my opinion would be that the program using libxdp should be
> > the one to implement that sort of policy; it keeps libxdp more simple
> > without needing to worry about parsing config files (and handling config
> > version changes in the code and the spec). For example, xdp-loader could
> > have a config file with priority overrides and people could use that
> > code if they wanted to do something similar.
> 
> Yeah, that's totally what would make sense for your deployment case. The
> design where libxdp reads a config file comes from my distro
> perspective: We want to build a system whereby different applications
> can each incorporate XDP functionality and co-exist; and the goal is to
> make libxdp the synchronisation point between them. I.e., we can say to
> application authors "just use libxdp when writing your application and
> it'll work", while at the same time empowering sysadmins to change the
> default application ordering.

Ah, yes, that perspective helps a lot; I understand much better why you
would want a centralized configuration file. My mind is actually kind of
melting now thinking about all the use cases.

Is the idea that the configuration file would have the final word? For
example, if there are multiple applications using libxdp (and possibly
even programatically overriding their priorities), could a sysadmin then
write a config file to dictate what they actually want? That would made
sense to me. If that's the goal, then I guess I would take back my
opinion about that policy not belonging in libxdp. I think application
writers would be less likely to use the configuration format (for
reasons we discussed above), but it does seem like a necessary mechanism
for sysadmins to orchestrate multiple applications that don't know about
each other. For my team's use case, we have the luxury of being the One
Application to rule all BPF programs.

> 
> By having that configuration be part of the library, applications can be
> free to use either the command-line loader or include the loading into
> their own user-space binary.
> 
> But since you are (notionally) both the application developer and system
> owner, that is less of a concern for you as you control the whole stack.
> 
> > Hopefully I'm even making sense, but like I said, I don't have strong
> > feelings about the format, as long as we are able to achieve our
> > required use case of programmatically setting the run configuration
> > values from a libxdp user program.
> 
> Sure, that you can certainly achieve with implementing what libxdp
> includes today. I'm just trying to make sure we explore any
> opportunities for standardising something useful so others can benefit
> from it as well; so I hope you'll forgive my probing :)

8< snip

> > We explicitly do not want defaults set by program authors. We want that
> > policy to be completely in the hands of the orchestration environment.
> 
> Right, OK. How does the admin configuring the orchestration system
> figure out which order to run programs in, BTW? Is this obvious from the
> nature of the programs, or do you document it out of band somewhere, or
> something like that?

We're a pretty huge organization... lots of DCs, public cloud, private
cloud, different kernel versions, sister companies, hundreds of
applications, etc. We want anyone to be able to write cool BPF programs
and userspace applications without needing awareness of what's running
before or after or if that order might change in the future. I'm sure
the desired order will be more obvious for some programs than others,
but we have administrators that can analyze the BPF programs, compose
multiple BPF programs together, and order and reorder them. We have a
team of people that can work with teams to resolve any interdependencies
if necessary.

As an example, we've done something similar for HTTP ingress and egress
Lua plugins in the past. We have dozens of teams that write Lua code to
do custom L7 things with HTTP requests and responses, and then we have a
UI where admins/ops folks can literally drag and drog the plugins into
the desired order. We wouldn't want teams making assumptions about what
order plugins should run in, either.

Hopefully that helps!

Thanks again,
Brian

Re: How to orchestrate multiple XDP programs

[Brian G. Merrell]

On 21/02/11 12:18PM, Toke Høiland-Jørgensen wrote:
> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
> 
> > On 21/01/29 01:02PM, Toke Høiland-Jørgensen wrote:
> >> Hi Brian
> >> 
> >> I've posted a first draft of this protocol description here:
> >> https://github.com/xdp-project/xdp-tools/blob/master/lib/libxdp/protocol.org
> >> 
> >> Please take a look and let me know what you think. And do feel free to
> >> point out any places that are unclear, as I said this is a first draft,
> >> and I'm expecting it to evolve as I get feedback from you and others :)
> >> 
> >> -Toke
> >> 
> >
> > Thanks so much for doing this Toke. There's a lot of great information.
> > I did one read-through, and didn't notice any surprises compared to the
> > code that I've read so far.
> 
> Awesome! :)

A question for anyone (sorry if it's a silly one)...

I did a second read-through of the protocol this evening. I wanted to
take a deeper look at the function calls that are referenced. Some of
them are BPF syscalls, which should be relatively straightforward to
interface with from Go. However, some functions like
bpf_get_link_xdp_info() appear to reside deep in the bowels of libbpf.
I'd really like to avoid needing cgo bindings, so my question is if
there some way to 1) interface with these functions that I'm just not
seeing, or 2) achieve what's necessary for implementing libxdp by only
utilizing syscalls.

I'm definitely open to insight from anyone with experience in this area.

Thanks,
Brian

Re: How to orchestrate multiple XDP programs

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> On 21/02/11 12:18PM, Toke Høiland-Jørgensen wrote:
>> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
>> 
>> > On 21/01/29 01:02PM, Toke Høiland-Jørgensen wrote:
>> >> Hi Brian
>> >> 
>> >> I've posted a first draft of this protocol description here:
>> >> https://github.com/xdp-project/xdp-tools/blob/master/lib/libxdp/protocol.org
>> >> 
>> >> Please take a look and let me know what you think. And do feel free to
>> >> point out any places that are unclear, as I said this is a first draft,
>> >> and I'm expecting it to evolve as I get feedback from you and others :)
>> >> 
>> >> -Toke
>> >> 
>> >
>> > Thanks so much for doing this Toke. There's a lot of great information.
>> > I did one read-through, and didn't notice any surprises compared to the
>> > code that I've read so far.
>> 
>> Awesome! :)
>
> A question for anyone (sorry if it's a silly one)...
>
> I did a second read-through of the protocol this evening. I wanted to
> take a deeper look at the function calls that are referenced. Some of
> them are BPF syscalls, which should be relatively straightforward to
> interface with from Go. However, some functions like
> bpf_get_link_xdp_info() appear to reside deep in the bowels of libbpf.
> I'd really like to avoid needing cgo bindings, so my question is if
> there some way to 1) interface with these functions that I'm just not
> seeing, or 2) achieve what's necessary for implementing libxdp by only
> utilizing syscalls.

bpf_get_link_xdp_info() is a wrapper around the kernel rt_netlink
interface. It issues an RTM_GETLINK with flags NLM_F_DUMP |
NLM_F_REQUEST and parses the IFLA_XDP attribute on the return value to
extract the program ID of the currently attached XDP program. I'm pretty
sure you can find an existing netlink library for Go, so this should be
pretty straight-forward to implement.

But point well taken that a document such as the protocol doc should
refer to the kernel interface and not the libbpf internals - I'll fix
that :)

-Toke

Re: How to orchestrate multiple XDP programs

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

>> Yeah, so what I was thinking was whether it would be useful to, for
>> instance, define a "bundle" format that contains the config data that
>> libxdp will understand. Could just be a JSON schema containing keys for
>> priority, chain call actions and a filename, so you can just point
>> xdp_program__open_file() at that and it will do the rest.
>> 
>> However, I'm still not quite sure I'm convinced that this will be
>> generally useful. As you say, you can just as well just set the values
>> programmatically after loading the file, and I suspect that different
>> deployments will end up having too much custom stuff around this that
>> they'll bother using such a facility anyway. WDYT?
>
> Yeah, my hunch is that different deployments will have custom config
> data and then getting the data into libxdp format will just be another
> data conversion step. I could be wrong.
>
> If it is decided to create a data format, then the JSON schema you
> described is pretty much exactly what I had in mind, too. For whatever
> that is worth!

Right, I'll mull this over some more but let's defer it for now.

>> > I know in a previous e-mail you mentioned having a config file with
>> > priority overrides. That's just not a use case that our team would want
>> > to use. And, my opinion would be that the program using libxdp should be
>> > the one to implement that sort of policy; it keeps libxdp more simple
>> > without needing to worry about parsing config files (and handling config
>> > version changes in the code and the spec). For example, xdp-loader could
>> > have a config file with priority overrides and people could use that
>> > code if they wanted to do something similar.
>> 
>> Yeah, that's totally what would make sense for your deployment case. The
>> design where libxdp reads a config file comes from my distro
>> perspective: We want to build a system whereby different applications
>> can each incorporate XDP functionality and co-exist; and the goal is to
>> make libxdp the synchronisation point between them. I.e., we can say to
>> application authors "just use libxdp when writing your application and
>> it'll work", while at the same time empowering sysadmins to change the
>> default application ordering.
>
> Ah, yes, that perspective helps a lot; I understand much better why you
> would want a centralized configuration file. My mind is actually kind of
> melting now thinking about all the use cases.
>
> Is the idea that the configuration file would have the final word? For
> example, if there are multiple applications using libxdp (and possibly
> even programatically overriding their priorities), could a sysadmin then
> write a config file to dictate what they actually want? That would made
> sense to me. If that's the goal, then I guess I would take back my
> opinion about that policy not belonging in libxdp. I think application
> writers would be less likely to use the configuration format (for
> reasons we discussed above), but it does seem like a necessary mechanism
> for sysadmins to orchestrate multiple applications that don't know about
> each other.

Yeah, that would be the idea.

> For my team's use case, we have the luxury of being the One
> Application to rule all BPF programs.

Yup, obviously being in full control of the system gives you some
options we don't quite have as a distro. But see below - I still think
there are some similarities.

>> By having that configuration be part of the library, applications can be
>> free to use either the command-line loader or include the loading into
>> their own user-space binary.
>> 
>> But since you are (notionally) both the application developer and system
>> owner, that is less of a concern for you as you control the whole stack.
>> 
>> > Hopefully I'm even making sense, but like I said, I don't have strong
>> > feelings about the format, as long as we are able to achieve our
>> > required use case of programmatically setting the run configuration
>> > values from a libxdp user program.
>> 
>> Sure, that you can certainly achieve with implementing what libxdp
>> includes today. I'm just trying to make sure we explore any
>> opportunities for standardising something useful so others can benefit
>> from it as well; so I hope you'll forgive my probing :)
>
> 8< snip
>
>> > We explicitly do not want defaults set by program authors. We want that
>> > policy to be completely in the hands of the orchestration environment.
>> 
>> Right, OK. How does the admin configuring the orchestration system
>> figure out which order to run programs in, BTW? Is this obvious from the
>> nature of the programs, or do you document it out of band somewhere, or
>> something like that?
>
> We're a pretty huge organization... lots of DCs, public cloud, private
> cloud, different kernel versions, sister companies, hundreds of
> applications, etc. We want anyone to be able to write cool BPF
> programs and userspace applications without needing awareness of
> what's running before or after or if that order might change in the
> future. I'm sure the desired order will be more obvious for some
> programs than others, but we have administrators that can analyze the
> BPF programs, compose multiple BPF programs together, and order and
> reorder them. We have a team of people that can work with teams to
> resolve any interdependencies if necessary.
>
> As an example, we've done something similar for HTTP ingress and
> egress Lua plugins in the past. We have dozens of teams that write Lua
> code to do custom L7 things with HTTP requests and responses, and then
> we have a UI where admins/ops folks can literally drag and drog the
> plugins into the desired order. We wouldn't want teams making
> assumptions about what order plugins should run in, either.


See, so this is the part that's actually analogous to what we want to do
as a distro. Except the people writing the cool BPF programs are
different software vendors and open source projects, not different
divisions within the same sprawling org. But in a sense the situation is
quite similar.

So thinking a bit more about the difference between your orchestration
system and the model I've been working from, I think the biggest
difference is not that you are assuming control of a system with an
orchestration system. In a sense a distro is also an orchestration
system bringing together different software from different sources.

No, I think the main difference is that in the model you described,
you're assuming that your orchestration system would install the XDP
program on behalf of the application as well as launch the userspace
bits. Whereas I'm assuming that an application that uses XDP will start
in userspace (launched by systemd, most likely), and will then load its
own XDP program after possibly doing some initialisation first (e.g.,
pre-populating maps, that sort of thing).

From what I've understood from what you explained about your setup, your
model could work with both models as well; so why are you assuming that
applications won't want to install their own XDP programs? :)

-Toke

Re: How to orchestrate multiple XDP programs

[Brian G. Merrell]

On 21/02/18 05:20PM, Toke Høiland-Jørgensen wrote:
> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> >> > We explicitly do not want defaults set by program authors. We want that
> >> > policy to be completely in the hands of the orchestration environment.
> >> 
> >> Right, OK. How does the admin configuring the orchestration system
> >> figure out which order to run programs in, BTW? Is this obvious from the
> >> nature of the programs, or do you document it out of band somewhere, or
> >> something like that?
> >
> > We're a pretty huge organization... lots of DCs, public cloud, private
> > cloud, different kernel versions, sister companies, hundreds of
> > applications, etc. We want anyone to be able to write cool BPF
> > programs and userspace applications without needing awareness of
> > what's running before or after or if that order might change in the
> > future. I'm sure the desired order will be more obvious for some
> > programs than others, but we have administrators that can analyze the
> > BPF programs, compose multiple BPF programs together, and order and
> > reorder them. We have a team of people that can work with teams to
> > resolve any interdependencies if necessary.
> >
> > As an example, we've done something similar for HTTP ingress and
> > egress Lua plugins in the past. We have dozens of teams that write Lua
> > code to do custom L7 things with HTTP requests and responses, and then
> > we have a UI where admins/ops folks can literally drag and drog the
> > plugins into the desired order. We wouldn't want teams making
> > assumptions about what order plugins should run in, either.
> 
> 
> See, so this is the part that's actually analogous to what we want to do
> as a distro. Except the people writing the cool BPF programs are
> different software vendors and open source projects, not different
> divisions within the same sprawling org. But in a sense the situation is
> quite similar.
> 
> So thinking a bit more about the difference between your orchestration
> system and the model I've been working from, I think the biggest
> difference is not that you are assuming control of a system with an
> orchestration system. In a sense a distro is also an orchestration
> system bringing together different software from different sources.
> 
> No, I think the main difference is that in the model you described,
> you're assuming that your orchestration system would install the XDP
> program on behalf of the application as well as launch the userspace
> bits.

Yes, that's right. This is the model we are implementing.

> Whereas I'm assuming that an application that uses XDP will start
> in userspace (launched by systemd, most likely), and will then load its
> own XDP program after possibly doing some initialisation first (e.g.,
> pre-populating maps, that sort of thing).
> 
> From what I've understood from what you explained about your setup, your
> model could work with both models as well; so why are you assuming that
> applications won't want to install their own XDP programs? :)

I would just say that in our organizations network and administration
environment, we ideally want a centralized orchestration tooling and
control plane that is used for all XDP (and tc) programs running on our
machines with our model described above.

That said, I do see your point about the possibility of using some other
application that runs its own XDP programs, and then, yes, we would
definitely want some way to control the priority. Ideally, the
application would have its own configuration to set priorities, but I do
think the system configuration file is a good way to ensure that the
sysadmin does have the power to override if necessary.

I think you're right that both models should be able to be used. Thanks
for the good discussion.

Thanks,
Brian

Re: How to orchestrate multiple XDP programs

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> On 21/02/18 05:20PM, Toke Høiland-Jørgensen wrote:
>> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
>
>> >> > We explicitly do not want defaults set by program authors. We want that
>> >> > policy to be completely in the hands of the orchestration environment.
>> >> 
>> >> Right, OK. How does the admin configuring the orchestration system
>> >> figure out which order to run programs in, BTW? Is this obvious from the
>> >> nature of the programs, or do you document it out of band somewhere, or
>> >> something like that?
>> >
>> > We're a pretty huge organization... lots of DCs, public cloud, private
>> > cloud, different kernel versions, sister companies, hundreds of
>> > applications, etc. We want anyone to be able to write cool BPF
>> > programs and userspace applications without needing awareness of
>> > what's running before or after or if that order might change in the
>> > future. I'm sure the desired order will be more obvious for some
>> > programs than others, but we have administrators that can analyze the
>> > BPF programs, compose multiple BPF programs together, and order and
>> > reorder them. We have a team of people that can work with teams to
>> > resolve any interdependencies if necessary.
>> >
>> > As an example, we've done something similar for HTTP ingress and
>> > egress Lua plugins in the past. We have dozens of teams that write Lua
>> > code to do custom L7 things with HTTP requests and responses, and then
>> > we have a UI where admins/ops folks can literally drag and drog the
>> > plugins into the desired order. We wouldn't want teams making
>> > assumptions about what order plugins should run in, either.
>> 
>> 
>> See, so this is the part that's actually analogous to what we want to do
>> as a distro. Except the people writing the cool BPF programs are
>> different software vendors and open source projects, not different
>> divisions within the same sprawling org. But in a sense the situation is
>> quite similar.
>> 
>> So thinking a bit more about the difference between your orchestration
>> system and the model I've been working from, I think the biggest
>> difference is not that you are assuming control of a system with an
>> orchestration system. In a sense a distro is also an orchestration
>> system bringing together different software from different sources.
>> 
>> No, I think the main difference is that in the model you described,
>> you're assuming that your orchestration system would install the XDP
>> program on behalf of the application as well as launch the userspace
>> bits.
>
> Yes, that's right. This is the model we are implementing.
>
>> Whereas I'm assuming that an application that uses XDP will start
>> in userspace (launched by systemd, most likely), and will then load its
>> own XDP program after possibly doing some initialisation first (e.g.,
>> pre-populating maps, that sort of thing).
>> 
>> From what I've understood from what you explained about your setup, your
>> model could work with both models as well; so why are you assuming that
>> applications won't want to install their own XDP programs? :)
>
> I would just say that in our organizations network and administration
> environment, we ideally want a centralized orchestration tooling and
> control plane that is used for all XDP (and tc) programs running on our
> machines with our model described above.

Right, sure, I'm not disputing this model is useful as well, I'm just
wondering about how you envision the details working. Say your
orchestration system installs an XDP program on behalf of an application
and then launches the userspace component (assuming one exists). How is
that userspace program supposed to obtain a file descriptor for the
map(s) used by the XDP program in order to communicate with it?

> That said, I do see your point about the possibility of using some
> other application that runs its own XDP programs, and then, yes, we
> would definitely want some way to control the priority. Ideally, the
> application would have its own configuration to set priorities, but I
> do think the system configuration file is a good way to ensure that
> the sysadmin does have the power to override if necessary.
>
> I think you're right that both models should be able to be used.
> Thanks for the good discussion.

Yeah, you too! Always good to have someone to bounce ideas off of :)

-Toke

Re: How to orchestrate multiple XDP programs

[Brian G. Merrell]

On 21/02/22 11:41PM, Toke Høiland-Jørgensen wrote:
> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
> > On 21/02/18 05:20PM, Toke Høiland-Jørgensen wrote:

> >> No, I think the main difference is that in the model you described,
> >> you're assuming that your orchestration system would install the XDP
> >> program on behalf of the application as well as launch the userspace
> >> bits.
> >
> > Yes, that's right. This is the model we are implementing.
> >
> >> Whereas I'm assuming that an application that uses XDP will start
> >> in userspace (launched by systemd, most likely), and will then load its
> >> own XDP program after possibly doing some initialisation first (e.g.,
> >> pre-populating maps, that sort of thing).
> >> 
> >> From what I've understood from what you explained about your setup, your
> >> model could work with both models as well; so why are you assuming that
> >> applications won't want to install their own XDP programs? :)
> >
> > I would just say that in our organizations network and administration
> > environment, we ideally want a centralized orchestration tooling and
> > control plane that is used for all XDP (and tc) programs running on our
> > machines with our model described above.
> 
> Right, sure, I'm not disputing this model is useful as well, I'm just
> wondering about how you envision the details working. Say your
> orchestration system installs an XDP program on behalf of an application
> and then launches the userspace component (assuming one exists). How is
> that userspace program supposed to obtain a file descriptor for the
> map(s) used by the XDP program in order to communicate with it?

OK, so this part is admittedly a little hand-wavy and a work in
progress. We're literally working on design and proof of concepts right
now, but this is basically what we're envisioning:

1. Orchestration tool gets all its JSON config data, which includes
   remote paths for BPF programs and any respective userspace
   programs.
2. Orchestration tool downloads BPF programs and loads them (using
   Go libxdp when it's available). Then (and this is where I'm going to
   start waving my hands) the orchestrator will need to gather any
   necessary map names/ids/fds information to be send to the userspace
   program. I'm just not exactly sure how easy/hard/possible this part
   is.
3. We start the userspace programs as separate processes and communicate
   with them via RPC (there's a nice Go plugin system for this[1]). Each
   userspace program implements an interface and we communicate the map
   info (among other things) over RPC to the userspace program when it
   starts.

I'm going to continue researching and fleshing out the details, but are
there any obvious problems with this approach? A backup plan is to have
the userspace programs do the loading of the BPF program, but it's not
obvious to me how that would be easier to obtain the file descriptor for
the map(s) vs. having the orchestrator figure it out and send it to the
userspace process.

If it works out that the orchestrator can load the BPF programs on
behalf of the userspace programs, then I think the primary benefit is
that the developer of the userspace program doesn't need to follow some
boilerplate to load the appropriate way--we've done all that for them.
It seems nice that the orchestrator could be the one interface with
libxdp (for the XDP case) without every userspace program needing to
doing it's own adding/removing (and thus dispatcher swapping), though I
would guess that's not really a problem at all.

I feel like I've gone out of the scope of libxdp in this e-mail, but you
did ask :) And I do appreciate any feedback or raising of red flags.

Thanks,
Brian

[1] https://github.com/hashicorp/go-plugin

Re: How to orchestrate multiple XDP programs

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> On 21/02/22 11:41PM, Toke Høiland-Jørgensen wrote:
>> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
>> > On 21/02/18 05:20PM, Toke Høiland-Jørgensen wrote:
>
>> >> No, I think the main difference is that in the model you described,
>> >> you're assuming that your orchestration system would install the XDP
>> >> program on behalf of the application as well as launch the userspace
>> >> bits.
>> >
>> > Yes, that's right. This is the model we are implementing.
>> >
>> >> Whereas I'm assuming that an application that uses XDP will start
>> >> in userspace (launched by systemd, most likely), and will then load its
>> >> own XDP program after possibly doing some initialisation first (e.g.,
>> >> pre-populating maps, that sort of thing).
>> >> 
>> >> From what I've understood from what you explained about your setup, your
>> >> model could work with both models as well; so why are you assuming that
>> >> applications won't want to install their own XDP programs? :)
>> >
>> > I would just say that in our organizations network and administration
>> > environment, we ideally want a centralized orchestration tooling and
>> > control plane that is used for all XDP (and tc) programs running on our
>> > machines with our model described above.
>> 
>> Right, sure, I'm not disputing this model is useful as well, I'm just
>> wondering about how you envision the details working. Say your
>> orchestration system installs an XDP program on behalf of an application
>> and then launches the userspace component (assuming one exists). How is
>> that userspace program supposed to obtain a file descriptor for the
>> map(s) used by the XDP program in order to communicate with it?
>
> OK, so this part is admittedly a little hand-wavy and a work in
> progress. We're literally working on design and proof of concepts right
> now, but this is basically what we're envisioning:
>
> 1. Orchestration tool gets all its JSON config data, which includes
>    remote paths for BPF programs and any respective userspace
>    programs.
> 2. Orchestration tool downloads BPF programs and loads them (using
>    Go libxdp when it's available). Then (and this is where I'm going to
>    start waving my hands) the orchestrator will need to gather any
>    necessary map names/ids/fds information to be send to the userspace
>    program. I'm just not exactly sure how easy/hard/possible this part
>    is.
> 3. We start the userspace programs as separate processes and communicate
>    with them via RPC (there's a nice Go plugin system for this[1]). Each
>    userspace program implements an interface and we communicate the map
>    info (among other things) over RPC to the userspace program when it
>    starts.
>
> I'm going to continue researching and fleshing out the details, but are
> there any obvious problems with this approach?

I think the basic idea can work (it's similar to systemd's socket
activation, which also passes the socket fd to the userspace process on
launch). However, there are a couple of things that become impossible
for the userspace process to do in this model:

- Modifying the BPF object before load: Libbpf does quite a few
  transformations on the bytecode to handle relocations, and it's also
  going to grow a full linker at some point. This is not a problem if
  the userspace program just lets libbpf do the default thing, but if it
  wants to customise the operations it becomes a problem. The obvious
  use case for this that comes to mind is dynamically omitting parts of
  the code for features that are not enabled (like we do in xdp-filter).

- Populating maps before load: This is necessary to use customised
  'const' global variables: The map backing these are frozen on load to
  allow the verifier to make strong assumptions about their content, so
  you can't modify them after the map is loaded.

- Atomic map population: Say you have an XDP program that reacts to
  traffic steering rules, and you start out with the program being
  attached to the interface and an empty map that userspace then has to
  populate. While the map is being populated, the XDP program will
  process some packets with an incomplete view of the final ruleset.
  Whereas if you can populate the map completely before attaching the
  program you can be sure that it's consistent. Depending on the nature
  of the application, this may lead to weird effects, or it may be
  mostly harmless.

Now, all of these could in principle be performed by the orchestrator on
behalf of the program, but that means you'll have to make the
orchestrator more complex, and you'll have to come up with a way to
express these operations in your configuration language.

> A backup plan is to have the userspace programs do the loading of the
> BPF program, but it's not obvious to me how that would be easier to
> obtain the file descriptor for the map(s) vs. having the orchestrator
> figure it out and send it to the userspace process.

Both approaches carry complexities with it, and I'm not sure there
really is a universal right answer. What tipped the scales for me were
the issues above. However, being in a more controlled environment, the
trade off may well be different for you.

> If it works out that the orchestrator can load the BPF programs on
> behalf of the userspace programs, then I think the primary benefit is
> that the developer of the userspace program doesn't need to follow
> some boilerplate to load the appropriate way--we've done all that for
> them. It seems nice that the orchestrator could be the one interface
> with libxdp (for the XDP case) without every userspace program needing
> to doing it's own adding/removing (and thus dispatcher swapping),
> though I would guess that's not really a problem at all.

Yeah, I do agree that it would be nicer if there was a clean interface
the application could talk to without having to muck about with
dispatchers. My hope is that by encapsulating all that in a library we
can pretend that there is :)

However, I can see how this perspective may also be different in a Go
world: With a C library in a distro we can ship the library as a
separate package and as long as we maintain ABI compatibility, we can
upgrade the library independently of the applications. Whereas with Go's
vendoring approach it becomes way harder to ensure that all applications
use the "right" version of the library. In that sense, a C library is
more of a "system service" whereas a Go library is more of an
"application sub-function".

> I feel like I've gone out of the scope of libxdp in this e-mail, but
> you did ask :) And I do appreciate any feedback or raising of red
> flags.

Only slightly :)
As outlined above all of this has gone into my thinking when designing
libxdp, so I appreciate the chance to get your perspectives on these
very real tradeoffs. So thanks again for taking the time to explain your
thought process!

-Toke