Re: How to orchestrate multiple XDP programs

["Toke Høiland-Jørgensen" <toke@redhat.com>]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> On 21/01/29 01:02PM, Toke Høiland-Jørgensen wrote:
>> Hi Brian
>> 
>> I've posted a first draft of this protocol description here:
>> https://github.com/xdp-project/xdp-tools/blob/master/lib/libxdp/protocol.org
>> 
>> Please take a look and let me know what you think. And do feel free to
>> point out any places that are unclear, as I said this is a first draft,
>> and I'm expecting it to evolve as I get feedback from you and others :)
>> 
>> -Toke
>> 
>
> Thanks so much for doing this Toke. There's a lot of great information.
> I did one read-through, and didn't notice any surprises compared to the
> code that I've read so far.

Awesome! :)

> One thing I have been a little concerned about is the XDP_RUN_CONFIG in
> the xdp program function. For our case--with multiple teams writing
> independent, composable xdp programs--we don't want the XDP_RUN_CONFIG
> policy to be in the xdp program. Instead, we want the Go orchestration
> tool to have that policy as part of its configuration data (e.g., what
> order to run the xdp program functions in). From what I can tell, it's
> possible to omit the XDP_RUN_CONFIG from the xdp program function, and
> instead set the values when loading the xdp dispatcher. That's great, and
> thanks for the foresight there. I just want to confirm that I'm
> understanding that correctly, because it's very important for us.

Yes. The values embedded into the program BTF are defaults, and can be
overridden on load. The idea is that an application will set a default
value (e.g., "I'm a firewall, so I want to run early" or "I want to
monitor traffic to the stack so I'll run late"), but if the sysadmin
wants to do things differently they can override the order. The
important bit being that ultimate control of run order is up to the
*user*, not the application developer.

The policy override stuff is not implemented yet, but I am planning to
implement it by having libxdp read a config file with priority overrides
(similar to how libc will read /etc/nsswitch.conf or /etc/hosts which
makes them work in all applications).

And of course, if you're writing an orchestration tool, then you *are*
the user, so having the tool override priorities is definitely in scope
(it'll just be an alternative way to set policy instead of a config
file). How are you planning to specify the effective run order? I am
also quite open to working on a compatible way that can work for both
your tool and libxdp :)

> Also, I do hope that the existing Go BTF libraries are good enough to do
> what's needed here, because if I'm understand correctly, that's how I'll
> need to approach setting the XDP_RUN_CONFIG values for our use case.

You'll need to *parse* BTF to *read* the XDP_RUN_CONFIG. Which is pretty
basic, really, you just need to walk the BTF reference tree. Feel free
to reuse the parsing code in libxdp; that is, in turn, adapted from the
.maps section parsing code in libbpf :)

> I've been tasked to work on a Go libxdp implementation this quarter, so
> I'll be starting on that soon and let you know if I have questions as I
> go. I'm also happy to coordinate with anyone else that's interested.

Sounds great! Will be interesting to see a second implementation;
independent implementations are the ultimate test of any specification :)

Please do keep me in the loop, and don't hesitate to ping me if there
are things that are unclear or that you feel are less-than-ideal in the
way things work. I'm also quite open to evolving the spec to meet
everyone's needs!

-Toke