From: "Brian G. Merrell" <brian.g.merrell@gmail.com>
To: "Toke Høiland-Jørgensen" <toke@redhat.com>
Cc: xdp-newbies@vger.kernel.org,
"Jesper Dangaard Brouer" <brouer@redhat.com>,
"Maciej Żenczykowski" <maze@google.com>,
"Lorenz Bauer" <lmb@cloudflare.com>,
"Andrey Ignatov" <rdna@fb.com>,
bpf@vger.kernel.org
Subject: Re: How to orchestrate multiple XDP programs
Date: Tue, 23 Feb 2021 01:54:22 -0700 [thread overview]
Message-ID: <20210223085422.tv2gk6olg66zcbwe@snout.localdomain> (raw)
In-Reply-To: <87v9ajg1yx.fsf@toke.dk>
On 21/02/22 11:41PM, Toke Høiland-Jørgensen wrote:
> "Brian G. Merrell" <brian.g.merrell@gmail.com> writes:
> > On 21/02/18 05:20PM, Toke Høiland-Jørgensen wrote:
> >> No, I think the main difference is that in the model you described,
> >> you're assuming that your orchestration system would install the XDP
> >> program on behalf of the application as well as launch the userspace
> >> bits.
> >
> > Yes, that's right. This is the model we are implementing.
> >
> >> Whereas I'm assuming that an application that uses XDP will start
> >> in userspace (launched by systemd, most likely), and will then load its
> >> own XDP program after possibly doing some initialisation first (e.g.,
> >> pre-populating maps, that sort of thing).
> >>
> >> From what I've understood from what you explained about your setup, your
> >> model could work with both models as well; so why are you assuming that
> >> applications won't want to install their own XDP programs? :)
> >
> > I would just say that in our organizations network and administration
> > environment, we ideally want a centralized orchestration tooling and
> > control plane that is used for all XDP (and tc) programs running on our
> > machines with our model described above.
>
> Right, sure, I'm not disputing this model is useful as well, I'm just
> wondering about how you envision the details working. Say your
> orchestration system installs an XDP program on behalf of an application
> and then launches the userspace component (assuming one exists). How is
> that userspace program supposed to obtain a file descriptor for the
> map(s) used by the XDP program in order to communicate with it?
OK, so this part is admittedly a little hand-wavy and a work in
progress. We're literally working on design and proof of concepts right
now, but this is basically what we're envisioning:
1. Orchestration tool gets all its JSON config data, which includes
remote paths for BPF programs and any respective userspace
programs.
2. Orchestration tool downloads BPF programs and loads them (using
Go libxdp when it's available). Then (and this is where I'm going to
start waving my hands) the orchestrator will need to gather any
necessary map names/ids/fds information to be send to the userspace
program. I'm just not exactly sure how easy/hard/possible this part
is.
3. We start the userspace programs as separate processes and communicate
with them via RPC (there's a nice Go plugin system for this[1]). Each
userspace program implements an interface and we communicate the map
info (among other things) over RPC to the userspace program when it
starts.
I'm going to continue researching and fleshing out the details, but are
there any obvious problems with this approach? A backup plan is to have
the userspace programs do the loading of the BPF program, but it's not
obvious to me how that would be easier to obtain the file descriptor for
the map(s) vs. having the orchestrator figure it out and send it to the
userspace process.
If it works out that the orchestrator can load the BPF programs on
behalf of the userspace programs, then I think the primary benefit is
that the developer of the userspace program doesn't need to follow some
boilerplate to load the appropriate way--we've done all that for them.
It seems nice that the orchestrator could be the one interface with
libxdp (for the XDP case) without every userspace program needing to
doing it's own adding/removing (and thus dispatcher swapping), though I
would guess that's not really a problem at all.
I feel like I've gone out of the scope of libxdp in this e-mail, but you
did ask :) And I do appreciate any feedback or raising of red flags.
Thanks,
Brian
[1] https://github.com/hashicorp/go-plugin
next prev parent reply other threads:[~2021-02-23 8:55 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20201201091203.ouqtpdmvvl2m2pga@snout.localdomain>
2020-12-01 12:08 ` How to orchestrate multiple XDP programs Toke Høiland-Jørgensen
2020-12-16 7:29 ` Brian G. Merrell
2020-12-16 12:45 ` Toke Høiland-Jørgensen
[not found] ` <87tur0x874.fsf@toke.dk>
2021-02-10 22:27 ` Brian G. Merrell
2021-02-11 11:18 ` Toke Høiland-Jørgensen
2021-02-12 6:51 ` Brian G. Merrell
2021-02-15 12:47 ` Toke Høiland-Jørgensen
2021-02-17 1:20 ` Brian G. Merrell
2021-02-17 15:53 ` Toke Høiland-Jørgensen
2021-02-17 22:27 ` Brian G. Merrell
2021-02-18 16:20 ` Toke Høiland-Jørgensen
2021-02-22 19:34 ` Brian G. Merrell
2021-02-22 22:41 ` Toke Høiland-Jørgensen
2021-02-23 8:54 ` Brian G. Merrell [this message]
2021-02-23 11:07 ` Toke Høiland-Jørgensen
2021-02-18 10:16 ` Brian G. Merrell
2021-02-18 11:00 ` Toke Høiland-Jørgensen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210223085422.tv2gk6olg66zcbwe@snout.localdomain \
--to=brian.g.merrell@gmail.com \
--cc=bpf@vger.kernel.org \
--cc=brouer@redhat.com \
--cc=lmb@cloudflare.com \
--cc=maze@google.com \
--cc=rdna@fb.com \
--cc=toke@redhat.com \
--cc=xdp-newbies@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).