Re: [PATCH bpf-next 04/10] bpf: tcp: Allow bpf prog to write and parse BPF TCP header option

[Martin KaFai Lau <kafai@fb.com>]

On Sun, Jun 28, 2020 at 11:24:27AM -0700, Alexei Starovoitov wrote:
> On Fri, Jun 26, 2020 at 10:55:26AM -0700, Martin KaFai Lau wrote:
> > 
> > Parsing BPF Header Option
> > ─────────────────────────
> > 
> > As mentioned earlier, the received SYN/SYNACK/ACK during the 3WHS
> > will be available to some specific CB (e.g. the *_ESTABLISHED_CB)
> > 
> > For established connection, if the kernel finds a bpf header
> > option (i.e. option with kind:254 and magic:0xeB9F) and the
> > the "PARSE_HDR_OPT_CB_FLAG" flag is set,  the
> > bpf prog will be called in the "BPF_SOCK_OPS_PARSE_HDR_OPT_CB" op.
> > The received skb will be available through sock_ops->skb_data
> > and the bpf header option offset will also be specified
> > in sock_ops->skb_bpf_hdr_opt_off.
> 
> TCP noob question:
> - can tcp header have two or more options with the same kind and magic?
> I scanned draft-ietf-tcpm-experimental-options-00.txt and it seems
> it's not prohibiting collisions.
> So should be ok?
I also think it is ok.  Regardless of kind, the kernel's tcp_parse_options()
seems to be ok on duplication also.

> Why I'm asking... I think existing bpf_sock_ops style of running
> multiple bpf progs is gonna be awkward to use.
> Picking the max of bpf_reserve_hdr_opt() from many calls and let
> parent bpf progs override children written headers feels a bit hackish.
> I feel the users will thank us if we design the progs to be more
> isolated and independent somehow.
> I was thinking may be each bpf prog will bpf_reserve_hdr_opt()
> and bpf_store_hdr_opt() into their own option?
> Then during option writing side the tcp header will have two or more
> options with the same kind and magic.
> Obviously it creates a headache during parsing. Which bpf prog
> should be called for each option?
> 
> I suspect tcp draft actually prefers all options to have unique kind+magic.
> Can we add an attribute to prog load time that will request particular magic ?
> Then only that _one_ program will be called for the given kind+magic.
> We can still have multiple progs attached to a cgroup (likely root cgroup)
> and different progs will take care of parsing and writing their own option.
> cgroup attaching side can make sure that multi progs have different magics.
Interesting idea.

If the magic can be specified at load time,
may be extend this for the "length" requirement too.  At load time,
both magic and length should be specified.  The total length can
be calculated during the attach time.  That will avoid making
an extra call to bpf prog to learn the length.

If we don't limit magic, I think we should discuss if we need to limit the
kind to 254 too.  How about we allow user to write any option kind?  That can
save 2 byte magic from the limited TCP option spaces.  At load
time, we can definitely reject the kind that the kernel is already
writing, e.g. timestamp, sack...etc.

When a tcp pkt is received at an established sk, this patch decides
if there is 0xeB9F option before calling into bpf prog.  That needs
to be adjusted as well.  It could be changed to: call into bpf prog
if there is option "kind" that the kernel cannot handle and the
PARSE_HDR_CB_FLAGS is set.

> Saving multiple bpf_hdr_opt_off in patch 2 for different magics becomes
> problematic. So clearly the implementation complexity shots through the roof
> with above proposal, but it feels more flexible and more user friendly?
> Not a strong opinion. The feature is great as-is.
If we go with the above route that multiple cgroup-bpf has multiple
kinds (or magics),  each of them has to parse the tcphdr to figure
out where is its own kind (or magic).  The intention of providing
bpf_hdr_opt_off in this patch is mostly for bpf-prog convenience only.
I think having it work nicer in multi bpf prog is the proper trade off.