Re: [PATCH bpf-next 04/10] bpf: tcp: Allow bpf prog to write and parse BPF TCP header option

[Martin KaFai Lau <kafai@fb.com>]

On Sun, Jun 28, 2020 at 05:34:48PM -0700, Martin KaFai Lau wrote:
> On Sun, Jun 28, 2020 at 11:24:27AM -0700, Alexei Starovoitov wrote:
> > On Fri, Jun 26, 2020 at 10:55:26AM -0700, Martin KaFai Lau wrote:
> > > 
> > > Parsing BPF Header Option
> > > ─────────────────────────
> > > 
> > > As mentioned earlier, the received SYN/SYNACK/ACK during the 3WHS
> > > will be available to some specific CB (e.g. the *_ESTABLISHED_CB)
> > > 
> > > For established connection, if the kernel finds a bpf header
> > > option (i.e. option with kind:254 and magic:0xeB9F) and the
> > > the "PARSE_HDR_OPT_CB_FLAG" flag is set,  the
> > > bpf prog will be called in the "BPF_SOCK_OPS_PARSE_HDR_OPT_CB" op.
> > > The received skb will be available through sock_ops->skb_data
> > > and the bpf header option offset will also be specified
> > > in sock_ops->skb_bpf_hdr_opt_off.
> > 
> > TCP noob question:
> > - can tcp header have two or more options with the same kind and magic?
> > I scanned draft-ietf-tcpm-experimental-options-00.txt and it seems
> > it's not prohibiting collisions.
> > So should be ok?
> I also think it is ok.  Regardless of kind, the kernel's tcp_parse_options()
> seems to be ok on duplication also.
> 
> > Why I'm asking... I think existing bpf_sock_ops style of running
> > multiple bpf progs is gonna be awkward to use.
> > Picking the max of bpf_reserve_hdr_opt() from many calls and let
> > parent bpf progs override children written headers feels a bit hackish.
> > I feel the users will thank us if we design the progs to be more
> > isolated and independent somehow.
> > I was thinking may be each bpf prog will bpf_reserve_hdr_opt()
> > and bpf_store_hdr_opt() into their own option?
> > Then during option writing side the tcp header will have two or more
> > options with the same kind and magic.
> > Obviously it creates a headache during parsing. Which bpf prog
> > should be called for each option?
> > 
> > I suspect tcp draft actually prefers all options to have unique kind+magic.
> > Can we add an attribute to prog load time that will request particular magic ?
> > Then only that _one_ program will be called for the given kind+magic.
> > We can still have multiple progs attached to a cgroup (likely root cgroup)
> > and different progs will take care of parsing and writing their own option.
> > cgroup attaching side can make sure that multi progs have different magics.
> Interesting idea.
> 
> If the magic can be specified at load time,
> may be extend this for the "length" requirement too.  At load time,
> both magic and length should be specified.  The total length can
> be calculated during the attach time.  That will avoid making
> an extra call to bpf prog to learn the length.
> 
> If we don't limit magic, I think we should discuss if we need to limit the
> kind to 254 too.  How about we allow user to write any option kind?  That can
> save 2 byte magic from the limited TCP option spaces.  At load
> time, we can definitely reject the kind that the kernel is already
> writing, e.g. timestamp, sack...etc.
I have thought more about allowing only one kind per bpf prog at load time.

I think it is not ideal for some common cases in 3WHS.  For example,
when rolling/testing out a newer option to replace the old one,
the bpf@server may want to see if client supports option-A or option-B
from the SYN and then reply SYNACK accordingly with either option-A
or option-B.  It will be easier if it allows one bpf prog to make
the decision instead of having two bpf progs (one for option-A and one
for option-B) and may require these two bpf progs to co-ordinate with
each other.  The option length will not be static also.

The prog load attribute can be extended to take >1 kinds or may be
some arraymap convention can be used to do this.  However,
I am not sure that worths it considering most of the usecases is only
in 3WHS and checking for kind duplication in runtime may not be too bad
considering the TCP option space is only 40bytes and the option
has to be 4 bytes aligned.

I am thinking to allow the bpf prog to write multiple option kinds
and check for kind uniqueness at runtime.  That will include
checking the options already written by the kernel.  In SYN, there are
usually 4 options: mss, sackOK, TS, and wscale.

The bpf_store_hdr_opt API will be changed to:

long bpf_store_hdr_opt(struct bpf_sock_ops *skops, u8 kind, const void *from, u32 len, u64 flags)

It writes one _complete_ option at a time and "u8 kind" is required.
No offset is needed because it does not allow going back to rewrite
something that has already been written.

Thoughts?