combining sockmap + ktls

[Willem de Bruijn <willemdebruijn.kernel@gmail.com>]

I've been playing with sockmap and ktls. They're fantastic tools.
Combining them I did run into a few issues. Would like to understand
whether (a) it's just me, else (b) whether these are known issues and
(c) some feedback on an initial hacky patch.

My test [1] sets up an echo request/response between a client and
server, optionally interposed by an "icept" guard process on each side
and optionally enabling ktls between the icept processes.

Without ktls, most variants of interpositioning {iptables, iptables +
splice(), iptables + sockmap splice, sk_msg to icept tx } work.

Only sk_msg redirection to icept ingress with BPF_F_INGRESS does not
if the destination socket has a verdict program. I *think* this is
intentional, judging from commit 552de9106882 ("bpf: sk_msg, fix
socket data_ready events") explicitly ensuring that the process gets
awoken on new data if a socket has a verdict program and another
socket redirects to it, as opposed to passing it to the program.

For this workload, more interesting is sk_msg directly to icept
egress, anyway. This works without ktls. Support for ktls is added in
commit d3b18ad31f93 ("tls: add bpf support to sk_msg handling"). The
relevant callback function tls_sw_sendpage_locked was not immediately
used and subsequently removed in commit cc1dbdfed023 ("Revert
"net/tls: remove unused function tls_sw_sendpage_locked""). It appears
to work once reverting that change, plus registering the function

        @@ -859,6 +861,7 @@ static int __init tls_register(void)

                tls_sw_proto_ops = inet_stream_ops;
                tls_sw_proto_ops.splice_read = tls_sw_splice_read;
        +       tls_sw_proto_ops.sendpage_locked   = tls_sw_sendpage_locked,

and additionally allowing MSG_NO_SHARED_FRAGS:

         int tls_sw_sendpage_locked(struct sock *sk, struct page *page,
                                    int offset, size_t size, int flags)
         {
               if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL |
        -                     MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY))
        +                     MSG_SENDPAGE_NOTLAST |
MSG_SENDPAGE_NOPOLICY | MSG_NO_SHARED_FRAGS))
                         return -ENOTSUPP;

and not registering parser+verdict programs on the destination socket.
Note that without ktls this mode also works with such programs
attached.

Lastly, sockmap splicing from icept ingress to egress (no sk_msg) also
stops working when I enable ktls on the egress socket. I'm taking a
look at that next. But this email is long enough already ;)

Thanks for having a look!

  Willem

[1] https://github.com/wdebruij/kerneltools/tree/icept.2

probably more readable is the stack of commits, one per feature:

  c86c112 icept: initial client/server test
  727a8ae icept: add iptables interception
  60c34b2 icept: add splice interception
  03a516a icept: add sockmap interception
  c9c6103 icept: run client and server in cgroup
  579bcae icept: add skmsg interception
  e1b0d17 icept: add kTLS