[PATCH net-next v3 0/7] tls: rx: decrypt from the TCP queue

[PATCH net-next v3 0/7] tls: rx: decrypt from the TCP queue

[Jakub Kicinski]

This is the final part of my TLS Rx rework. It switches from
strparser to decrypting data from skbs queued in TCP. We don't
need the full strparser for TLS, its needs are very basic.
This set gives us a small but measurable (6%) performance
improvement (continuous stream).

v2: drop the __exit marking for the unroll path
v3: drop tcp_recv_skb() in patch 5

Jakub Kicinski (7):
  tls: rx: wrap recv_pkt accesses in helpers
  tls: rx: factor SW handling out of tls_rx_one_record()
  tls: rx: don't free the output in case of zero-copy
  tls: rx: device: keep the zero copy status with offload
  tcp: allow tls to decrypt directly from the tcp rcv queue
  tls: rx: device: add input CoW helper
  tls: rx: do not use the standard strparser

 include/net/tcp.h    |   2 +
 include/net/tls.h    |  19 +-
 net/ipv4/tcp.c       |  42 +++-
 net/tls/tls.h        |  29 ++-
 net/tls/tls_device.c |  19 +-
 net/tls/tls_main.c   |  20 +-
 net/tls/tls_strp.c   | 488 ++++++++++++++++++++++++++++++++++++++++++-
 net/tls/tls_sw.c     | 228 +++++++++++---------
 8 files changed, 723 insertions(+), 124 deletions(-)

-- 
2.37.1

[PATCH net-next v3 1/7] tls: rx: wrap recv_pkt accesses in helpers

[Jakub Kicinski]

To allow for the logic to change later wrap accesses
which interrogate the input skb in helper functions.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: borisp@nvidia.com
CC: john.fastabend@gmail.com
---
 net/tls/tls.h    |  5 +++++
 net/tls/tls_sw.c | 11 ++++++-----
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/net/tls/tls.h b/net/tls/tls.h
index 3740740504e3..24bec1c5f1e8 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -142,6 +142,11 @@ static inline struct sk_buff *tls_strp_msg(struct tls_sw_context_rx *ctx)
 	return ctx->recv_pkt;
 }
 
+static inline bool tls_strp_msg_ready(struct tls_sw_context_rx *ctx)
+{
+	return ctx->recv_pkt;
+}
+
 #ifdef CONFIG_TLS_DEVICE
 int tls_device_init(void);
 void tls_device_cleanup(void);
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index ed5e6f1df9c7..cb99fc11997b 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -1289,7 +1289,7 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
 	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
 	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 
-	while (!ctx->recv_pkt) {
+	while (!tls_strp_msg_ready(ctx)) {
 		if (!sk_psock_queue_empty(psock))
 			return 0;
 
@@ -1298,7 +1298,7 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
 
 		if (!skb_queue_empty(&sk->sk_receive_queue)) {
 			__strp_unpause(&ctx->strp);
-			if (ctx->recv_pkt)
+			if (tls_strp_msg_ready(ctx))
 				break;
 		}
 
@@ -1314,7 +1314,8 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
 		add_wait_queue(sk_sleep(sk), &wait);
 		sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
 		sk_wait_event(sk, &timeo,
-			      ctx->recv_pkt || !sk_psock_queue_empty(psock),
+			      tls_strp_msg_ready(ctx) ||
+			      !sk_psock_queue_empty(psock),
 			      &wait);
 		sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
 		remove_wait_queue(sk_sleep(sk), &wait);
@@ -1907,7 +1908,7 @@ int tls_sw_recvmsg(struct sock *sk,
 	zc_capable = !bpf_strp_enabled && !is_kvec && !is_peek &&
 		ctx->zc_capable;
 	decrypted = 0;
-	while (len && (decrypted + copied < target || ctx->recv_pkt)) {
+	while (len && (decrypted + copied < target || tls_strp_msg_ready(ctx))) {
 		struct tls_decrypt_arg darg;
 		int to_decrypt, chunk;
 
@@ -2158,7 +2159,7 @@ bool tls_sw_sock_is_readable(struct sock *sk)
 		ingress_empty = list_empty(&psock->ingress_msg);
 	rcu_read_unlock();
 
-	return !ingress_empty || ctx->recv_pkt ||
+	return !ingress_empty || tls_strp_msg_ready(ctx) ||
 		!skb_queue_empty(&ctx->rx_list);
 }
 
-- 
2.37.1

[PATCH net-next v3 2/7] tls: rx: factor SW handling out of tls_rx_one_record()

[Jakub Kicinski]

After recent changes the SW side of tls_rx_one_record() can
be nicely encapsulated in its own function. Move the pad handling
as well. This will be useful for ->zc handling in tls_decrypt_device().

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: borisp@nvidia.com
CC: john.fastabend@gmail.com
---
 net/tls/tls_sw.c | 93 +++++++++++++++++++++++++++++-------------------
 1 file changed, 57 insertions(+), 36 deletions(-)

diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index cb99fc11997b..eed52f853418 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -1409,7 +1409,7 @@ tls_alloc_clrtxt_skb(struct sock *sk, struct sk_buff *skb,
 
 /* Decrypt handlers
  *
- * tls_decrypt_sg() and tls_decrypt_device() are decrypt handlers.
+ * tls_decrypt_sw() and tls_decrypt_device() are decrypt handlers.
  * They must transform the darg in/out argument are as follows:
  *       |          Input            |         Output
  * -------------------------------------------------------------------
@@ -1589,49 +1589,22 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov,
 }
 
 static int
-tls_decrypt_device(struct sock *sk, struct tls_context *tls_ctx,
-		   struct tls_decrypt_arg *darg)
-{
-	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
-	int err;
-
-	if (tls_ctx->rx_conf != TLS_HW)
-		return 0;
-
-	err = tls_device_decrypted(sk, tls_ctx);
-	if (err <= 0)
-		return err;
-
-	darg->zc = false;
-	darg->async = false;
-	darg->skb = tls_strp_msg(ctx);
-	ctx->recv_pkt = NULL;
-	return 1;
-}
-
-static int tls_rx_one_record(struct sock *sk, struct iov_iter *dest,
-			     struct tls_decrypt_arg *darg)
+tls_decrypt_sw(struct sock *sk, struct tls_context *tls_ctx,
+	       struct msghdr *msg, struct tls_decrypt_arg *darg)
 {
-	struct tls_context *tls_ctx = tls_get_ctx(sk);
 	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
 	struct tls_prot_info *prot = &tls_ctx->prot_info;
 	struct strp_msg *rxm;
 	int pad, err;
 
-	err = tls_decrypt_device(sk, tls_ctx, darg);
-	if (err < 0)
-		return err;
-	if (err)
-		goto decrypt_done;
-
-	err = tls_decrypt_sg(sk, dest, NULL, darg);
+	err = tls_decrypt_sg(sk, &msg->msg_iter, NULL, darg);
 	if (err < 0) {
 		if (err == -EBADMSG)
 			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSDECRYPTERROR);
 		return err;
 	}
-	if (darg->async)
-		goto decrypt_done;
+	/* keep going even for ->async, the code below is TLS 1.3 */
+
 	/* If opportunistic TLS 1.3 ZC failed retry without ZC */
 	if (unlikely(darg->zc && prot->version == TLS_1_3_VERSION &&
 		     darg->tail != TLS_RECORD_TYPE_DATA)) {
@@ -1639,10 +1612,9 @@ static int tls_rx_one_record(struct sock *sk, struct iov_iter *dest,
 		if (!darg->tail)
 			TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXNOPADVIOL);
 		TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSDECRYPTRETRY);
-		return tls_rx_one_record(sk, dest, darg);
+		return tls_decrypt_sw(sk, tls_ctx, msg, darg);
 	}
 
-decrypt_done:
 	if (darg->skb == ctx->recv_pkt)
 		ctx->recv_pkt = NULL;
 
@@ -1654,6 +1626,55 @@ static int tls_rx_one_record(struct sock *sk, struct iov_iter *dest,
 
 	rxm = strp_msg(darg->skb);
 	rxm->full_len -= pad;
+
+	return 0;
+}
+
+static int
+tls_decrypt_device(struct sock *sk, struct tls_context *tls_ctx,
+		   struct tls_decrypt_arg *darg)
+{
+	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
+	struct tls_prot_info *prot = &tls_ctx->prot_info;
+	struct strp_msg *rxm;
+	int pad, err;
+
+	if (tls_ctx->rx_conf != TLS_HW)
+		return 0;
+
+	err = tls_device_decrypted(sk, tls_ctx);
+	if (err <= 0)
+		return err;
+
+	pad = tls_padding_length(prot, tls_strp_msg(ctx), darg);
+	if (pad < 0)
+		return pad;
+
+	darg->zc = false;
+	darg->async = false;
+	darg->skb = tls_strp_msg(ctx);
+	ctx->recv_pkt = NULL;
+
+	rxm = strp_msg(darg->skb);
+	rxm->full_len -= pad;
+	return 1;
+}
+
+static int tls_rx_one_record(struct sock *sk, struct msghdr *msg,
+			     struct tls_decrypt_arg *darg)
+{
+	struct tls_context *tls_ctx = tls_get_ctx(sk);
+	struct tls_prot_info *prot = &tls_ctx->prot_info;
+	struct strp_msg *rxm;
+	int err;
+
+	err = tls_decrypt_device(sk, tls_ctx, darg);
+	if (!err)
+		err = tls_decrypt_sw(sk, tls_ctx, msg, darg);
+	if (err < 0)
+		return err;
+
+	rxm = strp_msg(darg->skb);
 	rxm->offset += prot->prepend_size;
 	rxm->full_len -= prot->overhead_size;
 	tls_advance_record_sn(sk, prot, &tls_ctx->rx);
@@ -1943,7 +1964,7 @@ int tls_sw_recvmsg(struct sock *sk,
 		else
 			darg.async = false;
 
-		err = tls_rx_one_record(sk, &msg->msg_iter, &darg);
+		err = tls_rx_one_record(sk, msg, &darg);
 		if (err < 0) {
 			tls_err_abort(sk, -EBADMSG);
 			goto recv_end;
-- 
2.37.1

[PATCH net-next v3 3/7] tls: rx: don't free the output in case of zero-copy

[Jakub Kicinski]

In the future we'll want to reuse the input skb in case of
zero-copy so we shouldn't always free darg.skb. Move the
freeing of darg.skb into the non-zc cases. All cases will
now free ctx->recv_pkt (inside let tls_rx_rec_done()).

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: borisp@nvidia.com
CC: john.fastabend@gmail.com
---
 net/tls/tls_sw.c | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index eed52f853418..fe38b49a2607 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -1416,6 +1416,8 @@ tls_alloc_clrtxt_skb(struct sock *sk, struct sk_buff *skb,
  *    zc | Zero-copy decrypt allowed | Zero-copy performed
  * async | Async decrypt allowed     | Async crypto used / in progress
  *   skb |            *              | Output skb
+ *
+ * If ZC decryption was performed darg.skb will point to the input skb.
  */
 
 /* This function decrypts the input skb into either out_iov or in out_sg
@@ -1615,12 +1617,10 @@ tls_decrypt_sw(struct sock *sk, struct tls_context *tls_ctx,
 		return tls_decrypt_sw(sk, tls_ctx, msg, darg);
 	}
 
-	if (darg->skb == ctx->recv_pkt)
-		ctx->recv_pkt = NULL;
-
 	pad = tls_padding_length(prot, darg->skb, darg);
 	if (pad < 0) {
-		consume_skb(darg->skb);
+		if (darg->skb != tls_strp_msg(ctx))
+			consume_skb(darg->skb);
 		return pad;
 	}
 
@@ -1890,7 +1890,6 @@ int tls_sw_recvmsg(struct sock *sk,
 	size_t flushed_at = 0;
 	struct strp_msg *rxm;
 	struct tls_msg *tlm;
-	struct sk_buff *skb;
 	ssize_t copied = 0;
 	bool async = false;
 	int target, err = 0;
@@ -1970,10 +1969,6 @@ int tls_sw_recvmsg(struct sock *sk,
 			goto recv_end;
 		}
 
-		skb = darg.skb;
-		rxm = strp_msg(skb);
-		tlm = tls_msg(skb);
-
 		async |= darg.async;
 
 		/* If the type of records being processed is not known yet,
@@ -1983,11 +1978,12 @@ int tls_sw_recvmsg(struct sock *sk,
 		 * is known just after record is dequeued from stream parser.
 		 * For tls1.3, we disable async.
 		 */
-		err = tls_record_content_type(msg, tlm, &control);
+		err = tls_record_content_type(msg, tls_msg(darg.skb), &control);
 		if (err <= 0) {
+			DEBUG_NET_WARN_ON_ONCE(darg.zc);
 			tls_rx_rec_done(ctx);
 put_on_rx_list_err:
-			__skb_queue_tail(&ctx->rx_list, skb);
+			__skb_queue_tail(&ctx->rx_list, darg.skb);
 			goto recv_end;
 		}
 
@@ -1996,11 +1992,15 @@ int tls_sw_recvmsg(struct sock *sk,
 				       decrypted + copied, &flushed_at);
 
 		/* TLS 1.3 may have updated the length by more than overhead */
+		rxm = strp_msg(darg.skb);
 		chunk = rxm->full_len;
 		tls_rx_rec_done(ctx);
 
 		if (!darg.zc) {
 			bool partially_consumed = chunk > len;
+			struct sk_buff *skb = darg.skb;
+
+			DEBUG_NET_WARN_ON_ONCE(darg.skb == ctx->recv_pkt);
 
 			if (async) {
 				/* TLS 1.2-only, to_decrypt must be text len */
@@ -2040,13 +2040,13 @@ int tls_sw_recvmsg(struct sock *sk,
 				rxm->full_len -= chunk;
 				goto put_on_rx_list;
 			}
+
+			consume_skb(skb);
 		}
 
 		decrypted += chunk;
 		len -= chunk;
 
-		consume_skb(skb);
-
 		/* Return full control message to userspace before trying
 		 * to parse another message type
 		 */
-- 
2.37.1

[PATCH net-next v3 4/7] tls: rx: device: keep the zero copy status with offload

[Jakub Kicinski]

The non-zero-copy path assumes a full skb with decrypted contents.
This means the device offload would have to CoW the data. Try
to keep the zero-copy status instead, copy the data to user space.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: borisp@nvidia.com
CC: john.fastabend@gmail.com
---
 net/tls/tls.h      |  1 +
 net/tls/tls_strp.c |  9 +++++++++
 net/tls/tls_sw.c   | 30 +++++++++++++++++++++++++-----
 3 files changed, 35 insertions(+), 5 deletions(-)

diff --git a/net/tls/tls.h b/net/tls/tls.h
index 24bec1c5f1e8..78c5d699bf75 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -127,6 +127,7 @@ int tls_sw_fallback_init(struct sock *sk,
 			 struct tls_offload_context_tx *offload_ctx,
 			 struct tls_crypto_info *crypto_info);
 
+struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx);
 int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
 		      struct sk_buff_head *dst);
 
diff --git a/net/tls/tls_strp.c b/net/tls/tls_strp.c
index 9ccab79a6e1e..40b177366121 100644
--- a/net/tls/tls_strp.c
+++ b/net/tls/tls_strp.c
@@ -4,6 +4,15 @@
 
 #include "tls.h"
 
+struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx)
+{
+	struct sk_buff *skb;
+
+	skb = ctx->recv_pkt;
+	ctx->recv_pkt = NULL;
+	return skb;
+}
+
 int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
 		      struct sk_buff_head *dst)
 {
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index fe38b49a2607..bd4486819e64 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -1631,8 +1631,8 @@ tls_decrypt_sw(struct sock *sk, struct tls_context *tls_ctx,
 }
 
 static int
-tls_decrypt_device(struct sock *sk, struct tls_context *tls_ctx,
-		   struct tls_decrypt_arg *darg)
+tls_decrypt_device(struct sock *sk, struct msghdr *msg,
+		   struct tls_context *tls_ctx, struct tls_decrypt_arg *darg)
 {
 	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
 	struct tls_prot_info *prot = &tls_ctx->prot_info;
@@ -1650,13 +1650,33 @@ tls_decrypt_device(struct sock *sk, struct tls_context *tls_ctx,
 	if (pad < 0)
 		return pad;
 
-	darg->zc = false;
 	darg->async = false;
 	darg->skb = tls_strp_msg(ctx);
-	ctx->recv_pkt = NULL;
+	/* ->zc downgrade check, in case TLS 1.3 gets here */
+	darg->zc &= !(prot->version == TLS_1_3_VERSION &&
+		      tls_msg(darg->skb)->control != TLS_RECORD_TYPE_DATA);
 
 	rxm = strp_msg(darg->skb);
 	rxm->full_len -= pad;
+
+	if (!darg->zc) {
+		/* Non-ZC case needs a real skb */
+		darg->skb = tls_strp_msg_detach(ctx);
+		if (!darg->skb)
+			return -ENOMEM;
+	} else {
+		unsigned int off, len;
+
+		/* In ZC case nobody cares about the output skb.
+		 * Just copy the data here. Note the skb is not fully trimmed.
+		 */
+		off = rxm->offset + prot->prepend_size;
+		len = rxm->full_len - prot->overhead_size;
+
+		err = skb_copy_datagram_msg(darg->skb, off, msg, len);
+		if (err)
+			return err;
+	}
 	return 1;
 }
 
@@ -1668,7 +1688,7 @@ static int tls_rx_one_record(struct sock *sk, struct msghdr *msg,
 	struct strp_msg *rxm;
 	int err;
 
-	err = tls_decrypt_device(sk, tls_ctx, darg);
+	err = tls_decrypt_device(sk, msg, tls_ctx, darg);
 	if (!err)
 		err = tls_decrypt_sw(sk, tls_ctx, msg, darg);
 	if (err < 0)
-- 
2.37.1

[PATCH net-next v3 5/7] tcp: allow tls to decrypt directly from the tcp rcv queue

[Jakub Kicinski]

Expose TCP rx queue accessor and cleanup, so that TLS can
decrypt directly from the TCP queue. The expectation
is that the caller can access the skb returned from
tcp_recv_skb() and up to inq bytes worth of data (some
of which may be in ->next skbs) and then call
tcp_read_done() when data has been consumed.
The socket lock must be held continuously across
those two operations.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
v3: drop unnecessay tcp_recv_skb() (Paolo)

CC: yoshfuji@linux-ipv6.org
CC: dsahern@kernel.org
---
 include/net/tcp.h |  2 ++
 net/ipv4/tcp.c    | 42 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/include/net/tcp.h b/include/net/tcp.h
index f9e7c85ea829..b8620be32f8f 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -673,6 +673,8 @@ void tcp_get_info(struct sock *, struct tcp_info *);
 int tcp_read_sock(struct sock *sk, read_descriptor_t *desc,
 		  sk_read_actor_t recv_actor);
 int tcp_read_skb(struct sock *sk, skb_read_actor_t recv_actor);
+struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off);
+void tcp_read_done(struct sock *sk, size_t len);
 
 void tcp_initialize_rcv_mss(struct sock *sk);
 
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index ba2bdc811374..dc7cc3ce6a53 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1635,7 +1635,7 @@ static void tcp_eat_recv_skb(struct sock *sk, struct sk_buff *skb)
 	__kfree_skb(skb);
 }
 
-static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
+struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
 {
 	struct sk_buff *skb;
 	u32 offset;
@@ -1658,6 +1658,7 @@ static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
 	}
 	return NULL;
 }
+EXPORT_SYMBOL(tcp_recv_skb);
 
 /*
  * This routine provides an alternative to tcp_recvmsg() for routines
@@ -1788,6 +1789,45 @@ int tcp_read_skb(struct sock *sk, skb_read_actor_t recv_actor)
 }
 EXPORT_SYMBOL(tcp_read_skb);
 
+void tcp_read_done(struct sock *sk, size_t len)
+{
+	struct tcp_sock *tp = tcp_sk(sk);
+	u32 seq = tp->copied_seq;
+	struct sk_buff *skb;
+	size_t left;
+	u32 offset;
+
+	if (sk->sk_state == TCP_LISTEN)
+		return;
+
+	left = len;
+	while (left && (skb = tcp_recv_skb(sk, seq, &offset)) != NULL) {
+		int used;
+
+		used = min_t(size_t, skb->len - offset, left);
+		seq += used;
+		left -= used;
+
+		if (skb->len > offset + used)
+			break;
+
+		if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) {
+			tcp_eat_recv_skb(sk, skb);
+			++seq;
+			break;
+		}
+		tcp_eat_recv_skb(sk, skb);
+	}
+	WRITE_ONCE(tp->copied_seq, seq);
+
+	tcp_rcv_space_adjust(sk);
+
+	/* Clean up data we have read: This will do ACK frames. */
+	if (left != len)
+		tcp_cleanup_rbuf(sk, len - left);
+}
+EXPORT_SYMBOL(tcp_read_done);
+
 int tcp_peek_len(struct socket *sock)
 {
 	return tcp_inq(sock->sk);
-- 
2.37.1

[PATCH net-next v3 6/7] tls: rx: device: add input CoW helper

[Jakub Kicinski]

Wrap the remaining skb_cow_data() into a helper, so it's easier
to replace down the lane. The new version will change the skb
so make sure relevant pointers get reloaded after the call.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: borisp@nvidia.com
CC: john.fastabend@gmail.com
---
 net/tls/tls.h        |  1 +
 net/tls/tls_device.c | 19 +++++++++----------
 net/tls/tls_strp.c   | 11 +++++++++++
 3 files changed, 21 insertions(+), 10 deletions(-)

diff --git a/net/tls/tls.h b/net/tls/tls.h
index 78c5d699bf75..154a3773e785 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -127,6 +127,7 @@ int tls_sw_fallback_init(struct sock *sk,
 			 struct tls_offload_context_tx *offload_ctx,
 			 struct tls_crypto_info *crypto_info);
 
+int tls_strp_msg_cow(struct tls_sw_context_rx *ctx);
 struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx);
 int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
 		      struct sk_buff_head *dst);
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index b1fcd61836d1..fc513c1806a0 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -894,27 +894,26 @@ static void tls_device_core_ctrl_rx_resync(struct tls_context *tls_ctx,
 static int
 tls_device_reencrypt(struct sock *sk, struct tls_sw_context_rx *sw_ctx)
 {
-	int err = 0, offset, copy, nsg, data_len, pos;
-	struct sk_buff *skb, *skb_iter, *unused;
+	int err, offset, copy, data_len, pos;
+	struct sk_buff *skb, *skb_iter;
 	struct scatterlist sg[1];
 	struct strp_msg *rxm;
 	char *orig_buf, *buf;
 
-	skb = tls_strp_msg(sw_ctx);
-	rxm = strp_msg(skb);
-	offset = rxm->offset;
-
+	rxm = strp_msg(tls_strp_msg(sw_ctx));
 	orig_buf = kmalloc(rxm->full_len + TLS_HEADER_SIZE +
 			   TLS_CIPHER_AES_GCM_128_IV_SIZE, sk->sk_allocation);
 	if (!orig_buf)
 		return -ENOMEM;
 	buf = orig_buf;
 
-	nsg = skb_cow_data(skb, 0, &unused);
-	if (unlikely(nsg < 0)) {
-		err = nsg;
+	err = tls_strp_msg_cow(sw_ctx);
+	if (unlikely(err))
 		goto free_buf;
-	}
+
+	skb = tls_strp_msg(sw_ctx);
+	rxm = strp_msg(skb);
+	offset = rxm->offset;
 
 	sg_init_table(sg, 1);
 	sg_set_buf(&sg[0], buf,
diff --git a/net/tls/tls_strp.c b/net/tls/tls_strp.c
index 40b177366121..d9bb4f23f01a 100644
--- a/net/tls/tls_strp.c
+++ b/net/tls/tls_strp.c
@@ -13,6 +13,17 @@ struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx)
 	return skb;
 }
 
+int tls_strp_msg_cow(struct tls_sw_context_rx *ctx)
+{
+	struct sk_buff *unused;
+	int nsg;
+
+	nsg = skb_cow_data(ctx->recv_pkt, 0, &unused);
+	if (nsg < 0)
+		return nsg;
+	return 0;
+}
+
 int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
 		      struct sk_buff_head *dst)
 {
-- 
2.37.1

[PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

TLS is a relatively poor fit for strparser. We pause the input
every time a message is received, wait for a read which will
decrypt the message, start the parser, repeat. strparser is
built to delineate the messages, wrap them in individual skbs
and let them float off into the stack or a different socket.
TLS wants the data pages and nothing else. There's no need
for TLS to keep cloning (and occasionally skb_unclone()'ing)
the TCP rx queue.

This patch uses a pre-allocated skb and attaches the skbs
from the TCP rx queue to it as frags. TLS is careful never
to modify the input skb without CoW'ing / detaching it first.

Since we call TCP rx queue cleanup directly we also get back
the benefit of skb deferred free.

Overall this results in a 6% gain in my benchmarks.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
CC: borisp@nvidia.com
CC: john.fastabend@gmail.com
---
 include/net/tls.h  |  19 +-
 net/tls/tls.h      |  24 ++-
 net/tls/tls_main.c |  20 +-
 net/tls/tls_strp.c | 484 +++++++++++++++++++++++++++++++++++++++++++--
 net/tls/tls_sw.c   |  80 ++++----
 5 files changed, 558 insertions(+), 69 deletions(-)

diff --git a/include/net/tls.h b/include/net/tls.h
index 181c496b01b8..abb050b0df83 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -108,18 +108,33 @@ struct tls_sw_context_tx {
 	unsigned long tx_bitmask;
 };
 
+struct tls_strparser {
+	struct sock *sk;
+
+	u32 mark : 8;
+	u32 stopped : 1;
+	u32 copy_mode : 1;
+	u32 msg_ready : 1;
+
+	struct strp_msg stm;
+
+	struct sk_buff *anchor;
+	struct work_struct work;
+};
+
 struct tls_sw_context_rx {
 	struct crypto_aead *aead_recv;
 	struct crypto_wait async_wait;
-	struct strparser strp;
 	struct sk_buff_head rx_list;	/* list of decrypted 'data' records */
 	void (*saved_data_ready)(struct sock *sk);
 
-	struct sk_buff *recv_pkt;
 	u8 reader_present;
 	u8 async_capable:1;
 	u8 zc_capable:1;
 	u8 reader_contended:1;
+
+	struct tls_strparser strp;
+
 	atomic_t decrypt_pending;
 	/* protect crypto_wait with decrypt_pending*/
 	spinlock_t decrypt_compl_lock;
diff --git a/net/tls/tls.h b/net/tls/tls.h
index 154a3773e785..0e840a0c3437 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -1,4 +1,5 @@
 /*
+ * Copyright (c) 2016 Tom Herbert <tom@herbertland.com>
  * Copyright (c) 2016-2017, Mellanox Technologies. All rights reserved.
  * Copyright (c) 2016-2017, Dave Watson <davejwatson@fb.com>. All rights reserved.
  *
@@ -127,10 +128,24 @@ int tls_sw_fallback_init(struct sock *sk,
 			 struct tls_offload_context_tx *offload_ctx,
 			 struct tls_crypto_info *crypto_info);
 
+int tls_strp_dev_init(void);
+void tls_strp_dev_exit(void);
+
+void tls_strp_done(struct tls_strparser *strp);
+void tls_strp_stop(struct tls_strparser *strp);
+int tls_strp_init(struct tls_strparser *strp, struct sock *sk);
+void tls_strp_data_ready(struct tls_strparser *strp);
+
+void tls_strp_check_rcv(struct tls_strparser *strp);
+void tls_strp_msg_done(struct tls_strparser *strp);
+
+int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb);
+void tls_rx_msg_ready(struct tls_strparser *strp);
+
+void tls_strp_msg_load(struct tls_strparser *strp, bool force_refresh);
 int tls_strp_msg_cow(struct tls_sw_context_rx *ctx);
 struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx);
-int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
-		      struct sk_buff_head *dst);
+int tls_strp_msg_hold(struct tls_strparser *strp, struct sk_buff_head *dst);
 
 static inline struct tls_msg *tls_msg(struct sk_buff *skb)
 {
@@ -141,12 +156,13 @@ static inline struct tls_msg *tls_msg(struct sk_buff *skb)
 
 static inline struct sk_buff *tls_strp_msg(struct tls_sw_context_rx *ctx)
 {
-	return ctx->recv_pkt;
+	DEBUG_NET_WARN_ON_ONCE(!ctx->strp.msg_ready || !ctx->strp.anchor->len);
+	return ctx->strp.anchor;
 }
 
 static inline bool tls_strp_msg_ready(struct tls_sw_context_rx *ctx)
 {
-	return ctx->recv_pkt;
+	return ctx->strp.msg_ready;
 }
 
 #ifdef CONFIG_TLS_DEVICE
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index 9703636cfc60..08ddf9d837ae 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -725,6 +725,10 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
 	if (tx) {
 		ctx->sk_write_space = sk->sk_write_space;
 		sk->sk_write_space = tls_write_space;
+	} else {
+		struct tls_sw_context_rx *rx_ctx = tls_sw_ctx_rx(ctx);
+
+		tls_strp_check_rcv(&rx_ctx->strp);
 	}
 	return 0;
 
@@ -1141,20 +1145,28 @@ static int __init tls_register(void)
 	if (err)
 		return err;
 
+	err = tls_strp_dev_init();
+	if (err)
+		goto err_pernet;
+
 	err = tls_device_init();
-	if (err) {
-		unregister_pernet_subsys(&tls_proc_ops);
-		return err;
-	}
+	if (err)
+		goto err_strp;
 
 	tcp_register_ulp(&tcp_tls_ulp_ops);
 
 	return 0;
+err_strp:
+	tls_strp_dev_exit();
+err_pernet:
+	unregister_pernet_subsys(&tls_proc_ops);
+	return err;
 }
 
 static void __exit tls_unregister(void)
 {
 	tcp_unregister_ulp(&tcp_tls_ulp_ops);
+	tls_strp_dev_exit();
 	tls_device_cleanup();
 	unregister_pernet_subsys(&tls_proc_ops);
 }
diff --git a/net/tls/tls_strp.c b/net/tls/tls_strp.c
index d9bb4f23f01a..b945288c312e 100644
--- a/net/tls/tls_strp.c
+++ b/net/tls/tls_strp.c
@@ -1,37 +1,493 @@
 // SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2016 Tom Herbert <tom@herbertland.com> */
 
 #include <linux/skbuff.h>
+#include <linux/workqueue.h>
+#include <net/strparser.h>
+#include <net/tcp.h>
+#include <net/sock.h>
+#include <net/tls.h>
 
 #include "tls.h"
 
-struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx)
+static struct workqueue_struct *tls_strp_wq;
+
+static void tls_strp_abort_strp(struct tls_strparser *strp, int err)
+{
+	if (strp->stopped)
+		return;
+
+	strp->stopped = 1;
+
+	/* Report an error on the lower socket */
+	strp->sk->sk_err = -err;
+	sk_error_report(strp->sk);
+}
+
+static void tls_strp_anchor_free(struct tls_strparser *strp)
 {
+	struct skb_shared_info *shinfo = skb_shinfo(strp->anchor);
+
+	DEBUG_NET_WARN_ON_ONCE(atomic_read(&shinfo->dataref) != 1);
+	shinfo->frag_list = NULL;
+	consume_skb(strp->anchor);
+	strp->anchor = NULL;
+}
+
+/* Create a new skb with the contents of input copied to its page frags */
+static struct sk_buff *tls_strp_msg_make_copy(struct tls_strparser *strp)
+{
+	struct strp_msg *rxm;
 	struct sk_buff *skb;
+	int i, err, offset;
+
+	skb = alloc_skb_with_frags(0, strp->anchor->len, TLS_PAGE_ORDER,
+				   &err, strp->sk->sk_allocation);
+	if (!skb)
+		return NULL;
+
+	offset = strp->stm.offset;
+	for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+		skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
 
-	skb = ctx->recv_pkt;
-	ctx->recv_pkt = NULL;
+		WARN_ON_ONCE(skb_copy_bits(strp->anchor, offset,
+					   skb_frag_address(frag),
+					   skb_frag_size(frag)));
+		offset += skb_frag_size(frag);
+	}
+
+	skb_copy_header(skb, strp->anchor);
+	rxm = strp_msg(skb);
+	rxm->offset = 0;
 	return skb;
 }
 
+/* Steal the input skb, input msg is invalid after calling this function */
+struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx)
+{
+	struct tls_strparser *strp = &ctx->strp;
+
+#ifdef CONFIG_TLS_DEVICE
+	DEBUG_NET_WARN_ON_ONCE(!strp->anchor->decrypted);
+#else
+	/* This function turns an input into an output,
+	 * that can only happen if we have offload.
+	 */
+	WARN_ON(1);
+#endif
+
+	if (strp->copy_mode) {
+		struct sk_buff *skb;
+
+		/* Replace anchor with an empty skb, this is a little
+		 * dangerous but __tls_cur_msg() warns on empty skbs
+		 * so hopefully we'll catch abuses.
+		 */
+		skb = alloc_skb(0, strp->sk->sk_allocation);
+		if (!skb)
+			return NULL;
+
+		swap(strp->anchor, skb);
+		return skb;
+	}
+
+	return tls_strp_msg_make_copy(strp);
+}
+
+/* Force the input skb to be in copy mode. The data ownership remains
+ * with the input skb itself (meaning unpause will wipe it) but it can
+ * be modified.
+ */
 int tls_strp_msg_cow(struct tls_sw_context_rx *ctx)
 {
-	struct sk_buff *unused;
-	int nsg;
+	struct tls_strparser *strp = &ctx->strp;
+	struct sk_buff *skb;
+
+	if (strp->copy_mode)
+		return 0;
+
+	skb = tls_strp_msg_make_copy(strp);
+	if (!skb)
+		return -ENOMEM;
+
+	tls_strp_anchor_free(strp);
+	strp->anchor = skb;
+
+	tcp_read_done(strp->sk, strp->stm.full_len);
+	strp->copy_mode = 1;
+
+	return 0;
+}
+
+/* Make a clone (in the skb sense) of the input msg to keep a reference
+ * to the underlying data. The reference-holding skbs get placed on
+ * @dst.
+ */
+int tls_strp_msg_hold(struct tls_strparser *strp, struct sk_buff_head *dst)
+{
+	struct skb_shared_info *shinfo = skb_shinfo(strp->anchor);
+
+	if (strp->copy_mode) {
+		struct sk_buff *skb;
+
+		WARN_ON_ONCE(!shinfo->nr_frags);
+
+		/* We can't skb_clone() the anchor, it gets wiped by unpause */
+		skb = alloc_skb(0, strp->sk->sk_allocation);
+		if (!skb)
+			return -ENOMEM;
+
+		__skb_queue_tail(dst, strp->anchor);
+		strp->anchor = skb;
+	} else {
+		struct sk_buff *iter, *clone;
+		int chunk, len, offset;
+
+		offset = strp->stm.offset;
+		len = strp->stm.full_len;
+		iter = shinfo->frag_list;
+
+		while (len > 0) {
+			if (iter->len <= offset) {
+				offset -= iter->len;
+				goto next;
+			}
+
+			chunk = iter->len - offset;
+			offset = 0;
+
+			clone = skb_clone(iter, strp->sk->sk_allocation);
+			if (!clone)
+				return -ENOMEM;
+			__skb_queue_tail(dst, clone);
+
+			len -= chunk;
+next:
+			iter = iter->next;
+		}
+	}
+
+	return 0;
+}
+
+static void tls_strp_flush_anchor_copy(struct tls_strparser *strp)
+{
+	struct skb_shared_info *shinfo = skb_shinfo(strp->anchor);
+	int i;
+
+	DEBUG_NET_WARN_ON_ONCE(atomic_read(&shinfo->dataref) != 1);
+
+	for (i = 0; i < shinfo->nr_frags; i++)
+		__skb_frag_unref(&shinfo->frags[i], false);
+	shinfo->nr_frags = 0;
+	strp->copy_mode = 0;
+}
+
+static int tls_strp_copyin(read_descriptor_t *desc, struct sk_buff *in_skb,
+			   unsigned int offset, size_t in_len)
+{
+	struct tls_strparser *strp = (struct tls_strparser *)desc->arg.data;
+	size_t sz, len, chunk;
+	struct sk_buff *skb;
+	skb_frag_t *frag;
+
+	if (strp->msg_ready)
+		return 0;
+
+	skb = strp->anchor;
+	frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE];
+
+	len = in_len;
+	/* First make sure we got the header */
+	if (!strp->stm.full_len) {
+		/* Assume one page is more than enough for headers */
+		chunk =	min_t(size_t, len, PAGE_SIZE - skb_frag_size(frag));
+		WARN_ON_ONCE(skb_copy_bits(in_skb, offset,
+					   skb_frag_address(frag) +
+					   skb_frag_size(frag),
+					   chunk));
+
+		sz = tls_rx_msg_size(strp, strp->anchor);
+		if (sz < 0) {
+			desc->error = sz;
+			return 0;
+		}
+
+		/* We may have over-read, sz == 0 is guaranteed under-read */
+		if (sz > 0)
+			chunk =	min_t(size_t, chunk, sz - skb->len);
+
+		skb->len += chunk;
+		skb->data_len += chunk;
+		skb_frag_size_add(frag, chunk);
+		frag++;
+		len -= chunk;
+		offset += chunk;
+
+		strp->stm.full_len = sz;
+		if (!strp->stm.full_len)
+			goto read_done;
+	}
+
+	/* Load up more data */
+	while (len && strp->stm.full_len > skb->len) {
+		chunk =	min_t(size_t, len, strp->stm.full_len - skb->len);
+		chunk = min_t(size_t, chunk, PAGE_SIZE - skb_frag_size(frag));
+		WARN_ON_ONCE(skb_copy_bits(in_skb, offset,
+					   skb_frag_address(frag) +
+					   skb_frag_size(frag),
+					   chunk));
+
+		skb->len += chunk;
+		skb->data_len += chunk;
+		skb_frag_size_add(frag, chunk);
+		frag++;
+		len -= chunk;
+		offset += chunk;
+	}
+
+	if (strp->stm.full_len == skb->len) {
+		desc->count = 0;
+
+		strp->msg_ready = 1;
+		tls_rx_msg_ready(strp);
+	}
+
+read_done:
+	return in_len - len;
+}
+
+static int tls_strp_read_copyin(struct tls_strparser *strp)
+{
+	struct socket *sock = strp->sk->sk_socket;
+	read_descriptor_t desc;
+
+	desc.arg.data = strp;
+	desc.error = 0;
+	desc.count = 1; /* give more than one skb per call */
+
+	/* sk should be locked here, so okay to do read_sock */
+	sock->ops->read_sock(strp->sk, &desc, tls_strp_copyin);
+
+	return desc.error;
+}
+
+static int tls_strp_read_short(struct tls_strparser *strp)
+{
+	struct skb_shared_info *shinfo;
+	struct page *page;
+	int need_spc, len;
+
+	/* If the rbuf is small or rcv window has collapsed to 0 we need
+	 * to read the data out. Otherwise the connection will stall.
+	 * Without pressure threshold of INT_MAX will never be ready.
+	 */
+	if (likely(!tcp_epollin_ready(strp->sk, INT_MAX)))
+		return 0;
+
+	shinfo = skb_shinfo(strp->anchor);
+	shinfo->frag_list = NULL;
+
+	/* If we don't know the length go max plus page for cipher overhead */
+	need_spc = strp->stm.full_len ?: TLS_MAX_PAYLOAD_SIZE + PAGE_SIZE;
+
+	for (len = need_spc; len > 0; len -= PAGE_SIZE) {
+		page = alloc_page(strp->sk->sk_allocation);
+		if (!page) {
+			tls_strp_flush_anchor_copy(strp);
+			return -ENOMEM;
+		}
+
+		skb_fill_page_desc(strp->anchor, shinfo->nr_frags++,
+				   page, 0, 0);
+	}
+
+	strp->copy_mode = 1;
+	strp->stm.offset = 0;
+
+	strp->anchor->len = 0;
+	strp->anchor->data_len = 0;
+	strp->anchor->truesize = round_up(need_spc, PAGE_SIZE);
+
+	tls_strp_read_copyin(strp);
+
+	return 0;
+}
+
+static void tls_strp_load_anchor_with_queue(struct tls_strparser *strp, int len)
+{
+	struct tcp_sock *tp = tcp_sk(strp->sk);
+	struct sk_buff *first;
+	u32 offset;
+
+	first = tcp_recv_skb(strp->sk, tp->copied_seq, &offset);
+	if (WARN_ON_ONCE(!first))
+		return;
+
+	/* Bestow the state onto the anchor */
+	strp->anchor->len = offset + len;
+	strp->anchor->data_len = offset + len;
+	strp->anchor->truesize = offset + len;
+
+	skb_shinfo(strp->anchor)->frag_list = first;
+
+	skb_copy_header(strp->anchor, first);
+	strp->anchor->destructor = NULL;
+
+	strp->stm.offset = offset;
+}
+
+void tls_strp_msg_load(struct tls_strparser *strp, bool force_refresh)
+{
+	struct strp_msg *rxm;
+	struct tls_msg *tlm;
+
+	DEBUG_NET_WARN_ON_ONCE(!strp->msg_ready);
+	DEBUG_NET_WARN_ON_ONCE(!strp->stm.full_len);
+
+	if (!strp->copy_mode && force_refresh) {
+		if (WARN_ON(tcp_inq(strp->sk) < strp->stm.full_len))
+			return;
+
+		tls_strp_load_anchor_with_queue(strp, strp->stm.full_len);
+	}
+
+	rxm = strp_msg(strp->anchor);
+	rxm->full_len	= strp->stm.full_len;
+	rxm->offset	= strp->stm.offset;
+	tlm = tls_msg(strp->anchor);
+	tlm->control	= strp->mark;
+}
+
+/* Called with lock held on lower socket */
+static int tls_strp_read_sock(struct tls_strparser *strp)
+{
+	int sz, inq;
+
+	inq = tcp_inq(strp->sk);
+	if (inq < 1)
+		return 0;
+
+	if (unlikely(strp->copy_mode))
+		return tls_strp_read_copyin(strp);
+
+	if (inq < strp->stm.full_len)
+		return tls_strp_read_short(strp);
+
+	if (!strp->stm.full_len) {
+		tls_strp_load_anchor_with_queue(strp, inq);
+
+		sz = tls_rx_msg_size(strp, strp->anchor);
+		if (sz < 0) {
+			tls_strp_abort_strp(strp, sz);
+			return sz;
+		}
+
+		strp->stm.full_len = sz;
+
+		if (!strp->stm.full_len || inq < strp->stm.full_len)
+			return tls_strp_read_short(strp);
+	}
+
+	strp->msg_ready = 1;
+	tls_rx_msg_ready(strp);
+
+	return 0;
+}
+
+void tls_strp_check_rcv(struct tls_strparser *strp)
+{
+	if (unlikely(strp->stopped) || strp->msg_ready)
+		return;
+
+	if (tls_strp_read_sock(strp) == -ENOMEM)
+		queue_work(tls_strp_wq, &strp->work);
+}
+
+/* Lower sock lock held */
+void tls_strp_data_ready(struct tls_strparser *strp)
+{
+	/* This check is needed to synchronize with do_tls_strp_work.
+	 * do_tls_strp_work acquires a process lock (lock_sock) whereas
+	 * the lock held here is bh_lock_sock. The two locks can be
+	 * held by different threads at the same time, but bh_lock_sock
+	 * allows a thread in BH context to safely check if the process
+	 * lock is held. In this case, if the lock is held, queue work.
+	 */
+	if (sock_owned_by_user_nocheck(strp->sk)) {
+		queue_work(tls_strp_wq, &strp->work);
+		return;
+	}
+
+	tls_strp_check_rcv(strp);
+}
+
+static void tls_strp_work(struct work_struct *w)
+{
+	struct tls_strparser *strp =
+		container_of(w, struct tls_strparser, work);
+
+	lock_sock(strp->sk);
+	tls_strp_check_rcv(strp);
+	release_sock(strp->sk);
+}
+
+void tls_strp_msg_done(struct tls_strparser *strp)
+{
+	WARN_ON(!strp->stm.full_len);
+
+	if (likely(!strp->copy_mode))
+		tcp_read_done(strp->sk, strp->stm.full_len);
+	else
+		tls_strp_flush_anchor_copy(strp);
+
+	strp->msg_ready = 0;
+	memset(&strp->stm, 0, sizeof(strp->stm));
+
+	tls_strp_check_rcv(strp);
+}
+
+void tls_strp_stop(struct tls_strparser *strp)
+{
+	strp->stopped = 1;
+}
+
+int tls_strp_init(struct tls_strparser *strp, struct sock *sk)
+{
+	memset(strp, 0, sizeof(*strp));
+
+	strp->sk = sk;
+
+	strp->anchor = alloc_skb(0, GFP_KERNEL);
+	if (!strp->anchor)
+		return -ENOMEM;
+
+	INIT_WORK(&strp->work, tls_strp_work);
 
-	nsg = skb_cow_data(ctx->recv_pkt, 0, &unused);
-	if (nsg < 0)
-		return nsg;
 	return 0;
 }
 
-int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
-		      struct sk_buff_head *dst)
+/* strp must already be stopped so that tls_strp_recv will no longer be called.
+ * Note that tls_strp_done is not called with the lower socket held.
+ */
+void tls_strp_done(struct tls_strparser *strp)
 {
-	struct sk_buff *clone;
+	WARN_ON(!strp->stopped);
 
-	clone = skb_clone(skb, sk->sk_allocation);
-	if (!clone)
+	cancel_work_sync(&strp->work);
+	tls_strp_anchor_free(strp);
+}
+
+int __init tls_strp_dev_init(void)
+{
+	tls_strp_wq = create_singlethread_workqueue("kstrp");
+	if (unlikely(!tls_strp_wq))
 		return -ENOMEM;
-	__skb_queue_tail(dst, clone);
+
 	return 0;
 }
+
+void tls_strp_dev_exit(void)
+{
+	destroy_workqueue(tls_strp_wq);
+}
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index bd4486819e64..0fc24a5ce208 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -1283,7 +1283,7 @@ int tls_sw_sendpage(struct sock *sk, struct page *page,
 
 static int
 tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
-		long timeo)
+		bool released, long timeo)
 {
 	struct tls_context *tls_ctx = tls_get_ctx(sk);
 	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
@@ -1297,7 +1297,7 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
 			return sock_error(sk);
 
 		if (!skb_queue_empty(&sk->sk_receive_queue)) {
-			__strp_unpause(&ctx->strp);
+			tls_strp_check_rcv(&ctx->strp);
 			if (tls_strp_msg_ready(ctx))
 				break;
 		}
@@ -1311,6 +1311,7 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
 		if (nonblock || !timeo)
 			return -EAGAIN;
 
+		released = true;
 		add_wait_queue(sk_sleep(sk), &wait);
 		sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
 		sk_wait_event(sk, &timeo,
@@ -1325,6 +1326,8 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
 			return sock_intr_errno(timeo);
 	}
 
+	tls_strp_msg_load(&ctx->strp, released);
+
 	return 1;
 }
 
@@ -1570,7 +1573,7 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov,
 	clear_skb = NULL;
 
 	if (unlikely(darg->async)) {
-		err = tls_strp_msg_hold(sk, skb, &ctx->async_hold);
+		err = tls_strp_msg_hold(&ctx->strp, &ctx->async_hold);
 		if (err)
 			__skb_queue_tail(&ctx->async_hold, darg->skb);
 		return err;
@@ -1734,9 +1737,7 @@ static int tls_record_content_type(struct msghdr *msg, struct tls_msg *tlm,
 
 static void tls_rx_rec_done(struct tls_sw_context_rx *ctx)
 {
-	consume_skb(ctx->recv_pkt);
-	ctx->recv_pkt = NULL;
-	__strp_unpause(&ctx->strp);
+	tls_strp_msg_done(&ctx->strp);
 }
 
 /* This function traverses the rx_list in tls receive context to copies the
@@ -1823,7 +1824,7 @@ static int process_rx_list(struct tls_sw_context_rx *ctx,
 	return copied ? : err;
 }
 
-static void
+static bool
 tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot,
 		       size_t len_left, size_t decrypted, ssize_t done,
 		       size_t *flushed_at)
@@ -1831,14 +1832,14 @@ tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot,
 	size_t max_rec;
 
 	if (len_left <= decrypted)
-		return;
+		return false;
 
 	max_rec = prot->overhead_size - prot->tail_size + TLS_MAX_PAYLOAD_SIZE;
 	if (done - *flushed_at < SZ_128K && tcp_inq(sk) > max_rec)
-		return;
+		return false;
 
 	*flushed_at = done;
-	sk_flush_backlog(sk);
+	return sk_flush_backlog(sk);
 }
 
 static long tls_rx_reader_lock(struct sock *sk, struct tls_sw_context_rx *ctx,
@@ -1916,6 +1917,7 @@ int tls_sw_recvmsg(struct sock *sk,
 	long timeo;
 	bool is_kvec = iov_iter_is_kvec(&msg->msg_iter);
 	bool is_peek = flags & MSG_PEEK;
+	bool released = true;
 	bool bpf_strp_enabled;
 	bool zc_capable;
 
@@ -1952,7 +1954,8 @@ int tls_sw_recvmsg(struct sock *sk,
 		struct tls_decrypt_arg darg;
 		int to_decrypt, chunk;
 
-		err = tls_rx_rec_wait(sk, psock, flags & MSG_DONTWAIT, timeo);
+		err = tls_rx_rec_wait(sk, psock, flags & MSG_DONTWAIT, released,
+				      timeo);
 		if (err <= 0) {
 			if (psock) {
 				chunk = sk_msg_recvmsg(sk, psock, msg, len,
@@ -1968,8 +1971,8 @@ int tls_sw_recvmsg(struct sock *sk,
 
 		memset(&darg.inargs, 0, sizeof(darg.inargs));
 
-		rxm = strp_msg(ctx->recv_pkt);
-		tlm = tls_msg(ctx->recv_pkt);
+		rxm = strp_msg(tls_strp_msg(ctx));
+		tlm = tls_msg(tls_strp_msg(ctx));
 
 		to_decrypt = rxm->full_len - prot->overhead_size;
 
@@ -2008,8 +2011,9 @@ int tls_sw_recvmsg(struct sock *sk,
 		}
 
 		/* periodically flush backlog, and feed strparser */
-		tls_read_flush_backlog(sk, prot, len, to_decrypt,
-				       decrypted + copied, &flushed_at);
+		released = tls_read_flush_backlog(sk, prot, len, to_decrypt,
+						  decrypted + copied,
+						  &flushed_at);
 
 		/* TLS 1.3 may have updated the length by more than overhead */
 		rxm = strp_msg(darg.skb);
@@ -2020,7 +2024,7 @@ int tls_sw_recvmsg(struct sock *sk,
 			bool partially_consumed = chunk > len;
 			struct sk_buff *skb = darg.skb;
 
-			DEBUG_NET_WARN_ON_ONCE(darg.skb == ctx->recv_pkt);
+			DEBUG_NET_WARN_ON_ONCE(darg.skb == tls_strp_msg(ctx));
 
 			if (async) {
 				/* TLS 1.2-only, to_decrypt must be text len */
@@ -2034,6 +2038,7 @@ int tls_sw_recvmsg(struct sock *sk,
 			}
 
 			if (bpf_strp_enabled) {
+				released = true;
 				err = sk_psock_tls_strp_read(psock, skb);
 				if (err != __SK_PASS) {
 					rxm->offset = rxm->offset + rxm->full_len;
@@ -2140,7 +2145,7 @@ ssize_t tls_sw_splice_read(struct socket *sock,  loff_t *ppos,
 		struct tls_decrypt_arg darg;
 
 		err = tls_rx_rec_wait(sk, NULL, flags & SPLICE_F_NONBLOCK,
-				      timeo);
+				      true, timeo);
 		if (err <= 0)
 			goto splice_read_end;
 
@@ -2204,19 +2209,17 @@ bool tls_sw_sock_is_readable(struct sock *sk)
 		!skb_queue_empty(&ctx->rx_list);
 }
 
-static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
+int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb)
 {
 	struct tls_context *tls_ctx = tls_get_ctx(strp->sk);
 	struct tls_prot_info *prot = &tls_ctx->prot_info;
 	char header[TLS_HEADER_SIZE + MAX_IV_SIZE];
-	struct strp_msg *rxm = strp_msg(skb);
-	struct tls_msg *tlm = tls_msg(skb);
 	size_t cipher_overhead;
 	size_t data_len = 0;
 	int ret;
 
 	/* Verify that we have a full TLS header, or wait for more data */
-	if (rxm->offset + prot->prepend_size > skb->len)
+	if (strp->stm.offset + prot->prepend_size > skb->len)
 		return 0;
 
 	/* Sanity-check size of on-stack buffer. */
@@ -2226,11 +2229,11 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
 	}
 
 	/* Linearize header to local buffer */
-	ret = skb_copy_bits(skb, rxm->offset, header, prot->prepend_size);
+	ret = skb_copy_bits(skb, strp->stm.offset, header, prot->prepend_size);
 	if (ret < 0)
 		goto read_failure;
 
-	tlm->control = header[0];
+	strp->mark = header[0];
 
 	data_len = ((header[4] & 0xFF) | (header[3] << 8));
 
@@ -2257,7 +2260,7 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
 	}
 
 	tls_device_rx_resync_new_rec(strp->sk, data_len + TLS_HEADER_SIZE,
-				     TCP_SKB_CB(skb)->seq + rxm->offset);
+				     TCP_SKB_CB(skb)->seq + strp->stm.offset);
 	return data_len + TLS_HEADER_SIZE;
 
 read_failure:
@@ -2266,14 +2269,11 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
 	return ret;
 }
 
-static void tls_queue(struct strparser *strp, struct sk_buff *skb)
+void tls_rx_msg_ready(struct tls_strparser *strp)
 {
-	struct tls_context *tls_ctx = tls_get_ctx(strp->sk);
-	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
-
-	ctx->recv_pkt = skb;
-	strp_pause(strp);
+	struct tls_sw_context_rx *ctx;
 
+	ctx = container_of(strp, struct tls_sw_context_rx, strp);
 	ctx->saved_data_ready(strp->sk);
 }
 
@@ -2283,7 +2283,7 @@ static void tls_data_ready(struct sock *sk)
 	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
 	struct sk_psock *psock;
 
-	strp_data_ready(&ctx->strp);
+	tls_strp_data_ready(&ctx->strp);
 
 	psock = sk_psock_get(sk);
 	if (psock) {
@@ -2359,13 +2359,11 @@ void tls_sw_release_resources_rx(struct sock *sk)
 	kfree(tls_ctx->rx.iv);
 
 	if (ctx->aead_recv) {
-		kfree_skb(ctx->recv_pkt);
-		ctx->recv_pkt = NULL;
 		__skb_queue_purge(&ctx->rx_list);
 		crypto_free_aead(ctx->aead_recv);
-		strp_stop(&ctx->strp);
+		tls_strp_stop(&ctx->strp);
 		/* If tls_sw_strparser_arm() was not called (cleanup paths)
-		 * we still want to strp_stop(), but sk->sk_data_ready was
+		 * we still want to tls_strp_stop(), but sk->sk_data_ready was
 		 * never swapped.
 		 */
 		if (ctx->saved_data_ready) {
@@ -2380,7 +2378,7 @@ void tls_sw_strparser_done(struct tls_context *tls_ctx)
 {
 	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
 
-	strp_done(&ctx->strp);
+	tls_strp_done(&ctx->strp);
 }
 
 void tls_sw_free_ctx_rx(struct tls_context *tls_ctx)
@@ -2453,8 +2451,6 @@ void tls_sw_strparser_arm(struct sock *sk, struct tls_context *tls_ctx)
 	rx_ctx->saved_data_ready = sk->sk_data_ready;
 	sk->sk_data_ready = tls_data_ready;
 	write_unlock_bh(&sk->sk_callback_lock);
-
-	strp_check_rcv(&rx_ctx->strp);
 }
 
 void tls_update_rx_zc_capable(struct tls_context *tls_ctx)
@@ -2474,7 +2470,6 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx)
 	struct tls_sw_context_rx *sw_ctx_rx = NULL;
 	struct cipher_context *cctx;
 	struct crypto_aead **aead;
-	struct strp_callbacks cb;
 	u16 nonce_size, tag_size, iv_size, rec_seq_size, salt_size;
 	struct crypto_tfm *tfm;
 	char *iv, *rec_seq, *key, *salt, *cipher_name;
@@ -2708,12 +2703,7 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx)
 			crypto_info->version != TLS_1_3_VERSION &&
 			!!(tfm->__crt_alg->cra_flags & CRYPTO_ALG_ASYNC);
 
-		/* Set up strparser */
-		memset(&cb, 0, sizeof(cb));
-		cb.rcv_msg = tls_queue;
-		cb.parse_msg = tls_read_size;
-
-		strp_init(&sw_ctx_rx->strp, sk, &cb);
+		tls_strp_init(&sw_ctx_rx->strp, sk);
 	}
 
 	goto out;
-- 
2.37.1

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Paolo Abeni]

Hello,

On Fri, 2022-07-22 at 16:50 -0700, Jakub Kicinski wrote:
> TLS is a relatively poor fit for strparser. We pause the input
> every time a message is received, wait for a read which will
> decrypt the message, start the parser, repeat. strparser is
> built to delineate the messages, wrap them in individual skbs
> and let them float off into the stack or a different socket.
> TLS wants the data pages and nothing else. There's no need
> for TLS to keep cloning (and occasionally skb_unclone()'ing)
> the TCP rx queue.
> 
> This patch uses a pre-allocated skb and attaches the skbs
> from the TCP rx queue to it as frags. TLS is careful never
> to modify the input skb without CoW'ing / detaching it first.
> 
> Since we call TCP rx queue cleanup directly we also get back
> the benefit of skb deferred free.
> 
> Overall this results in a 6% gain in my benchmarks.
> 
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>

I'm sorry for the incremental feedback, I have a few minor comment
below, hopefully nothing blocking.

> ---
> CC: borisp@nvidia.com
> CC: john.fastabend@gmail.com
> ---
>  include/net/tls.h  |  19 +-
>  net/tls/tls.h      |  24 ++-
>  net/tls/tls_main.c |  20 +-
>  net/tls/tls_strp.c | 484 +++++++++++++++++++++++++++++++++++++++++++--
>  net/tls/tls_sw.c   |  80 ++++----
>  5 files changed, 558 insertions(+), 69 deletions(-)
> 
> diff --git a/include/net/tls.h b/include/net/tls.h
> index 181c496b01b8..abb050b0df83 100644
> --- a/include/net/tls.h
> +++ b/include/net/tls.h
> @@ -108,18 +108,33 @@ struct tls_sw_context_tx {
>  	unsigned long tx_bitmask;
>  };
>  
> +struct tls_strparser {
> +	struct sock *sk;
> +
> +	u32 mark : 8;
> +	u32 stopped : 1;
> +	u32 copy_mode : 1;
> +	u32 msg_ready : 1;
> +
> +	struct strp_msg stm;
> +
> +	struct sk_buff *anchor;
> +	struct work_struct work;
> +};
> +
>  struct tls_sw_context_rx {
>  	struct crypto_aead *aead_recv;
>  	struct crypto_wait async_wait;
> -	struct strparser strp;
>  	struct sk_buff_head rx_list;	/* list of decrypted 'data' records */
>  	void (*saved_data_ready)(struct sock *sk);
>  
> -	struct sk_buff *recv_pkt;
>  	u8 reader_present;
>  	u8 async_capable:1;
>  	u8 zc_capable:1;
>  	u8 reader_contended:1;
> +
> +	struct tls_strparser strp;
> +
>  	atomic_t decrypt_pending;
>  	/* protect crypto_wait with decrypt_pending*/
>  	spinlock_t decrypt_compl_lock;
> diff --git a/net/tls/tls.h b/net/tls/tls.h
> index 154a3773e785..0e840a0c3437 100644
> --- a/net/tls/tls.h
> +++ b/net/tls/tls.h
> @@ -1,4 +1,5 @@
>  /*
> + * Copyright (c) 2016 Tom Herbert <tom@herbertland.com>

It's a little strange to me the above line ??! digging this file
history, you created it out of include/net/tls.h and the latter was
originally authored by Dave Watson (modulo ENOCOFFEE here...)

[...]

> diff --git a/net/tls/tls_strp.c b/net/tls/tls_strp.c
> index d9bb4f23f01a..b945288c312e 100644
> --- a/net/tls/tls_strp.c
> +++ b/net/tls/tls_strp.c
> @@ -1,37 +1,493 @@
>  // SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright (c) 2016 Tom Herbert <tom@herbertland.com> */

Same here ...

>  #include <linux/skbuff.h>
> +#include <linux/workqueue.h>
> +#include <net/strparser.h>
> +#include <net/tcp.h>
> +#include <net/sock.h>
> +#include <net/tls.h>
>  
>  #include "tls.h"
>  
> -struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx)
> +static struct workqueue_struct *tls_strp_wq;
> +
> +static void tls_strp_abort_strp(struct tls_strparser *strp, int err)
> +{
> +	if (strp->stopped)
> +		return;
> +
> +	strp->stopped = 1;
> +
> +	/* Report an error on the lower socket */
> +	strp->sk->sk_err = -err;
> +	sk_error_report(strp->sk);
> +}
> +
> +static void tls_strp_anchor_free(struct tls_strparser *strp)
>  {
> +	struct skb_shared_info *shinfo = skb_shinfo(strp->anchor);
> +
> +	DEBUG_NET_WARN_ON_ONCE(atomic_read(&shinfo->dataref) != 1);
> +	shinfo->frag_list = NULL;
> +	consume_skb(strp->anchor);
> +	strp->anchor = NULL;
> +}
> +
> +/* Create a new skb with the contents of input copied to its page frags */
> +static struct sk_buff *tls_strp_msg_make_copy(struct tls_strparser *strp)
> +{
> +	struct strp_msg *rxm;
>  	struct sk_buff *skb;
> +	int i, err, offset;
> +
> +	skb = alloc_skb_with_frags(0, strp->anchor->len, TLS_PAGE_ORDER,
> +				   &err, strp->sk->sk_allocation);
> +	if (!skb)
> +		return NULL;
> +
> +	offset = strp->stm.offset;
> +	for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
> +		skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
>  
> -	skb = ctx->recv_pkt;
> -	ctx->recv_pkt = NULL;
> +		WARN_ON_ONCE(skb_copy_bits(strp->anchor, offset,
> +					   skb_frag_address(frag),
> +					   skb_frag_size(frag)));
> +		offset += skb_frag_size(frag);
> +	}
> +
> +	skb_copy_header(skb, strp->anchor);
> +	rxm = strp_msg(skb);
> +	rxm->offset = 0;
>  	return skb;
>  }
>  
> +/* Steal the input skb, input msg is invalid after calling this function */
> +struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx)
> +{
> +	struct tls_strparser *strp = &ctx->strp;
> +
> +#ifdef CONFIG_TLS_DEVICE
> +	DEBUG_NET_WARN_ON_ONCE(!strp->anchor->decrypted);
> +#else
> +	/* This function turns an input into an output,
> +	 * that can only happen if we have offload.
> +	 */
> +	WARN_ON(1);
> +#endif
> +
> +	if (strp->copy_mode) {
> +		struct sk_buff *skb;
> +
> +		/* Replace anchor with an empty skb, this is a little
> +		 * dangerous but __tls_cur_msg() warns on empty skbs
> +		 * so hopefully we'll catch abuses.
> +		 */
> +		skb = alloc_skb(0, strp->sk->sk_allocation);
> +		if (!skb)
> +			return NULL;
> +
> +		swap(strp->anchor, skb);
> +		return skb;
> +	}
> +
> +	return tls_strp_msg_make_copy(strp);
> +}
> +
> +/* Force the input skb to be in copy mode. The data ownership remains
> + * with the input skb itself (meaning unpause will wipe it) but it can
> + * be modified.
> + */
>  int tls_strp_msg_cow(struct tls_sw_context_rx *ctx)
>  {
> -	struct sk_buff *unused;
> -	int nsg;
> +	struct tls_strparser *strp = &ctx->strp;
> +	struct sk_buff *skb;
> +
> +	if (strp->copy_mode)
> +		return 0;
> +
> +	skb = tls_strp_msg_make_copy(strp);
> +	if (!skb)
> +		return -ENOMEM;
> +
> +	tls_strp_anchor_free(strp);
> +	strp->anchor = skb;
> +
> +	tcp_read_done(strp->sk, strp->stm.full_len);
> +	strp->copy_mode = 1;
> +
> +	return 0;
> +}
> +
> +/* Make a clone (in the skb sense) of the input msg to keep a reference
> + * to the underlying data. The reference-holding skbs get placed on
> + * @dst.
> + */
> +int tls_strp_msg_hold(struct tls_strparser *strp, struct sk_buff_head *dst)
> +{
> +	struct skb_shared_info *shinfo = skb_shinfo(strp->anchor);
> +
> +	if (strp->copy_mode) {
> +		struct sk_buff *skb;
> +
> +		WARN_ON_ONCE(!shinfo->nr_frags);
> +
> +		/* We can't skb_clone() the anchor, it gets wiped by unpause */
> +		skb = alloc_skb(0, strp->sk->sk_allocation);
> +		if (!skb)
> +			return -ENOMEM;
> +
> +		__skb_queue_tail(dst, strp->anchor);
> +		strp->anchor = skb;
> +	} else {
> +		struct sk_buff *iter, *clone;
> +		int chunk, len, offset;
> +
> +		offset = strp->stm.offset;
> +		len = strp->stm.full_len;
> +		iter = shinfo->frag_list;
> +
> +		while (len > 0) {
> +			if (iter->len <= offset) {
> +				offset -= iter->len;
> +				goto next;
> +			}
> +
> +			chunk = iter->len - offset;
> +			offset = 0;
> +
> +			clone = skb_clone(iter, strp->sk->sk_allocation);
> +			if (!clone)
> +				return -ENOMEM;
> +			__skb_queue_tail(dst, clone);
> +
> +			len -= chunk;
> +next:
> +			iter = iter->next;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static void tls_strp_flush_anchor_copy(struct tls_strparser *strp)
> +{
> +	struct skb_shared_info *shinfo = skb_shinfo(strp->anchor);
> +	int i;
> +
> +	DEBUG_NET_WARN_ON_ONCE(atomic_read(&shinfo->dataref) != 1);
> +
> +	for (i = 0; i < shinfo->nr_frags; i++)
> +		__skb_frag_unref(&shinfo->frags[i], false);
> +	shinfo->nr_frags = 0;
> +	strp->copy_mode = 0;
> +}
> +
> +static int tls_strp_copyin(read_descriptor_t *desc, struct sk_buff *in_skb,
> +			   unsigned int offset, size_t in_len)
> +{
> +	struct tls_strparser *strp = (struct tls_strparser *)desc->arg.data;
> +	size_t sz, len, chunk;
> +	struct sk_buff *skb;
> +	skb_frag_t *frag;
> +
> +	if (strp->msg_ready)
> +		return 0;
> +
> +	skb = strp->anchor;
> +	frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE];

I'm wondering if TSOv2 GRO packets can reach here? Even without TSO v2,
I *think* the TCP stack is allowed to grow queued skbs above 64K via
tcp_queue_rcv()/tcp_try_coalesce().

> +
> +	len = in_len;
> +	/* First make sure we got the header */
> +	if (!strp->stm.full_len) {
> +		/* Assume one page is more than enough for headers */
> +		chunk =	min_t(size_t, len, PAGE_SIZE - skb_frag_size(frag));
> +		WARN_ON_ONCE(skb_copy_bits(in_skb, offset,
> +					   skb_frag_address(frag) +
> +					   skb_frag_size(frag),
> +					   chunk));
> +
> +		sz = tls_rx_msg_size(strp, strp->anchor);
> +		if (sz < 0) {
> +			desc->error = sz;
> +			return 0;
> +		}
> +
> +		/* We may have over-read, sz == 0 is guaranteed under-read */
> +		if (sz > 0)
> +			chunk =	min_t(size_t, chunk, sz - skb->len);
> +
> +		skb->len += chunk;
> +		skb->data_len += chunk;
> +		skb_frag_size_add(frag, chunk);
> +		frag++;
> +		len -= chunk;
> +		offset += chunk;
> +
> +		strp->stm.full_len = sz;
> +		if (!strp->stm.full_len)
> +			goto read_done;
> +	}
> +
> +	/* Load up more data */
> +	while (len && strp->stm.full_len > skb->len) {
> +		chunk =	min_t(size_t, len, strp->stm.full_len - skb->len);
> +		chunk = min_t(size_t, chunk, PAGE_SIZE - skb_frag_size(frag));
> +		WARN_ON_ONCE(skb_copy_bits(in_skb, offset,
> +					   skb_frag_address(frag) +
> +					   skb_frag_size(frag),
> +					   chunk));
> +
> +		skb->len += chunk;
> +		skb->data_len += chunk;
> +		skb_frag_size_add(frag, chunk);
> +		frag++;
> +		len -= chunk;
> +		offset += chunk;
> +	}
> +
> +	if (strp->stm.full_len == skb->len) {
> +		desc->count = 0;
> +
> +		strp->msg_ready = 1;
> +		tls_rx_msg_ready(strp);
> +	}
> +
> +read_done:
> +	return in_len - len;
> +}
> +
> +static int tls_strp_read_copyin(struct tls_strparser *strp)
> +{
> +	struct socket *sock = strp->sk->sk_socket;
> +	read_descriptor_t desc;
> +
> +	desc.arg.data = strp;
> +	desc.error = 0;
> +	desc.count = 1; /* give more than one skb per call */
> +
> +	/* sk should be locked here, so okay to do read_sock */
> +	sock->ops->read_sock(strp->sk, &desc, tls_strp_copyin);

If you are concerned by indirect calls/retpoline, you can use directly
tcp_read_sock here, as read_sock is always tcp_read_sock since commit
965b57b469a589d64d81b1688b38dcb537011bb0. Or you can use
indirect_call_wrapper.h

> +
> +	return desc.error;
> +}
> +
> +static int tls_strp_read_short(struct tls_strparser *strp)
> +{
> +	struct skb_shared_info *shinfo;
> +	struct page *page;
> +	int need_spc, len;
> +
> +	/* If the rbuf is small or rcv window has collapsed to 0 we need
> +	 * to read the data out. Otherwise the connection will stall.
> +	 * Without pressure threshold of INT_MAX will never be ready.
> +	 */
> +	if (likely(!tcp_epollin_ready(strp->sk, INT_MAX)))
> +		return 0;
> +
> +	shinfo = skb_shinfo(strp->anchor);
> +	shinfo->frag_list = NULL;
> +
> +	/* If we don't know the length go max plus page for cipher overhead */
> +	need_spc = strp->stm.full_len ?: TLS_MAX_PAYLOAD_SIZE + PAGE_SIZE;
> +
> +	for (len = need_spc; len > 0; len -= PAGE_SIZE) {
> +		page = alloc_page(strp->sk->sk_allocation);
> +		if (!page) {
> +			tls_strp_flush_anchor_copy(strp);
> +			return -ENOMEM;
> +		}
> +
> +		skb_fill_page_desc(strp->anchor, shinfo->nr_frags++,
> +				   page, 0, 0);
> +	}
> +
> +	strp->copy_mode = 1;
> +	strp->stm.offset = 0;
> +
> +	strp->anchor->len = 0;
> +	strp->anchor->data_len = 0;
> +	strp->anchor->truesize = round_up(need_spc, PAGE_SIZE);
> +
> +	tls_strp_read_copyin(strp);
> +
> +	return 0;
> +}
> +
> +static void tls_strp_load_anchor_with_queue(struct tls_strparser *strp, int len)
> +{
> +	struct tcp_sock *tp = tcp_sk(strp->sk);
> +	struct sk_buff *first;
> +	u32 offset;
> +
> +	first = tcp_recv_skb(strp->sk, tp->copied_seq, &offset);
> +	if (WARN_ON_ONCE(!first))
> +		return;
> +
> +	/* Bestow the state onto the anchor */
> +	strp->anchor->len = offset + len;
> +	strp->anchor->data_len = offset + len;
> +	strp->anchor->truesize = offset + len;
> +
> +	skb_shinfo(strp->anchor)->frag_list = first;
> +
> +	skb_copy_header(strp->anchor, first);
> +	strp->anchor->destructor = NULL;
> +
> +	strp->stm.offset = offset;
> +}
> +
> +void tls_strp_msg_load(struct tls_strparser *strp, bool force_refresh)
> +{
> +	struct strp_msg *rxm;
> +	struct tls_msg *tlm;
> +
> +	DEBUG_NET_WARN_ON_ONCE(!strp->msg_ready);
> +	DEBUG_NET_WARN_ON_ONCE(!strp->stm.full_len);
> +
> +	if (!strp->copy_mode && force_refresh) {
> +		if (WARN_ON(tcp_inq(strp->sk) < strp->stm.full_len))
> +			return;
> +
> +		tls_strp_load_anchor_with_queue(strp, strp->stm.full_len);
> +	}
> +
> +	rxm = strp_msg(strp->anchor);
> +	rxm->full_len	= strp->stm.full_len;
> +	rxm->offset	= strp->stm.offset;
> +	tlm = tls_msg(strp->anchor);
> +	tlm->control	= strp->mark;
> +}
> +
> +/* Called with lock held on lower socket */
> +static int tls_strp_read_sock(struct tls_strparser *strp)
> +{
> +	int sz, inq;
> +
> +	inq = tcp_inq(strp->sk);
> +	if (inq < 1)
> +		return 0;
> +
> +	if (unlikely(strp->copy_mode))
> +		return tls_strp_read_copyin(strp);
> +
> +	if (inq < strp->stm.full_len)
> +		return tls_strp_read_short(strp);
> +
> +	if (!strp->stm.full_len) {
> +		tls_strp_load_anchor_with_queue(strp, inq);
> +
> +		sz = tls_rx_msg_size(strp, strp->anchor);
> +		if (sz < 0) {
> +			tls_strp_abort_strp(strp, sz);
> +			return sz;
> +		}
> +
> +		strp->stm.full_len = sz;
> +
> +		if (!strp->stm.full_len || inq < strp->stm.full_len)
> +			return tls_strp_read_short(strp);
> +	}
> +
> +	strp->msg_ready = 1;
> +	tls_rx_msg_ready(strp);
> +
> +	return 0;
> +}
> +
> +void tls_strp_check_rcv(struct tls_strparser *strp)
> +{
> +	if (unlikely(strp->stopped) || strp->msg_ready)
> +		return;
> +
> +	if (tls_strp_read_sock(strp) == -ENOMEM)
> +		queue_work(tls_strp_wq, &strp->work);
> +}
> +
> +/* Lower sock lock held */
> +void tls_strp_data_ready(struct tls_strparser *strp)
> +{
> +	/* This check is needed to synchronize with do_tls_strp_work.
> +	 * do_tls_strp_work acquires a process lock (lock_sock) whereas
> +	 * the lock held here is bh_lock_sock. The two locks can be
> +	 * held by different threads at the same time, but bh_lock_sock
> +	 * allows a thread in BH context to safely check if the process
> +	 * lock is held. In this case, if the lock is held, queue work.
> +	 */
> +	if (sock_owned_by_user_nocheck(strp->sk)) {
> +		queue_work(tls_strp_wq, &strp->work);
> +		return;
> +	}
> +
> +	tls_strp_check_rcv(strp);
> +}
> +
> +static void tls_strp_work(struct work_struct *w)
> +{
> +	struct tls_strparser *strp =
> +		container_of(w, struct tls_strparser, work);
> +
> +	lock_sock(strp->sk);
> +	tls_strp_check_rcv(strp);
> +	release_sock(strp->sk);
> +}
> +
> +void tls_strp_msg_done(struct tls_strparser *strp)
> +{
> +	WARN_ON(!strp->stm.full_len);
> +
> +	if (likely(!strp->copy_mode))
> +		tcp_read_done(strp->sk, strp->stm.full_len);
> +	else
> +		tls_strp_flush_anchor_copy(strp);
> +
> +	strp->msg_ready = 0;
> +	memset(&strp->stm, 0, sizeof(strp->stm));
> +
> +	tls_strp_check_rcv(strp);
> +}
> +
> +void tls_strp_stop(struct tls_strparser *strp)
> +{
> +	strp->stopped = 1;
> +}
> +
> +int tls_strp_init(struct tls_strparser *strp, struct sock *sk)
> +{
> +	memset(strp, 0, sizeof(*strp));
> +
> +	strp->sk = sk;
> +
> +	strp->anchor = alloc_skb(0, GFP_KERNEL);
> +	if (!strp->anchor)
> +		return -ENOMEM;
> +
> +	INIT_WORK(&strp->work, tls_strp_work);
>  
> -	nsg = skb_cow_data(ctx->recv_pkt, 0, &unused);
> -	if (nsg < 0)
> -		return nsg;
>  	return 0;
>  }
>  
> -int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
> -		      struct sk_buff_head *dst)
> +/* strp must already be stopped so that tls_strp_recv will no longer be called.
> + * Note that tls_strp_done is not called with the lower socket held.
> + */
> +void tls_strp_done(struct tls_strparser *strp)
>  {
> -	struct sk_buff *clone;
> +	WARN_ON(!strp->stopped);
>  
> -	clone = skb_clone(skb, sk->sk_allocation);
> -	if (!clone)
> +	cancel_work_sync(&strp->work);
> +	tls_strp_anchor_free(strp);
> +}
> +
> +int __init tls_strp_dev_init(void)
> +{
> +	tls_strp_wq = create_singlethread_workqueue("kstrp");

I guess it's better to change the name to avoid confusing with plain
strparser ?!?

Out of sheer ignorance and not related to this patch: If I read
correctly, the above means that multiple tls flows on top of different
TCP sockets will use a single CPU, isn't that a relevant bottle-neck?
isn't enough to rely on queue_work() to submit the work on the same CPU
that just did the TCP stack processing? 

> +	if (unlikely(!tls_strp_wq))
>  		return -ENOMEM;
> -	__skb_queue_tail(dst, clone);
> +
>  	return 0;
>  }
> +
> +void tls_strp_dev_exit(void)
> +{
> +	destroy_workqueue(tls_strp_wq);
> +}
> diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
> index bd4486819e64..0fc24a5ce208 100644
> --- a/net/tls/tls_sw.c
> +++ b/net/tls/tls_sw.c
> @@ -1283,7 +1283,7 @@ int tls_sw_sendpage(struct sock *sk, struct page *page,
>  
>  static int
>  tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
> -		long timeo)
> +		bool released, long timeo)
>  {
>  	struct tls_context *tls_ctx = tls_get_ctx(sk);
>  	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
> @@ -1297,7 +1297,7 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
>  			return sock_error(sk);
>  
>  		if (!skb_queue_empty(&sk->sk_receive_queue)) {
> -			__strp_unpause(&ctx->strp);
> +			tls_strp_check_rcv(&ctx->strp);
>  			if (tls_strp_msg_ready(ctx))
>  				break;
>  		}
> @@ -1311,6 +1311,7 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
>  		if (nonblock || !timeo)
>  			return -EAGAIN;
>  
> +		released = true;
>  		add_wait_queue(sk_sleep(sk), &wait);
>  		sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
>  		sk_wait_event(sk, &timeo,
> @@ -1325,6 +1326,8 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
>  			return sock_intr_errno(timeo);
>  	}
>  
> +	tls_strp_msg_load(&ctx->strp, released);
> +
>  	return 1;
>  }
>  
> @@ -1570,7 +1573,7 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov,
>  	clear_skb = NULL;
>  
>  	if (unlikely(darg->async)) {
> -		err = tls_strp_msg_hold(sk, skb, &ctx->async_hold);
> +		err = tls_strp_msg_hold(&ctx->strp, &ctx->async_hold);
>  		if (err)
>  			__skb_queue_tail(&ctx->async_hold, darg->skb);
>  		return err;
> @@ -1734,9 +1737,7 @@ static int tls_record_content_type(struct msghdr *msg, struct tls_msg *tlm,
>  
>  static void tls_rx_rec_done(struct tls_sw_context_rx *ctx)
>  {
> -	consume_skb(ctx->recv_pkt);
> -	ctx->recv_pkt = NULL;
> -	__strp_unpause(&ctx->strp);
> +	tls_strp_msg_done(&ctx->strp);
>  }
>  
>  /* This function traverses the rx_list in tls receive context to copies the
> @@ -1823,7 +1824,7 @@ static int process_rx_list(struct tls_sw_context_rx *ctx,
>  	return copied ? : err;
>  }
>  
> -static void
> +static bool
>  tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot,
>  		       size_t len_left, size_t decrypted, ssize_t done,
>  		       size_t *flushed_at)
> @@ -1831,14 +1832,14 @@ tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot,
>  	size_t max_rec;
>  
>  	if (len_left <= decrypted)
> -		return;
> +		return false;
>  
>  	max_rec = prot->overhead_size - prot->tail_size + TLS_MAX_PAYLOAD_SIZE;
>  	if (done - *flushed_at < SZ_128K && tcp_inq(sk) > max_rec)
> -		return;
> +		return false;
>  
>  	*flushed_at = done;
> -	sk_flush_backlog(sk);
> +	return sk_flush_backlog(sk);
>  }
>  
>  static long tls_rx_reader_lock(struct sock *sk, struct tls_sw_context_rx *ctx,
> @@ -1916,6 +1917,7 @@ int tls_sw_recvmsg(struct sock *sk,
>  	long timeo;
>  	bool is_kvec = iov_iter_is_kvec(&msg->msg_iter);
>  	bool is_peek = flags & MSG_PEEK;
> +	bool released = true;
>  	bool bpf_strp_enabled;
>  	bool zc_capable;
>  
> @@ -1952,7 +1954,8 @@ int tls_sw_recvmsg(struct sock *sk,
>  		struct tls_decrypt_arg darg;
>  		int to_decrypt, chunk;
>  
> -		err = tls_rx_rec_wait(sk, psock, flags & MSG_DONTWAIT, timeo);
> +		err = tls_rx_rec_wait(sk, psock, flags & MSG_DONTWAIT, released,
> +				      timeo);
>  		if (err <= 0) {
>  			if (psock) {
>  				chunk = sk_msg_recvmsg(sk, psock, msg, len,
> @@ -1968,8 +1971,8 @@ int tls_sw_recvmsg(struct sock *sk,
>  
>  		memset(&darg.inargs, 0, sizeof(darg.inargs));
>  
> -		rxm = strp_msg(ctx->recv_pkt);
> -		tlm = tls_msg(ctx->recv_pkt);
> +		rxm = strp_msg(tls_strp_msg(ctx));
> +		tlm = tls_msg(tls_strp_msg(ctx));
>  
>  		to_decrypt = rxm->full_len - prot->overhead_size;
>  
> @@ -2008,8 +2011,9 @@ int tls_sw_recvmsg(struct sock *sk,
>  		}
>  
>  		/* periodically flush backlog, and feed strparser */
> -		tls_read_flush_backlog(sk, prot, len, to_decrypt,
> -				       decrypted + copied, &flushed_at);
> +		released = tls_read_flush_backlog(sk, prot, len, to_decrypt,
> +						  decrypted + copied,
> +						  &flushed_at);
>  
>  		/* TLS 1.3 may have updated the length by more than overhead */
>  		rxm = strp_msg(darg.skb);
> @@ -2020,7 +2024,7 @@ int tls_sw_recvmsg(struct sock *sk,
>  			bool partially_consumed = chunk > len;
>  			struct sk_buff *skb = darg.skb;
>  
> -			DEBUG_NET_WARN_ON_ONCE(darg.skb == ctx->recv_pkt);
> +			DEBUG_NET_WARN_ON_ONCE(darg.skb == tls_strp_msg(ctx));
>  
>  			if (async) {
>  				/* TLS 1.2-only, to_decrypt must be text len */
> @@ -2034,6 +2038,7 @@ int tls_sw_recvmsg(struct sock *sk,
>  			}
>  
>  			if (bpf_strp_enabled) {
> +				released = true;
>  				err = sk_psock_tls_strp_read(psock, skb);
>  				if (err != __SK_PASS) {
>  					rxm->offset = rxm->offset + rxm->full_len;
> @@ -2140,7 +2145,7 @@ ssize_t tls_sw_splice_read(struct socket *sock,  loff_t *ppos,
>  		struct tls_decrypt_arg darg;
>  
>  		err = tls_rx_rec_wait(sk, NULL, flags & SPLICE_F_NONBLOCK,
> -				      timeo);
> +				      true, timeo);
>  		if (err <= 0)
>  			goto splice_read_end;
>  
> @@ -2204,19 +2209,17 @@ bool tls_sw_sock_is_readable(struct sock *sk)
>  		!skb_queue_empty(&ctx->rx_list);
>  }
>  
> -static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
> +int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb)
>  {
>  	struct tls_context *tls_ctx = tls_get_ctx(strp->sk);
>  	struct tls_prot_info *prot = &tls_ctx->prot_info;
>  	char header[TLS_HEADER_SIZE + MAX_IV_SIZE];
> -	struct strp_msg *rxm = strp_msg(skb);
> -	struct tls_msg *tlm = tls_msg(skb);
>  	size_t cipher_overhead;
>  	size_t data_len = 0;
>  	int ret;
>  
>  	/* Verify that we have a full TLS header, or wait for more data */
> -	if (rxm->offset + prot->prepend_size > skb->len)
> +	if (strp->stm.offset + prot->prepend_size > skb->len)
>  		return 0;
>  
>  	/* Sanity-check size of on-stack buffer. */
> @@ -2226,11 +2229,11 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
>  	}
>  
>  	/* Linearize header to local buffer */
> -	ret = skb_copy_bits(skb, rxm->offset, header, prot->prepend_size);
> +	ret = skb_copy_bits(skb, strp->stm.offset, header, prot->prepend_size);
>  	if (ret < 0)
>  		goto read_failure;
>  
> -	tlm->control = header[0];
> +	strp->mark = header[0];
>  
>  	data_len = ((header[4] & 0xFF) | (header[3] << 8));
>  
> @@ -2257,7 +2260,7 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
>  	}
>  
>  	tls_device_rx_resync_new_rec(strp->sk, data_len + TLS_HEADER_SIZE,
> -				     TCP_SKB_CB(skb)->seq + rxm->offset);
> +				     TCP_SKB_CB(skb)->seq + strp->stm.offset);
>  	return data_len + TLS_HEADER_SIZE;
>  
>  read_failure:
> @@ -2266,14 +2269,11 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
>  	return ret;
>  }
>  
> -static void tls_queue(struct strparser *strp, struct sk_buff *skb)
> +void tls_rx_msg_ready(struct tls_strparser *strp)
>  {
> -	struct tls_context *tls_ctx = tls_get_ctx(strp->sk);
> -	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
> -
> -	ctx->recv_pkt = skb;
> -	strp_pause(strp);
> +	struct tls_sw_context_rx *ctx;
>  
> +	ctx = container_of(strp, struct tls_sw_context_rx, strp);
>  	ctx->saved_data_ready(strp->sk);
>  }
>  
> @@ -2283,7 +2283,7 @@ static void tls_data_ready(struct sock *sk)
>  	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
>  	struct sk_psock *psock;
>  
> -	strp_data_ready(&ctx->strp);
> +	tls_strp_data_ready(&ctx->strp);
>  
>  	psock = sk_psock_get(sk);
>  	if (psock) {
> @@ -2359,13 +2359,11 @@ void tls_sw_release_resources_rx(struct sock *sk)
>  	kfree(tls_ctx->rx.iv);
>  
>  	if (ctx->aead_recv) {
> -		kfree_skb(ctx->recv_pkt);
> -		ctx->recv_pkt = NULL;
>  		__skb_queue_purge(&ctx->rx_list);
>  		crypto_free_aead(ctx->aead_recv);
> -		strp_stop(&ctx->strp);
> +		tls_strp_stop(&ctx->strp);
>  		/* If tls_sw_strparser_arm() was not called (cleanup paths)
> -		 * we still want to strp_stop(), but sk->sk_data_ready was
> +		 * we still want to tls_strp_stop(), but sk->sk_data_ready was
>  		 * never swapped.
>  		 */
>  		if (ctx->saved_data_ready) {
> @@ -2380,7 +2378,7 @@ void tls_sw_strparser_done(struct tls_context *tls_ctx)
>  {
>  	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
>  
> -	strp_done(&ctx->strp);
> +	tls_strp_done(&ctx->strp);
>  }
>  
>  void tls_sw_free_ctx_rx(struct tls_context *tls_ctx)
> @@ -2453,8 +2451,6 @@ void tls_sw_strparser_arm(struct sock *sk, struct tls_context *tls_ctx)
>  	rx_ctx->saved_data_ready = sk->sk_data_ready;
>  	sk->sk_data_ready = tls_data_ready;
>  	write_unlock_bh(&sk->sk_callback_lock);
> -
> -	strp_check_rcv(&rx_ctx->strp);
>  }
>  
>  void tls_update_rx_zc_capable(struct tls_context *tls_ctx)
> @@ -2474,7 +2470,6 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx)
>  	struct tls_sw_context_rx *sw_ctx_rx = NULL;
>  	struct cipher_context *cctx;
>  	struct crypto_aead **aead;
> -	struct strp_callbacks cb;
>  	u16 nonce_size, tag_size, iv_size, rec_seq_size, salt_size;
>  	struct crypto_tfm *tfm;
>  	char *iv, *rec_seq, *key, *salt, *cipher_name;
> @@ -2708,12 +2703,7 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx)
>  			crypto_info->version != TLS_1_3_VERSION &&
>  			!!(tfm->__crt_alg->cra_flags & CRYPTO_ALG_ASYNC);
>  
> -		/* Set up strparser */
> -		memset(&cb, 0, sizeof(cb));
> -		cb.rcv_msg = tls_queue;
> -		cb.parse_msg = tls_read_size;
> -
> -		strp_init(&sw_ctx_rx->strp, sk, &cb);
> +		tls_strp_init(&sw_ctx_rx->strp, sk);
>  	}
>  
>  	goto out;

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

On Tue, 26 Jul 2022 11:27:36 +0200 Paolo Abeni wrote:
> On Fri, 2022-07-22 at 16:50 -0700, Jakub Kicinski wrote:
> > diff --git a/net/tls/tls.h b/net/tls/tls.h
> > index 154a3773e785..0e840a0c3437 100644
> > --- a/net/tls/tls.h
> > +++ b/net/tls/tls.h
> > @@ -1,4 +1,5 @@
> >  /*
> > + * Copyright (c) 2016 Tom Herbert <tom@herbertland.com>  
> 
> It's a little strange to me the above line ??! digging this file
> history, you created it out of include/net/tls.h and the latter was
> originally authored by Dave Watson (modulo ENOCOFFEE here...)
> 
> > +++ b/net/tls/tls_strp.c
> > @@ -1,37 +1,493 @@
> >  // SPDX-License-Identifier: GPL-2.0-only
> > +/* Copyright (c) 2016 Tom Herbert <tom@herbertland.com> */  
> 
> Same here ...

I tried to add the Copyrights as I copied some code around, since I'm
lazy around legal stuff. I think I copied parts of the strparser
at some point and the structure definition (workqueue handling?). 
I'd rather keep too many copyrights than too few, tho. 
The semi-custom license is more annoying :(

> > +static int tls_strp_copyin(read_descriptor_t *desc, struct sk_buff *in_skb,
> > +			   unsigned int offset, size_t in_len)
> > +{
> > +	struct tls_strparser *strp = (struct tls_strparser *)desc->arg.data;
> > +	size_t sz, len, chunk;
> > +	struct sk_buff *skb;
> > +	skb_frag_t *frag;
> > +
> > +	if (strp->msg_ready)
> > +		return 0;
> > +
> > +	skb = strp->anchor;
> > +	frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE];  
> 
> I'm wondering if TSOv2 GRO packets can reach here? Even without TSO v2,
> I *think* the TCP stack is allowed to grow queued skbs above 64K via
> tcp_queue_rcv()/tcp_try_coalesce().

I don't think the TSO skbs can get here, the @skb is completely
constructed by me and the length is bounded by max TLS record size
(16k + overheads). We should be safe to use 4k pages, I think.

> > +static int tls_strp_read_copyin(struct tls_strparser *strp)
> > +{
> > +	struct socket *sock = strp->sk->sk_socket;
> > +	read_descriptor_t desc;
> > +
> > +	desc.arg.data = strp;
> > +	desc.error = 0;
> > +	desc.count = 1; /* give more than one skb per call */
> > +
> > +	/* sk should be locked here, so okay to do read_sock */
> > +	sock->ops->read_sock(strp->sk, &desc, tls_strp_copyin);  
> 
> If you are concerned by indirect calls/retpoline, you can use directly
> tcp_read_sock here, as read_sock is always tcp_read_sock since commit
> 965b57b469a589d64d81b1688b38dcb537011bb0. Or you can use
> indirect_call_wrapper.h

This is a slowpath which only gets triggered if we are so rbuf
constrained that TCP will not be able to buffer a full record.
Otherwise we try to avoid doing any copying/read_sock at all.

> > -int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
> > -		      struct sk_buff_head *dst)
> > +/* strp must already be stopped so that tls_strp_recv will no longer be called.
> > + * Note that tls_strp_done is not called with the lower socket held.
> > + */
> > +void tls_strp_done(struct tls_strparser *strp)
> >  {
> > -	struct sk_buff *clone;
> > +	WARN_ON(!strp->stopped);
> >  
> > -	clone = skb_clone(skb, sk->sk_allocation);
> > -	if (!clone)
> > +	cancel_work_sync(&strp->work);
> > +	tls_strp_anchor_free(strp);
> > +}
> > +
> > +int __init tls_strp_dev_init(void)
> > +{
> > +	tls_strp_wq = create_singlethread_workqueue("kstrp");  
> 
> I guess it's better to change the name to avoid confusing with plain
> strparser ?!?
> 
> Out of sheer ignorance and not related to this patch: If I read
> correctly, the above means that multiple tls flows on top of different
> TCP sockets will use a single CPU, isn't that a relevant bottle-neck?
> isn't enough to rely on queue_work() to submit the work on the same CPU
> that just did the TCP stack processing? 

Oh yeah, this is a slow/rare path too but you're right. I copied this
code from strparser without thinking, not sure what the motivation was
there.

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Paolo Abeni]

On Tue, 2022-07-26 at 10:01 -0700, Jakub Kicinski wrote:
> On Tue, 26 Jul 2022 11:27:36 +0200 Paolo Abeni wrote:
> > On Fri, 2022-07-22 at 16:50 -0700, Jakub Kicinski wrote:
> > > diff --git a/net/tls/tls.h b/net/tls/tls.h
> > > index 154a3773e785..0e840a0c3437 100644
> > > --- a/net/tls/tls.h
> > > +++ b/net/tls/tls.h
> > > @@ -1,4 +1,5 @@
> > >  /*
> > > + * Copyright (c) 2016 Tom Herbert <tom@herbertland.com>  
> > 
> > It's a little strange to me the above line ??! digging this file
> > history, you created it out of include/net/tls.h and the latter was
> > originally authored by Dave Watson (modulo ENOCOFFEE here...)
> > 
> > > +++ b/net/tls/tls_strp.c
> > > @@ -1,37 +1,493 @@
> > >  // SPDX-License-Identifier: GPL-2.0-only
> > > +/* Copyright (c) 2016 Tom Herbert <tom@herbertland.com> */  
> > 
> > Same here ...
> 
> I tried to add the Copyrights as I copied some code around, since I'm
> lazy around legal stuff. I think I copied parts of the strparser
> at some point and the structure definition (workqueue handling?). 
> I'd rather keep too many copyrights than too few, tho. 
> The semi-custom license is more annoying :(
> 
> > > +static int tls_strp_copyin(read_descriptor_t *desc, struct sk_buff *in_skb,
> > > +			   unsigned int offset, size_t in_len)
> > > +{
> > > +	struct tls_strparser *strp = (struct tls_strparser *)desc->arg.data;
> > > +	size_t sz, len, chunk;
> > > +	struct sk_buff *skb;
> > > +	skb_frag_t *frag;
> > > +
> > > +	if (strp->msg_ready)
> > > +		return 0;
> > > +
> > > +	skb = strp->anchor;
> > > +	frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE];  
> > 
> > I'm wondering if TSOv2 GRO packets can reach here? Even without TSO v2,
> > I *think* the TCP stack is allowed to grow queued skbs above 64K via
> > tcp_queue_rcv()/tcp_try_coalesce().
> 
> I don't think the TSO skbs can get here, the @skb is completely
> constructed by me and the length is bounded by max TLS record size
> (16k + overheads). We should be safe to use 4k pages, I think.
> 
> > > +static int tls_strp_read_copyin(struct tls_strparser *strp)
> > > +{
> > > +	struct socket *sock = strp->sk->sk_socket;
> > > +	read_descriptor_t desc;
> > > +
> > > +	desc.arg.data = strp;
> > > +	desc.error = 0;
> > > +	desc.count = 1; /* give more than one skb per call */
> > > +
> > > +	/* sk should be locked here, so okay to do read_sock */
> > > +	sock->ops->read_sock(strp->sk, &desc, tls_strp_copyin);  
> > 
> > If you are concerned by indirect calls/retpoline, you can use directly
> > tcp_read_sock here, as read_sock is always tcp_read_sock since commit
> > 965b57b469a589d64d81b1688b38dcb537011bb0. Or you can use
> > indirect_call_wrapper.h
> 
> This is a slowpath which only gets triggered if we are so rbuf
> constrained that TCP will not be able to buffer a full record.
> Otherwise we try to avoid doing any copying/read_sock at all.
> 
> > > -int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
> > > -		      struct sk_buff_head *dst)
> > > +/* strp must already be stopped so that tls_strp_recv will no longer be called.
> > > + * Note that tls_strp_done is not called with the lower socket held.
> > > + */
> > > +void tls_strp_done(struct tls_strparser *strp)
> > >  {
> > > -	struct sk_buff *clone;
> > > +	WARN_ON(!strp->stopped);
> > >  
> > > -	clone = skb_clone(skb, sk->sk_allocation);
> > > -	if (!clone)
> > > +	cancel_work_sync(&strp->work);
> > > +	tls_strp_anchor_free(strp);
> > > +}
> > > +
> > > +int __init tls_strp_dev_init(void)
> > > +{
> > > +	tls_strp_wq = create_singlethread_workqueue("kstrp");  
> > 
> > I guess it's better to change the name to avoid confusing with plain
> > strparser ?!?
> > 
> > Out of sheer ignorance and not related to this patch: If I read
> > correctly, the above means that multiple tls flows on top of different
> > TCP sockets will use a single CPU, isn't that a relevant bottle-neck?
> > isn't enough to rely on queue_work() to submit the work on the same CPU
> > that just did the TCP stack processing? 
> 
> Oh yeah, this is a slow/rare path too but you're right. I copied this
> code from strparser without thinking, not sure what the motivation was
> there.

I'm fine with your replies here. I'm ok with the code as-is. I think
this last bits could be updated with a later patch, if needed.

Acked-by: Paolo Abeni <pabeni@redhat.com>

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

On 7/23/2022 2:50 AM, Jakub Kicinski wrote:
> TLS is a relatively poor fit for strparser. We pause the input
> every time a message is received, wait for a read which will
> decrypt the message, start the parser, repeat. strparser is
> built to delineate the messages, wrap them in individual skbs
> and let them float off into the stack or a different socket.
> TLS wants the data pages and nothing else. There's no need
> for TLS to keep cloning (and occasionally skb_unclone()'ing)
> the TCP rx queue.
> 
> This patch uses a pre-allocated skb and attaches the skbs
> from the TCP rx queue to it as frags. TLS is careful never
> to modify the input skb without CoW'ing / detaching it first.
> 
> Since we call TCP rx queue cleanup directly we also get back
> the benefit of skb deferred free.
> 
> Overall this results in a 6% gain in my benchmarks.
> 
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
> CC: borisp@nvidia.com
> CC: john.fastabend@gmail.com
> ---
>   include/net/tls.h  |  19 +-
>   net/tls/tls.h      |  24 ++-
>   net/tls/tls_main.c |  20 +-
>   net/tls/tls_strp.c | 484 +++++++++++++++++++++++++++++++++++++++++++--
>   net/tls/tls_sw.c   |  80 ++++----
>   5 files changed, 558 insertions(+), 69 deletions(-)
> 

Hi Jakub,

The device offload flow got broken, we started getting the call trace 
below in our regressions tests.

Bisecting points to this one as the offending commit.

I taking a look, but I'm less familiar with this change.
Probably you have a direction?

Regards,
Tariq

  [  407.560799] rcu: INFO: rcu_sched self-detected stall on CPU
  [  407.561734] rcu: 	1-....: (5248 ticks this GP) 
idle=51b/1/0x4000000000000000 softirq=41347/41347 fqs=2625
  [  407.563101] 	(t=5250 jiffies g=65669 q=4492 ncpus=10)
  [  407.563859] NMI backtrace for cpu 1
  [  407.564430] CPU: 1 PID: 45266 Comm: iperf Not tainted 
5.19.0-rc7_for-upstream_min-debug_5f35d2896553 #1
  [  407.565766] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), 
BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
  [  407.567319] Call Trace:
  [  407.567772]  <IRQ>
  [  407.568168]  dump_stack_lvl+0x34/0x44
  [  407.568763]  nmi_cpu_backtrace.cold+0x30/0x70
  [  407.569462]  ? lapic_can_unplug_cpu+0x70/0x70
  [  407.570146]  nmi_trigger_cpumask_backtrace+0xef/0x100
  [  407.570897]  trigger_single_cpu_backtrace+0x24/0x27
  [  407.571635]  rcu_dump_cpu_stacks+0xa0/0xd9
  [  407.572278]  rcu_sched_clock_irq.cold+0x111/0x2d3
  [  407.573004]  update_process_times+0x5b/0x90
  [  407.584227]  tick_sched_timer+0x88/0xa0
  [  407.584855]  ? tick_sched_do_timer+0xf0/0xf0
  [  407.585515]  __hrtimer_run_queues+0x139/0x290
  [  407.586199]  hrtimer_interrupt+0x10e/0x240
  [  407.586844]  __sysvec_apic_timer_interrupt+0x56/0xd0
  [  407.587575]  sysvec_apic_timer_interrupt+0x6d/0x90
  [  407.588293]  </IRQ>
  [  407.588699]  <TASK>
  [  407.589118]  asm_sysvec_apic_timer_interrupt+0x16/0x20
  [  407.589886] RIP: 0010:tls_device_decrypted+0x7a/0x2e0
  [  407.590641] Code: 83 e7 01 45 0f b6 e7 41 83 f7 01 48 85 d2 45 0f 
b6 ff 74 1b 0f b6 82 83 00 00 00 48 8b 12 c0 e8 06 41 21 c4 83 f0 01 41 
21 c7 <48> 85 d2 75 e5 8b 05 d3 71 ca 00 85 c0 0f 8f 4b 01 00 00 48 8b 85
  [  407.593152] RSP: 0018:ffff8881a113bb60 EFLAGS: 00000202
  [  407.593927] RAX: 0000000000000003 RBX: ffff88810e2d6000 RCX: 
0000000000000000
  [  407.594913] RDX: ffff8881792b80d8 RSI: ffff88814ef46800 RDI: 
ffff8881792b8000
  [  407.595904] RBP: ffff88814ef46800 R08: 7fffffffffffffff R09: 
0000000000000001
  [  407.596907] R10: ffff888105cd2200 R11: 0000000000022dd1 R12: 
0000000000000000
  [  407.597901] R13: ffff888108f45000 R14: ffff8881792b8000 R15: 
0000000000000001
  [  407.598897]  ? tls_rx_rec_wait+0x225/0x250
  [  407.599539]  tls_rx_one_record+0xe4/0x2d0
  [  407.600169]  tls_sw_recvmsg+0x380/0x910
  [  407.600773]  ? 0xffffffff81000000
  [  407.601336]  inet6_recvmsg+0x47/0xc0
  [  407.601932]  ____sys_recvmsg+0x109/0x120
  [  407.602554]  ? _copy_from_user+0x26/0x60
  [  407.603171]  ? iovec_from_user+0x4a/0x150
  [  407.603803]  ___sys_recvmsg+0xa4/0xe0
  [  407.604394]  ? mlx5e_ktls_add_rx+0x349/0x430 [mlx5_core]
  [  407.605262]  ? mlx5e_ktls_add_rx+0x3af/0x430 [mlx5_core]
  [  407.606082]  ? tls_device_attach+0x60/0xe0
  [  407.606721]  ? tls_set_device_offload_rx+0x11b/0x220
  [  407.607464]  __sys_recvmsg+0x4e/0x90
  [  407.608050]  do_syscall_64+0x3d/0x90
  [  407.608631]  entry_SYSCALL_64_after_hwframe+0x46/0xb0
  [  407.609400] RIP: 0033:0x7f7a41f856dd

> diff --git a/include/net/tls.h b/include/net/tls.h
> index 181c496b01b8..abb050b0df83 100644
> --- a/include/net/tls.h
> +++ b/include/net/tls.h
> @@ -108,18 +108,33 @@ struct tls_sw_context_tx {
>   	unsigned long tx_bitmask;
>   };
>   
> +struct tls_strparser {
> +	struct sock *sk;
> +
> +	u32 mark : 8;
> +	u32 stopped : 1;
> +	u32 copy_mode : 1;
> +	u32 msg_ready : 1;
> +
> +	struct strp_msg stm;
> +
> +	struct sk_buff *anchor;
> +	struct work_struct work;
> +};
> +
>   struct tls_sw_context_rx {
>   	struct crypto_aead *aead_recv;
>   	struct crypto_wait async_wait;
> -	struct strparser strp;
>   	struct sk_buff_head rx_list;	/* list of decrypted 'data' records */
>   	void (*saved_data_ready)(struct sock *sk);
>   
> -	struct sk_buff *recv_pkt;
>   	u8 reader_present;
>   	u8 async_capable:1;
>   	u8 zc_capable:1;
>   	u8 reader_contended:1;
> +
> +	struct tls_strparser strp;
> +
>   	atomic_t decrypt_pending;
>   	/* protect crypto_wait with decrypt_pending*/
>   	spinlock_t decrypt_compl_lock;
> diff --git a/net/tls/tls.h b/net/tls/tls.h
> index 154a3773e785..0e840a0c3437 100644
> --- a/net/tls/tls.h
> +++ b/net/tls/tls.h
> @@ -1,4 +1,5 @@
>   /*
> + * Copyright (c) 2016 Tom Herbert <tom@herbertland.com>
>    * Copyright (c) 2016-2017, Mellanox Technologies. All rights reserved.
>    * Copyright (c) 2016-2017, Dave Watson <davejwatson@fb.com>. All rights reserved.
>    *
> @@ -127,10 +128,24 @@ int tls_sw_fallback_init(struct sock *sk,
>   			 struct tls_offload_context_tx *offload_ctx,
>   			 struct tls_crypto_info *crypto_info);
>   
> +int tls_strp_dev_init(void);
> +void tls_strp_dev_exit(void);
> +
> +void tls_strp_done(struct tls_strparser *strp);
> +void tls_strp_stop(struct tls_strparser *strp);
> +int tls_strp_init(struct tls_strparser *strp, struct sock *sk);
> +void tls_strp_data_ready(struct tls_strparser *strp);
> +
> +void tls_strp_check_rcv(struct tls_strparser *strp);
> +void tls_strp_msg_done(struct tls_strparser *strp);
> +
> +int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb);
> +void tls_rx_msg_ready(struct tls_strparser *strp);
> +
> +void tls_strp_msg_load(struct tls_strparser *strp, bool force_refresh);
>   int tls_strp_msg_cow(struct tls_sw_context_rx *ctx);
>   struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx);
> -int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
> -		      struct sk_buff_head *dst);
> +int tls_strp_msg_hold(struct tls_strparser *strp, struct sk_buff_head *dst);
>   
>   static inline struct tls_msg *tls_msg(struct sk_buff *skb)
>   {
> @@ -141,12 +156,13 @@ static inline struct tls_msg *tls_msg(struct sk_buff *skb)
>   
>   static inline struct sk_buff *tls_strp_msg(struct tls_sw_context_rx *ctx)
>   {
> -	return ctx->recv_pkt;
> +	DEBUG_NET_WARN_ON_ONCE(!ctx->strp.msg_ready || !ctx->strp.anchor->len);
> +	return ctx->strp.anchor;
>   }
>   
>   static inline bool tls_strp_msg_ready(struct tls_sw_context_rx *ctx)
>   {
> -	return ctx->recv_pkt;
> +	return ctx->strp.msg_ready;
>   }
>   
>   #ifdef CONFIG_TLS_DEVICE
> diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
> index 9703636cfc60..08ddf9d837ae 100644
> --- a/net/tls/tls_main.c
> +++ b/net/tls/tls_main.c
> @@ -725,6 +725,10 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
>   	if (tx) {
>   		ctx->sk_write_space = sk->sk_write_space;
>   		sk->sk_write_space = tls_write_space;
> +	} else {
> +		struct tls_sw_context_rx *rx_ctx = tls_sw_ctx_rx(ctx);
> +
> +		tls_strp_check_rcv(&rx_ctx->strp);
>   	}
>   	return 0;
>   
> @@ -1141,20 +1145,28 @@ static int __init tls_register(void)
>   	if (err)
>   		return err;
>   
> +	err = tls_strp_dev_init();
> +	if (err)
> +		goto err_pernet;
> +
>   	err = tls_device_init();
> -	if (err) {
> -		unregister_pernet_subsys(&tls_proc_ops);
> -		return err;
> -	}
> +	if (err)
> +		goto err_strp;
>   
>   	tcp_register_ulp(&tcp_tls_ulp_ops);
>   
>   	return 0;
> +err_strp:
> +	tls_strp_dev_exit();
> +err_pernet:
> +	unregister_pernet_subsys(&tls_proc_ops);
> +	return err;
>   }
>   
>   static void __exit tls_unregister(void)
>   {
>   	tcp_unregister_ulp(&tcp_tls_ulp_ops);
> +	tls_strp_dev_exit();
>   	tls_device_cleanup();
>   	unregister_pernet_subsys(&tls_proc_ops);
>   }
> diff --git a/net/tls/tls_strp.c b/net/tls/tls_strp.c
> index d9bb4f23f01a..b945288c312e 100644
> --- a/net/tls/tls_strp.c
> +++ b/net/tls/tls_strp.c
> @@ -1,37 +1,493 @@
>   // SPDX-License-Identifier: GPL-2.0-only
> +/* Copyright (c) 2016 Tom Herbert <tom@herbertland.com> */
>   
>   #include <linux/skbuff.h>
> +#include <linux/workqueue.h>
> +#include <net/strparser.h>
> +#include <net/tcp.h>
> +#include <net/sock.h>
> +#include <net/tls.h>
>   
>   #include "tls.h"
>   
> -struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx)
> +static struct workqueue_struct *tls_strp_wq;
> +
> +static void tls_strp_abort_strp(struct tls_strparser *strp, int err)
> +{
> +	if (strp->stopped)
> +		return;
> +
> +	strp->stopped = 1;
> +
> +	/* Report an error on the lower socket */
> +	strp->sk->sk_err = -err;
> +	sk_error_report(strp->sk);
> +}
> +
> +static void tls_strp_anchor_free(struct tls_strparser *strp)
>   {
> +	struct skb_shared_info *shinfo = skb_shinfo(strp->anchor);
> +
> +	DEBUG_NET_WARN_ON_ONCE(atomic_read(&shinfo->dataref) != 1);
> +	shinfo->frag_list = NULL;
> +	consume_skb(strp->anchor);
> +	strp->anchor = NULL;
> +}
> +
> +/* Create a new skb with the contents of input copied to its page frags */
> +static struct sk_buff *tls_strp_msg_make_copy(struct tls_strparser *strp)
> +{
> +	struct strp_msg *rxm;
>   	struct sk_buff *skb;
> +	int i, err, offset;
> +
> +	skb = alloc_skb_with_frags(0, strp->anchor->len, TLS_PAGE_ORDER,
> +				   &err, strp->sk->sk_allocation);
> +	if (!skb)
> +		return NULL;
> +
> +	offset = strp->stm.offset;
> +	for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
> +		skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
>   
> -	skb = ctx->recv_pkt;
> -	ctx->recv_pkt = NULL;
> +		WARN_ON_ONCE(skb_copy_bits(strp->anchor, offset,
> +					   skb_frag_address(frag),
> +					   skb_frag_size(frag)));
> +		offset += skb_frag_size(frag);
> +	}
> +
> +	skb_copy_header(skb, strp->anchor);
> +	rxm = strp_msg(skb);
> +	rxm->offset = 0;
>   	return skb;
>   }
>   
> +/* Steal the input skb, input msg is invalid after calling this function */
> +struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx)
> +{
> +	struct tls_strparser *strp = &ctx->strp;
> +
> +#ifdef CONFIG_TLS_DEVICE
> +	DEBUG_NET_WARN_ON_ONCE(!strp->anchor->decrypted);
> +#else
> +	/* This function turns an input into an output,
> +	 * that can only happen if we have offload.
> +	 */
> +	WARN_ON(1);
> +#endif
> +
> +	if (strp->copy_mode) {
> +		struct sk_buff *skb;
> +
> +		/* Replace anchor with an empty skb, this is a little
> +		 * dangerous but __tls_cur_msg() warns on empty skbs
> +		 * so hopefully we'll catch abuses.
> +		 */
> +		skb = alloc_skb(0, strp->sk->sk_allocation);
> +		if (!skb)
> +			return NULL;
> +
> +		swap(strp->anchor, skb);
> +		return skb;
> +	}
> +
> +	return tls_strp_msg_make_copy(strp);
> +}
> +
> +/* Force the input skb to be in copy mode. The data ownership remains
> + * with the input skb itself (meaning unpause will wipe it) but it can
> + * be modified.
> + */
>   int tls_strp_msg_cow(struct tls_sw_context_rx *ctx)
>   {
> -	struct sk_buff *unused;
> -	int nsg;
> +	struct tls_strparser *strp = &ctx->strp;
> +	struct sk_buff *skb;
> +
> +	if (strp->copy_mode)
> +		return 0;
> +
> +	skb = tls_strp_msg_make_copy(strp);
> +	if (!skb)
> +		return -ENOMEM;
> +
> +	tls_strp_anchor_free(strp);
> +	strp->anchor = skb;
> +
> +	tcp_read_done(strp->sk, strp->stm.full_len);
> +	strp->copy_mode = 1;
> +
> +	return 0;
> +}
> +
> +/* Make a clone (in the skb sense) of the input msg to keep a reference
> + * to the underlying data. The reference-holding skbs get placed on
> + * @dst.
> + */
> +int tls_strp_msg_hold(struct tls_strparser *strp, struct sk_buff_head *dst)
> +{
> +	struct skb_shared_info *shinfo = skb_shinfo(strp->anchor);
> +
> +	if (strp->copy_mode) {
> +		struct sk_buff *skb;
> +
> +		WARN_ON_ONCE(!shinfo->nr_frags);
> +
> +		/* We can't skb_clone() the anchor, it gets wiped by unpause */
> +		skb = alloc_skb(0, strp->sk->sk_allocation);
> +		if (!skb)
> +			return -ENOMEM;
> +
> +		__skb_queue_tail(dst, strp->anchor);
> +		strp->anchor = skb;
> +	} else {
> +		struct sk_buff *iter, *clone;
> +		int chunk, len, offset;
> +
> +		offset = strp->stm.offset;
> +		len = strp->stm.full_len;
> +		iter = shinfo->frag_list;
> +
> +		while (len > 0) {
> +			if (iter->len <= offset) {
> +				offset -= iter->len;
> +				goto next;
> +			}
> +
> +			chunk = iter->len - offset;
> +			offset = 0;
> +
> +			clone = skb_clone(iter, strp->sk->sk_allocation);
> +			if (!clone)
> +				return -ENOMEM;
> +			__skb_queue_tail(dst, clone);
> +
> +			len -= chunk;
> +next:
> +			iter = iter->next;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static void tls_strp_flush_anchor_copy(struct tls_strparser *strp)
> +{
> +	struct skb_shared_info *shinfo = skb_shinfo(strp->anchor);
> +	int i;
> +
> +	DEBUG_NET_WARN_ON_ONCE(atomic_read(&shinfo->dataref) != 1);
> +
> +	for (i = 0; i < shinfo->nr_frags; i++)
> +		__skb_frag_unref(&shinfo->frags[i], false);
> +	shinfo->nr_frags = 0;
> +	strp->copy_mode = 0;
> +}
> +
> +static int tls_strp_copyin(read_descriptor_t *desc, struct sk_buff *in_skb,
> +			   unsigned int offset, size_t in_len)
> +{
> +	struct tls_strparser *strp = (struct tls_strparser *)desc->arg.data;
> +	size_t sz, len, chunk;
> +	struct sk_buff *skb;
> +	skb_frag_t *frag;
> +
> +	if (strp->msg_ready)
> +		return 0;
> +
> +	skb = strp->anchor;
> +	frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE];
> +
> +	len = in_len;
> +	/* First make sure we got the header */
> +	if (!strp->stm.full_len) {
> +		/* Assume one page is more than enough for headers */
> +		chunk =	min_t(size_t, len, PAGE_SIZE - skb_frag_size(frag));
> +		WARN_ON_ONCE(skb_copy_bits(in_skb, offset,
> +					   skb_frag_address(frag) +
> +					   skb_frag_size(frag),
> +					   chunk));
> +
> +		sz = tls_rx_msg_size(strp, strp->anchor);
> +		if (sz < 0) {
> +			desc->error = sz;
> +			return 0;
> +		}
> +
> +		/* We may have over-read, sz == 0 is guaranteed under-read */
> +		if (sz > 0)
> +			chunk =	min_t(size_t, chunk, sz - skb->len);
> +
> +		skb->len += chunk;
> +		skb->data_len += chunk;
> +		skb_frag_size_add(frag, chunk);
> +		frag++;
> +		len -= chunk;
> +		offset += chunk;
> +
> +		strp->stm.full_len = sz;
> +		if (!strp->stm.full_len)
> +			goto read_done;
> +	}
> +
> +	/* Load up more data */
> +	while (len && strp->stm.full_len > skb->len) {
> +		chunk =	min_t(size_t, len, strp->stm.full_len - skb->len);
> +		chunk = min_t(size_t, chunk, PAGE_SIZE - skb_frag_size(frag));
> +		WARN_ON_ONCE(skb_copy_bits(in_skb, offset,
> +					   skb_frag_address(frag) +
> +					   skb_frag_size(frag),
> +					   chunk));
> +
> +		skb->len += chunk;
> +		skb->data_len += chunk;
> +		skb_frag_size_add(frag, chunk);
> +		frag++;
> +		len -= chunk;
> +		offset += chunk;
> +	}
> +
> +	if (strp->stm.full_len == skb->len) {
> +		desc->count = 0;
> +
> +		strp->msg_ready = 1;
> +		tls_rx_msg_ready(strp);
> +	}
> +
> +read_done:
> +	return in_len - len;
> +}
> +
> +static int tls_strp_read_copyin(struct tls_strparser *strp)
> +{
> +	struct socket *sock = strp->sk->sk_socket;
> +	read_descriptor_t desc;
> +
> +	desc.arg.data = strp;
> +	desc.error = 0;
> +	desc.count = 1; /* give more than one skb per call */
> +
> +	/* sk should be locked here, so okay to do read_sock */
> +	sock->ops->read_sock(strp->sk, &desc, tls_strp_copyin);
> +
> +	return desc.error;
> +}
> +
> +static int tls_strp_read_short(struct tls_strparser *strp)
> +{
> +	struct skb_shared_info *shinfo;
> +	struct page *page;
> +	int need_spc, len;
> +
> +	/* If the rbuf is small or rcv window has collapsed to 0 we need
> +	 * to read the data out. Otherwise the connection will stall.
> +	 * Without pressure threshold of INT_MAX will never be ready.
> +	 */
> +	if (likely(!tcp_epollin_ready(strp->sk, INT_MAX)))
> +		return 0;
> +
> +	shinfo = skb_shinfo(strp->anchor);
> +	shinfo->frag_list = NULL;
> +
> +	/* If we don't know the length go max plus page for cipher overhead */
> +	need_spc = strp->stm.full_len ?: TLS_MAX_PAYLOAD_SIZE + PAGE_SIZE;
> +
> +	for (len = need_spc; len > 0; len -= PAGE_SIZE) {
> +		page = alloc_page(strp->sk->sk_allocation);
> +		if (!page) {
> +			tls_strp_flush_anchor_copy(strp);
> +			return -ENOMEM;
> +		}
> +
> +		skb_fill_page_desc(strp->anchor, shinfo->nr_frags++,
> +				   page, 0, 0);
> +	}
> +
> +	strp->copy_mode = 1;
> +	strp->stm.offset = 0;
> +
> +	strp->anchor->len = 0;
> +	strp->anchor->data_len = 0;
> +	strp->anchor->truesize = round_up(need_spc, PAGE_SIZE);
> +
> +	tls_strp_read_copyin(strp);
> +
> +	return 0;
> +}
> +
> +static void tls_strp_load_anchor_with_queue(struct tls_strparser *strp, int len)
> +{
> +	struct tcp_sock *tp = tcp_sk(strp->sk);
> +	struct sk_buff *first;
> +	u32 offset;
> +
> +	first = tcp_recv_skb(strp->sk, tp->copied_seq, &offset);
> +	if (WARN_ON_ONCE(!first))
> +		return;
> +
> +	/* Bestow the state onto the anchor */
> +	strp->anchor->len = offset + len;
> +	strp->anchor->data_len = offset + len;
> +	strp->anchor->truesize = offset + len;
> +
> +	skb_shinfo(strp->anchor)->frag_list = first;
> +
> +	skb_copy_header(strp->anchor, first);
> +	strp->anchor->destructor = NULL;
> +
> +	strp->stm.offset = offset;
> +}
> +
> +void tls_strp_msg_load(struct tls_strparser *strp, bool force_refresh)
> +{
> +	struct strp_msg *rxm;
> +	struct tls_msg *tlm;
> +
> +	DEBUG_NET_WARN_ON_ONCE(!strp->msg_ready);
> +	DEBUG_NET_WARN_ON_ONCE(!strp->stm.full_len);
> +
> +	if (!strp->copy_mode && force_refresh) {
> +		if (WARN_ON(tcp_inq(strp->sk) < strp->stm.full_len))
> +			return;
> +
> +		tls_strp_load_anchor_with_queue(strp, strp->stm.full_len);
> +	}
> +
> +	rxm = strp_msg(strp->anchor);
> +	rxm->full_len	= strp->stm.full_len;
> +	rxm->offset	= strp->stm.offset;
> +	tlm = tls_msg(strp->anchor);
> +	tlm->control	= strp->mark;
> +}
> +
> +/* Called with lock held on lower socket */
> +static int tls_strp_read_sock(struct tls_strparser *strp)
> +{
> +	int sz, inq;
> +
> +	inq = tcp_inq(strp->sk);
> +	if (inq < 1)
> +		return 0;
> +
> +	if (unlikely(strp->copy_mode))
> +		return tls_strp_read_copyin(strp);
> +
> +	if (inq < strp->stm.full_len)
> +		return tls_strp_read_short(strp);
> +
> +	if (!strp->stm.full_len) {
> +		tls_strp_load_anchor_with_queue(strp, inq);
> +
> +		sz = tls_rx_msg_size(strp, strp->anchor);
> +		if (sz < 0) {
> +			tls_strp_abort_strp(strp, sz);
> +			return sz;
> +		}
> +
> +		strp->stm.full_len = sz;
> +
> +		if (!strp->stm.full_len || inq < strp->stm.full_len)
> +			return tls_strp_read_short(strp);
> +	}
> +
> +	strp->msg_ready = 1;
> +	tls_rx_msg_ready(strp);
> +
> +	return 0;
> +}
> +
> +void tls_strp_check_rcv(struct tls_strparser *strp)
> +{
> +	if (unlikely(strp->stopped) || strp->msg_ready)
> +		return;
> +
> +	if (tls_strp_read_sock(strp) == -ENOMEM)
> +		queue_work(tls_strp_wq, &strp->work);
> +}
> +
> +/* Lower sock lock held */
> +void tls_strp_data_ready(struct tls_strparser *strp)
> +{
> +	/* This check is needed to synchronize with do_tls_strp_work.
> +	 * do_tls_strp_work acquires a process lock (lock_sock) whereas
> +	 * the lock held here is bh_lock_sock. The two locks can be
> +	 * held by different threads at the same time, but bh_lock_sock
> +	 * allows a thread in BH context to safely check if the process
> +	 * lock is held. In this case, if the lock is held, queue work.
> +	 */
> +	if (sock_owned_by_user_nocheck(strp->sk)) {
> +		queue_work(tls_strp_wq, &strp->work);
> +		return;
> +	}
> +
> +	tls_strp_check_rcv(strp);
> +}
> +
> +static void tls_strp_work(struct work_struct *w)
> +{
> +	struct tls_strparser *strp =
> +		container_of(w, struct tls_strparser, work);
> +
> +	lock_sock(strp->sk);
> +	tls_strp_check_rcv(strp);
> +	release_sock(strp->sk);
> +}
> +
> +void tls_strp_msg_done(struct tls_strparser *strp)
> +{
> +	WARN_ON(!strp->stm.full_len);
> +
> +	if (likely(!strp->copy_mode))
> +		tcp_read_done(strp->sk, strp->stm.full_len);
> +	else
> +		tls_strp_flush_anchor_copy(strp);
> +
> +	strp->msg_ready = 0;
> +	memset(&strp->stm, 0, sizeof(strp->stm));
> +
> +	tls_strp_check_rcv(strp);
> +}
> +
> +void tls_strp_stop(struct tls_strparser *strp)
> +{
> +	strp->stopped = 1;
> +}
> +
> +int tls_strp_init(struct tls_strparser *strp, struct sock *sk)
> +{
> +	memset(strp, 0, sizeof(*strp));
> +
> +	strp->sk = sk;
> +
> +	strp->anchor = alloc_skb(0, GFP_KERNEL);
> +	if (!strp->anchor)
> +		return -ENOMEM;
> +
> +	INIT_WORK(&strp->work, tls_strp_work);
>   
> -	nsg = skb_cow_data(ctx->recv_pkt, 0, &unused);
> -	if (nsg < 0)
> -		return nsg;
>   	return 0;
>   }
>   
> -int tls_strp_msg_hold(struct sock *sk, struct sk_buff *skb,
> -		      struct sk_buff_head *dst)
> +/* strp must already be stopped so that tls_strp_recv will no longer be called.
> + * Note that tls_strp_done is not called with the lower socket held.
> + */
> +void tls_strp_done(struct tls_strparser *strp)
>   {
> -	struct sk_buff *clone;
> +	WARN_ON(!strp->stopped);
>   
> -	clone = skb_clone(skb, sk->sk_allocation);
> -	if (!clone)
> +	cancel_work_sync(&strp->work);
> +	tls_strp_anchor_free(strp);
> +}
> +
> +int __init tls_strp_dev_init(void)
> +{
> +	tls_strp_wq = create_singlethread_workqueue("kstrp");
> +	if (unlikely(!tls_strp_wq))
>   		return -ENOMEM;
> -	__skb_queue_tail(dst, clone);
> +
>   	return 0;
>   }
> +
> +void tls_strp_dev_exit(void)
> +{
> +	destroy_workqueue(tls_strp_wq);
> +}
> diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
> index bd4486819e64..0fc24a5ce208 100644
> --- a/net/tls/tls_sw.c
> +++ b/net/tls/tls_sw.c
> @@ -1283,7 +1283,7 @@ int tls_sw_sendpage(struct sock *sk, struct page *page,
>   
>   static int
>   tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
> -		long timeo)
> +		bool released, long timeo)
>   {
>   	struct tls_context *tls_ctx = tls_get_ctx(sk);
>   	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
> @@ -1297,7 +1297,7 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
>   			return sock_error(sk);
>   
>   		if (!skb_queue_empty(&sk->sk_receive_queue)) {
> -			__strp_unpause(&ctx->strp);
> +			tls_strp_check_rcv(&ctx->strp);
>   			if (tls_strp_msg_ready(ctx))
>   				break;
>   		}
> @@ -1311,6 +1311,7 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
>   		if (nonblock || !timeo)
>   			return -EAGAIN;
>   
> +		released = true;
>   		add_wait_queue(sk_sleep(sk), &wait);
>   		sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
>   		sk_wait_event(sk, &timeo,
> @@ -1325,6 +1326,8 @@ tls_rx_rec_wait(struct sock *sk, struct sk_psock *psock, bool nonblock,
>   			return sock_intr_errno(timeo);
>   	}
>   
> +	tls_strp_msg_load(&ctx->strp, released);
> +
>   	return 1;
>   }
>   
> @@ -1570,7 +1573,7 @@ static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov,
>   	clear_skb = NULL;
>   
>   	if (unlikely(darg->async)) {
> -		err = tls_strp_msg_hold(sk, skb, &ctx->async_hold);
> +		err = tls_strp_msg_hold(&ctx->strp, &ctx->async_hold);
>   		if (err)
>   			__skb_queue_tail(&ctx->async_hold, darg->skb);
>   		return err;
> @@ -1734,9 +1737,7 @@ static int tls_record_content_type(struct msghdr *msg, struct tls_msg *tlm,
>   
>   static void tls_rx_rec_done(struct tls_sw_context_rx *ctx)
>   {
> -	consume_skb(ctx->recv_pkt);
> -	ctx->recv_pkt = NULL;
> -	__strp_unpause(&ctx->strp);
> +	tls_strp_msg_done(&ctx->strp);
>   }
>   
>   /* This function traverses the rx_list in tls receive context to copies the
> @@ -1823,7 +1824,7 @@ static int process_rx_list(struct tls_sw_context_rx *ctx,
>   	return copied ? : err;
>   }
>   
> -static void
> +static bool
>   tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot,
>   		       size_t len_left, size_t decrypted, ssize_t done,
>   		       size_t *flushed_at)
> @@ -1831,14 +1832,14 @@ tls_read_flush_backlog(struct sock *sk, struct tls_prot_info *prot,
>   	size_t max_rec;
>   
>   	if (len_left <= decrypted)
> -		return;
> +		return false;
>   
>   	max_rec = prot->overhead_size - prot->tail_size + TLS_MAX_PAYLOAD_SIZE;
>   	if (done - *flushed_at < SZ_128K && tcp_inq(sk) > max_rec)
> -		return;
> +		return false;
>   
>   	*flushed_at = done;
> -	sk_flush_backlog(sk);
> +	return sk_flush_backlog(sk);
>   }
>   
>   static long tls_rx_reader_lock(struct sock *sk, struct tls_sw_context_rx *ctx,
> @@ -1916,6 +1917,7 @@ int tls_sw_recvmsg(struct sock *sk,
>   	long timeo;
>   	bool is_kvec = iov_iter_is_kvec(&msg->msg_iter);
>   	bool is_peek = flags & MSG_PEEK;
> +	bool released = true;
>   	bool bpf_strp_enabled;
>   	bool zc_capable;
>   
> @@ -1952,7 +1954,8 @@ int tls_sw_recvmsg(struct sock *sk,
>   		struct tls_decrypt_arg darg;
>   		int to_decrypt, chunk;
>   
> -		err = tls_rx_rec_wait(sk, psock, flags & MSG_DONTWAIT, timeo);
> +		err = tls_rx_rec_wait(sk, psock, flags & MSG_DONTWAIT, released,
> +				      timeo);
>   		if (err <= 0) {
>   			if (psock) {
>   				chunk = sk_msg_recvmsg(sk, psock, msg, len,
> @@ -1968,8 +1971,8 @@ int tls_sw_recvmsg(struct sock *sk,
>   
>   		memset(&darg.inargs, 0, sizeof(darg.inargs));
>   
> -		rxm = strp_msg(ctx->recv_pkt);
> -		tlm = tls_msg(ctx->recv_pkt);
> +		rxm = strp_msg(tls_strp_msg(ctx));
> +		tlm = tls_msg(tls_strp_msg(ctx));
>   
>   		to_decrypt = rxm->full_len - prot->overhead_size;
>   
> @@ -2008,8 +2011,9 @@ int tls_sw_recvmsg(struct sock *sk,
>   		}
>   
>   		/* periodically flush backlog, and feed strparser */
> -		tls_read_flush_backlog(sk, prot, len, to_decrypt,
> -				       decrypted + copied, &flushed_at);
> +		released = tls_read_flush_backlog(sk, prot, len, to_decrypt,
> +						  decrypted + copied,
> +						  &flushed_at);
>   
>   		/* TLS 1.3 may have updated the length by more than overhead */
>   		rxm = strp_msg(darg.skb);
> @@ -2020,7 +2024,7 @@ int tls_sw_recvmsg(struct sock *sk,
>   			bool partially_consumed = chunk > len;
>   			struct sk_buff *skb = darg.skb;
>   
> -			DEBUG_NET_WARN_ON_ONCE(darg.skb == ctx->recv_pkt);
> +			DEBUG_NET_WARN_ON_ONCE(darg.skb == tls_strp_msg(ctx));
>   
>   			if (async) {
>   				/* TLS 1.2-only, to_decrypt must be text len */
> @@ -2034,6 +2038,7 @@ int tls_sw_recvmsg(struct sock *sk,
>   			}
>   
>   			if (bpf_strp_enabled) {
> +				released = true;
>   				err = sk_psock_tls_strp_read(psock, skb);
>   				if (err != __SK_PASS) {
>   					rxm->offset = rxm->offset + rxm->full_len;
> @@ -2140,7 +2145,7 @@ ssize_t tls_sw_splice_read(struct socket *sock,  loff_t *ppos,
>   		struct tls_decrypt_arg darg;
>   
>   		err = tls_rx_rec_wait(sk, NULL, flags & SPLICE_F_NONBLOCK,
> -				      timeo);
> +				      true, timeo);
>   		if (err <= 0)
>   			goto splice_read_end;
>   
> @@ -2204,19 +2209,17 @@ bool tls_sw_sock_is_readable(struct sock *sk)
>   		!skb_queue_empty(&ctx->rx_list);
>   }
>   
> -static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
> +int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb)
>   {
>   	struct tls_context *tls_ctx = tls_get_ctx(strp->sk);
>   	struct tls_prot_info *prot = &tls_ctx->prot_info;
>   	char header[TLS_HEADER_SIZE + MAX_IV_SIZE];
> -	struct strp_msg *rxm = strp_msg(skb);
> -	struct tls_msg *tlm = tls_msg(skb);
>   	size_t cipher_overhead;
>   	size_t data_len = 0;
>   	int ret;
>   
>   	/* Verify that we have a full TLS header, or wait for more data */
> -	if (rxm->offset + prot->prepend_size > skb->len)
> +	if (strp->stm.offset + prot->prepend_size > skb->len)
>   		return 0;
>   
>   	/* Sanity-check size of on-stack buffer. */
> @@ -2226,11 +2229,11 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
>   	}
>   
>   	/* Linearize header to local buffer */
> -	ret = skb_copy_bits(skb, rxm->offset, header, prot->prepend_size);
> +	ret = skb_copy_bits(skb, strp->stm.offset, header, prot->prepend_size);
>   	if (ret < 0)
>   		goto read_failure;
>   
> -	tlm->control = header[0];
> +	strp->mark = header[0];
>   
>   	data_len = ((header[4] & 0xFF) | (header[3] << 8));
>   
> @@ -2257,7 +2260,7 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
>   	}
>   
>   	tls_device_rx_resync_new_rec(strp->sk, data_len + TLS_HEADER_SIZE,
> -				     TCP_SKB_CB(skb)->seq + rxm->offset);
> +				     TCP_SKB_CB(skb)->seq + strp->stm.offset);
>   	return data_len + TLS_HEADER_SIZE;
>   
>   read_failure:
> @@ -2266,14 +2269,11 @@ static int tls_read_size(struct strparser *strp, struct sk_buff *skb)
>   	return ret;
>   }
>   
> -static void tls_queue(struct strparser *strp, struct sk_buff *skb)
> +void tls_rx_msg_ready(struct tls_strparser *strp)
>   {
> -	struct tls_context *tls_ctx = tls_get_ctx(strp->sk);
> -	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
> -
> -	ctx->recv_pkt = skb;
> -	strp_pause(strp);
> +	struct tls_sw_context_rx *ctx;
>   
> +	ctx = container_of(strp, struct tls_sw_context_rx, strp);
>   	ctx->saved_data_ready(strp->sk);
>   }
>   
> @@ -2283,7 +2283,7 @@ static void tls_data_ready(struct sock *sk)
>   	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
>   	struct sk_psock *psock;
>   
> -	strp_data_ready(&ctx->strp);
> +	tls_strp_data_ready(&ctx->strp);
>   
>   	psock = sk_psock_get(sk);
>   	if (psock) {
> @@ -2359,13 +2359,11 @@ void tls_sw_release_resources_rx(struct sock *sk)
>   	kfree(tls_ctx->rx.iv);
>   
>   	if (ctx->aead_recv) {
> -		kfree_skb(ctx->recv_pkt);
> -		ctx->recv_pkt = NULL;
>   		__skb_queue_purge(&ctx->rx_list);
>   		crypto_free_aead(ctx->aead_recv);
> -		strp_stop(&ctx->strp);
> +		tls_strp_stop(&ctx->strp);
>   		/* If tls_sw_strparser_arm() was not called (cleanup paths)
> -		 * we still want to strp_stop(), but sk->sk_data_ready was
> +		 * we still want to tls_strp_stop(), but sk->sk_data_ready was
>   		 * never swapped.
>   		 */
>   		if (ctx->saved_data_ready) {
> @@ -2380,7 +2378,7 @@ void tls_sw_strparser_done(struct tls_context *tls_ctx)
>   {
>   	struct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);
>   
> -	strp_done(&ctx->strp);
> +	tls_strp_done(&ctx->strp);
>   }
>   
>   void tls_sw_free_ctx_rx(struct tls_context *tls_ctx)
> @@ -2453,8 +2451,6 @@ void tls_sw_strparser_arm(struct sock *sk, struct tls_context *tls_ctx)
>   	rx_ctx->saved_data_ready = sk->sk_data_ready;
>   	sk->sk_data_ready = tls_data_ready;
>   	write_unlock_bh(&sk->sk_callback_lock);
> -
> -	strp_check_rcv(&rx_ctx->strp);
>   }
>   
>   void tls_update_rx_zc_capable(struct tls_context *tls_ctx)
> @@ -2474,7 +2470,6 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx)
>   	struct tls_sw_context_rx *sw_ctx_rx = NULL;
>   	struct cipher_context *cctx;
>   	struct crypto_aead **aead;
> -	struct strp_callbacks cb;
>   	u16 nonce_size, tag_size, iv_size, rec_seq_size, salt_size;
>   	struct crypto_tfm *tfm;
>   	char *iv, *rec_seq, *key, *salt, *cipher_name;
> @@ -2708,12 +2703,7 @@ int tls_set_sw_offload(struct sock *sk, struct tls_context *ctx, int tx)
>   			crypto_info->version != TLS_1_3_VERSION &&
>   			!!(tfm->__crt_alg->cra_flags & CRYPTO_ALG_ASYNC);
>   
> -		/* Set up strparser */
> -		memset(&cb, 0, sizeof(cb));
> -		cb.rcv_msg = tls_queue;
> -		cb.parse_msg = tls_read_size;
> -
> -		strp_init(&sw_ctx_rx->strp, sk, &cb);
> +		tls_strp_init(&sw_ctx_rx->strp, sk);
>   	}
>   
>   	goto out;

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

On Tue, 2 Aug 2022 17:54:01 +0300 Tariq Toukan wrote:
> Hi Jakub,
> 
> The device offload flow got broken, we started getting the call trace 
> below in our regressions tests.
> 
> Bisecting points to this one as the offending commit.
> 
> I taking a look, but I'm less familiar with this change.
> Probably you have a direction?

Sorry about that, I'll take a look. Would you be able to run it thru
decode_stacktrace before I get to the office and start digging?

>   [  407.560799] rcu: INFO: rcu_sched self-detected stall on CPU
>   [  407.561734] rcu: 	1-....: (5248 ticks this GP) 
> idle=51b/1/0x4000000000000000 softirq=41347/41347 fqs=2625
>   [  407.563101] 	(t=5250 jiffies g=65669 q=4492 ncpus=10)
>   [  407.563859] NMI backtrace for cpu 1
>   [  407.564430] CPU: 1 PID: 45266 Comm: iperf Not tainted 
> 5.19.0-rc7_for-upstream_min-debug_5f35d2896553 #1
>   [  407.565766] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), 
> BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
>   [  407.567319] Call Trace:
>   [  407.567772]  <IRQ>
>   [  407.568168]  dump_stack_lvl+0x34/0x44
>   [  407.568763]  nmi_cpu_backtrace.cold+0x30/0x70
>   [  407.569462]  ? lapic_can_unplug_cpu+0x70/0x70
>   [  407.570146]  nmi_trigger_cpumask_backtrace+0xef/0x100
>   [  407.570897]  trigger_single_cpu_backtrace+0x24/0x27
>   [  407.571635]  rcu_dump_cpu_stacks+0xa0/0xd9
>   [  407.572278]  rcu_sched_clock_irq.cold+0x111/0x2d3
>   [  407.573004]  update_process_times+0x5b/0x90
>   [  407.584227]  tick_sched_timer+0x88/0xa0
>   [  407.584855]  ? tick_sched_do_timer+0xf0/0xf0
>   [  407.585515]  __hrtimer_run_queues+0x139/0x290
>   [  407.586199]  hrtimer_interrupt+0x10e/0x240
>   [  407.586844]  __sysvec_apic_timer_interrupt+0x56/0xd0
>   [  407.587575]  sysvec_apic_timer_interrupt+0x6d/0x90
>   [  407.588293]  </IRQ>
>   [  407.588699]  <TASK>
>   [  407.589118]  asm_sysvec_apic_timer_interrupt+0x16/0x20
>   [  407.589886] RIP: 0010:tls_device_decrypted+0x7a/0x2e0
>   [  407.590641] Code: 83 e7 01 45 0f b6 e7 41 83 f7 01 48 85 d2 45 0f 
> b6 ff 74 1b 0f b6 82 83 00 00 00 48 8b 12 c0 e8 06 41 21 c4 83 f0 01 41 
> 21 c7 <48> 85 d2 75 e5 8b 05 d3 71 ca 00 85 c0 0f 8f 4b 01 00 00 48 8b 85
>   [  407.593152] RSP: 0018:ffff8881a113bb60 EFLAGS: 00000202
>   [  407.593927] RAX: 0000000000000003 RBX: ffff88810e2d6000 RCX: 
> 0000000000000000
>   [  407.594913] RDX: ffff8881792b80d8 RSI: ffff88814ef46800 RDI: 
> ffff8881792b8000
>   [  407.595904] RBP: ffff88814ef46800 R08: 7fffffffffffffff R09: 
> 0000000000000001
>   [  407.596907] R10: ffff888105cd2200 R11: 0000000000022dd1 R12: 
> 0000000000000000
>   [  407.597901] R13: ffff888108f45000 R14: ffff8881792b8000 R15: 
> 0000000000000001
>   [  407.598897]  ? tls_rx_rec_wait+0x225/0x250
>   [  407.599539]  tls_rx_one_record+0xe4/0x2d0
>   [  407.600169]  tls_sw_recvmsg+0x380/0x910
>   [  407.600773]  ? 0xffffffff81000000
>   [  407.601336]  inet6_recvmsg+0x47/0xc0
>   [  407.601932]  ____sys_recvmsg+0x109/0x120
>   [  407.602554]  ? _copy_from_user+0x26/0x60
>   [  407.603171]  ? iovec_from_user+0x4a/0x150
>   [  407.603803]  ___sys_recvmsg+0xa4/0xe0
>   [  407.604394]  ? mlx5e_ktls_add_rx+0x349/0x430 [mlx5_core]
>   [  407.605262]  ? mlx5e_ktls_add_rx+0x3af/0x430 [mlx5_core]
>   [  407.606082]  ? tls_device_attach+0x60/0xe0
>   [  407.606721]  ? tls_set_device_offload_rx+0x11b/0x220
>   [  407.607464]  __sys_recvmsg+0x4e/0x90
>   [  407.608050]  do_syscall_64+0x3d/0x90
>   [  407.608631]  entry_SYSCALL_64_after_hwframe+0x46/0xb0
>   [  407.609400] RIP: 0033:0x7f7a41f856dd

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

On Tue, 2 Aug 2022 17:54:01 +0300 Tariq Toukan wrote:
>   [  407.589886] RIP: 0010:tls_device_decrypted+0x7a/0x2e0

Sorry, got distracted yesterday. This?

--->8--------------------
tls: rx: device: bound the frag walk

We can't do skb_walk_frags() on the input skbs, because
the input skbs is really just a pointer to the tcp read
queue. We need to bound the "is decrypted" check by the
amount of data in the message.

Note that the walk in tls_device_reencrypt() is after a
CoW so the skb there is safe to walk. Actually in the
current implementation it can't have frags at all, but
whatever, maybe one day it will.

Reported-by: Tariq Toukan <tariqt@nvidia.com>
Fixes: 84c61fe1a75b ("tls: rx: do not use the standard strparser")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
 net/tls/tls_device.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index e3e6cf75aa03..6ed41474bdf8 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -984,11 +984,17 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
 	int is_decrypted = skb->decrypted;
 	int is_encrypted = !is_decrypted;
 	struct sk_buff *skb_iter;
+	int left;
 
+	left = rxm->full_len - skb->len;
 	/* Check if all the data is decrypted already */
-	skb_walk_frags(skb, skb_iter) {
+	skb_iter = skb_shinfo(skb)->frag_list;
+	while (skb_iter && left > 0) {
 		is_decrypted &= skb_iter->decrypted;
 		is_encrypted &= !skb_iter->decrypted;
+
+		left -= skb_iter->len;
+		skb_iter = skb_iter->next;
 	}
 
 	trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,
-- 
2.37.1

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

On 8/4/2022 4:24 AM, Jakub Kicinski wrote:
> On Tue, 2 Aug 2022 17:54:01 +0300 Tariq Toukan wrote:
>>    [  407.589886] RIP: 0010:tls_device_decrypted+0x7a/0x2e0
> 
> Sorry, got distracted yesterday. This?

I also had issues yesterday with the decode script.
For some reason, it didn't work for me.
Probably a missing env variable, or a kernel config.

...
? tls_rx_rec_wait (tls_sw.c:?)
tls_rx_one_record (tls_sw.c:?)
tls_sw_recvmsg (??:?)
? __wait_for_common (build_utility.c:?)
? usleep_range_state (??:?)
inet6_recvmsg (??:?)
____sys_recvmsg (socket.c:?)
? _copy_from_user (??:?)
? iovec_from_user (??:?)
___sys_recvmsg (socket.c:?)
WARNING! Modules path isn't set, but is needed to parse this symbol
? post_static_params+0x15c/0x320 mlx5_core
WARNING! Modules path isn't set, but is needed to parse this symbol
? mlx5e_ktls_add_rx+0x3d1/0x5d0 mlx5_core
WARNING! Modules path isn't set, but is needed to parse this symbol
? mlx5e_ktls_add_rx+0x45d/0x5d0 mlx5_core
...

> 
> --->8--------------------
> tls: rx: device: bound the frag walk
> 
> We can't do skb_walk_frags() on the input skbs, because
> the input skbs is really just a pointer to the tcp read
> queue. We need to bound the "is decrypted" check by the
> amount of data in the message.
> 
> Note that the walk in tls_device_reencrypt() is after a
> CoW so the skb there is safe to walk. Actually in the
> current implementation it can't have frags at all, but
> whatever, maybe one day it will.
> 
> Reported-by: Tariq Toukan <tariqt@nvidia.com>
> Fixes: 84c61fe1a75b ("tls: rx: do not use the standard strparser")
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
>   net/tls/tls_device.c | 8 +++++++-
>   1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
> index e3e6cf75aa03..6ed41474bdf8 100644
> --- a/net/tls/tls_device.c
> +++ b/net/tls/tls_device.c
> @@ -984,11 +984,17 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
>   	int is_decrypted = skb->decrypted;
>   	int is_encrypted = !is_decrypted;
>   	struct sk_buff *skb_iter;
> +	int left;
>   
> +	left = rxm->full_len - skb->len;
>   	/* Check if all the data is decrypted already */
> -	skb_walk_frags(skb, skb_iter) {
> +	skb_iter = skb_shinfo(skb)->frag_list;
> +	while (skb_iter && left > 0) {
>   		is_decrypted &= skb_iter->decrypted;
>   		is_encrypted &= !skb_iter->decrypted;
> +
> +		left -= skb_iter->len;
> +		skb_iter = skb_iter->next;
>   	}
>   
>   	trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,

Thanks!
Testing it. I'll update.

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

On 8/4/2022 4:24 AM, Jakub Kicinski wrote:
> On Tue, 2 Aug 2022 17:54:01 +0300 Tariq Toukan wrote:
>>    [  407.589886] RIP: 0010:tls_device_decrypted+0x7a/0x2e0
> 
> Sorry, got distracted yesterday. This?
> 
> --->8--------------------
> tls: rx: device: bound the frag walk
> 
> We can't do skb_walk_frags() on the input skbs, because
> the input skbs is really just a pointer to the tcp read
> queue. We need to bound the "is decrypted" check by the
> amount of data in the message.
> 
> Note that the walk in tls_device_reencrypt() is after a
> CoW so the skb there is safe to walk. Actually in the
> current implementation it can't have frags at all, but
> whatever, maybe one day it will.
> 
> Reported-by: Tariq Toukan <tariqt@nvidia.com>
> Fixes: 84c61fe1a75b ("tls: rx: do not use the standard strparser")
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
>   net/tls/tls_device.c | 8 +++++++-
>   1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
> index e3e6cf75aa03..6ed41474bdf8 100644
> --- a/net/tls/tls_device.c
> +++ b/net/tls/tls_device.c
> @@ -984,11 +984,17 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
>   	int is_decrypted = skb->decrypted;
>   	int is_encrypted = !is_decrypted;
>   	struct sk_buff *skb_iter;
> +	int left;
>   
> +	left = rxm->full_len - skb->len;
>   	/* Check if all the data is decrypted already */
> -	skb_walk_frags(skb, skb_iter) {
> +	skb_iter = skb_shinfo(skb)->frag_list;
> +	while (skb_iter && left > 0) {
>   		is_decrypted &= skb_iter->decrypted;
>   		is_encrypted &= !skb_iter->decrypted;
> +
> +		left -= skb_iter->len;
> +		skb_iter = skb_iter->next;
>   	}
>   
>   	trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,

Now we see a different trace:

------------[ cut here ]------------
WARNING: CPU: 4 PID: 45887 at net/tls/tls_strp.c:53 
tls_strp_msg_make_copy+0x10e/0x120
Modules linked in: sch_netem iptable_raw bonding rdma_ucm nf_tables 
ib_ipoib ib_umad ip_gre ip6_gre gre ip6_tunnel tunnel6 ipip tunnel4 
mlx5_vfio_pci vfio_pci_core vfio_virqfd vfio_iommu_type1 vfio geneve 
mlx5_ib ib_uverbs mlx5_core openvswitch nsh xt_conntrack xt_MASQUERADE 
nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat 
br_netfilter rpcrdma ib_iser libiscsi scsi_transport_iscsi rdma_cm iw_cm 
ib_cm overlay ib_core fuse [last unloaded: ib_uverbs]
CPU: 4 PID: 45887 Comm: iperf Not tainted 
5.19.0-rc8_net_next_mlx5_18607aa #1
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:tls_strp_msg_make_copy+0x10e/0x120
Code: 41 c7 47 44 00 00 00 00 48 8b 44 24 08 65 48 2b 04 25 28 00 00 00 
75 16 48 83 c4 10 4c 89 f8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b eb 
a5 e8 79 0a 28 00 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00
RSP: 0018:ffff88810574fb28 EFLAGS: 00010282
RAX: 00000000fffffff2 RBX: ffff8881136b8cd0 RCX: 0000000000000872
RDX: ffff8881c4bb6000 RSI: 0000000000004855 RDI: ffff88810c4ab500
RBP: 0000000000004855 R08: 00000000000042f0 R09: ffff88810c4aa000
R10: 0000000000000001 R11: 0000000000001000 R12: 0000160000000000
R13: 0000000000000001 R14: ffff8881ada4f8d0 R15: ffff88810c4aa000
FS:  00007f5efe4fe700(0000) GS:ffff88846fa00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5ef4004000 CR3: 00000001ee214005 CR4: 0000000000370ea0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
  <TASK>
  tls_rx_one_record+0x1f2/0x310
  tls_sw_recvmsg+0x327/0x8d0
  inet6_recvmsg+0x62/0x220
  ____sys_recvmsg+0x109/0x120
  ? _copy_from_user+0x44/0x80
  ? iovec_from_user+0x4a/0x150
  ___sys_recvmsg+0xa4/0xe0
  ? find_held_lock+0x2b/0x80
  ? __fget_files+0xb9/0x190
  ? __fget_files+0xd3/0x190
  __sys_recvmsg+0x4e/0x90
  do_syscall_64+0x3d/0x90
  entry_SYSCALL_64_after_hwframe+0x46/0xb0
RIP: 0033:0x7f5effe086dd
Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 5a ef ff ff 8b 54 24 
1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2f 00 00 00 0f 05 <48> 3d 00 
f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 8e ef ff ff 48
RSP: 002b:00007f5efe4fda70 EFLAGS: 00000293 ORIG_RAX: 000000000000002f
RAX: ffffffffffffffda RBX: 00007f5ef4022be0 RCX: 00007f5effe086dd
RDX: 0000000000000000 RSI: 00007f5efe4fdad0 RDI: 0000000000000004
RBP: 0000000000004130 R08: 0000000000000000 R09: 00007f5efe4fdc38
R10: 0000000000000000 R11: 0000000000000293 R12: 00007f5ef40284e3
R13: 00007f5efe4fe660 R14: 00007f5efe4fdb58 R15: 00007f5ef4020ba0
  </TASK>
irq event stamp: 2999
hardirqs last  enabled at (3017): [<ffffffff811daf27>] 
__up_console_sem+0x67/0x70
hardirqs last disabled at (3030): [<ffffffff811daf0c>] 
__up_console_sem+0x4c/0x70
softirqs last  enabled at (2680): [<ffffffff81170d47>] 
irq_exit_rcu+0x97/0xd0
softirqs last disabled at (2671): [<ffffffff81170d47>] 
irq_exit_rcu+0x97/0xd0
---[ end trace 0000000000000000 ]---

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

On Thu, 4 Aug 2022 11:05:18 +0300 Tariq Toukan wrote:
> >   	trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,  
> 
> Now we see a different trace:
> 
> ------------[ cut here ]------------
> WARNING: CPU: 4 PID: 45887 at net/tls/tls_strp.c:53 

Is this with 1.2 or 1.3?

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

On 8/4/2022 6:35 PM, Jakub Kicinski wrote:
> On Thu, 4 Aug 2022 11:05:18 +0300 Tariq Toukan wrote:
>>>    	trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,
>>
>> Now we see a different trace:
>>
>> ------------[ cut here ]------------
>> WARNING: CPU: 4 PID: 45887 at net/tls/tls_strp.c:53
> 
> Is this with 1.2 or 1.3?

Repro with 1.2.
We didn't try with 1.3.

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

On Thu, 4 Aug 2022 11:05:18 +0300 Tariq Toukan wrote:
> >   	trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,  
> 
> Now we see a different trace:
> 
> ------------[ cut here ]------------
> WARNING: CPU: 4 PID: 45887 at net/tls/tls_strp.c:53 

OK, if you find another I promise I'll try to hassle a machine with
offload from somewhere... here's the fix for the new one:

--->8----------------
tls: rx: device: don't try to copy too much on detach

Another device offload bug, we use the length of the output
skb as an indication of how much data to copy. But that skb
is sized to offset + record length, and we start from offset.
So we end up double-counting the offset which leads to
skb_copy_bits() returning -EFAULT.

Reported-by: Tariq Toukan <tariqt@nvidia.com>
Fixes: 84c61fe1a75b ("tls: rx: do not use the standard strparser")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
 net/tls/tls_strp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/tls/tls_strp.c b/net/tls/tls_strp.c
index f0b7c9122fba..9b79e334dbd9 100644
--- a/net/tls/tls_strp.c
+++ b/net/tls/tls_strp.c
@@ -41,7 +41,7 @@ static struct sk_buff *tls_strp_msg_make_copy(struct tls_strparser *strp)
 	struct sk_buff *skb;
 	int i, err, offset;
 
-	skb = alloc_skb_with_frags(0, strp->anchor->len, TLS_PAGE_ORDER,
+	skb = alloc_skb_with_frags(0, strp->stm.full_len, TLS_PAGE_ORDER,
 				   &err, strp->sk->sk_allocation);
 	if (!skb)
 		return NULL;
-- 
2.37.1

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

On 8/4/2022 6:59 PM, Jakub Kicinski wrote:
> On Thu, 4 Aug 2022 11:05:18 +0300 Tariq Toukan wrote:
>>>    	trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,
>>
>> Now we see a different trace:
>>
>> ------------[ cut here ]------------
>> WARNING: CPU: 4 PID: 45887 at net/tls/tls_strp.c:53
> 
> OK, if you find another I promise I'll try to hassle a machine with
> offload from somewhere... here's the fix for the new one:
> 
> --->8----------------
> tls: rx: device: don't try to copy too much on detach
> 
> Another device offload bug, we use the length of the output
> skb as an indication of how much data to copy. But that skb
> is sized to offset + record length, and we start from offset.
> So we end up double-counting the offset which leads to
> skb_copy_bits() returning -EFAULT.
> 
> Reported-by: Tariq Toukan <tariqt@nvidia.com>
> Fixes: 84c61fe1a75b ("tls: rx: do not use the standard strparser")
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
>   net/tls/tls_strp.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/net/tls/tls_strp.c b/net/tls/tls_strp.c
> index f0b7c9122fba..9b79e334dbd9 100644
> --- a/net/tls/tls_strp.c
> +++ b/net/tls/tls_strp.c
> @@ -41,7 +41,7 @@ static struct sk_buff *tls_strp_msg_make_copy(struct tls_strparser *strp)
>   	struct sk_buff *skb;
>   	int i, err, offset;
>   
> -	skb = alloc_skb_with_frags(0, strp->anchor->len, TLS_PAGE_ORDER,
> +	skb = alloc_skb_with_frags(0, strp->stm.full_len, TLS_PAGE_ORDER,
>   				   &err, strp->sk->sk_allocation);
>   	if (!skb)
>   		return NULL;

Hi Jakub,
Thanks for the patch.
We're testing it and I'll update.

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

On 8/7/2022 9:01 AM, Tariq Toukan wrote:
> 
> 
> On 8/4/2022 6:59 PM, Jakub Kicinski wrote:
>> On Thu, 4 Aug 2022 11:05:18 +0300 Tariq Toukan wrote:
>>>>        trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - 
>>>> rxm->full_len,
>>>
>>> Now we see a different trace:
>>>
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 4 PID: 45887 at net/tls/tls_strp.c:53
>>
>> OK, if you find another I promise I'll try to hassle a machine with
>> offload from somewhere... here's the fix for the new one:
>>
>> --->8----------------
>> tls: rx: device: don't try to copy too much on detach
>>
>> Another device offload bug, we use the length of the output
>> skb as an indication of how much data to copy. But that skb
>> is sized to offset + record length, and we start from offset.
>> So we end up double-counting the offset which leads to
>> skb_copy_bits() returning -EFAULT.
>>
>> Reported-by: Tariq Toukan <tariqt@nvidia.com>
>> Fixes: 84c61fe1a75b ("tls: rx: do not use the standard strparser")
>> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
>> ---
>>   net/tls/tls_strp.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/net/tls/tls_strp.c b/net/tls/tls_strp.c
>> index f0b7c9122fba..9b79e334dbd9 100644
>> --- a/net/tls/tls_strp.c
>> +++ b/net/tls/tls_strp.c
>> @@ -41,7 +41,7 @@ static struct sk_buff *tls_strp_msg_make_copy(struct 
>> tls_strparser *strp)
>>       struct sk_buff *skb;
>>       int i, err, offset;
>> -    skb = alloc_skb_with_frags(0, strp->anchor->len, TLS_PAGE_ORDER,
>> +    skb = alloc_skb_with_frags(0, strp->stm.full_len, TLS_PAGE_ORDER,
>>                      &err, strp->sk->sk_allocation);
>>       if (!skb)
>>           return NULL;
> 
> Hi Jakub,
> Thanks for the patch.
> We're testing it and I'll update.

Trace is gone. But we got TlsDecryptError counter increasing by 2.

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

On Mon, 8 Aug 2022 08:24:05 +0300 Tariq Toukan wrote:
> > Thanks for the patch.
> > We're testing it and I'll update.  
> 
> Trace is gone. But we got TlsDecryptError counter increasing by 2.

Does the connection break? Is there a repro?

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

On 8/8/2022 9:24 PM, Jakub Kicinski wrote:
> On Mon, 8 Aug 2022 08:24:05 +0300 Tariq Toukan wrote:
>>> Thanks for the patch.
>>> We're testing it and I'll update.
>>
>> Trace is gone. But we got TlsDecryptError counter increasing by 2.
> 
> Does the connection break? Is there a repro?

Hi Jakub,

Please ignore my previous email regarding the TlsDecryptError. It was a 
false alarm.

Fix looks goods.

Tested-by: Ran Rozenstein <ranro@nvidia.com>

Regards,
Tariq

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

On 23/07/2022 2:50, Jakub Kicinski wrote:
> TLS is a relatively poor fit for strparser. We pause the input
> every time a message is received, wait for a read which will
> decrypt the message, start the parser, repeat. strparser is
> built to delineate the messages, wrap them in individual skbs
> and let them float off into the stack or a different socket.
> TLS wants the data pages and nothing else. There's no need
> for TLS to keep cloning (and occasionally skb_unclone()'ing)
> the TCP rx queue.
> 
> This patch uses a pre-allocated skb and attaches the skbs
> from the TCP rx queue to it as frags. TLS is careful never
> to modify the input skb without CoW'ing / detaching it first.
> 
> Since we call TCP rx queue cleanup directly we also get back
> the benefit of skb deferred free.
> 
> Overall this results in a 6% gain in my benchmarks.
> 
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>

Hi Jakub,

A few fixes were introduced for this patch, but it seems to still cause 
issues.

I'm running simple client/server test with wrk and nginx and TLS RX 
device offload on.
It fails with TlsDecryptError on the client side for the large file 
(256000b), while succeeding for the small one (10000b). See repro 
details below.

I narrowed the issue down to this offending patch, by applying a few 
reverts (had to solve trivial conflicts):

Revert "tls: rx: do not use the standard strparser"
Revert "tls: rx: fix the false positive warning"
Revert "tls: rx: device: bound the frag walk"
Revert "tls: rx: device: don't try to copy too much on detach"
Revert "tls: rx: react to strparser initialization errors"
Revert "tls: strp: make sure the TCP skbs do not have overlapping data"

The error is caught by the if below, inside gcmaes_decrypt.

  819         /* Compare generated tag with passed in tag. */ 

  820         if (crypto_memneq(auth_tag_msg, auth_tag, auth_tag_len)) { 

  821                 memzero_explicit(auth_tag, sizeof(auth_tag)); 

  822                 return -EBADMSG; 

  823         }

but I couldn't yet identify the root cause.

Any idea what could it be?

Reproduction:

$ ethtool --features eth2 tls-hw-rx-offload on

$ wrk_openssl_3_0_0 -b2.2.2.2 -t10 -c1000 -d5 --timeout 5s 
https://2.2.2.3:20443/10000b.img
Running 5s test @ https://2.2.2.3:20443/10000b.img
   10 threads and 1000 connections
Ready 8/10
Ready 2/10
Ready 9/10
Ready 3/10
Ready 4/10
Ready 6/10
Ready 0/10
Ready 1/10
Ready 7/10
Ready 5/10
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
   Thread Stats   Avg      Stdev     Max   +/- Stdev
     Latency   112.97ms   19.05ms 160.31ms   80.71%
     Req/Sec     0.88k   122.34     1.07k    61.86%
   37495 requests in 5.06s, 366.24MB read
Requests/sec:   7415.86
Transfer/sec:     72.44MB

$ cat /proc/net/tls_stat
TlsCurrTxSw                             0
TlsCurrRxSw                             0
TlsCurrTxDevice                         0
TlsCurrRxDevice                         0
TlsTxSw                                 0
TlsRxSw                                 0
TlsTxDevice                             1000
TlsRxDevice                             1000
TlsDecryptError                         0
TlsRxDeviceResync                       1000
TlsDecryptRetry                         0
TlsRxNoPadViolation                     0

$ wrk_openssl_3_0_0 -b2.2.2.2 -t10 -c1000 -d5 --timeout 5s 
https://2.2.2.3:20443/256000b.img
Running 5s test @ https://2.2.2.3:20443/256000b.img
   10 threads and 1000 connections
Ready 6/10
Ready 2/10
Ready 7/10
Ready 9/10
Ready 1/10
Ready 4/10
Ready 8/10
Ready 3/10
Ready 5/10
Ready 0/10
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
Ready all
   Thread Stats   Avg      Stdev     Max   +/- Stdev
     Latency     3.94s   106.78ms   4.10s    50.00%
     Req/Sec     0.67      0.82     2.00     83.33%
   38 requests in 7.71s, 154.14MB read
   Socket errors: connect 0, read 1810, write 0, timeout 0
Requests/sec:      4.93
Transfer/sec:     19.98MB

$ cat /proc/net/tls_stat
TlsCurrTxSw                             0
TlsCurrRxSw                             0
TlsCurrTxDevice                         0
TlsCurrRxDevice                         0
TlsTxSw                                 0
TlsRxSw                                 0
TlsTxDevice                             3810
TlsRxDevice                             3810
TlsDecryptError                         3634
TlsRxDeviceResync                       3060
TlsDecryptRetry                         0
TlsRxNoPadViolation                     0

More monitors:

$ bpftrace -e 'kretprobe:generic_gcmaes_decrypt { @retval[retval] = 
count(); }'
Attaching 1 probe...
^C

@retval[4294967222]: 2067
@retval[0]: 26143


$ bpftrace -e 'kprobe:generic_gcmaes_decrypt { @[kstack] = count(); }'
Attaching 1 probe...
^C

@[
     generic_gcmaes_decrypt+1
     tls_decrypt_sg+1258
     tls_rx_one_record+72
     tls_sw_recvmsg+870
     inet_recvmsg+71
     sock_recvmsg+68
     ____sys_recvmsg+111
     ___sys_recvmsg+126
     __sys_recvmsg+78
     do_syscall_64+61
     entry_SYSCALL_64_after_hwframe+70
]: 27623

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

On Thu, 9 Mar 2023 17:15:26 +0200 Tariq Toukan wrote:
> A few fixes were introduced for this patch, but it seems to still cause 
> issues.
> 
> I'm running simple client/server test with wrk and nginx and TLS RX 
> device offload on.
> It fails with TlsDecryptError on the client side for the large file 
> (256000b), while succeeding for the small one (10000b). See repro 
> details below.
> 
> I narrowed the issue down to this offending patch, by applying a few 
> reverts (had to solve trivial conflicts):

What's the sequence of records in terms of being offloaded vs fall back?
Could you whip up a simple ring buffer to see if previous records were
offloaded and what the skb geometries where?

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

[-- Attachment #1: Type: text/plain, Size: 2580 bytes --]



On 09/03/2023 19:54, Jakub Kicinski wrote:
> On Thu, 9 Mar 2023 17:15:26 +0200 Tariq Toukan wrote:
>> A few fixes were introduced for this patch, but it seems to still cause
>> issues.
>>
>> I'm running simple client/server test with wrk and nginx and TLS RX
>> device offload on.
>> It fails with TlsDecryptError on the client side for the large file
>> (256000b), while succeeding for the small one (10000b). See repro
>> details below.
>>
>> I narrowed the issue down to this offending patch, by applying a few
>> reverts (had to solve trivial conflicts):
> 
> What's the sequence of records in terms of being offloaded vs fall back?
> Could you whip up a simple ring buffer to see if previous records were
> offloaded and what the skb geometries where?

Interesting. All records go through the sw fallback.

Command:
$ wrk_openssl_3_0_0 -b2.2.2.2 -t1 -c1 -d2 --timeout 5s 
https://2.2.2.3:20443/256000b.img

Debug code:
@@ -1712,8 +1723,13 @@ static int tls_rx_one_record(struct sock *sk, 
struct msghdr *msg,
         int err;

         err = tls_decrypt_device(sk, msg, tls_ctx, darg);
-       if (!err)
+       if (!err) {
                 err = tls_decrypt_sw(sk, tls_ctx, msg, darg);
+               printk("sk: %p, tls_decrypt_sw, err = %d\n", sk, err);
+       } else {
+               printk("sk: %p, tls_decrypt_device, err = %d\n", sk, err);
+       }
+       skb_dump(KERN_ERR, darg->skb, false);
         if (err < 0)
                 return err;

dmesg output including skb geometries is attached.

# cat /proc/net/tls_stat
TlsCurrTxSw                             0
TlsCurrRxSw                             0
TlsCurrTxDevice                         0
TlsCurrRxDevice                         0
TlsTxSw                                 0
TlsRxSw                                 0
TlsTxDevice                             12
TlsRxDevice                             12
TlsDecryptError                         22
TlsRxDeviceResync                       12
TlsDecryptRetry                         0
TlsRxNoPadViolation                     0

We start from out-of-sync state, so we expect to go through the sw 
fallback in the beginning.
I do see that the resync succeeds and device-offloaded packets start to 
flow, example ethtool output (1 sec interval):

       rx_tls_decrypted_packets: 356
         rx_tls_decrypted_bytes: 517,160 Bps          = 4.13 Mbps
          rx_tls_resync_req_pkt: 6
        rx_tls_resync_req_start: 6
          rx_tls_resync_req_end: 6
           rx_tls_resync_res_ok: 6
                     rx_tls_ctx: 6
                     rx_tls_del: 6

[-- Attachment #2: dmesg_tls_rx_decrypt_err.txt --]
[-- Type: text/plain, Size: 131169 bytes --]

[ 2579.783070] mlx5_core 0000:08:00.0: firmware version: 22.36.1010
[ 2579.783824] mlx5_core 0000:08:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0000:00:02.7 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 2580.113710] mlx5_core 0000:08:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 2580.115133] mlx5_core 0000:08:00.0: E-Switch: Total vports 2, per vport: max uc(128) max mc(2048)
[ 2580.142190] mlx5_core 0000:08:00.0: Port module event: module 0, Cable plugged
[ 2580.145306] mlx5_core 0000:08:00.0: mlx5_pcie_event:301:(pid 31122): PCIe slot advertised sufficient power (75W).
[ 2580.162126] mlx5_core 0000:08:00.0: mlx5e: IPSec ESP acceleration enabled
[ 2580.256370] mlx5_core 0000:08:00.0: Supported tc offload range - chains: 4294967294, prios: 4294967295
[ 2580.266364] mlx5_core 0000:08:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 2580.270988] mlx5_core 0000:08:00.1: firmware version: 22.36.1010
[ 2580.272199] mlx5_core 0000:08:00.1: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0000:00:02.7 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 2580.614487] mlx5_core 0000:08:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 2580.616122] mlx5_core 0000:08:00.1: E-Switch: Total vports 2, per vport: max uc(128) max mc(2048)
[ 2580.642233] mlx5_core 0000:08:00.1: Port module event: module 1, Cable plugged
[ 2580.644670] mlx5_core 0000:08:00.1: mlx5_pcie_event:301:(pid 31122): PCIe slot advertised sufficient power (75W).
[ 2580.659391] mlx5_core 0000:08:00.1: mlx5e: IPSec ESP acceleration enabled
[ 2580.764903] mlx5_core 0000:08:00.1: Supported tc offload range - chains: 4294967294, prios: 4294967295
[ 2580.774865] mlx5_core 0000:08:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 2584.458976] mlx5_core 0000:08:00.0 eth2: Link up
[ 2584.465661] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[ 2609.303033] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.303570] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x23a86512 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.305831] skb frag:     00000000: 00 9b 08 03 81 88 ff ff 00 00 00 00 00 14 00 00
[ 2609.306535] skb frag:     00000010: 0c 76 67 4b 70 ff 4e e1 a1 ef a6 04 4b ea ff ff
[ 2609.307252] skb frag:     00000020: 40 26 0c 04 00 ea ff ff 00 53 ae 04 00
[ 2609.354206] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.355676] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.362752] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.364245] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.373896] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2609.374597] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.375287] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.375987] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.376685] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.377370] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.378070] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.378756] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.379447] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.380131] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.380826] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.381519] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.382223] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.382956] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.383449] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.385723] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 33 34 35
[ 2609.386426] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.387120] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.387812] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.388515] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.389210] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.389911] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.390614] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.391310] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.392020] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.392743] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.393438] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.394143] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.394871] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.395374] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.397653] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 35 36 37
[ 2609.398378] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.399080] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.399766] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.400469] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.401164] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.401856] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.402563] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.403259] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.403955] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.404650] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.405345] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.406041] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.406770] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.407265] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.409554] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 37 38 39
[ 2609.410273] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.410959] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.411647] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.412332] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.413040] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.413729] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.414430] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.415118] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.415807] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.416508] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.417192] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.417886] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.418611] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.419126] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.421385] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 39 30 31
[ 2609.422095] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.422786] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.423481] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.424168] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.424864] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.425557] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.426258] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.426955] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.427646] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.428339] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.429038] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.429731] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.430460] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2609.430962] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.433261] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2609.433970] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.434667] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.435357] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.436051] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.436751] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.437445] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.438149] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.438841] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.439531] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.447110] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.447803] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.448503] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.449209] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2609.449713] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.451981] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2609.452684] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.453380] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.454077] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.454760] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.455454] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.456147] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.456837] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.457529] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.458230] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.458921] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.459616] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.460308] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.484034] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.485052] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x238a29f1 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.488767] skb frag:     00000000: 2e 5e b7 f3 f8 1a f2 c7 55 1a d2 2e 81 14 00 00
[ 2609.489896] skb frag:     00000010: 0c 50 98 3f c6 fa db ea 36 7e 00 f7 d5 00 00 00
[ 2609.491049] skb frag:     00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 2609.534148] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.535667] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.541443] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.541972] skb len=16898 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.544384] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.544914] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=11))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.547347] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 33 34 35
[ 2609.548078] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.548822] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.549557] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.550306] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.551038] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.551782] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.552543] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.553262] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.553966] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.554660] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.555359] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.556048] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.556773] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.557262] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.559541] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 35 36 37
[ 2609.560453] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.561172] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.561881] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.562635] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.563389] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.564160] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.564904] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.565645] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.566454] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.567216] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.567994] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.568728] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.569454] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.569960] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.572215] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 37 38 39
[ 2609.572925] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.573611] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.574315] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.575001] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.575692] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.576384] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.577083] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.577778] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.578472] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.579165] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.579856] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.580561] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.581277] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.581769] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=17))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.584041] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 39 30 31
[ 2609.584739] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.585425] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.586129] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.586818] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.587504] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.588197] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.588891] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.589580] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.597164] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.597848] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.598546] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.599232] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.599949] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2609.600461] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=17))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=1 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.602718] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2609.603406] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.604090] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.604785] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.605476] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.606171] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.606856] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.607546] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.608232] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.608929] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.609621] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.610324] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.611015] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.611728] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2609.612229] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=17))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=1 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.614517] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2609.615210] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.615897] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.616594] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.617279] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.617980] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.618671] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.619361] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.620052] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.620746] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.621439] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.622134] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.622826] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.647343] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.648832] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0xfbfa5696 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.655460] skb frag:     00000000: 4e fb 3b a5 de 72 f9 0b 8e f1 84 ee f1 14 00 00
[ 2609.656849] skb frag:     00000010: 0c 10 91 c6 e9 b5 9d 69 8e 3e 61 62 b5 00 00 00
[ 2609.657542] skb frag:     00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 2609.702112] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.702610] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.704942] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.705838] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.710043] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2609.711316] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.712609] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.713456] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.714159] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.714847] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.715543] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.716238] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.716932] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.717628] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.718329] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.719021] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.719713] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.720464] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.720951] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.723222] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 33 34 35
[ 2609.723914] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.724617] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.725312] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.726016] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.726710] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.727399] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.728092] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.728791] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.729487] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.730203] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.730894] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.731590] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.732312] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.732815] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=20))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.735085] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 35 36 37
[ 2609.735778] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.736469] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.737163] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.737853] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.738553] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.739242] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.739928] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.740633] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.741327] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.748890] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.749586] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.750286] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.751013] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.751503] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=20))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.753770] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 37 38 39
[ 2609.754469] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.755158] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.755845] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.756533] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.757223] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.757909] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.758608] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.759294] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.759985] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.760688] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.761393] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.762114] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.762837] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.763335] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=4))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.765627] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 39 30 31
[ 2609.766346] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.767042] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.767738] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.768439] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.769136] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.769823] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.770543] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.771242] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.771930] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.772633] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.773324] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.774027] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.774744] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2609.775248] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=4))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.777523] skb linear:   00000000: 60 05 0b a1 00 90 11 01 fe 80 00 00 00 00 00 00
[ 2609.778228] skb linear:   00000010: 02 50 56 ff fe 30 15 51 ff 02 00 00 00 00 00 00
[ 2609.778919] skb linear:   00000020: 00 00 00 00 00 00 00 fb 14 e9 14 e9 00 90 e6 2d
[ 2609.779610] skb linear:   00000030: 00 00 84 00 00 00 00 04 00 00 00 00 12 67 65 6e
[ 2609.780304] skb linear:   00000040: 2d 6c 2d 76 72 74 2d 30 33 34 30 2d 30 38 31 05
[ 2609.781009] skb linear:   00000050: 6c 6f 63 61 6c 00 00 1c 00 01 00 00 00 3c 00 10
[ 2609.781706] skb linear:   00000060: fd fd fd fd 00 10 02 37 00 00 00 00 10 00 46 22
[ 2609.782401] skb linear:   00000070: c0 0c 00 1c 00 01 00 00 00 3c 00 10 fd fd fd fd
[ 2609.783093] skb linear:   00000080: 00 10 02 37 02 50 56 ff fe 30 15 51 c0 0c 00 1c
[ 2609.783797] skb linear:   00000090: 00 01 00 00 00 3c 00 10 fe 80 00 00 00 00 00 00
[ 2609.784500] skb linear:   000000a0: 02 50 56 ff fe 30 15 51 c0 0c 00 01 00 01 00 00
[ 2609.785197] skb linear:   000000b0: 00 3c 00 04 0a ed 15 51
[ 2609.785722] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2609.786237] skb len=184 headroom=26 headlen=184 tailroom=3566
               mac=(12,14) net=(26,40) trans=66
               shinfo(txflags=0 nr_frags=0 gso(size=0 type=0 segs=0))
               csum(0x0 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=0 l4=0) proto=0x86dd pkttype=0 iif=3
[ 2609.788464] dev name=eth1 feat=0x0080010000174a29
[ 2609.788905] skb linear:   00000000: 60 05 0b a1 00 90 11 01 fe 80 00 00 00 00 00 00
[ 2609.789592] skb linear:   00000010: 02 50 56 ff fe 30 15 51 ff 02 00 00 00 00 00 00
[ 2609.790292] skb linear:   00000020: 00 00 00 00 00 00 00 fb 14 e9 14 e9 00 90 e6 2d
[ 2609.790986] skb linear:   00000030: 00 00 84 00 00 00 00 04 00 00 00 00 12 67 65 6e
[ 2609.791680] skb linear:   00000040: 2d 6c 2d 76 72 74 2d 30 33 34 30 2d 30 38 31 05
[ 2609.792371] skb linear:   00000050: 6c 6f 63 61 6c 00 00 1c 00 01 00 00 00 3c 00 10
[ 2609.793065] skb linear:   00000060: fd fd fd fd 00 10 02 37 00 00 00 00 10 00 46 22
[ 2609.793757] skb linear:   00000070: c0 0c 00 1c 00 01 00 00 00 3c 00 10 fd fd fd fd
[ 2609.794453] skb linear:   00000080: 00 10 02 37 02 50 56 ff fe 30 15 51 c0 0c 00 1c
[ 2609.795144] skb linear:   00000090: 00 01 00 00 00 3c 00 10 fe 80 00 00 00 00 00 00
[ 2609.795836] skb linear:   000000a0: 02 50 56 ff fe 30 15 51 c0 0c 00 01 00 01 00 00
[ 2609.796538] skb linear:   000000b0: 00 3c 00 04 0a ed 15 51
[ 2609.819404] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.820961] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x6a26e4dc sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.826023] skb frag:     00000000: 51 14 59 e9 a9 c0 c5 d9 17 d7 ff b2 a1 14 00 00
[ 2609.826962] skb frag:     00000010: 0c 38 98 a8 a2 c1 45 b5 87 bb 22 0c 9c 00 00 00
[ 2609.827779] skb frag:     00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 2609.870154] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.871890] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.876846] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.877797] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.882238] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2609.883589] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.884800] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.885488] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.886193] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.886882] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.887577] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.888263] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.888965] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.889657] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.890363] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.891057] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.891748] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.892488] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.892982] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.902111] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 33 34 35
[ 2609.902799] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.903486] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.904178] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.904882] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.905582] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.906281] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.906971] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.907659] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.908345] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.909042] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.909729] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.910430] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.911149] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.911646] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=0 segs=16))
               csum(0x28f2 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.913899] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 35 36 37
[ 2609.914603] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.915285] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.915981] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.916673] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.917360] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.918062] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.918748] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.919442] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.920130] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.920822] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.921509] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.922208] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.922929] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.923420] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=0 segs=16))
               csum(0x28f2 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.925702] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 37 38 39
[ 2609.926408] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.927099] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.927787] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.928486] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.929177] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.929872] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.930570] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.931258] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.931953] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.932654] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.933344] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.934045] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.934773] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2609.935260] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2609.937540] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 39 30 31
[ 2609.938246] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.938938] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.939625] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.940314] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.941013] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.941699] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.942404] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.943097] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.943794] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.944487] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.945182] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.945884] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.946618] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2609.947124] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=1 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.949411] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2609.950120] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.950814] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.951512] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.952204] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.952905] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.953604] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.954314] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.955006] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.955694] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.956389] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.957087] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.957787] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.958502] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2609.959006] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=1 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2609.961294] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2609.961999] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.962688] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.963379] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.964069] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.964762] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.965457] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.966155] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.973720] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2609.974425] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2609.975115] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2609.975811] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2609.976502] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2609.999410] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.000532] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0xc981cf3c sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.004638] skb frag:     00000000: 51 14 59 e9 a9 c0 c5 d9 17 d7 ff b2 a1 14 00 00
[ 2610.005905] skb frag:     00000010: 0c 66 da a6 2e a6 9d fa 83 2e 49 7c e7 00 00 00
[ 2610.007173] skb frag:     00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 2610.050132] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.051767] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.059340] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.060956] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.065852] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2610.066554] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.067241] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.067930] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.068619] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.069322] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.070027] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.070719] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.071416] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.072112] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.072820] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.073511] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.074218] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.074959] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.075454] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.077716] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 33 34 35
[ 2610.078423] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.079107] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.079792] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.080484] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.081167] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.081864] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.082562] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.083305] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.084049] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.084823] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.085578] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.086339] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.087155] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.087706] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.090163] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 35 36 37
[ 2610.090917] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.091659] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.092345] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.093056] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.093746] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.094448] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.095140] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.095830] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.096527] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.097213] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.097907] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.098603] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.099331] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.099820] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.102137] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 37 38 39
[ 2610.102829] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.103520] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.104214] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.104908] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.105605] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.106307] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.107004] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.107696] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.108392] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.109091] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.109782] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.110494] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.111216] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.111709] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.114030] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 39 30 31
[ 2610.114721] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.115418] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.116115] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.116812] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.117506] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.118204] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.118892] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.119586] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.120272] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.127863] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.128566] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.129250] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.129985] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2610.130484] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.132748] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2610.133434] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.134141] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.134827] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.135509] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.136200] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.136887] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.137591] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.138291] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.138985] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.139674] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.140366] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.141060] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.141768] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2610.142286] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=1)
               hash(0x0 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.144551] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2610.145244] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.145937] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.146633] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.147325] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.148017] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.148724] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.149418] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.150119] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.150805] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.151498] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.152191] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.152899] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.174046] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.175380] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x785a04c6 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.181158] skb frag:     00000000: d1 aa c6 19 6c 4e ec 9a d9 03 b6 e8 0e 14 00 00
[ 2610.181849] skb frag:     00000010: 0c 25 33 a5 e2 1b 48 f7 54 dc 78 17 e9 00 00 00
[ 2610.182548] skb frag:     00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 2610.226191] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.227954] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.233862] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.234406] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.236824] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2610.237558] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.238309] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.239044] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.239785] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.240531] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.241270] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.242014] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.242758] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.243496] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.244229] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.244979] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.245671] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.246411] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.246899] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.249172] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 33 34 35
[ 2610.249863] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.250569] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.251257] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.251951] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.252655] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.253342] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.254044] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.254743] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.255447] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.256138] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.256852] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.257546] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.258287] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.258815] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.261106] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 35 36 37
[ 2610.261812] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.262527] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.263223] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.263915] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.264634] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.265323] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.266032] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.266725] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.267415] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.268107] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.275685] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.276377] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.277113] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.277601] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.279865] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 37 38 39
[ 2610.280557] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.281251] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.281947] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.282636] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.283321] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.284010] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.284717] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.285408] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.286128] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.286819] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.287511] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.288201] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.288930] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.289445] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.291709] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 39 30 31
[ 2610.292397] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.293099] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.293792] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.294488] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.295183] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.295874] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.296570] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.297258] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.297964] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.298651] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.299338] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.300030] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.300758] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2610.301264] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=0 valid=0 level=1)
               hash(0x0 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.303537] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2610.304237] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.304944] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.305641] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.306341] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.307039] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.307731] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.308429] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.309129] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.309822] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.310524] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.311212] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.311910] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.312633] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2610.313137] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=0 valid=0 level=1)
               hash(0x0 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.315411] skb frag:     00000000: 08 e0 31 03 81 88 ff ff 00 00 00 00 00 31 32 33
[ 2610.316107] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.316801] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.317491] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.318192] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.318883] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.319583] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.320275] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.320976] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.321667] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.322378] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.323072] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.323763] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.355230] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.356572] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x901e34b2 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.361429] skb frag:     00000000: 1d 58 27 0e 11 39 f5 9d 0e 3d 9f 15 0c 14 00 00
[ 2610.362287] skb frag:     00000010: 0c da 66 29 30 73 8a f0 21 a4 97 5b 00 00 00 00
[ 2610.363127] skb frag:     00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 2610.406284] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.407916] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.413615] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.414120] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.416388] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.417081] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.417780] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.418491] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.419196] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.419901] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.420596] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.421290] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.421993] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.422690] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.423384] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.424082] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.424799] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.432457] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.432950] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.435231] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 33 34 35
[ 2610.435922] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.436619] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.437302] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.438010] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.438701] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.439392] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.440077] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.440774] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.441462] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.442163] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.442853] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.443547] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.444280] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.444792] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=14))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.447072] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 35 36 37
[ 2610.447763] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.448464] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.449153] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.449849] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.450550] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.451243] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.451938] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.452639] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.453337] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.454041] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.454735] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.455423] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.456148] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.456647] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=6))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.458925] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 37 38 39
[ 2610.459620] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.460314] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.461014] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.461707] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.462415] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.463105] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.463800] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.464508] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.465203] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.465900] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.466597] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.467289] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.468009] sk: 00000000a55afc21, tls_decrypt_sw, err = 0
[ 2610.468505] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.470784] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 39 30 31
[ 2610.471482] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.472173] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.472876] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.473567] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.474270] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.474965] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.475657] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.476350] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.477042] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.477736] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.478434] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.479130] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.479849] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2610.480348] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=2)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.482629] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.483320] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.484013] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.484723] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.485419] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.486122] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.486818] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.487506] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.488203] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.488900] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.489590] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.490295] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.490986] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.491697] sk: 00000000a55afc21, tls_decrypt_sw, err = -74
[ 2610.492196] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=2)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.494481] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.495168] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.495860] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.496558] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.497260] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.497957] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.498650] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.506293] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.506988] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.507680] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.508376] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.509070] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.509760] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.546224] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.547158] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x2bb0e4f7 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.550890] skb frag:     00000000: 2e 67 ae 4a 0a 35 9b 07 e8 9c 91 3e 94 14 00 00
[ 2610.552115] skb frag:     00000010: 0c d9 bc 43 d6 a4 a7 31 25 43 d8 34 36 00 00 00
[ 2610.553273] skb frag:     00000020: 72 78 39 5f 63 73 75 6d 5f 75 6e 6e 65
[ 2610.598171] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.598739] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.601025] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.601514] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.603788] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.604493] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.605188] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.605883] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.606581] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.607270] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.607964] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.608658] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.609353] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.610060] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.610750] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.611444] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.612133] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.612882] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.613385] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.615937] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 33 34 35
[ 2610.617017] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.618159] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.619262] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.620150] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.620857] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.621543] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.622255] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.622944] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.623641] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.624327] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.625033] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.625728] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.626486] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.626985] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=0 segs=16))
               csum(0x28f2 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.629244] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 35 36 37
[ 2610.629954] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.630642] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.631333] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.632020] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.632728] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.633417] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.634127] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.634826] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.635518] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.636213] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.636907] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.637601] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.638345] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.638843] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=0 segs=16))
               csum(0x28f2 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.641107] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 37 38 39
[ 2610.641801] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.642514] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.643204] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.643898] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.644595] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.645287] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.646002] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.646695] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.647385] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.648080] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.648774] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.649466] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.650206] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.650704] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.653003] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 39 30 31
[ 2610.653693] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.654400] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.655088] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.655782] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.656481] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.657177] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.657866] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.665491] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.666189] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.666885] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.667570] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.668266] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.668994] sk: 00000000d4b8ce89, tls_decrypt_sw, err = -74
[ 2610.669501] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=2)
               hash(0x0 sw=1 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.671774] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.672485] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.673171] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.673866] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.674562] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.675250] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.675947] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.676641] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.677337] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.678035] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.678729] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.679419] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.680108] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.680818] sk: 00000000d4b8ce89, tls_decrypt_sw, err = -74
[ 2610.681319] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=2)
               hash(0x0 sw=1 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.683591] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.684280] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.684985] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.685672] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.686379] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.687069] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.687763] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.688454] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.689154] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.689845] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.690544] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.691234] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.691923] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.714603] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.715855] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x8e333708 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.721187] skb frag:     00000000: ea 07 ef 6a cd 55 8d 77 21 20 9e 96 d1 14 00 00
[ 2610.721950] skb frag:     00000010: 0c 83 c8 76 12 23 4c 07 eb 89 d3 10 46 00 00 00
[ 2610.722733] skb frag:     00000020: 72 78 39 5f 63 73 75 6d 5f 75 6e 6e 65
[ 2610.766136] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.767105] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.771139] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.771996] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.775968] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.777196] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.778432] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.779652] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.780760] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.781452] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.782152] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.782849] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.783541] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.784238] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.784935] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.785626] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.786329] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.787058] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.787553] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.789830] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 33 34 35
[ 2610.790533] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.791220] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.791913] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.792612] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.793306] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.794007] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.794697] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.795392] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.796077] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.796776] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.797467] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.798172] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.798891] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.799386] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.801701] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 35 36 37
[ 2610.802407] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.803102] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.803797] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.804498] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.805190] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.805885] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.806586] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.807283] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.807973] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.815578] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.816274] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.816974] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.817693] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.818192] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=11))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.820464] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 37 38 39
[ 2610.821157] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.821849] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.822549] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.823238] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.823930] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.824625] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.825319] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.826020] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.826710] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.827411] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.828100] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.828799] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.829520] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.830027] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.832289] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 39 30 31
[ 2610.832994] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.833686] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.834389] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.835085] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.835777] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.836481] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.837171] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.837868] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.838570] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.839263] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.839956] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.840661] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.841382] sk: 00000000d4b8ce89, tls_decrypt_sw, err = -74
[ 2610.841883] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=2)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.844144] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.844845] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.845534] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.846229] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.846918] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.847605] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.848296] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.848991] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.849678] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.850384] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.851069] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.851760] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.852457] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.853179] sk: 00000000d4b8ce89, tls_decrypt_sw, err = -74
[ 2610.853682] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=2)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.855950] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.856645] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.857353] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.858056] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.858749] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.859440] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.860130] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.860844] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.861535] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.862252] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.862943] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.863637] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.864331] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.888633] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.890173] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0x984b6827 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2610.896878] skb frag:     00000000: b2 39 65 79 77 dc bd ff a7 8c 5d b4 66 14 00 00
[ 2610.897985] skb frag:     00000010: 0c 8d 74 1c 47 fd 4c 3d 35 80 0f a1 c2 00 00 00
[ 2610.899157] skb frag:     00000020: 72 78 39 5f 63 73 75 6d 5f 75 6e 6e 65
[ 2610.942127] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.942998] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.946445] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.947184] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.950605] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2610.951655] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.952696] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.953744] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.954798] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.955834] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.956908] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.957595] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.958301] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.958994] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.959687] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.967313] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.968009] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.968760] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.969259] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.971530] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 33 34 35
[ 2610.972218] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.972913] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.973606] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.974303] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.974996] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.975686] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.976380] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.977074] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.977767] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.978466] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.979155] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.979847] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.980577] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.981080] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.983356] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 35 36 37
[ 2610.984049] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.984751] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.985448] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.986149] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.986843] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.987541] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.988234] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.988937] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.989628] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.990338] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.991031] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.991725] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.992449] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2610.992944] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2610.995224] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 37 38 39
[ 2610.995917] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2610.996625] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2610.997315] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2610.998014] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2610.998704] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2610.999399] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.000088] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.000786] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.001476] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.002174] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.002868] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.003556] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.004280] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.004782] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.007064] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 39 30 31
[ 2611.007755] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.008465] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.009158] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.009849] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.010549] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.011237] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.011929] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.012617] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.013308] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.014004] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.014692] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.015378] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.016098] sk: 00000000d4b8ce89, tls_decrypt_sw, err = -74
[ 2611.016604] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=2)
               hash(0x0 sw=1 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2611.018867] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2611.019556] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.020243] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.020940] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.021628] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.022334] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.023023] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.023719] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.024427] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.025127] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.025822] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.026521] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.027216] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.027927] sk: 00000000d4b8ce89, tls_decrypt_sw, err = -74
[ 2611.028442] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=2)
               hash(0x0 sw=1 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2611.030725] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2611.031416] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.032106] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.032804] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.033497] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.041113] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.041811] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.042506] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.043193] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.043887] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.044592] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.045285] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.045981] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.071471] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.072994] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0xd928c678 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2611.077789] skb frag:     00000000: 54 e4 22 f1 15 8f 76 30 3b eb db 9f 75 14 00 00
[ 2611.078575] skb frag:     00000010: 0c cb be 8f a6 e1 3e cf a1 c1 c6 a4 3c 00 00 00
[ 2611.079358] skb frag:     00000020: 72 78 39 5f 63 73 75 6d 5f 75 6e 6e 65
[ 2611.122107] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.123742] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.129671] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.130915] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.136722] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2611.137549] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.138249] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.138945] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.139637] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.140331] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.141024] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.141722] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.142427] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.143114] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.143807] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.144500] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.145198] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.145945] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.146438] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.148722] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 33 34 35
[ 2611.149415] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.150115] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.151119] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.152204] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.153308] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.154431] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.155471] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.156160] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.156877] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.157570] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.158277] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.158970] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.159712] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.160206] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=15))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.162513] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 35 36 37
[ 2611.163209] skb frag:     00000010: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.163906] skb frag:     00000020: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.164602] skb frag:     00000030: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.165294] skb frag:     00000040: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.166002] skb frag:     00000050: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.166697] skb frag:     00000060: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.167388] skb frag:     00000070: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.168082] skb frag:     00000080: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.168795] skb frag:     00000090: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.169484] skb frag:     000000a0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.170188] skb frag:     000000b0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.170876] skb frag:     000000c0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.171602] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.172092] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=11))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.174391] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 37 38 39
[ 2611.175080] skb frag:     00000010: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.175774] skb frag:     00000020: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.176474] skb frag:     00000030: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.177166] skb frag:     00000040: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.177859] skb frag:     00000050: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.178570] skb frag:     00000060: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.179269] skb frag:     00000070: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.179962] skb frag:     00000080: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.180673] skb frag:     00000090: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.181368] skb frag:     000000a0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.182072] skb frag:     000000b0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.182767] skb frag:     000000c0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.183495] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.183990] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.186277] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 39 30 31
[ 2611.186976] skb frag:     00000010: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.187667] skb frag:     00000020: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.188360] skb frag:     00000030: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.189065] skb frag:     00000040: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.189762] skb frag:     00000050: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.190461] skb frag:     00000060: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.198069] skb frag:     00000070: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.198767] skb frag:     00000080: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.199455] skb frag:     00000090: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.200143] skb frag:     000000a0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.200844] skb frag:     000000b0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.201533] skb frag:     000000c0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.202269] sk: 00000000d4b8ce89, tls_decrypt_sw, err = -74
[ 2611.202769] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=3)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.205034] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2611.205721] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.206429] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.207114] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.207806] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.208520] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.209205] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.209897] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.210589] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.211278] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.211965] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.212659] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.213345] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.214064] sk: 00000000d4b8ce89, tls_decrypt_sw, err = -74
[ 2611.214560] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=13))
               csum(0x100062 ip_summed=0 complete_sw=1 valid=1 level=3)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.216838] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 31 32 33
[ 2611.217531] skb frag:     00000010: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.218231] skb frag:     00000020: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.218922] skb frag:     00000030: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.219614] skb frag:     00000040: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.220308] skb frag:     00000050: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.221002] skb frag:     00000060: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.221696] skb frag:     00000070: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.222395] skb frag:     00000080: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.223095] skb frag:     00000090: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.223787] skb frag:     000000a0: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.224480] skb frag:     000000b0: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.225175] skb frag:     000000c0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.254987] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.256469] skb len=45 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=1 gso(size=0 type=0 segs=0))
               csum(0xa3f7 ip_summed=0 complete_sw=0 valid=1 level=0)
               hash(0xea879801 sw=0 l4=1) proto=0x0800 pkttype=0 iif=11
[ 2611.262626] skb frag:     00000000: 2f 35 b7 cd 14 0f 2d 5f c5 95 de 5f 01 14 00 00
[ 2611.264542] skb frag:     00000010: 0c ea a1 06 ce 06 04 b4 c5 e5 3e 3e 09 00 00 00
[ 2611.265342] skb frag:     00000020: 72 78 39 5f 63 73 75 6d 5f 75 6e 6e 65
[ 2611.310108] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.310776] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=5))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.313073] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.313561] skb len=16898 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=0 gso(size=1448 type=1 segs=10))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.315843] sk: 00000000d4b8ce89, tls_decrypt_sw, err = 0
[ 2611.316333] skb len=16413 headroom=0 headlen=0 tailroom=0
               mac=(64,14) net=(78,20) trans=98
               shinfo(txflags=0 nr_frags=2 gso(size=1448 type=1 segs=14))
               csum(0x100062 ip_summed=3 complete_sw=0 valid=1 level=0)
               hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=11
[ 2611.318605] skb frag:     00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 33 34 35
[ 2611.319295] skb frag:     00000010: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.319981] skb frag:     00000020: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.320671] skb frag:     00000030: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.321363] skb frag:     00000040: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.322059] skb frag:     00000050: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.322748] skb frag:     00000060: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.323432] skb frag:     00000070: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37
[ 2611.324124] skb frag:     00000080: 38 39 30 31 32 33 34 35 36 37 38 39 30 31 32 33
[ 2611.324827] skb frag:     00000090: 34 35 36 37 38 39 30 31 32 33 34 35 36 37 38 39
[ 2611.325527] skb frag:     000000a0: 30 31 32 33 34 35 36 37 38 39 30 31 32 33 34 35
[ 2611.326230] skb frag:     000000b0: 36 37 38 39 30 31 32 33 34 35 36 37 38 39 30 31
[ 2611.326923] skb frag:     000000c0: 32 33 34 35 36 37 38 39 30 31 32 33 34 35 36 37

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

On Sun, 12 Mar 2023 19:59:57 +0200 Tariq Toukan wrote:
> On 09/03/2023 19:54, Jakub Kicinski wrote:
> > On Thu, 9 Mar 2023 17:15:26 +0200 Tariq Toukan wrote:  
> >> A few fixes were introduced for this patch, but it seems to still cause
> >> issues.
> >>
> >> I'm running simple client/server test with wrk and nginx and TLS RX
> >> device offload on.
> >> It fails with TlsDecryptError on the client side for the large file
> >> (256000b), while succeeding for the small one (10000b). See repro
> >> details below.
> >>
> >> I narrowed the issue down to this offending patch, by applying a few
> >> reverts (had to solve trivial conflicts):  
> > 
> > What's the sequence of records in terms of being offloaded vs fall back?
> > Could you whip up a simple ring buffer to see if previous records were
> > offloaded and what the skb geometries where?  
> 
> Interesting. All records go through the sw fallback.
> 
> Command:
> $ wrk_openssl_3_0_0 -b2.2.2.2 -t1 -c1 -d2 --timeout 5s 
> https://2.2.2.3:20443/256000b.img

Is wrk_openssl_3_0_0 a nginx command? Any CX6 DX card can do this?

> Debug code:
> @@ -1712,8 +1723,13 @@ static int tls_rx_one_record(struct sock *sk, 
> struct msghdr *msg,
>          int err;
> 
>          err = tls_decrypt_device(sk, msg, tls_ctx, darg);
> -       if (!err)
> +       if (!err) {
>                  err = tls_decrypt_sw(sk, tls_ctx, msg, darg);
> +               printk("sk: %p, tls_decrypt_sw, err = %d\n", sk, err);
> +       } else {
> +               printk("sk: %p, tls_decrypt_device, err = %d\n", sk, err);
> +       }
> +       skb_dump(KERN_ERR, darg->skb, false);
>          if (err < 0)
>                  return err;
> 
> dmesg output including skb geometries is attached.

Hm, could you add to the debug the addresses of the fragments 
(and decrypted status) of the data queued to TCP by the driver?
And then the frag addresses in skb_dump() ?

tls_decrypt_sw() will also get used in partially decrypted records, 
right?

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Tariq Toukan]

[-- Attachment #1: Type: text/plain, Size: 4812 bytes --]



On 13/03/2023 20:22, Jakub Kicinski wrote:
> On Sun, 12 Mar 2023 19:59:57 +0200 Tariq Toukan wrote:
>> On 09/03/2023 19:54, Jakub Kicinski wrote:
>>> On Thu, 9 Mar 2023 17:15:26 +0200 Tariq Toukan wrote:
>>>> A few fixes were introduced for this patch, but it seems to still cause
>>>> issues.
>>>>
>>>> I'm running simple client/server test with wrk and nginx and TLS RX
>>>> device offload on.
>>>> It fails with TlsDecryptError on the client side for the large file
>>>> (256000b), while succeeding for the small one (10000b). See repro
>>>> details below.
>>>>
>>>> I narrowed the issue down to this offending patch, by applying a few
>>>> reverts (had to solve trivial conflicts):
>>>
>>> What's the sequence of records in terms of being offloaded vs fall back?
>>> Could you whip up a simple ring buffer to see if previous records were
>>> offloaded and what the skb geometries where?
>>
>> Interesting. All records go through the sw fallback.
>>
>> Command:
>> $ wrk_openssl_3_0_0 -b2.2.2.2 -t1 -c1 -d2 --timeout 5s
>> https://2.2.2.3:20443/256000b.img
> 
> Is wrk_openssl_3_0_0 a nginx command? Any CX6 DX card can do this?
> 

It's the standard wrk client, nothing special here.
We simply rename it to indicate that it was compiled against a specific 
openssl version.

>> Debug code:
>> @@ -1712,8 +1723,13 @@ static int tls_rx_one_record(struct sock *sk,
>> struct msghdr *msg,
>>           int err;
>>
>>           err = tls_decrypt_device(sk, msg, tls_ctx, darg);
>> -       if (!err)
>> +       if (!err) {
>>                   err = tls_decrypt_sw(sk, tls_ctx, msg, darg);
>> +               printk("sk: %p, tls_decrypt_sw, err = %d\n", sk, err);
>> +       } else {
>> +               printk("sk: %p, tls_decrypt_device, err = %d\n", sk, err);
>> +       }
>> +       skb_dump(KERN_ERR, darg->skb, false);
>>           if (err < 0)
>>                   return err;
>>
>> dmesg output including skb geometries is attached.
> 
> Hm, could you add to the debug the addresses of the fragments
> (and decrypted status) of the data queued to TCP by the driver?
> And then the frag addresses in skb_dump() ?
> 

I used the debug code below. Find dmesg output attached.
I disabled GRO to reduce complexity.

# cat /proc/net/tls_stat
TlsCurrTxSw                             0
TlsCurrRxSw                             0
TlsCurrTxDevice                         1
TlsCurrRxDevice                         1
TlsTxSw                                 0
TlsRxSw                                 0
TlsTxDevice                             2
TlsRxDevice                             2
TlsDecryptError                         2
TlsRxDeviceResync                       2
TlsDecryptRetry                         0
TlsRxNoPadViolation                     0

Insights:

In this repro, we always go through tls_decrypt_sw, never to 
tls_decrypt_device.

The corrupted SKB is the one with mixed offloaded and non-offloaded 
decryption. You can clearly see this by its len (2330624 bytes!), 
followed by a kernel panic.

Calling skb_dump on the corrupted skb also hits the kernel panic.

Code:

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c 
b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 3f7b63d6616b..864288b5d412 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -2262,6 +2262,7 @@ static void mlx5e_handle_rx_cqe_mpwrq(struct 
mlx5e_rq *rq, struct mlx5_cqe64 *cq
                         goto mpwrq_cqe_out;
                 }

+       printk("driver calling napi_gro_receive for skb %p, is decrypt = 
%d\n", skb, skb->decrypted);
         napi_gro_receive(rq->cq.napi, skb);

  mpwrq_cqe_out:
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 782d3701b86f..3858299196d3 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -1710,10 +1710,19 @@ static int tls_rx_one_record(struct sock *sk, 
struct msghdr *msg,
         struct tls_prot_info *prot = &tls_ctx->prot_info;
         struct strp_msg *rxm;
         int err;
+       int i;

         err = tls_decrypt_device(sk, msg, tls_ctx, darg);
-       if (!err)
+       if (!err) {
                 err = tls_decrypt_sw(sk, tls_ctx, msg, darg);
+               printk("sk: %p, tls_decrypt_sw, err = %d\n", sk, err);
+       } else {
+               printk("sk: %p, tls_decrypt_device, err = %d\n", sk, err);
+       }
+       printk("skb: %p, len = %d, nfrags = %d\n", darg->skb, 
darg->skb->len, skb_shinfo(darg->skb)->nr_frags);
+       for (i = 0; i < skb_shinfo(darg->skb)->nr_frags; i++)
+               printk("frag#%d: %p, len %d\n", i, 
&skb_shinfo(darg->skb)->frags[i],
+ 
skb_frag_size(&skb_shinfo(darg->skb)->frags[i]));
         if (err < 0)
                 return err;

> tls_decrypt_sw() will also get used in partially decrypted records,
> right?

Right.

[-- Attachment #2: dmesg_tls_rx_decrypt_err_2.txt --]
[-- Type: text/plain, Size: 41837 bytes --]

[  476.401920] driver calling napi_gro_receive for skb 00000000172dacb5, is decrypt = 0
[  476.404309] driver calling napi_gro_receive for skb 00000000a2a5da4b, is decrypt = 0
[  476.406468] driver calling napi_gro_receive for skb 00000000a488e380, is decrypt = 0
[  476.409170] driver calling napi_gro_receive for skb 0000000010f82300, is decrypt = 0
[  476.416379] driver calling napi_gro_receive for skb 0000000010f82300, is decrypt = 0
[  476.417646] driver calling napi_gro_receive for skb 00000000a488e380, is decrypt = 0
[  476.418720] driver calling napi_gro_receive for skb 000000003772defa, is decrypt = 0
[  476.421004] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.421974] skb: 000000000c841177, len = 45, nfrags = 1
[  476.422865] frag#0: 00000000a5edf33c, len 45
[  476.426485] driver calling napi_gro_receive for skb 00000000a488e380, is decrypt = 0
[  476.427679] driver calling napi_gro_receive for skb 00000000a7267d4f, is decrypt = 0
[  476.428905] driver calling napi_gro_receive for skb 0000000077f97780, is decrypt = 0
[  476.430083] driver calling napi_gro_receive for skb 00000000059b4377, is decrypt = 0
[  476.431305] driver calling napi_gro_receive for skb 000000002be0b2b9, is decrypt = 0
[  476.432499] driver calling napi_gro_receive for skb 00000000675232c6, is decrypt = 0
[  476.433694] driver calling napi_gro_receive for skb 000000008574bfc0, is decrypt = 0
[  476.434923] driver calling napi_gro_receive for skb 000000003e9ad851, is decrypt = 0
[  476.436198] driver calling napi_gro_receive for skb 00000000c0674ee0, is decrypt = 0
[  476.437259] driver calling napi_gro_receive for skb 000000006e139894, is decrypt = 0
[  476.442763] driver calling napi_gro_receive for skb 000000001951ef02, is decrypt = 0
[  476.479911] driver calling napi_gro_receive for skb 00000000c8099318, is decrypt = 0
[  476.480644] driver calling napi_gro_receive for skb 00000000734f25bb, is decrypt = 0
[  476.481360] driver calling napi_gro_receive for skb 0000000028bbd832, is decrypt = 0
[  476.482124] driver calling napi_gro_receive for skb 00000000836b993d, is decrypt = 0
[  476.482846] driver calling napi_gro_receive for skb 000000002ccf5af5, is decrypt = 0
[  476.483566] driver calling napi_gro_receive for skb 000000009e72ffb7, is decrypt = 0
[  476.484349] driver calling napi_gro_receive for skb 0000000073698823, is decrypt = 0
[  476.485140] driver calling napi_gro_receive for skb 0000000056cd838c, is decrypt = 0
[  476.485936] driver calling napi_gro_receive for skb 0000000095ebfd81, is decrypt = 0
[  476.486006] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.486379] driver calling napi_gro_receive for skb 000000005ab97059, is decrypt = 0
[  476.487538] skb: 00000000cd6f327f, len = 16413, nfrags = 0
[  476.487984] driver calling napi_gro_receive for skb 000000006ce49e36, is decrypt = 0
[  476.487987] driver calling napi_gro_receive for skb 00000000d010301f, is decrypt = 0
[  476.487989] driver calling napi_gro_receive for skb 000000009bde62fb, is decrypt = 0
[  476.487992] driver calling napi_gro_receive for skb 0000000098e3b44b, is decrypt = 0
[  476.487994] driver calling napi_gro_receive for skb 0000000090a6546e, is decrypt = 0
[  476.487997] driver calling napi_gro_receive for skb 00000000baf7e4a4, is decrypt = 0
[  476.488011] driver calling napi_gro_receive for skb 000000001f86ffbf, is decrypt = 0
[  476.489305] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.489638] driver calling napi_gro_receive for skb 0000000047eebe9e, is decrypt = 0
[  476.491256] skb: 00000000cd6f327f, len = 32826, nfrags = 0
[  476.491701] driver calling napi_gro_receive for skb 000000007222c0fb, is decrypt = 0
[  476.499849] driver calling napi_gro_receive for skb 00000000ae17e442, is decrypt = 0
[  476.507486] driver calling napi_gro_receive for skb 00000000ef6f5f3b, is decrypt = 0
[  476.508209] driver calling napi_gro_receive for skb 00000000fd53ae4a, is decrypt = 0
[  476.508946] driver calling napi_gro_receive for skb 000000004b03b630, is decrypt = 0
[  476.509673] driver calling napi_gro_receive for skb 000000002ccf5af5, is decrypt = 0
[  476.510407] driver calling napi_gro_receive for skb 000000009e72ffb7, is decrypt = 0
[  476.510430] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.510849] driver calling napi_gro_receive for skb 0000000073698823, is decrypt = 0
[  476.511156] skb: 00000000cd6f327f, len = 23175, nfrags = 0
[  476.511600] driver calling napi_gro_receive for skb 0000000090a6546e, is decrypt = 0
[  476.513533] driver calling napi_gro_receive for skb 00000000baf7e4a4, is decrypt = 0
[  476.514246] driver calling napi_gro_receive for skb 0000000095ebfd81, is decrypt = 0
[  476.515077] driver calling napi_gro_receive for skb 000000007778a9f6, is decrypt = 0
[  476.515913] driver calling napi_gro_receive for skb 0000000068bd6dc4, is decrypt = 0
[  476.516697] driver calling napi_gro_receive for skb 00000000abee4c02, is decrypt = 0
[  476.517600] driver calling napi_gro_receive for skb 00000000be2a3447, is decrypt = 0
[  476.518313] driver calling napi_gro_receive for skb 00000000f13be044, is decrypt = 0
[  476.519026] driver calling napi_gro_receive for skb 0000000096ed4dd8, is decrypt = 0
[  476.519759] driver calling napi_gro_receive for skb 000000001285fc6b, is decrypt = 0
[  476.520597] driver calling napi_gro_receive for skb 000000002ecd82af, is decrypt = 0
[  476.521337] driver calling napi_gro_receive for skb 000000009d88e7f7, is decrypt = 0
[  476.522111] driver calling napi_gro_receive for skb 00000000a5391a61, is decrypt = 0
[  476.522933] driver calling napi_gro_receive for skb 000000002f93a7cf, is decrypt = 0
[  476.523703] driver calling napi_gro_receive for skb 00000000a19450ed, is decrypt = 0
[  476.523724] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.524147] driver calling napi_gro_receive for skb 0000000091e1c36b, is decrypt = 0
[  476.524457] skb: 00000000cd6f327f, len = 39588, nfrags = 0
[  476.524939] driver calling napi_gro_receive for skb 00000000fa963b74, is decrypt = 0
[  476.526884] driver calling napi_gro_receive for skb 00000000044342a6, is decrypt = 0
[  476.527613] driver calling napi_gro_receive for skb 000000000c50032e, is decrypt = 0
[  476.528336] driver calling napi_gro_receive for skb 00000000e94ed795, is decrypt = 0
[  476.529123] driver calling napi_gro_receive for skb 00000000059b4377, is decrypt = 0
[  476.529846] driver calling napi_gro_receive for skb 000000002be0b2b9, is decrypt = 0
[  476.530587] driver calling napi_gro_receive for skb 00000000675232c6, is decrypt = 0
[  476.530610] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.531031] driver calling napi_gro_receive for skb 000000008574bfc0, is decrypt = 0
[  476.531335] skb: 00000000cd6f327f, len = 29937, nfrags = 0
[  476.531895] driver calling napi_gro_receive for skb 000000003e9ad851, is decrypt = 0
[  476.531898] driver calling napi_gro_receive for skb 00000000c0674ee0, is decrypt = 0
[  476.534563] driver calling napi_gro_receive for skb 000000006e139894, is decrypt = 0
[  476.535305] driver calling napi_gro_receive for skb 000000001951ef02, is decrypt = 0
[  476.536027] driver calling napi_gro_receive for skb 0000000090a6546e, is decrypt = 0
[  476.536753] driver calling napi_gro_receive for skb 00000000abee4c02, is decrypt = 0
[  476.537482] driver calling napi_gro_receive for skb 000000007778a9f6, is decrypt = 0
[  476.538195] driver calling napi_gro_receive for skb 00000000be2a3447, is decrypt = 0
[  476.538990] driver calling napi_gro_receive for skb 0000000096ed4dd8, is decrypt = 0
[  476.539715] driver calling napi_gro_receive for skb 000000001285fc6b, is decrypt = 0
[  476.540469] driver calling napi_gro_receive for skb 00000000f13be044, is decrypt = 0
[  476.541215] driver calling napi_gro_receive for skb 000000002ecd82af, is decrypt = 0
[  476.541930] driver calling napi_gro_receive for skb 000000009d88e7f7, is decrypt = 0
[  476.542663] driver calling napi_gro_receive for skb 00000000a5391a61, is decrypt = 0
[  476.543437] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.543440] driver calling napi_gro_receive for skb 00000000c8099318, is decrypt = 0
[  476.543749] skb: 00000000cd6f327f, len = 20286, nfrags = 0
[  476.544193] driver calling napi_gro_receive for skb 000000007c40f21b, is decrypt = 0
[  476.544196] driver calling napi_gro_receive for skb 00000000a488e380, is decrypt = 0
[  476.546559] driver calling napi_gro_receive for skb 0000000056cd838c, is decrypt = 0
[  476.547276] driver calling napi_gro_receive for skb 0000000073698823, is decrypt = 0
[  476.548079] driver calling napi_gro_receive for skb 000000002ccf5af5, is decrypt = 0
[  476.548800] driver calling napi_gro_receive for skb 0000000095ebfd81, is decrypt = 0
[  476.549517] driver calling napi_gro_receive for skb 000000000c50032e, is decrypt = 0
[  476.550314] driver calling napi_gro_receive for skb 00000000fa963b74, is decrypt = 0
[  476.550335] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.550757] driver calling napi_gro_receive for skb 00000000e94ed795, is decrypt = 0
[  476.551063] skb: 00000000cd6f327f, len = 36699, nfrags = 0
[  476.551515] driver calling napi_gro_receive for skb 00000000059b4377, is decrypt = 0
[  476.553441] driver calling napi_gro_receive for skb 000000002be0b2b9, is decrypt = 0
[  476.554159] driver calling napi_gro_receive for skb 00000000abee4c02, is decrypt = 0
[  476.554867] driver calling napi_gro_receive for skb 000000007778a9f6, is decrypt = 0
[  476.555699] driver calling napi_gro_receive for skb 00000000be2a3447, is decrypt = 0
[  476.556445] driver calling napi_gro_receive for skb 0000000096ed4dd8, is decrypt = 0
[  476.557216] driver calling napi_gro_receive for skb 00000000f13be044, is decrypt = 0
[  476.557243] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.557658] driver calling napi_gro_receive for skb 000000002ecd82af, is decrypt = 0
[  476.557962] skb: 00000000cd6f327f, len = 29944, nfrags = 0
[  476.558403] driver calling napi_gro_receive for skb 000000009d88e7f7, is decrypt = 0
[  476.560348] driver calling napi_gro_receive for skb 00000000a5391a61, is decrypt = 0
[  476.561078] driver calling napi_gro_receive for skb 00000000c8099318, is decrypt = 0
[  476.561787] driver calling napi_gro_receive for skb 000000007c40f21b, is decrypt = 0
[  476.562568] driver calling napi_gro_receive for skb 00000000a488e380, is decrypt = 0
[  476.563303] driver calling napi_gro_receive for skb 0000000056cd838c, is decrypt = 0
[  476.564038] driver calling napi_gro_receive for skb 0000000073698823, is decrypt = 0
[  476.564749] driver calling napi_gro_receive for skb 000000002ccf5af5, is decrypt = 0
[  476.565464] driver calling napi_gro_receive for skb 0000000095ebfd81, is decrypt = 0
[  476.566190] driver calling napi_gro_receive for skb 000000000c50032e, is decrypt = 0
[  476.566908] driver calling napi_gro_receive for skb 000000007c40f21b, is decrypt = 0
[  476.567630] driver calling napi_gro_receive for skb 00000000a488e380, is decrypt = 0
[  476.568349] driver calling napi_gro_receive for skb 00000000a5391a61, is decrypt = 0
[  476.569117] driver calling napi_gro_receive for skb 00000000c8099318, is decrypt = 0
[  476.570012] driver calling napi_gro_receive for skb 00000000d2461240, is decrypt = 0
[  476.570026] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.570457] driver calling napi_gro_receive for skb 00000000684810c4, is decrypt = 0
[  476.570764] skb: 00000000cd6f327f, len = 20293, nfrags = 0
[  476.571213] driver calling napi_gro_receive for skb 00000000c2b6cd74, is decrypt = 0
[  476.573203] driver calling napi_gro_receive for skb 0000000055af9806, is decrypt = 0
[  476.573988] driver calling napi_gro_receive for skb 00000000157ff1db, is decrypt = 0
[  476.574767] driver calling napi_gro_receive for skb 00000000ca656a3b, is decrypt = 0
[  476.575521] driver calling napi_gro_receive for skb 0000000023f6592e, is decrypt = 0
[  476.583185] driver calling napi_gro_receive for skb 00000000b0f00ff7, is decrypt = 0
[  476.583926] driver calling napi_gro_receive for skb 00000000c3bdc5bc, is decrypt = 0
[  476.583942] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.584365] driver calling napi_gro_receive for skb 000000000ddf859f, is decrypt = 0
[  476.584678] skb: 00000000cd6f327f, len = 16434, nfrags = 0
[  476.585117] driver calling napi_gro_receive for skb 00000000afb015d0, is decrypt = 0
[  476.587084] driver calling napi_gro_receive for skb 00000000451fb132, is decrypt = 0
[  476.587824] driver calling napi_gro_receive for skb 0000000095ebfd81, is decrypt = 0
[  476.588536] driver calling napi_gro_receive for skb 00000000abee4c02, is decrypt = 0
[  476.589254] driver calling napi_gro_receive for skb 00000000be2a3447, is decrypt = 0
[  476.590020] driver calling napi_gro_receive for skb 0000000096ed4dd8, is decrypt = 0
[  476.590787] driver calling napi_gro_receive for skb 00000000f13be044, is decrypt = 0
[  476.591496] driver calling napi_gro_receive for skb 00000000d2461240, is decrypt = 0
[  476.592214] driver calling napi_gro_receive for skb 00000000684810c4, is decrypt = 0
[  476.592940] driver calling napi_gro_receive for skb 00000000c2b6cd74, is decrypt = 0
[  476.593666] driver calling napi_gro_receive for skb 00000000ca656a3b, is decrypt = 0
[  476.594585] driver calling napi_gro_receive for skb 0000000023f6592e, is decrypt = 0
[  476.595298] driver calling napi_gro_receive for skb 00000000b0f00ff7, is decrypt = 0
[  476.596081] driver calling napi_gro_receive for skb 0000000055af9806, is decrypt = 0
[  476.596823] driver calling napi_gro_receive for skb 00000000157ff1db, is decrypt = 0
[  476.596832] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.597271] driver calling napi_gro_receive for skb 000000002ecd82af, is decrypt = 0
[  476.597579] skb: 00000000cd6f327f, len = 32847, nfrags = 0
[  476.598022] driver calling napi_gro_receive for skb 000000007c40f21b, is decrypt = 0
[  476.599966] driver calling napi_gro_receive for skb 00000000a488e380, is decrypt = 0
[  476.600745] driver calling napi_gro_receive for skb 00000000ca656a3b, is decrypt = 0
[  476.601522] driver calling napi_gro_receive for skb 0000000023f6592e, is decrypt = 0
[  476.602278] driver calling napi_gro_receive for skb 00000000b0f00ff7, is decrypt = 0
[  476.603007] driver calling napi_gro_receive for skb 0000000055af9806, is decrypt = 0
[  476.603766] driver calling napi_gro_receive for skb 0000000012da5d4f, is decrypt = 0
[  476.603769] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.604209] driver calling napi_gro_receive for skb 00000000dbd83ebf, is decrypt = 0
[  476.604520] skb: 00000000cd6f327f, len = 23196, nfrags = 0
[  476.604959] driver calling napi_gro_receive for skb 00000000187e61fa, is decrypt = 0
[  476.606877] driver calling napi_gro_receive for skb 000000003b677506, is decrypt = 0
[  476.607584] driver calling napi_gro_receive for skb 00000000044342a6, is decrypt = 0
[  476.608287] driver calling napi_gro_receive for skb 000000001285fc6b, is decrypt = 0
[  476.608999] driver calling napi_gro_receive for skb 000000007778a9f6, is decrypt = 0
[  476.609697] driver calling napi_gro_receive for skb 000000000c50032e, is decrypt = 0
[  476.610413] driver calling napi_gro_receive for skb 00000000684810c4, is decrypt = 0
[  476.611125] driver calling napi_gro_receive for skb 00000000be2a3447, is decrypt = 0
[  476.611832] driver calling napi_gro_receive for skb 0000000096ed4dd8, is decrypt = 0
[  476.612545] driver calling napi_gro_receive for skb 00000000f13be044, is decrypt = 0
[  476.613244] driver calling napi_gro_receive for skb 00000000fa963b74, is decrypt = 0
[  476.613951] driver calling napi_gro_receive for skb 000000002be0b2b9, is decrypt = 0
[  476.614651] driver calling napi_gro_receive for skb 00000000e94ed795, is decrypt = 0
[  476.615355] driver calling napi_gro_receive for skb 00000000059b4377, is decrypt = 0
[  476.616071] driver calling napi_gro_receive for skb 0000000056cd838c, is decrypt = 0
[  476.616078] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.616507] driver calling napi_gro_receive for skb 0000000073698823, is decrypt = 0
[  476.616820] skb: 00000000cd6f327f, len = 16441, nfrags = 0
[  476.617256] driver calling napi_gro_receive for skb 000000002ccf5af5, is decrypt = 0
[  476.619173] driver calling napi_gro_receive for skb 0000000095ebfd81, is decrypt = 0
[  476.619879] driver calling napi_gro_receive for skb 000000001285fc6b, is decrypt = 0
[  476.620591] driver calling napi_gro_receive for skb 0000000096ed4dd8, is decrypt = 1
[  476.621293] driver calling napi_gro_receive for skb 00000000f13be044, is decrypt = 1
[  476.622001] driver calling napi_gro_receive for skb 000000000c50032e, is decrypt = 1
[  476.622712] driver calling napi_gro_receive for skb 00000000044342a6, is decrypt = 1
[  476.622720] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.623147] driver calling napi_gro_receive for skb 00000000fa963b74, is decrypt = 1
[  476.623459] skb: 00000000cd6f327f, len = 32854, nfrags = 0
[  476.623903] driver calling napi_gro_receive for skb 000000002be0b2b9, is decrypt = 1
[  476.625827] driver calling napi_gro_receive for skb 00000000e94ed795, is decrypt = 1
[  476.626532] driver calling napi_gro_receive for skb 00000000059b4377, is decrypt = 1
[  476.627232] driver calling napi_gro_receive for skb 00000000ca656a3b, is decrypt = 1
[  476.627936] driver calling napi_gro_receive for skb 00000000c3bdc5bc, is decrypt = 1
[  476.628648] driver calling napi_gro_receive for skb 000000000ddf859f, is decrypt = 1
[  476.629351] driver calling napi_gro_receive for skb 00000000afb015d0, is decrypt = 1
[  476.630058] driver calling napi_gro_receive for skb 00000000451fb132, is decrypt = 1
[  476.630760] driver calling napi_gro_receive for skb 00000000d2461240, is decrypt = 1
[  476.631461] driver calling napi_gro_receive for skb 00000000684810c4, is decrypt = 1
[  476.632174] driver calling napi_gro_receive for skb 0000000056cd838c, is decrypt = 1
[  476.632887] driver calling napi_gro_receive for skb 0000000073698823, is decrypt = 1
[  476.633588] driver calling napi_gro_receive for skb 000000002ccf5af5, is decrypt = 1
[  476.634294] driver calling napi_gro_receive for skb 0000000095ebfd81, is decrypt = 1
[  476.635015] sk: 000000001f13dce9, tls_decrypt_sw, err = -74
[  476.635537] skb: 00000000332e86ab, len = 2330624, nfrags = 0
[  476.660494] driver calling napi_gro_receive for skb 00000000699d87d6, is decrypt = 0
[  476.663310] driver calling napi_gro_receive for skb 00000000ba8ba02d, is decrypt = 0
[  476.666011] driver calling napi_gro_receive for skb 000000008049abe9, is decrypt = 0
[  476.670328] driver calling napi_gro_receive for skb 000000008049abe9, is decrypt = 0
[  476.671112] driver calling napi_gro_receive for skb 00000000ba8ba02d, is decrypt = 0
[  476.671904] driver calling napi_gro_receive for skb 0000000032bd7692, is decrypt = 0
[  476.673678] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.675110] skb: 00000000584135ab, len = 45, nfrags = 1
[  476.676512] frag#0: 00000000593a5a7c, len 45
[  476.678040] driver calling napi_gro_receive for skb 00000000ba8ba02d, is decrypt = 0
[  476.679214] driver calling napi_gro_receive for skb 00000000c5bc3d6c, is decrypt = 0
[  476.680446] driver calling napi_gro_receive for skb 00000000e02cd0e2, is decrypt = 0
[  476.681660] driver calling napi_gro_receive for skb 0000000075887a2f, is decrypt = 0
[  476.682851] driver calling napi_gro_receive for skb 000000007bf4c6c4, is decrypt = 0
[  476.684047] driver calling napi_gro_receive for skb 00000000384b4a6e, is decrypt = 0
[  476.685242] driver calling napi_gro_receive for skb 000000001825ba75, is decrypt = 0
[  476.686429] driver calling napi_gro_receive for skb 0000000033864df9, is decrypt = 0
[  476.687638] driver calling napi_gro_receive for skb 000000002b404bb4, is decrypt = 0
[  476.688842] driver calling napi_gro_receive for skb 000000000a8d9aba, is decrypt = 0
[  476.694760] driver calling napi_gro_receive for skb 0000000033233bcc, is decrypt = 0
[  476.727942] driver calling napi_gro_receive for skb 0000000094822866, is decrypt = 0
[  476.730468] driver calling napi_gro_receive for skb 00000000f3aef5ea, is decrypt = 0
[  476.732978] driver calling napi_gro_receive for skb 000000004172ec3c, is decrypt = 0
[  476.735088] driver calling napi_gro_receive for skb 0000000075534fcd, is decrypt = 0
[  476.737080] driver calling napi_gro_receive for skb 0000000028e91c60, is decrypt = 0
[  476.737802] driver calling napi_gro_receive for skb 00000000507304a0, is decrypt = 0
[  476.738526] driver calling napi_gro_receive for skb 00000000925c5b10, is decrypt = 0
[  476.739244] driver calling napi_gro_receive for skb 00000000003d1c40, is decrypt = 0
[  476.740007] driver calling napi_gro_receive for skb 00000000569b18b8, is decrypt = 0
[  476.740056] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.740447] driver calling napi_gro_receive for skb 00000000d44dbed8, is decrypt = 0
[  476.741589] skb: 0000000063c0ec85, len = 16413, nfrags = 0
[  476.742035] driver calling napi_gro_receive for skb 0000000072650410, is decrypt = 0
[  476.744706] driver calling napi_gro_receive for skb 000000000ce80127, is decrypt = 0
[  476.745421] driver calling napi_gro_receive for skb 00000000b322a911, is decrypt = 0
[  476.746144] driver calling napi_gro_receive for skb 0000000037002252, is decrypt = 0
[  476.746852] driver calling napi_gro_receive for skb 0000000055e2448e, is decrypt = 0
[  476.747573] driver calling napi_gro_receive for skb 00000000a092f228, is decrypt = 0
[  476.748316] driver calling napi_gro_receive for skb 000000006d079c0d, is decrypt = 0
[  476.748400] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.748773] driver calling napi_gro_receive for skb 0000000089d6435f, is decrypt = 0
[  476.749809] skb: 0000000063c0ec85, len = 32826, nfrags = 0
[  476.750250] driver calling napi_gro_receive for skb 0000000064339cef, is decrypt = 0
[  476.752928] driver calling napi_gro_receive for skb 00000000192574b8, is decrypt = 0
[  476.753665] driver calling napi_gro_receive for skb 00000000062ae971, is decrypt = 0
[  476.754385] driver calling napi_gro_receive for skb 00000000cb945c61, is decrypt = 0
[  476.755100] driver calling napi_gro_receive for skb 000000005561e897, is decrypt = 0
[  476.755830] driver calling napi_gro_receive for skb 000000008fc16595, is decrypt = 0
[  476.756569] driver calling napi_gro_receive for skb 000000004b488174, is decrypt = 0
[  476.756592] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.757022] driver calling napi_gro_receive for skb 00000000829f1f28, is decrypt = 0
[  476.757326] skb: 0000000063c0ec85, len = 23175, nfrags = 0
[  476.757771] driver calling napi_gro_receive for skb 00000000253355ff, is decrypt = 0
[  476.759681] driver calling napi_gro_receive for skb 00000000cded4fcf, is decrypt = 0
[  476.760417] driver calling napi_gro_receive for skb 000000002daab721, is decrypt = 0
[  476.761147] driver calling napi_gro_receive for skb 00000000ab9145f6, is decrypt = 0
[  476.761874] driver calling napi_gro_receive for skb 000000009c673c53, is decrypt = 0
[  476.762596] driver calling napi_gro_receive for skb 00000000b322a911, is decrypt = 0
[  476.763336] driver calling napi_gro_receive for skb 000000000ce80127, is decrypt = 0
[  476.764062] driver calling napi_gro_receive for skb 000000002462a0ef, is decrypt = 0
[  476.764802] driver calling napi_gro_receive for skb 0000000047b14a1a, is decrypt = 0
[  476.765527] driver calling napi_gro_receive for skb 000000008a44954d, is decrypt = 0
[  476.766250] driver calling napi_gro_receive for skb 00000000a7267d4f, is decrypt = 0
[  476.766970] driver calling napi_gro_receive for skb 0000000077f97780, is decrypt = 0
[  476.767689] driver calling napi_gro_receive for skb 000000005ab97059, is decrypt = 0
[  476.768423] driver calling napi_gro_receive for skb 000000006ce49e36, is decrypt = 0
[  476.769174] driver calling napi_gro_receive for skb 00000000d010301f, is decrypt = 0
[  476.769183] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.769618] driver calling napi_gro_receive for skb 000000009bde62fb, is decrypt = 0
[  476.769927] skb: 0000000063c0ec85, len = 39588, nfrags = 0
[  476.770369] driver calling napi_gro_receive for skb 000000008fd055e0, is decrypt = 0
[  476.772307] driver calling napi_gro_receive for skb 000000006bdce441, is decrypt = 0
[  476.773057] driver calling napi_gro_receive for skb 0000000044afc35e, is decrypt = 0
[  476.773787] driver calling napi_gro_receive for skb 00000000986d5c90, is decrypt = 0
[  476.774508] driver calling napi_gro_receive for skb 000000004a4abfc0, is decrypt = 0
[  476.775237] driver calling napi_gro_receive for skb 00000000b322a911, is decrypt = 0
[  476.775986] driver calling napi_gro_receive for skb 000000000ce80127, is decrypt = 0
[  476.776018] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.776429] driver calling napi_gro_receive for skb 000000002462a0ef, is decrypt = 0
[  476.776751] skb: 0000000063c0ec85, len = 29937, nfrags = 0
[  476.777199] driver calling napi_gro_receive for skb 0000000047b14a1a, is decrypt = 0
[  476.779122] driver calling napi_gro_receive for skb 00000000a7267d4f, is decrypt = 0
[  476.779875] driver calling napi_gro_receive for skb 0000000077f97780, is decrypt = 0
[  476.780623] driver calling napi_gro_receive for skb 000000005ab97059, is decrypt = 0
[  476.781353] driver calling napi_gro_receive for skb 000000006ce49e36, is decrypt = 0
[  476.782068] driver calling napi_gro_receive for skb 000000008a44954d, is decrypt = 0
[  476.782813] driver calling napi_gro_receive for skb 00000000ab9b12d1, is decrypt = 0
[  476.783531] driver calling napi_gro_receive for skb 0000000053ca597a, is decrypt = 0
[  476.784260] driver calling napi_gro_receive for skb 000000000cb4b9fd, is decrypt = 0
[  476.785005] driver calling napi_gro_receive for skb 00000000bd214bfd, is decrypt = 0
[  476.785725] driver calling napi_gro_receive for skb 00000000a00ec12e, is decrypt = 0
[  476.786455] driver calling napi_gro_receive for skb 000000006d079c0d, is decrypt = 0
[  476.787171] driver calling napi_gro_receive for skb 0000000089d6435f, is decrypt = 0
[  476.787904] driver calling napi_gro_receive for skb 000000003ccb2025, is decrypt = 0
[  476.788657] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.788672] driver calling napi_gro_receive for skb 0000000094822866, is decrypt = 0
[  476.788981] skb: 0000000063c0ec85, len = 20286, nfrags = 0
[  476.789421] driver calling napi_gro_receive for skb 0000000054cf4de7, is decrypt = 0
[  476.791084] driver calling napi_gro_receive for skb 00000000ba8ba02d, is decrypt = 0
[  476.791785] driver calling napi_gro_receive for skb 00000000003d1c40, is decrypt = 0
[  476.792561] driver calling napi_gro_receive for skb 00000000829f1f28, is decrypt = 0
[  476.793275] driver calling napi_gro_receive for skb 00000000bc8bcdf1, is decrypt = 0
[  476.793975] driver calling napi_gro_receive for skb 00000000fb4007fe, is decrypt = 0
[  476.794679] driver calling napi_gro_receive for skb 00000000c3d9ed74, is decrypt = 0
[  476.795404] driver calling napi_gro_receive for skb 000000007f98c29b, is decrypt = 0
[  476.795435] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.795858] driver calling napi_gro_receive for skb 00000000851ea100, is decrypt = 0
[  476.796172] skb: 0000000063c0ec85, len = 36699, nfrags = 0
[  476.796618] driver calling napi_gro_receive for skb 00000000cdf2ba92, is decrypt = 0
[  476.798535] driver calling napi_gro_receive for skb 0000000026e454cb, is decrypt = 0
[  476.799237] driver calling napi_gro_receive for skb 00000000796d8a38, is decrypt = 0
[  476.799994] driver calling napi_gro_receive for skb 0000000022c91ace, is decrypt = 0
[  476.800729] driver calling napi_gro_receive for skb 00000000943517ed, is decrypt = 0
[  476.801432] driver calling napi_gro_receive for skb 000000003116ab69, is decrypt = 0
[  476.802161] driver calling napi_gro_receive for skb 000000009db3b29b, is decrypt = 0
[  476.802182] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.802598] driver calling napi_gro_receive for skb 000000000565bae4, is decrypt = 0
[  476.802916] skb: 0000000063c0ec85, len = 31392, nfrags = 0
[  476.803349] driver calling napi_gro_receive for skb 0000000082bd8d50, is decrypt = 0
[  476.812305] driver calling napi_gro_receive for skb 00000000ebc5a357, is decrypt = 0
[  476.813028] driver calling napi_gro_receive for skb 00000000a00ec12e, is decrypt = 0
[  476.813726] driver calling napi_gro_receive for skb 000000006d079c0d, is decrypt = 0
[  476.814420] driver calling napi_gro_receive for skb 0000000089d6435f, is decrypt = 0
[  476.815119] driver calling napi_gro_receive for skb 00000000bc8bcdf1, is decrypt = 0
[  476.815846] driver calling napi_gro_receive for skb 0000000022c91ace, is decrypt = 0
[  476.816607] driver calling napi_gro_receive for skb 000000001486c3df, is decrypt = 0
[  476.817331] driver calling napi_gro_receive for skb 00000000d5b67e7d, is decrypt = 0
[  476.818033] driver calling napi_gro_receive for skb 00000000d2527282, is decrypt = 0
[  476.818728] driver calling napi_gro_receive for skb 000000000dfa85d4, is decrypt = 0
[  476.819429] driver calling napi_gro_receive for skb 0000000091e6478d, is decrypt = 0
[  476.820179] driver calling napi_gro_receive for skb 00000000292ff769, is decrypt = 0
[  476.820895] driver calling napi_gro_receive for skb 00000000556510b3, is decrypt = 0
[  476.821611] driver calling napi_gro_receive for skb 00000000fb4007fe, is decrypt = 0
[  476.821624] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.822044] driver calling napi_gro_receive for skb 0000000026e454cb, is decrypt = 0
[  476.822359] skb: 0000000063c0ec85, len = 21741, nfrags = 0
[  476.822794] driver calling napi_gro_receive for skb 00000000943517ed, is decrypt = 0
[  476.824793] driver calling napi_gro_receive for skb 000000003116ab69, is decrypt = 0
[  476.825492] driver calling napi_gro_receive for skb 000000000ce80127, is decrypt = 0
[  476.826194] driver calling napi_gro_receive for skb 000000009db3b29b, is decrypt = 0
[  476.826888] driver calling napi_gro_receive for skb 000000000565bae4, is decrypt = 0
[  476.827592] driver calling napi_gro_receive for skb 0000000082bd8d50, is decrypt = 0
[  476.828359] driver calling napi_gro_receive for skb 00000000ebc5a357, is decrypt = 0
[  476.828368] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.828818] driver calling napi_gro_receive for skb 00000000bc8bcdf1, is decrypt = 0
[  476.829127] skb: 0000000063c0ec85, len = 16434, nfrags = 0
[  476.829563] driver calling napi_gro_receive for skb 0000000022c91ace, is decrypt = 0
[  476.831473] driver calling napi_gro_receive for skb 000000001486c3df, is decrypt = 0
[  476.832231] driver calling napi_gro_receive for skb 00000000d5b67e7d, is decrypt = 0
[  476.832936] driver calling napi_gro_receive for skb 000000000dfa85d4, is decrypt = 0
[  476.833649] driver calling napi_gro_receive for skb 0000000091e6478d, is decrypt = 0
[  476.834356] driver calling napi_gro_receive for skb 00000000292ff769, is decrypt = 0
[  476.835065] driver calling napi_gro_receive for skb 00000000556510b3, is decrypt = 0
[  476.835769] driver calling napi_gro_receive for skb 00000000a00ec12e, is decrypt = 0
[  476.836519] driver calling napi_gro_receive for skb 000000006d079c0d, is decrypt = 0
[  476.837232] driver calling napi_gro_receive for skb 0000000089d6435f, is decrypt = 0
[  476.837935] driver calling napi_gro_receive for skb 00000000a7267d4f, is decrypt = 0
[  476.838634] driver calling napi_gro_receive for skb 0000000094822866, is decrypt = 0
[  476.839335] driver calling napi_gro_receive for skb 0000000054cf4de7, is decrypt = 0
[  476.840061] driver calling napi_gro_receive for skb 00000000ba8ba02d, is decrypt = 0
[  476.840829] driver calling napi_gro_receive for skb 000000000ce80127, is decrypt = 0
[  476.840835] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.841267] driver calling napi_gro_receive for skb 0000000082bd8d50, is decrypt = 0
[  476.841578] skb: 0000000063c0ec85, len = 32847, nfrags = 0
[  476.842013] driver calling napi_gro_receive for skb 00000000943517ed, is decrypt = 0
[  476.843981] driver calling napi_gro_receive for skb 000000009db3b29b, is decrypt = 0
[  476.844699] driver calling napi_gro_receive for skb 000000000565bae4, is decrypt = 0
[  476.845403] driver calling napi_gro_receive for skb 00000000ebc5a357, is decrypt = 0
[  476.846100] driver calling napi_gro_receive for skb 00000000bc8bcdf1, is decrypt = 0
[  476.846802] driver calling napi_gro_receive for skb 0000000022c91ace, is decrypt = 0
[  476.847521] driver calling napi_gro_receive for skb 0000000095afe878, is decrypt = 0
[  476.847528] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.847988] driver calling napi_gro_receive for skb 0000000042259aef, is decrypt = 0
[  476.848296] skb: 0000000063c0ec85, len = 23196, nfrags = 0
[  476.848748] driver calling napi_gro_receive for skb 000000005d49c6d6, is decrypt = 0
[  476.850669] driver calling napi_gro_receive for skb 000000005be7b58d, is decrypt = 0
[  476.851380] driver calling napi_gro_receive for skb 000000006bdce441, is decrypt = 0
[  476.852114] driver calling napi_gro_receive for skb 000000000cb4b9fd, is decrypt = 0
[  476.852829] driver calling napi_gro_receive for skb 00000000796d8a38, is decrypt = 0
[  476.853536] driver calling napi_gro_receive for skb 00000000d2527282, is decrypt = 0
[  476.854238] driver calling napi_gro_receive for skb 000000001486c3df, is decrypt = 0
[  476.854942] driver calling napi_gro_receive for skb 00000000d5b67e7d, is decrypt = 0
[  476.855642] driver calling napi_gro_receive for skb 0000000091e6478d, is decrypt = 0
[  476.856411] driver calling napi_gro_receive for skb 00000000292ff769, is decrypt = 0
[  476.857134] driver calling napi_gro_receive for skb 000000009bde62fb, is decrypt = 0
[  476.857843] driver calling napi_gro_receive for skb 0000000044afc35e, is decrypt = 0
[  476.858549] driver calling napi_gro_receive for skb 00000000986d5c90, is decrypt = 0
[  476.859261] driver calling napi_gro_receive for skb 000000004a4abfc0, is decrypt = 0
[  476.859992] driver calling napi_gro_receive for skb 000000008fd055e0, is decrypt = 0
[  476.859999] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.860475] driver calling napi_gro_receive for skb 000000002462a0ef, is decrypt = 0
[  476.860802] skb: 0000000063c0ec85, len = 16441, nfrags = 0
[  476.861251] driver calling napi_gro_receive for skb 0000000077f97780, is decrypt = 0
[  476.863177] driver calling napi_gro_receive for skb 000000005ab97059, is decrypt = 0
[  476.863915] driver calling napi_gro_receive for skb 000000006ce49e36, is decrypt = 0
[  476.864645] driver calling napi_gro_receive for skb 000000008a44954d, is decrypt = 1
[  476.865340] driver calling napi_gro_receive for skb 0000000047b14a1a, is decrypt = 1
[  476.866046] driver calling napi_gro_receive for skb 00000000a7267d4f, is decrypt = 1
[  476.866768] driver calling napi_gro_receive for skb 0000000054cf4de7, is decrypt = 1
[  476.866783] sk: 000000001f13dce9, tls_decrypt_sw, err = 0
[  476.867205] driver calling napi_gro_receive for skb 00000000ba8ba02d, is decrypt = 1
[  476.867522] skb: 0000000063c0ec85, len = 32854, nfrags = 0
[  476.867976] driver calling napi_gro_receive for skb 000000005d49c6d6, is decrypt = 1
[  476.869914] driver calling napi_gro_receive for skb 000000006bdce441, is decrypt = 1
[  476.870635] driver calling napi_gro_receive for skb 000000000cb4b9fd, is decrypt = 1
[  476.871333] driver calling napi_gro_receive for skb 000000000ce80127, is decrypt = 1
[  476.872100] driver calling napi_gro_receive for skb 0000000094822866, is decrypt = 1
[  476.872826] driver calling napi_gro_receive for skb 0000000095afe878, is decrypt = 1
[  476.873545] driver calling napi_gro_receive for skb 0000000042259aef, is decrypt = 1
[  476.874251] driver calling napi_gro_receive for skb 000000005be7b58d, is decrypt = 1
[  476.874948] driver calling napi_gro_receive for skb 0000000044afc35e, is decrypt = 1
[  476.875653] driver calling napi_gro_receive for skb 00000000986d5c90, is decrypt = 1
[  476.876414] driver calling napi_gro_receive for skb 0000000077f97780, is decrypt = 1
[  476.877131] driver calling napi_gro_receive for skb 000000005ab97059, is decrypt = 1
[  476.884790] driver calling napi_gro_receive for skb 00000000d010301f, is decrypt = 1
[  476.885523] driver calling napi_gro_receive for skb 000000009bde62fb, is decrypt = 1
[  476.886247] sk: 000000001f13dce9, tls_decrypt_sw, err = -74
[  476.886775] BUG: kernel NULL pointer dereference, address: 0000000000000108
[  476.887381] #PF: supervisor read access in kernel mode
[  476.887859] #PF: error_code(0x0000) - not-present page
[  476.888355] PGD 10d7d7067 P4D 10d7d7067 PUD 10d7d9067 PMD 0 
[  476.893564] Oops: 0000 [#1] SMP
[  476.893899] CPU: 2 PID: 11095 Comm: wrk_openssl_3_0 Not tainted 6.2.0+ #17
[  476.894493] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
[  476.895446] RIP: 0010:tls_rx_one_record+0x93/0x370 [tls]
[  476.895943] Code: 0f 84 54 02 00 00 44 29 66 40 45 31 e4 44 89 e2 4c 89 ee 48 c7 c7 50 7b 24 a0 e8 58 8b f6 e0 48 8b 73 08 48 c7 c7 a0 7b 24 a0 <48> 8b 96 c8 00 00 00 8b 86 c0 00 00 00 0f b6 4c 02 02 8b 56 70 e8
[  476.897607] RSP: 0018:ffff88811dadfba0 EFLAGS: 00010246
[  476.898168] RAX: 000000000000002f RBX: ffff88811dadfc58 RCX: 0000000000000000
[  476.898890] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffffffffa0247ba0
[  476.899613] RBP: ffff88814f588200 R08: ffffffff8345c2c8 R09: 0000000000000003
[  476.900351] R10: ffffffff8265c2e0 R11: ffffffff8315c2e0 R12: 00000000ffffffb6
[  476.901080] R13: ffff888131cb71c0 R14: ffff88810340cc00 R15: ffff88811dadfec0
[  476.901801] FS:  00007f635578d640(0000) GS:ffff88846f900000(0000) knlGS:0000000000000000
[  476.902651] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  476.903256] CR2: 0000000000000108 CR3: 00000001216f5006 CR4: 0000000000370ea0
[  476.903991] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  476.904721] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  476.905455] Call Trace:
[  476.905795]  <TASK>
[  476.906102]  tls_sw_recvmsg+0x366/0x8f0 [tls]
[  476.906606]  inet_recvmsg+0x47/0xc0
[  476.907035]  sock_recvmsg+0x44/0x70
[  476.907455]  ____sys_recvmsg+0x6f/0x140
[  476.907908]  ? iovec_from_user.part.0+0x44/0x170
[  476.908426]  ? __import_iovec+0x42/0x150
[  476.908908]  ? import_iovec+0x7/0x10
[  476.909272]  ? copy_msghdr_from_user+0x6d/0xa0
[  476.909700]  ___sys_recvmsg+0x7e/0xc0
[  476.910075]  ? ep_done_scan+0x16/0xc0
[  476.910446]  ? do_epoll_wait+0xd2/0x6b0
[  476.910829]  __sys_recvmsg+0x4e/0x90
[  476.911195]  do_syscall_64+0x3d/0x90
[  476.911559]  entry_SYSCALL_64_after_hwframe+0x46/0xb0
[  476.912035] RIP: 0033:0x7f635cb1323d
[  476.912399] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 da 6e f7 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2f 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 1e 6f f7 ff 48
[  476.913937] RSP: 002b:00007f635578ca20 EFLAGS: 00000293 ORIG_RAX: 000000000000002f
[  476.914620] RAX: ffffffffffffffda RBX: 0000000000004145 RCX: 00007f635cb1323d
[  476.915235] RDX: 0000000000000000 RSI: 00007f635578ca80 RDI: 0000000000000004
[  476.915864] RBP: 00007f6350020a50 R08: 0000000000000000 R09: 00007f635578cbe8
[  476.916478] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f6350018843
[  476.917108] R13: 00007f635578d5a0 R14: 00007f635578cb08 R15: 0000000000000000
[  476.917725]  </TASK>
[  476.917986] Modules linked in: mlx5_ib mlx5_core tls xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat br_netfilter overlay rpcrdma rdma_ucm ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm ib_cm ib_uverbs ib_core zram zsmalloc fuse [last unloaded: tls]
[  476.920201] CR2: 0000000000000108
[  476.920546] ---[ end trace 0000000000000000 ]---
[  476.920989] RIP: 0010:tls_rx_one_record+0x93/0x370 [tls]
[  476.921474] Code: 0f 84 54 02 00 00 44 29 66 40 45 31 e4 44 89 e2 4c 89 ee 48 c7 c7 50 7b 24 a0 e8 58 8b f6 e0 48 8b 73 08 48 c7 c7 a0 7b 24 a0 <48> 8b 96 c8 00 00 00 8b 86 c0 00 00 00 0f b6 4c 02 02 8b 56 70 e8
[  476.922992] RSP: 0018:ffff88811dadfba0 EFLAGS: 00010246
[  476.923476] RAX: 000000000000002f RBX: ffff88811dadfc58 RCX: 0000000000000000
[  476.924094] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffffffffa0247ba0
[  476.924711] RBP: ffff88814f588200 R08: ffffffff8345c2c8 R09: 0000000000000003
[  476.925324] R10: ffffffff8265c2e0 R11: ffffffff8315c2e0 R12: 00000000ffffffb6
[  476.925935] R13: ffff888131cb71c0 R14: ffff88810340cc00 R15: ffff88811dadfec0
[  476.926547] FS:  00007f635578d640(0000) GS:ffff88846f900000(0000) knlGS:0000000000000000
[  476.927267] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  476.927786] CR2: 0000000000000108 CR3: 00000001216f5006 CR4: 0000000000370ea0
[  476.928410] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  476.929038] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  476.929650] note: wrk_openssl_3_0[11095] exited with irqs disabled
[  477.030807] driver calling napi_gro_receive for skb 00000000a00ec12e, is decrypt = 0
[  477.310813] driver calling napi_gro_receive for skb 0000000089d6435f, is decrypt = 0
[  477.854843] driver calling napi_gro_receive for skb 000000006d079c0d, is decrypt = 0
[  478.942824] driver calling napi_gro_receive for skb 000000004a4abfc0, is decrypt = 0
[  481.150820] driver calling napi_gro_receive for skb 000000003116ab69, is decrypt = 0
[  485.502809] driver calling napi_gro_receive for skb 00000000fb4007fe, is decrypt = 0

Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser

[Jakub Kicinski]

On Wed, 15 Mar 2023 22:26:55 +0200 Tariq Toukan wrote:
> +       printk("skb: %p, len = %d, nfrags = %d\n", darg->skb, 
> darg->skb->len, skb_shinfo(darg->skb)->nr_frags);
> +       for (i = 0; i < skb_shinfo(darg->skb)->nr_frags; i++)
> +               printk("frag#%d: %p, len %d\n", i, 

Not enough info, I tried my best to get the CX6 DX I have to work, 
but mlx5_crypto_create_dek_key() returns -EINVAL fw 22.32.1206
IDK what the story is but it's quite frustrating to try to help
the offload vendors (NXP before you) and not be able to repro even 
when in theory one has the right HW :|

What do I need to do get a CX6 DX which reports

# ethtool -k eth0 | grep tls
tls-hw-tx-offload: on
tls-hw-rx-offload: on

to actually offload TLS 1.2?

Re: [PATCH net-next v3 0/7] tls: rx: decrypt from the TCP queue

[patchwork-bot+netdevbpf]

Hello:

This series was applied to netdev/net-next.git (master)
by Jakub Kicinski <kuba@kernel.org>:

On Fri, 22 Jul 2022 16:50:26 -0700 you wrote:
> This is the final part of my TLS Rx rework. It switches from
> strparser to decrypting data from skbs queued in TCP. We don't
> need the full strparser for TLS, its needs are very basic.
> This set gives us a small but measurable (6%) performance
> improvement (continuous stream).
> 
> v2: drop the __exit marking for the unroll path
> v3: drop tcp_recv_skb() in patch 5
> 
> [...]

Here is the summary with links:
  - [net-next,v3,1/7] tls: rx: wrap recv_pkt accesses in helpers
    https://git.kernel.org/netdev/net-next/c/b92a13d488de
  - [net-next,v3,2/7] tls: rx: factor SW handling out of tls_rx_one_record()
    https://git.kernel.org/netdev/net-next/c/dd47ed3620e6
  - [net-next,v3,3/7] tls: rx: don't free the output in case of zero-copy
    https://git.kernel.org/netdev/net-next/c/b93f5700164d
  - [net-next,v3,4/7] tls: rx: device: keep the zero copy status with offload
    https://git.kernel.org/netdev/net-next/c/d4e5db645221
  - [net-next,v3,5/7] tcp: allow tls to decrypt directly from the tcp rcv queue
    https://git.kernel.org/netdev/net-next/c/3f92a64e44e5
  - [net-next,v3,6/7] tls: rx: device: add input CoW helper
    https://git.kernel.org/netdev/net-next/c/8b3c59a7a0be
  - [net-next,v3,7/7] tls: rx: do not use the standard strparser
    https://git.kernel.org/netdev/net-next/c/84c61fe1a75b

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html