On 09/03/2023 19:54, Jakub Kicinski wrote:
> On Thu, 9 Mar 2023 17:15:26 +0200 Tariq Toukan wrote:
>> A few fixes were introduced for this patch, but it seems to still cause
>> issues.
>>
>> I'm running simple client/server test with wrk and nginx and TLS RX
>> device offload on.
>> It fails with TlsDecryptError on the client side for the large file
>> (256000b), while succeeding for the small one (10000b). See repro
>> details below.
>>
>> I narrowed the issue down to this offending patch, by applying a few
>> reverts (had to solve trivial conflicts):
> 
> What's the sequence of records in terms of being offloaded vs fall back?
> Could you whip up a simple ring buffer to see if previous records were
> offloaded and what the skb geometries where?

Interesting. All records go through the sw fallback.

Command:
$ wrk_openssl_3_0_0 -b2.2.2.2 -t1 -c1 -d2 --timeout 5s 
https://2.2.2.3:20443/256000b.img

Debug code:
@@ -1712,8 +1723,13 @@ static int tls_rx_one_record(struct sock *sk, 
struct msghdr *msg,
         int err;

         err = tls_decrypt_device(sk, msg, tls_ctx, darg);
-       if (!err)
+       if (!err) {
                 err = tls_decrypt_sw(sk, tls_ctx, msg, darg);
+               printk("sk: %p, tls_decrypt_sw, err = %d\n", sk, err);
+       } else {
+               printk("sk: %p, tls_decrypt_device, err = %d\n", sk, err);
+       }
+       skb_dump(KERN_ERR, darg->skb, false);
         if (err < 0)
                 return err;

dmesg output including skb geometries is attached.

# cat /proc/net/tls_stat
TlsCurrTxSw                             0
TlsCurrRxSw                             0
TlsCurrTxDevice                         0
TlsCurrRxDevice                         0
TlsTxSw                                 0
TlsRxSw                                 0
TlsTxDevice                             12
TlsRxDevice                             12
TlsDecryptError                         22
TlsRxDeviceResync                       12
TlsDecryptRetry                         0
TlsRxNoPadViolation                     0

We start from out-of-sync state, so we expect to go through the sw 
fallback in the beginning.
I do see that the resync succeeds and device-offloaded packets start to 
flow, example ethtool output (1 sec interval):

       rx_tls_decrypted_packets: 356
         rx_tls_decrypted_bytes: 517,160 Bps          = 4.13 Mbps
          rx_tls_resync_req_pkt: 6
        rx_tls_resync_req_start: 6
          rx_tls_resync_req_end: 6
           rx_tls_resync_res_ok: 6
                     rx_tls_ctx: 6
                     rx_tls_del: 6