[ISSUE][4.20.6] mlx5 and checksum failures

From: Ian Kumlien <ian.kumlien@gmail.com>
To: Linux Kernel Network Developers <netdev@vger.kernel.org>
Subject: [ISSUE][4.20.6] mlx5 and checksum failures
Date: Wed, 6 Feb 2019 17:16:47 +0100	[thread overview]
Message-ID: <CAA85sZtE7Gv8mKL5tUh8AJ4yG9xd_HZh9svWkHXm=j7VohD1Cw@mail.gmail.com> (raw)

Hi,

I'm hitting an issue that i think is fixed by the following patch,
i haven't verified it yet - but it looks like it should go on the
stable queue(?)

(And yes, I did look, and couldn't find it ;))

commit e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a
Author: Cong Wang <xiyou.wangcong@gmail.com>
Date:   Mon Dec 3 22:14:04 2018 -0800

    net/mlx5e: Force CHECKSUM_UNNECESSARY for short ethernet frames

    When an ethernet frame is padded to meet the minimum ethernet frame
    size, the padding octets are not covered by the hardware checksum.
    Fortunately the padding octets are usually zero's, which don't affect
    checksum. However, we have a switch which pads non-zero octets, this
    causes kernel hardware checksum fault repeatedly.

    Prior to:
    commit '88078d98d1bb ("net: pskb_trim_rcsum() and
CHECKSUM_COMPLETE ...")'
    skb checksum was forced to be CHECKSUM_NONE when padding is detected.
    After it, we need to keep skb->csum updated, like what we do for RXFCS.
    However, fixing up CHECKSUM_COMPLETE requires to verify and parse IP
    headers, it is not worthy the effort as the packets are so small that
    CHECKSUM_COMPLETE can't save anything.

    Fixes: 88078d98d1bb ("net: pskb_trim_rcsum() and CHECKSUM_COMPLETE
are friends"),
    Cc: Eric Dumazet <edumazet@google.com>
    Cc: Tariq Toukan <tariqt@mellanox.com>
    Cc: Nikola Ciprich <nikola.ciprich@linuxbox.cz>
    Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
    Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 1d0bb5ff8c26..f86e4804e83e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -732,6 +732,8 @@ static u8 get_ip_proto(struct sk_buff *skb, int
network_depth, __be16 proto)
                                            ((struct ipv6hdr
*)ip_p)->nexthdr;
 }

+#define short_frame(size) ((size) <= ETH_ZLEN + ETH_FCS_LEN)
+
 static inline void mlx5e_handle_csum(struct net_device *netdev,
                                     struct mlx5_cqe64 *cqe,
                                     struct mlx5e_rq *rq,
@@ -754,6 +756,17 @@ static inline void mlx5e_handle_csum(struct
net_device *netdev,
        if (unlikely(test_bit(MLX5E_RQ_STATE_NO_CSUM_COMPLETE, &rq->state)))
                goto csum_unnecessary;

+       /* CQE csum doesn't cover padding octets in short ethernet
+        * frames. And the pad field is appended prior to calculating
+        * and appending the FCS field.
+        *
+        * Detecting these padded frames requires to verify and parse
+        * IP headers, so we simply force all those small frames to be
+        * CHECKSUM_UNNECESSARY even if they are not padded.
+        */
+       if (short_frame(skb->len))
+               goto csum_unnecessary;
+
        if (likely(is_last_ethertype_ip(skb, &network_depth, &proto))) {
                if (unlikely(get_ip_proto(skb, network_depth, proto)
== IPPROTO_SCTP))
                        goto csum_unnecessary;
---

Kernel log:
[ 3226.017424] bond0: hw csum failure
[ 3226.018387] CPU: 13 PID: 0 Comm: swapper/13 Tainted: G          I
    4.20.6-1.el7.elrepo.x86_64 #1
[ 3226.020928] Hardware name: HP ProLiant DL380 G6, BIOS P62 01/22/2015
[ 3226.022649] Call Trace:
[ 3226.023409]  <IRQ>
[ 3226.024039]  dump_stack+0x63/0x88
[ 3226.025066]  netdev_rx_csum_fault+0x3a/0x40
[ 3226.026208]  __skb_checksum_complete+0xd5/0xe0
[ 3226.027418]  nf_ip_checksum+0xc9/0xf0
[ 3226.028474]  nf_checksum+0x2d/0x40
[ 3226.029504]  tcp_packet+0x2ce/0xa20 [nf_conntrack]
[ 3226.030913]  ? tcp_v4_do_rcv+0x77/0x1f0
[ 3226.032094]  ? sock_put+0x19/0x20
[ 3226.033070]  ? nf_ct_deliver_cached_events+0xd0/0x110 [nf_conntrack]
[ 3226.034754]  nf_conntrack_in+0x140/0x510 [nf_conntrack]
[ 3226.036228]  ipv4_conntrack_in+0x14/0x20 [nf_conntrack]
[ 3226.037646]  nf_hook_slow+0x42/0xc0
[ 3226.038626]  ip_rcv+0xb5/0xd0
[ 3226.039480]  ? ip_local_deliver_finish+0x1e0/0x1e0
[ 3226.040767]  __netif_receive_skb_one_core+0x57/0x80
[ 3226.042155]  __netif_receive_skb+0x18/0x60
[ 3226.043275]  netif_receive_skb_internal+0x45/0xf0
[ 3226.044530]  napi_gro_receive+0xd0/0xf0
[ 3226.045665]  mlx5e_handle_rx_cqe+0x1e6/0x540 [mlx5_core]
[ 3226.047167]  mlx5e_poll_rx_cq+0xd6/0x9c0 [mlx5_core]
[ 3226.048516]  mlx5e_napi_poll+0xc2/0xcd0 [mlx5_core]
[ 3226.049836]  ? mlx5_eq_int+0x4b4/0x6c0 [mlx5_core]
[ 3226.051118]  net_rx_action+0x289/0x3d0
[ 3226.052257]  __do_softirq+0xd5/0x2a2
[ 3226.053277]  irq_exit+0xe8/0x100
[ 3226.054183]  do_IRQ+0x59/0xe0
[ 3226.055014]  common_interrupt+0xf/0xf
[ 3226.056038]  </IRQ>
[ 3226.056722] RIP: 0010:cpuidle_enter_state+0xba/0x2f0
[ 3226.058087] Code: d0 95 7e e8 38 07 a1 ff 41 8b 5c 24 04 49 89 c6
66 66 66 66 90 31 ff e8 34 19 a1 ff 80 7d cf 00 0f 85 8c 01 00 00 fb
66 66 90 <66> 66 90 45 85 ed 0f 88 94 01 00 00 4c 2b 75 c0 48 ba cf f7
53 e3
[ 3226.062925] RSP: 0018:ffffc9000c547e50 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffffd6
[ 3226.064974] RAX: ffff88a3df7a2dc0 RBX: 000000000000000d RCX: 000000000000001f
[ 3226.066866] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000000
[ 3226.068747] RBP: ffffc9000c547e90 R08: 0000000000000002 R09: ffffffcdc506f2e7
[ 3226.070622] R10: 0000000000000018 R11: 071c71c71c71c71c R12: ffffe8ffffb96f00
[ 3226.072525] R13: 0000000000000004 R14: 000002ef1d9f1e10 R15: ffff88a3d8900000
[ 3226.074479]  cpuidle_enter+0x17/0x20
[ 3226.075463]  call_cpuidle+0x23/0x40
[ 3226.076412]  do_idle+0x1db/0x280
[ 3226.077323]  cpu_startup_entry+0x1d/0x30
[ 3226.078417]  start_secondary+0x1ae/0x200
[ 3226.079490]  secondary_startup_64+0xa4/0xb0
