soft lockup in __inet_lookup_established() function

soft lockup in __inet_lookup_established() function

[Jason Xing]

Hello,

I've encountered one big issue which causes infinite loop in
__inet_lookup_established() function until I reboot manually. it's
happening randomly among thousands of machines with the 4.19 kernel
running. Once the soft lockup issue is triggered, whatever I try I
still cannot ping or ssh to the machine anymore until reboot.

Does anyone have any clue on how to dig into this part of code?  I
highly suspect that it has something to do with the corruption of
nulls_list, so the lookup of sk could never break the infinite loop of
hashinfo.

These call traces are totally identical attached below:
[1048271.465028] watchdog: BUG: soft lockup - CPU#20 stuck for 22s!
[swapper/20:0]
[1048271.473669] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel
udp_diag tcp_diag inet_diag nf_conntrack_netlink nfnetlink
br_netfilter bridge stp llc xt_statistic xt_nat ipt_MASQUERADE
ipt_REJECT nf_reject_ipv4 xt_mark xt_addrtype xt_comment xt_conntrack
...
[1048271.553597] RIP: 0010:__inet_lookup_established+0x5a/0x190
...
[1048271.660309] Call Trace:
[1048271.663135]  <IRQ>
[1048271.665432]  tcp_v4_early_demux+0xaa/0x150
[1048271.669812]  ip_rcv_finish+0x171/0x410
[1048271.673941]  ip_rcv+0x273/0x362
[1048271.677360]  ? inet_add_protocol.cold.1+0x1e/0x1e
[1048271.682354]  __netif_receive_skb_core+0xac2/0xbb0
[1048271.687351]  ? inet_gro_receive+0x22a/0x2d0
[1048271.692001]  ? ktime_get_with_offset+0x4d/0xc0
[1048271.696725]  netif_receive_skb_internal+0x42/0xf0
[1048271.701717]  napi_gro_receive+0xba/0xe0
[1048271.705839]  receive_buf+0x165/0xa50 [virtio_net]
[1048271.710839]  ? receive_buf+0x165/0xa50 [virtio_net]
[1048271.716053]  ? vring_unmap_one+0x16/0x80
[1048271.720308]  ? detach_buf+0x69/0x110
[1048271.724218]  virtnet_poll+0xc0/0x2ea [virtio_net]
[1048271.729202]  net_rx_action+0x149/0x3b0
[1048271.733234]  __do_softirq+0xe3/0x30a
[1048271.737095]  irq_exit+0x100/0x110
[1048271.740882]  do_IRQ+0x85/0xd0
[1048271.744143]  common_interrupt+0xf/0xf
[1048271.748104]  </IRQ>


thanks,
jason

Re: soft lockup in __inet_lookup_established() function

[Jason Xing]

Does anyone have some suggestions? I've been haunted for a while.

thanks,
jason

On Thu, Apr 29, 2021 at 8:15 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
>
> Hello,
>
> I've encountered one big issue which causes infinite loop in
> __inet_lookup_established() function until I reboot manually. it's
> happening randomly among thousands of machines with the 4.19 kernel
> running. Once the soft lockup issue is triggered, whatever I try I
> still cannot ping or ssh to the machine anymore until reboot.
>
> Does anyone have any clue on how to dig into this part of code?  I
> highly suspect that it has something to do with the corruption of
> nulls_list, so the lookup of sk could never break the infinite loop of
> hashinfo.
>
> These call traces are totally identical attached below:
> [1048271.465028] watchdog: BUG: soft lockup - CPU#20 stuck for 22s!
> [swapper/20:0]
> [1048271.473669] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel
> udp_diag tcp_diag inet_diag nf_conntrack_netlink nfnetlink
> br_netfilter bridge stp llc xt_statistic xt_nat ipt_MASQUERADE
> ipt_REJECT nf_reject_ipv4 xt_mark xt_addrtype xt_comment xt_conntrack
> ...
> [1048271.553597] RIP: 0010:__inet_lookup_established+0x5a/0x190
> ...
> [1048271.660309] Call Trace:
> [1048271.663135]  <IRQ>
> [1048271.665432]  tcp_v4_early_demux+0xaa/0x150
> [1048271.669812]  ip_rcv_finish+0x171/0x410
> [1048271.673941]  ip_rcv+0x273/0x362
> [1048271.677360]  ? inet_add_protocol.cold.1+0x1e/0x1e
> [1048271.682354]  __netif_receive_skb_core+0xac2/0xbb0
> [1048271.687351]  ? inet_gro_receive+0x22a/0x2d0
> [1048271.692001]  ? ktime_get_with_offset+0x4d/0xc0
> [1048271.696725]  netif_receive_skb_internal+0x42/0xf0
> [1048271.701717]  napi_gro_receive+0xba/0xe0
> [1048271.705839]  receive_buf+0x165/0xa50 [virtio_net]
> [1048271.710839]  ? receive_buf+0x165/0xa50 [virtio_net]
> [1048271.716053]  ? vring_unmap_one+0x16/0x80
> [1048271.720308]  ? detach_buf+0x69/0x110
> [1048271.724218]  virtnet_poll+0xc0/0x2ea [virtio_net]
> [1048271.729202]  net_rx_action+0x149/0x3b0
> [1048271.733234]  __do_softirq+0xe3/0x30a
> [1048271.737095]  irq_exit+0x100/0x110
> [1048271.740882]  do_IRQ+0x85/0xd0
> [1048271.744143]  common_interrupt+0xf/0xf
> [1048271.748104]  </IRQ>
>
>
> thanks,
> jason