From: "Jason A. Donenfeld" <Jason@zx2c4.com>
To: Netdev <netdev@vger.kernel.org>, Willem de Bruijn <willemb@google.com>
Cc: LKML <linux-kernel@vger.kernel.org>
Subject: possible stack corruption in icmp_send (__stack_chk_fail)
Date: Wed, 17 Feb 2021 19:12:00 +0100 [thread overview]
Message-ID: <CAHmME9qfXFZKZfO-uc7GC3xguSq99_CqrTtzmgp_984MSfNbgA@mail.gmail.com> (raw)
Hi Netdev & Willem,
I've received a report of stack corruption -- via the stack protector
check -- in icmp_send. I was sent a vmcore, and was able to extract
the OOPS from there. However, I've been unable to produce the bug and
I don't see where it'd be in the code. That might point to a more
sinister problem, or I'm simply just not seeing it. Apparently the
reporter reproduces it every 40 or so minutes, and has seen it happen
since at least ~5.10. Willem - I'm emailing you because it seems like
you were making a lot of changes to the icmp code around then, and
perhaps you have an intuition. For example, some of the error handling
code takes a pointer to a stack buffer (_objh and such), and maybe
that's problematic? I'm not quite sure. The vmcore, along with the
various kernel binaries I hunted down are here:
https://data.zx2c4.com/icmp_send-crash-e03b4a42-706a-43bf-bc40-1f15966b3216.tar.xz
. The extracted dmesg follows below, in case you or anyone has a
pointer. I've been staring at this for a while and don't see it.
Jason
Kernel panic - not syncing: stack-protector: Kernel stack is corrupted
in: __icmp_send+0x5bd/0x5c0
CPU: 0 PID: 959 Comm: kworker/0:2 Kdump: loaded Not tainted
5.11.0-051100-lowlatency #202102142330
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org 04/01/2014
Workqueue: wg-crypt-wg0 wg_packet_decrypt_worker [wireguard]
Call Trace:
<IRQ>
show_stack+0x52/0x58
dump_stack+0x70/0x8b
panic+0x108/0x2ea
? ip_push_pending_frames+0x42/0x90
? __icmp_send+0x5bd/0x5c0
__stack_chk_fail+0x14/0x20
__icmp_send+0x5bd/0x5c0
icmp_ndo_send+0x148/0x160
wg_xmit+0x359/0x450 [wireguard]
? harmonize_features+0x19/0x80
xmit_one.constprop.0+0x9f/0x190
dev_hard_start_xmit+0x43/0x90
sch_direct_xmit+0x11d/0x340
__qdisc_run+0x66/0xc0
__dev_xmit_skb+0xd5/0x340
__dev_queue_xmit+0x32b/0x4d0
? nf_conntrack_double_lock.constprop.0+0x97/0x140 [nf_conntrack]
dev_queue_xmit+0x10/0x20
neigh_connected_output+0xcb/0xf0
ip_finish_output2+0x17f/0x470
__ip_finish_output+0x9b/0x140
? ipv4_confirm+0x4a/0x80 [nf_conntrack]
ip_finish_output+0x2d/0xb0
ip_output+0x78/0x110
? __ip_finish_output+0x140/0x140
ip_forward_finish+0x58/0x90
ip_forward+0x40a/0x4d0
? ip4_key_hashfn+0xb0/0xb0
ip_sublist_rcv_finish+0x3d/0x50
ip_list_rcv_finish.constprop.0+0x163/0x190
ip_sublist_rcv+0x37/0xb0
? ip_rcv_finish_core.constprop.0+0x310/0x310
ip_list_rcv+0xf5/0x120
__netif_receive_skb_list_core+0x228/0x250
__netif_receive_skb_list+0x102/0x170
? dev_gro_receive+0x1b5/0x370
netif_receive_skb_list_internal+0xca/0x190
napi_complete_done+0x7a/0x1a0
wg_packet_rx_poll+0x384/0x400 [wireguard]
napi_poll+0x92/0x200
net_rx_action+0xb8/0x1c0
__do_softirq+0xce/0x2b3
asm_call_irq_on_stack+0x12/0x20
</IRQ>
do_softirq_own_stack+0x3d/0x50
do_softirq+0x66/0x80
__local_bh_enable_ip+0x62/0x70
_raw_spin_unlock_bh+0x1e/0x20
wg_packet_decrypt_worker+0xf6/0x190 [wireguard]
process_one_work+0x217/0x3e0
worker_thread+0x4d/0x350
? rescuer_thread+0x390/0x390
kthread+0x145/0x170
? __kthread_bind_mask+0x70/0x70
ret_from_fork+0x22/0x30
Kernel Offset: 0x2000000 from 0xffffffff81000000 (relocation range:
0xffffffff80000000-0xffffffffbfffffff)
next reply other threads:[~2021-02-17 18:13 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-17 18:12 Jason A. Donenfeld [this message]
2021-02-17 22:27 ` possible stack corruption in icmp_send (__stack_chk_fail) Willem de Bruijn
2021-02-17 22:55 ` Jason A. Donenfeld
2021-02-17 23:05 ` Willem de Bruijn
2021-02-17 23:18 ` Jason A. Donenfeld
2021-02-17 23:37 ` Willem de Bruijn
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAHmME9qfXFZKZfO-uc7GC3xguSq99_CqrTtzmgp_984MSfNbgA@mail.gmail.com \
--to=jason@zx2c4.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).