WireGuard Archive on lore.kernel.org
 help / color / Atom feed
* Ryzen 3000 CPU soft lockup
@ 2019-07-22  5:09 trulyliu
  2019-08-25 15:51 ` Jason A. Donenfeld
  0 siblings, 1 reply; 3+ messages in thread
From: trulyliu @ 2019-07-22  5:09 UTC (permalink / raw)
  To: wireguard

[-- Attachment #1.1: Type: text/plain, Size: 4086 bytes --]

I just upgrade my cpu to Ryzen 3700x, my mainboard is Asus B450-I.  I have
updated the latest bios.
After upgarde, wireguard constantly cause CPU soft lockup.

[   79.397005] wireguard: WireGuard 0.0.20190601 loaded. See
www.wireguard.com for information.
[   79.397005] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <
Jason@zx2c4.com>. All Rights Reserved.
[  224.295881] watchdog: BUG: soft lockup - CPU#2 stuck for 22s!
[kworker/u64:4:304]
[  224.295884] Modules linked in: wireguard(OE) ip6_udp_tunnel udp_tunnel
rfcomm pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE)
xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_n
etlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat
nf_conntrack libcrc32c br_netfilter bridge stp llc aufs overlay cmac bnep
snd
_hda_codec_hdmi binfmt_misc nls_iso8859_1 edac_mce_amd
snd_hda_codec_realtek snd_hda_codec_generic kvm snd_hda_intel arc4
snd_hda_codec irqbypass snd_hda_core crct10dif_pclmul snd_hwdep
crc32_pclmul snd_pcm ghash_clmul
ni_intel snd_seq_midi r8822be(C) snd_seq_midi_event pcbc snd_rawmidi btusb
snd_seq btrtl aesni_intel mac80211 btbcm btintel snd_seq_device snd_timer
bluetooth
[  224.295921]  aes_x86_64 eeepc_wmi asus_wmi crypto_simd sparse_keymap snd
glue_helper video wmi_bmof joydev input_leds ecdh_generic cryptd cfg80211
soundcore shpchp wmi mac_hid nvidia_uvm(OE) sch_fq_codel msr parport
_pc ppdev lp sunrpc parport ip_tables x_tables autofs4 btrfs xor
zstd_compress raid6_pq hid_generic usbhid hid nvidia_drm(POE)
nvidia_modeset(POE) nvidia(POE) drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fop
s drm igb ipmi_devintf ipmi_msghandler nvme dca i2c_algo_bit i2c_piix4 ahci
ptp nvme_core pps_core libahci gpio_amdpt gpio_generic
[  224.295952] CPU: 2 PID: 304 Comm: kworker/u64:4 Tainted: P         C OE
   4.15.0-54-generic #58-Ubuntu
[  224.295953] Hardware name: System manufacturer System Product Name/ROG
STRIX B450-I GAMING, BIOS 2501 07/12/2019
[  224.295961] Workqueue: wg-kex-wg0 wg_packet_handshake_send_worker
[wireguard]
[  224.295965] RIP: 0010:get_random_u32+0x24/0xe0
[  224.295967] RSP: 0018:ffffa20888307ca8 EFLAGS: 00000203 ORIG_RAX:
ffffffffffffff11
[  224.295968] RAX: 00000000ffffffff RBX: ffff921d74985ac8 RCX:
0000000000000000
[  224.295969] RDX: 000000000000000a RSI: 00000000fffffe01 RDI:
ffffffffc2386d02
[  224.295970] RBP: ffffa20888307cc0 R08: ffffffffc23a8ca0 R09:
ffffa20888307c58
[  224.295971] R10: 0000000000000000 R11: 02d4998003202790 R12:
ffff921cb5720000
[  224.295972] R13: ffff921cb5730000 R14: ffff921d74985ad0 R15:
ffffa20888307e0c
[  224.295973] FS:  0000000000000000(0000) GS:ffff921e3e280000(0000)
knlGS:0000000000000000
[  224.295974] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  224.295975] CR2: 00007ffbffff3308 CR3: 000000074f19c000 CR4:
0000000000340ee0
[  224.295976] Call Trace:
[  224.295982]  ? wg_index_hashtable_insert+0x71/0x110 [wireguard]
[  224.295985]  wg_index_hashtable_insert+0x62/0x110 [wireguard]
[  224.295988]  wg_noise_handshake_create_initiation+0x2d2/0x2f0
[wireguard]
[  224.295992]  wg_packet_send_handshake_initiation+0x8c/0x130 [wireguard]
[  224.295995]  wg_packet_handshake_send_worker+0x19/0x30 [wireguard]
[  224.295998]  process_one_work+0x1de/0x410
[  224.295999]  worker_thread+0x32/0x410
[  224.296001]  kthread+0x121/0x140
[  224.296003]  ? process_one_work+0x410/0x410
[  224.296005]  ? kthread_create_worker_on_cpu+0x70/0x70
[  224.296008]  ret_from_fork+0x22/0x40
[  224.296009] Code: 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54
53 48 83 ec 08 0f 1f 44 00 00 ba 0a 00 00 00 eb 05 ba 0a 00 00 00 0f c7 f0
<72> 46 83 ea 01 75 f6 48 8b 75 08 48 c7 c2 b0 52 30 ba 48 c7 c7

Kernel version: Linux 5.0.0-20-generic
I also tried WireGuard 0.0.20190702, the same CPU lockup.
I know this maybe the bug of CPU or kernel itself.
But this lockup only occurs in wireguard,  If stop wireguard, the CPU soft
lockup disappear.

Could anyone look into this issue?




-- 
trulyliu@gmail.com

[-- Attachment #1.2: Type: text/html, Size: 5175 bytes --]

<div dir="ltr">I just upgrade my cpu to Ryzen 3700x, my mainboard is Asus B450-I.  I have updated the latest bios.<div>After upgarde, wireguard constantly cause CPU soft lockup.<br></div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">[   79.397005] wireguard: WireGuard 0.0.20190601 loaded. See <a href="http://www.wireguard.com">www.wireguard.com</a> for information.
</span><br>[   79.397005] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld &lt;<a href="mailto:Jason@zx2c4.com">Jason@zx2c4.com</a>&gt;. All Rights Reserved.
<br>[  224.295881] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u64:4:304]
<br>[  224.295884] Modules linked in: wireguard(OE) ip6_udp_tunnel udp_tunnel rfcomm pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_n<br>etlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc aufs overlay cmac bnep snd<br>_hda_codec_hdmi binfmt_misc nls_iso8859_1 edac_mce_amd snd_hda_codec_realtek snd_hda_codec_generic kvm snd_hda_intel arc4 snd_hda_codec irqbypass snd_hda_core crct10dif_pclmul snd_hwdep crc32_pclmul snd_pcm ghash_clmul<br>ni_intel snd_seq_midi r8822be(C) snd_seq_midi_event pcbc snd_rawmidi btusb snd_seq btrtl aesni_intel mac80211 btbcm btintel snd_seq_device snd_timer bluetooth
<br>[  224.295921]  aes_x86_64 eeepc_wmi asus_wmi crypto_simd sparse_keymap snd glue_helper video wmi_bmof joydev input_leds ecdh_generic cryptd cfg80211 soundcore shpchp wmi mac_hid nvidia_uvm(OE) sch_fq_codel msr parport<br>_pc ppdev lp sunrpc parport ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq hid_generic usbhid hid nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fop<br>s drm igb ipmi_devintf ipmi_msghandler nvme dca i2c_algo_bit i2c_piix4 ahci ptp nvme_core pps_core libahci gpio_amdpt gpio_generic
<br>[  224.295952] CPU: 2 PID: 304 Comm: kworker/u64:4 Tainted: P         C OE    4.15.0-54-generic #58-Ubuntu
<br>[  224.295953] Hardware name: System manufacturer System Product Name/ROG STRIX B450-I GAMING, BIOS 2501 07/12/2019
<br>[  224.295961] Workqueue: wg-kex-wg0 wg_packet_handshake_send_worker [wireguard]
<br>[  224.295965] RIP: 0010:get_random_u32+0x24/0xe0
<br>[  224.295967] RSP: 0018:ffffa20888307ca8 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff11
<br>[  224.295968] RAX: 00000000ffffffff RBX: ffff921d74985ac8 RCX: 0000000000000000
<br>[  224.295969] RDX: 000000000000000a RSI: 00000000fffffe01 RDI: ffffffffc2386d02
<br>[  224.295970] RBP: ffffa20888307cc0 R08: ffffffffc23a8ca0 R09: ffffa20888307c58
<br>[  224.295971] R10: 0000000000000000 R11: 02d4998003202790 R12: ffff921cb5720000
<br>[  224.295972] R13: ffff921cb5730000 R14: ffff921d74985ad0 R15: ffffa20888307e0c
<br>[  224.295973] FS:  0000000000000000(0000) GS:ffff921e3e280000(0000) knlGS:0000000000000000
<br>[  224.295974] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<br>[  224.295975] CR2: 00007ffbffff3308 CR3: 000000074f19c000 CR4: 0000000000340ee0
<br>[  224.295976] Call Trace:
<br>[  224.295982]  ? wg_index_hashtable_insert+0x71/0x110 [wireguard]
<br>[  224.295985]  wg_index_hashtable_insert+0x62/0x110 [wireguard]
<br>[  224.295988]  wg_noise_handshake_create_initiation+0x2d2/0x2f0 [wireguard]
<br>[  224.295992]  wg_packet_send_handshake_initiation+0x8c/0x130 [wireguard]
<br>[  224.295995]  wg_packet_handshake_send_worker+0x19/0x30 [wireguard]
<br>[  224.295998]  process_one_work+0x1de/0x410
<br>[  224.295999]  worker_thread+0x32/0x410
<br>[  224.296001]  kthread+0x121/0x140
<br>[  224.296003]  ? process_one_work+0x410/0x410
<br>[  224.296005]  ? kthread_create_worker_on_cpu+0x70/0x70
<br>[  224.296008]  ret_from_fork+0x22/0x40
<br>[  224.296009] Code: 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 53 48 83 ec 08 0f 1f 44 00 00 ba 0a 00 00 00 eb 05 ba 0a 00 00 00 0f c7 f0 &lt;72&gt; 46 83 ea 01 75 f6 48 8b 75 08 48 c7 c2 b0 52 30 ba 48 c7 c7 <br>
<br></span><div>Kernel version: <span style="color:rgb(0,0,0);font-family:monospace">Linux 5.0.0-20-generic</span></div><div>I also tried <span style="color:rgb(0,0,0);font-family:monospace">WireGuard 0.0.20190702, the same CPU lockup.</span><br></div><div><span style="color:rgb(0,0,0);font-family:monospace">I know this maybe the bug of CPU or kernel itself.</span></div><div><span style="color:rgb(0,0,0);font-family:monospace">But this lockup only occurs in wireguard,  If stop wireguard, the CPU soft lockup disappear.</span></div><div><br></div><div>Could anyone look into this issue?</div><div><br></div><div><br></div><div><span style="color:rgb(0,0,0);font-family:monospace"><br></span></div><div><span style="color:rgb(0,0,0);font-family:monospace"><br></span></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><a href="mailto:trulyliu@gmail.com" target="_blank">trulyliu@gmail.com</a><br></div></div></div>

[-- Attachment #2: Type: text/plain, Size: 148 bytes --]

_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Ryzen 3000 CPU soft lockup
  2019-07-22  5:09 Ryzen 3000 CPU soft lockup trulyliu
@ 2019-08-25 15:51 ` Jason A. Donenfeld
  2019-08-27  5:41   ` trulyliu
  0 siblings, 1 reply; 3+ messages in thread
From: Jason A. Donenfeld @ 2019-08-25 15:51 UTC (permalink / raw)
  To: trulyliu; +Cc: WireGuard mailing list

WireGuard uses get_random_u32 which uses rdrand. AMD shipped with a
broken rdrand. Update your CPU's microcode or your motherboard's BIOS
and all should be good.
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Ryzen 3000 CPU soft lockup
  2019-08-25 15:51 ` Jason A. Donenfeld
@ 2019-08-27  5:41   ` trulyliu
  0 siblings, 0 replies; 3+ messages in thread
From: trulyliu @ 2019-08-27  5:41 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

[-- Attachment #1.1: Type: text/plain, Size: 618 bytes --]

ASUS  does not follow AMD microcode release in time, the latest motherboard
bios are using two month ago AMD
microcode.
Tried CPU's microcode with kernel, didn't work either.
fortunately, found a solution from arch linux comunity.  Appending nordrand
kernel parameter in grub resolved the issue.
Hope this can be documented some where to help others.


Jason A. Donenfeld <Jason@zx2c4.com> 于 2019年8月25日周日 23:51写道:

> WireGuard uses get_random_u32 which uses rdrand. AMD shipped with a
> broken rdrand. Update your CPU's microcode or your motherboard's BIOS
> and all should be good.
>

[-- Attachment #1.2: Type: text/html, Size: 1040 bytes --]

<div dir="auto"><div>ASUS  does not follow AMD microcode release in time, the latest motherboard bios are using two month ago AMD </div><div dir="auto">microcode.</div><div dir="auto">Tried CPU&#39;s microcode with kernel, didn&#39;t work either.</div><div dir="auto">fortunately, found a solution from arch linux comunity.  Appending nordrand kernel parameter in grub resolved the issue.</div><div dir="auto">Hope this can be documented some where to help others.</div><div dir="auto"><br></div><div dir="auto"><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">Jason A. Donenfeld &lt;<a href="mailto:Jason@zx2c4.com">Jason@zx2c4.com</a>&gt; 于 2019年8月25日周日 23:51写道:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">WireGuard uses get_random_u32 which uses rdrand. AMD shipped with a<br>
broken rdrand. Update your CPU&#39;s microcode or your motherboard&#39;s BIOS<br>
and all should be good.<br>
</blockquote></div></div></div>

[-- Attachment #2: Type: text/plain, Size: 148 bytes --]

_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, back to index

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-22  5:09 Ryzen 3000 CPU soft lockup trulyliu
2019-08-25 15:51 ` Jason A. Donenfeld
2019-08-27  5:41   ` trulyliu

WireGuard Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/wireguard/0 wireguard/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 wireguard wireguard/ https://lore.kernel.org/wireguard \
		wireguard@lists.zx2c4.com
	public-inbox-index wireguard

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/com.zx2c4.lists.wireguard


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git