WireGuard Archive on lore.kernel.org
 help / color / Atom feed
From: Ryan Whelan <rcwhelan@gmail.com>
To: WireGuard mailing list <wireguard@lists.zx2c4.com>
Subject: Wireguard Bug?
Date: Sun, 12 May 2019 09:44:26 -0400
Message-ID: <CAM3m09Q3LCCFRJEe6J1UO0mtTiuVOYEiiyhEJJppDnuDnZZTwA@mail.gmail.com> (raw)

[-- Attachment #1.1: Type: text/plain, Size: 6725 bytes --]

I am building a system which coordinates the meshing of wireguard devices.
Currently, all the devices are running on an embedded platform (AMD Geode
LX500) and works as expected.  However, when introducing a 64bit KVM host
for testing, all the 32bit hosts running on the Geode platform, report the
following warning and drop offline temporarily.  The CPU usage on KVM host
spikes and all the CPU time is spent on the kernel threads servicing the WG
interfaces.

I'm using kernel 4.19.41 and have seen the issue with both the last WG
snapshot as well building from master.

Is this a bug in WG?

May 10 18:36:25 buildroot kern.warn kernel: WARNING: CPU: 0 PID: 9 at
kernel/workqueue.c:1442 __queue_work+0x1d4/0x2aa
May 10 18:36:25 buildroot kern.warn kernel: CPU: 0 PID: 9 Comm: ksoftirqd/0
Tainted: G        W         4.19.41-banana #1
May 10 18:36:25 buildroot kern.warn kernel: EIP: __queue_work+0x1d4/0x2aa
May 10 18:36:25 buildroot kern.warn kernel: Code: c1 e8 4e 09 1d 00 c7 05
30 d3 5f c1 00 00 00 00 c7 45 e4 00 00 00 00 e9 76 fe ff ff 8b 3d 80 3c 60
c1 89 78 10 e9 32 ff ff ff <0f> 0b b8 01 00 00 00 e8 1c 82 00 00 a1 a8 9e
5f c1 85 c0 0f 85 3b
May 10 18:36:25 buildroot kern.warn kernel: EAX: 00000000 EBX: cfb63044
ECX: 00000003 EDX: cfb63048
May 10 18:36:25 buildroot kern.warn kernel: ESI: cfdfbb00 EDI: cfab29c0
EBP: cf847d38 ESP: cf847d1c
May 10 18:36:25 buildroot kern.warn kernel: DS: 007b ES: 007b FS: 0000 GS:
00e0 SS: 0068 EFLAGS: 00010086
May 10 18:36:25 buildroot kern.warn kernel: CR0: 80050033 CR2: a7bb5058
CR3: 0eebe000 CR4: 00000090
May 10 18:36:25 buildroot kern.warn kernel: Call Trace:
May 10 18:36:25 buildroot kern.warn kernel:  queue_work_on+0x19/0x1d
May 10 18:36:25 buildroot kern.warn kernel:  wg_packet_receive+0x4a5/0x557
May 10 18:36:25 buildroot kern.warn kernel:  ? send6+0x1fc/0x1fc
May 10 18:36:25 buildroot kern.warn kernel:  wg_receive+0x16/0x25
May 10 18:36:25 buildroot kern.warn kernel:  udp_queue_rcv_skb+0x254/0x324
May 10 18:36:25 buildroot kern.warn kernel:
udp_unicast_rcv_skb.isra.12+0x68/0x7a
May 10 18:36:25 buildroot kern.warn kernel:  __udp4_lib_rcv+0x413/0x88b
May 10 18:36:25 buildroot kern.warn kernel:  udp_rcv+0x12/0x14
May 10 18:36:25 buildroot kern.warn kernel:
ip_local_deliver_finish+0x82/0x1ff
May 10 18:36:25 buildroot kern.warn kernel:  ip_local_deliver+0xc0/0xcb
May 10 18:36:25 buildroot kern.warn kernel:  ?
ip_sublist_rcv_finish+0x41/0x41
May 10 18:36:25 buildroot kern.warn kernel:  ip_rcv_finish+0x24/0x2a
May 10 18:36:25 buildroot kern.warn kernel:  ip_rcv+0xa1/0xaf
May 10 18:36:25 buildroot kern.warn kernel:  ?
ip_rcv_finish_core.isra.0+0x331/0x331
May 10 18:36:25 buildroot kern.warn kernel:
__netif_receive_skb_one_core+0x3f/0x59
May 10 18:36:25 buildroot kern.warn kernel:  __netif_receive_skb+0x16/0x4f
May 10 18:36:25 buildroot kern.warn kernel:
netif_receive_skb_internal+0x26/0xaf
May 10 18:36:25 buildroot kern.warn kernel:  netif_receive_skb+0x8/0xa
May 10 18:36:25 buildroot kern.warn kernel:  rhine_napipoll+0x5ef/0x9c6
May 10 18:36:25 buildroot kern.warn kernel:  net_rx_action+0x197/0x24d
May 10 18:36:25 buildroot kern.warn kernel:  __do_softirq+0xd6/0x1ae
May 10 18:36:25 buildroot kern.warn kernel:  run_ksoftirqd+0x21/0x24
May 10 18:36:25 buildroot kern.warn kernel:  smpboot_thread_fn+0x137/0x1ea
May 10 18:36:25 buildroot kern.warn kernel:  kthread+0xbe/0xea
May 10 18:36:25 buildroot kern.warn kernel:  ? sort_range+0x18/0x18
May 10 18:36:25 buildroot kern.warn kernel:  ?
__kthread_create_on_node+0x13e/0x13e
May 10 18:36:25 buildroot kern.warn kernel:  ret_from_fork+0x19/0x24
May 10 18:36:25 buildroot kern.warn kernel: ---[ end trace c200a14cd22c0ee1
]---
May 10 18:36:25 buildroot kern.warn kernel: WARNING: CPU: 0 PID: 9 at
kernel/workqueue.c:1442 __queue_work+0x1d4/0x2aa
May 10 18:36:25 buildroot kern.warn kernel: CPU: 0 PID: 9 Comm: ksoftirqd/0
Tainted: G        W         4.19.41-banana #1
May 10 18:36:25 buildroot kern.warn kernel: EIP: __queue_work+0x1d4/0x2aa
May 10 18:36:25 buildroot kern.warn kernel: Code: c1 e8 4e 09 1d 00 c7 05
30 d3 5f c1 00 00 00 00 c7 45 e4 00 00 00 00 e9 76 fe ff ff 8b 3d 80 3c 60
c1 89 78 10 e9 32 ff ff ff <0f> 0b b8 01 00 00 00 e8 1c 82 00 00 a1 a8 9e
5f c1 85 c0 0f 85 3b
May 10 18:36:25 buildroot kern.warn kernel: EAX: 00000000 EBX: cfb62044
ECX: 00000003 EDX: cfb62048
May 10 18:36:25 buildroot kern.warn kernel: ESI: cfdfbb00 EDI: cfab29c0
EBP: cf847d38 ESP: cf847d1c
May 10 18:36:25 buildroot kern.warn kernel: DS: 007b ES: 007b FS: 0000 GS:
00e0 SS: 0068 EFLAGS: 00010086
May 10 18:36:25 buildroot kern.warn kernel: CR0: 80050033 CR2: a7bb5058
CR3: 0eebe000 CR4: 00000090
May 10 18:36:25 buildroot kern.warn kernel: Call Trace:
May 10 18:36:25 buildroot kern.warn kernel:  queue_work_on+0x19/0x1d
May 10 18:36:25 buildroot kern.warn kernel:  wg_packet_receive+0x4a5/0x557
May 10 18:36:25 buildroot kern.warn kernel:  ? send6+0x1fc/0x1fc
May 10 18:36:25 buildroot kern.warn kernel:  wg_receive+0x16/0x25
May 10 18:36:25 buildroot kern.warn kernel:  udp_queue_rcv_skb+0x254/0x324
May 10 18:36:25 buildroot kern.warn kernel:
udp_unicast_rcv_skb.isra.12+0x68/0x7a
May 10 18:36:25 buildroot kern.warn kernel:  __udp4_lib_rcv+0x413/0x88b
May 10 18:36:25 buildroot kern.warn kernel:  udp_rcv+0x12/0x14
May 10 18:36:25 buildroot kern.warn kernel:
ip_local_deliver_finish+0x82/0x1ff
May 10 18:36:25 buildroot kern.warn kernel:  ip_local_deliver+0xc0/0xcb
May 10 18:36:25 buildroot kern.warn kernel:  ?
ip_sublist_rcv_finish+0x41/0x41
May 10 18:36:25 buildroot kern.warn kernel:  ip_rcv_finish+0x24/0x2a
May 10 18:36:25 buildroot kern.warn kernel:  ip_rcv+0xa1/0xaf
May 10 18:36:25 buildroot kern.warn kernel:  ?
ip_rcv_finish_core.isra.0+0x331/0x331
May 10 18:36:25 buildroot kern.warn kernel:
__netif_receive_skb_one_core+0x3f/0x59
May 10 18:36:25 buildroot kern.warn kernel:  __netif_receive_skb+0x16/0x4f
May 10 18:36:25 buildroot kern.warn kernel:
netif_receive_skb_internal+0x26/0xaf
May 10 18:36:25 buildroot kern.warn kernel:  netif_receive_skb+0x8/0xa
May 10 18:36:25 buildroot kern.warn kernel:  rhine_napipoll+0x5ef/0x9c6
May 10 18:36:25 buildroot kern.warn kernel:  net_rx_action+0x197/0x24d
May 10 18:36:25 buildroot kern.warn kernel:  __do_softirq+0xd6/0x1ae
May 10 18:36:25 buildroot kern.warn kernel:  run_ksoftirqd+0x21/0x24
May 10 18:36:25 buildroot kern.warn kernel:  smpboot_thread_fn+0x137/0x1ea
May 10 18:36:25 buildroot kern.warn kernel:  kthread+0xbe/0xea
May 10 18:36:25 buildroot kern.warn kernel:  ? sort_range+0x18/0x18
May 10 18:36:25 buildroot kern.warn kernel:  ?
__kthread_create_on_node+0x13e/0x13e
May 10 18:36:25 buildroot kern.warn kernel:  ret_from_fork+0x19/0x24
May 10 18:36:25 buildroot kern.warn kernel: ---[ end trace c200a14cd22c0ee2
]---

[-- Attachment #1.2: Type: text/html, Size: 7700 bytes --]

<div dir="ltr"><div dir="ltr"><div>I am building a system which coordinates the meshing of wireguard devices.  Currently, all the devices are running on an embedded platform (AMD Geode LX500) and works as expected.  However, when introducing a 64bit KVM host for testing, all the 32bit hosts running on the Geode platform, report the following warning and drop offline temporarily.  The CPU usage on KVM host spikes and all the CPU time is spent on the kernel threads servicing the WG interfaces.</div><div><br></div><div>I&#39;m using kernel 4.19.41 and have seen the issue with both the last WG snapshot as well building from master.</div><div><br></div><div>Is this a bug in WG?  </div><div><br></div><div>May 10 18:36:25 buildroot kern.warn kernel: WARNING: CPU: 0 PID: 9 at kernel/workqueue.c:1442 __queue_work+0x1d4/0x2aa</div><div>May 10 18:36:25 buildroot kern.warn kernel: CPU: 0 PID: 9 Comm: ksoftirqd/0 Tainted: G        W         4.19.41-banana #1</div><div>May 10 18:36:25 buildroot kern.warn kernel: EIP: __queue_work+0x1d4/0x2aa</div><div>May 10 18:36:25 buildroot kern.warn kernel: Code: c1 e8 4e 09 1d 00 c7 05 30 d3 5f c1 00 00 00 00 c7 45 e4 00 00 00 00 e9 76 fe ff ff 8b 3d 80 3c 60 c1 89 78 10 e9 32 ff ff ff &lt;0f&gt; 0b b8 01 00 00 00 e8 1c 82 00 00 a1 a8 9e 5f c1 85 c0 0f 85 3b</div><div>May 10 18:36:25 buildroot kern.warn kernel: EAX: 00000000 EBX: cfb63044 ECX: 00000003 EDX: cfb63048</div><div>May 10 18:36:25 buildroot kern.warn kernel: ESI: cfdfbb00 EDI: cfab29c0 EBP: cf847d38 ESP: cf847d1c</div><div>May 10 18:36:25 buildroot kern.warn kernel: DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010086</div><div>May 10 18:36:25 buildroot kern.warn kernel: CR0: 80050033 CR2: a7bb5058 CR3: 0eebe000 CR4: 00000090</div><div>May 10 18:36:25 buildroot kern.warn kernel: Call Trace:</div><div>May 10 18:36:25 buildroot kern.warn kernel:  queue_work_on+0x19/0x1d</div><div>May 10 18:36:25 buildroot kern.warn kernel:  wg_packet_receive+0x4a5/0x557</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? send6+0x1fc/0x1fc</div><div>May 10 18:36:25 buildroot kern.warn kernel:  wg_receive+0x16/0x25</div><div>May 10 18:36:25 buildroot kern.warn kernel:  udp_queue_rcv_skb+0x254/0x324</div><div>May 10 18:36:25 buildroot kern.warn kernel:  udp_unicast_rcv_skb.isra.12+0x68/0x7a</div><div>May 10 18:36:25 buildroot kern.warn kernel:  __udp4_lib_rcv+0x413/0x88b</div><div>May 10 18:36:25 buildroot kern.warn kernel:  udp_rcv+0x12/0x14</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ip_local_deliver_finish+0x82/0x1ff</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ip_local_deliver+0xc0/0xcb</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? ip_sublist_rcv_finish+0x41/0x41</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ip_rcv_finish+0x24/0x2a</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ip_rcv+0xa1/0xaf</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? ip_rcv_finish_core.isra.0+0x331/0x331</div><div>May 10 18:36:25 buildroot kern.warn kernel:  __netif_receive_skb_one_core+0x3f/0x59</div><div>May 10 18:36:25 buildroot kern.warn kernel:  __netif_receive_skb+0x16/0x4f</div><div>May 10 18:36:25 buildroot kern.warn kernel:  netif_receive_skb_internal+0x26/0xaf</div><div>May 10 18:36:25 buildroot kern.warn kernel:  netif_receive_skb+0x8/0xa</div><div>May 10 18:36:25 buildroot kern.warn kernel:  rhine_napipoll+0x5ef/0x9c6</div><div>May 10 18:36:25 buildroot kern.warn kernel:  net_rx_action+0x197/0x24d</div><div>May 10 18:36:25 buildroot kern.warn kernel:  __do_softirq+0xd6/0x1ae</div><div>May 10 18:36:25 buildroot kern.warn kernel:  run_ksoftirqd+0x21/0x24</div><div>May 10 18:36:25 buildroot kern.warn kernel:  smpboot_thread_fn+0x137/0x1ea</div><div>May 10 18:36:25 buildroot kern.warn kernel:  kthread+0xbe/0xea</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? sort_range+0x18/0x18</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? __kthread_create_on_node+0x13e/0x13e</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ret_from_fork+0x19/0x24</div><div>May 10 18:36:25 buildroot kern.warn kernel: ---[ end trace c200a14cd22c0ee1 ]---</div><div>May 10 18:36:25 buildroot kern.warn kernel: WARNING: CPU: 0 PID: 9 at kernel/workqueue.c:1442 __queue_work+0x1d4/0x2aa</div><div>May 10 18:36:25 buildroot kern.warn kernel: CPU: 0 PID: 9 Comm: ksoftirqd/0 Tainted: G        W         4.19.41-banana #1</div><div>May 10 18:36:25 buildroot kern.warn kernel: EIP: __queue_work+0x1d4/0x2aa</div><div>May 10 18:36:25 buildroot kern.warn kernel: Code: c1 e8 4e 09 1d 00 c7 05 30 d3 5f c1 00 00 00 00 c7 45 e4 00 00 00 00 e9 76 fe ff ff 8b 3d 80 3c 60 c1 89 78 10 e9 32 ff ff ff &lt;0f&gt; 0b b8 01 00 00 00 e8 1c 82 00 00 a1 a8 9e 5f c1 85 c0 0f 85 3b</div><div>May 10 18:36:25 buildroot kern.warn kernel: EAX: 00000000 EBX: cfb62044 ECX: 00000003 EDX: cfb62048</div><div>May 10 18:36:25 buildroot kern.warn kernel: ESI: cfdfbb00 EDI: cfab29c0 EBP: cf847d38 ESP: cf847d1c</div><div>May 10 18:36:25 buildroot kern.warn kernel: DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010086</div><div>May 10 18:36:25 buildroot kern.warn kernel: CR0: 80050033 CR2: a7bb5058 CR3: 0eebe000 CR4: 00000090</div><div>May 10 18:36:25 buildroot kern.warn kernel: Call Trace:</div><div>May 10 18:36:25 buildroot kern.warn kernel:  queue_work_on+0x19/0x1d</div><div>May 10 18:36:25 buildroot kern.warn kernel:  wg_packet_receive+0x4a5/0x557</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? send6+0x1fc/0x1fc</div><div>May 10 18:36:25 buildroot kern.warn kernel:  wg_receive+0x16/0x25</div><div>May 10 18:36:25 buildroot kern.warn kernel:  udp_queue_rcv_skb+0x254/0x324</div><div>May 10 18:36:25 buildroot kern.warn kernel:  udp_unicast_rcv_skb.isra.12+0x68/0x7a</div><div>May 10 18:36:25 buildroot kern.warn kernel:  __udp4_lib_rcv+0x413/0x88b</div><div>May 10 18:36:25 buildroot kern.warn kernel:  udp_rcv+0x12/0x14</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ip_local_deliver_finish+0x82/0x1ff</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ip_local_deliver+0xc0/0xcb</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? ip_sublist_rcv_finish+0x41/0x41</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ip_rcv_finish+0x24/0x2a</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ip_rcv+0xa1/0xaf</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? ip_rcv_finish_core.isra.0+0x331/0x331</div><div>May 10 18:36:25 buildroot kern.warn kernel:  __netif_receive_skb_one_core+0x3f/0x59</div><div>May 10 18:36:25 buildroot kern.warn kernel:  __netif_receive_skb+0x16/0x4f</div><div>May 10 18:36:25 buildroot kern.warn kernel:  netif_receive_skb_internal+0x26/0xaf</div><div>May 10 18:36:25 buildroot kern.warn kernel:  netif_receive_skb+0x8/0xa</div><div>May 10 18:36:25 buildroot kern.warn kernel:  rhine_napipoll+0x5ef/0x9c6</div><div>May 10 18:36:25 buildroot kern.warn kernel:  net_rx_action+0x197/0x24d</div><div>May 10 18:36:25 buildroot kern.warn kernel:  __do_softirq+0xd6/0x1ae</div><div>May 10 18:36:25 buildroot kern.warn kernel:  run_ksoftirqd+0x21/0x24</div><div>May 10 18:36:25 buildroot kern.warn kernel:  smpboot_thread_fn+0x137/0x1ea</div><div>May 10 18:36:25 buildroot kern.warn kernel:  kthread+0xbe/0xea</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? sort_range+0x18/0x18</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ? __kthread_create_on_node+0x13e/0x13e</div><div>May 10 18:36:25 buildroot kern.warn kernel:  ret_from_fork+0x19/0x24</div><div>May 10 18:36:25 buildroot kern.warn kernel: ---[ end trace c200a14cd22c0ee2 ]---</div><div><br></div></div></div>

[-- Attachment #2: Type: text/plain, Size: 148 bytes --]

_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

             reply index

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-05-12 13:44 Ryan Whelan [this message]
2019-05-12 15:41 ` Jason A. Donenfeld
2019-05-12 23:02 ` Lonnie Abelbeck
2019-06-14 11:56 ` Jason A. Donenfeld
2019-05-17  6:34 WireGuard Bug? . .
2019-05-18 17:03 ` Lucian Cristian

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAM3m09Q3LCCFRJEe6J1UO0mtTiuVOYEiiyhEJJppDnuDnZZTwA@mail.gmail.com \
    --to=rcwhelan@gmail.com \
    --cc=wireguard@lists.zx2c4.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

WireGuard Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/wireguard/0 wireguard/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 wireguard wireguard/ https://lore.kernel.org/wireguard \
		wireguard@lists.zx2c4.com zx2c4-wireguard@archiver.kernel.org
	public-inbox-index wireguard


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/com.zx2c4.lists.wireguard


AGPL code for this site: git clone https://public-inbox.org/ public-inbox