WireGuard Archive on lore.kernel.org
 help / color / Atom feed
* Kernel module sends infinite netlink messages on v0.0.20180802
@ 2018-08-08 19:36 Matt Layher
  2018-08-09  0:11 ` Jason A. Donenfeld
  0 siblings, 1 reply; 5+ messages in thread
From: Matt Layher @ 2018-08-08 19:36 UTC (permalink / raw)
  To: WireGuard mailing list

Hi all,

While working on wireguardctrl, I found what I believe to be a bug with 
the kernel module today.  I'm using v0.0.20180802.  At first I assumed 
that my code was doing something wrong, but I'm able to make "wg show" 
hang forever as well, so I believe this to be a problem with the kernel 
module itself.

System information:

matt@nerr-2:~$ dmesg | grep wireguard
[ 1075.085912] wireguard: module verification failed: signature and/or 
required key missing - tainting kernel
[ 1075.086235] wireguard: WireGuard 0.0.20180802 loaded. See 
www.wireguard.com for information.
[ 1075.086235] wireguard: Copyright (C) 2015-2018 Jason A. Donenfeld 
<Jason@zx2c4.com>. All Rights Reserved.

matt@nerr-2:~$ uname -a
Linux nerr-2 4.15.0-30-generic #32-Ubuntu SMP Thu Jul 26 17:42:43 UTC 
2018 x86_64 x86_64 x86_64 GNU/Linux

Here are the steps to reproduce the issue:

Grab my "wgnlbug" Go source program and build it: 
https://github.com/mdlayher/wireguardctrl/blob/master/cmd/wgnlbug/main.go

$ go install github.com/mdlayher/wireguardctrl/cmd/wgnlbug

Reset wg0 to a clean state:

$ sudo ip link del dev wg0 && sudo ip link add dev wg0 type wireguard

Attempt to add multiple peers with 511 addresses each (the actual CIDR 
is hard-coded for both and doesn't seem to matter).  Note that you have 
to Ctrl+C the program or it'll hang forever.

$ sudo time ./bin/wgnlbug -n 2
before: wg0
^CCommand terminated by signal 2
1.29user 2.62system 0:02.74elapsed 142%CPU (0avgtext+0avgdata 
385236maxresident)k
0inputs+0outputs (0major+98292minor)pagefaults 0swaps

At this point, "wg show" appears to hang forever until something sends 
it a KILL (kernel maybe?) as well:

$ sudo time wg show
Command terminated by signal 9
20.88user 40.39system 1:03.31elapsed 96%CPU (0avgtext+0avgdata 
12233204maxresident)k
16128inputs+0outputs (92major+3058349minor)pagefaults 0swaps

A look at strace reveals what appears to be an infinite stream of 
multi-part netlink messages with identical sequence numbers:

$ sudo strace wg show
...
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4072, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\x06\x00\x06\x00\x00\x00\x00\x00\x08\x00\x07\x00\x00\x00\x00\x00\x08\x00\x01\x00\x81\x00\x00\x00\x08\x00\x02\x00"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4072
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4068, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\xd0\x0f\x08\x00\xcc\x0f\x00\x00\x24\x00\x01\x00\xc6\x24\x8a\x34\xcc\x3c\x4a\x23\x00\xd4\x94\x8d\xec\x58\xc6\x7c"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4068
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4072, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\x06\x00\x06\x00\x00\x00\x00\x00\x08\x00\x07\x00\x00\x00\x00\x00\x08\x00\x01\x00\x81\x00\x00\x00\x08\x00\x02\x00"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4072
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4068, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\xd0\x0f\x08\x00\xcc\x0f\x00\x00\x24\x00\x01\x00\xc6\x24\x8a\x34\xcc\x3c\x4a\x23\x00\xd4\x94\x8d\xec\x58\xc6\x7c"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4068
recvmsg(3, ^C{msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4072, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\x06\x00\x06\x00\x00\x00\x00\x00\x08\x00\x07\x00\x00\x00\x00\x00\x08\x00\x01\x00\x81\x00\x00\x00\x08\x00\x02\x00"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4072
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
strace: Process 946 detached

Hope this is helpful.  If it isn't a kernel module problem, I'd be 
curious to see what both my code and "wg" are doing that causes this.  
It seems to be reproducible 100% of the time on my system.

- Matt Layher

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Kernel module sends infinite netlink messages on v0.0.20180802
  2018-08-08 19:36 Kernel module sends infinite netlink messages on v0.0.20180802 Matt Layher
@ 2018-08-09  0:11 ` Jason A. Donenfeld
  2018-08-09  0:30   ` Matt Layher
  0 siblings, 1 reply; 5+ messages in thread
From: Jason A. Donenfeld @ 2018-08-09  0:11 UTC (permalink / raw)
  To: Matt Layher; +Cc: WireGuard mailing list

Thanks for letting me know. I've got a simpler reproducer now:

    #!/bin/bash

    a=( )
    for i in {1..197}; do
            a+=( abcd::$i )
    done

    s="$IFS"
    IFS=,
    a="${a[*]}"
    IFS="$s"

    ip link del wg0
    ip link add wg0 type wireguard
    wg set wg0 peer "$(wg genkey)" allowed-ips "$a"
    wg set wg0 peer "$(wg genkey)" allowed-ips "$a"
    wg

I'll have this fixed shortly and will ping back on this thread.

Jason

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Kernel module sends infinite netlink messages on v0.0.20180802
  2018-08-09  0:11 ` Jason A. Donenfeld
@ 2018-08-09  0:30   ` Matt Layher
  2018-08-09  2:20     ` Jason A. Donenfeld
  0 siblings, 1 reply; 5+ messages in thread
From: Matt Layher @ 2018-08-09  0:30 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

Excellent! That's much more concise.

- Matt

On Wed, Aug 8, 2018, 8:11 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:

> Thanks for letting me know. I've got a simpler reproducer now:
>
>     #!/bin/bash
>
>     a=( )
>     for i in {1..197}; do
>             a+=( abcd::$i )
>     done
>
>     s="$IFS"
>     IFS=,
>     a="${a[*]}"
>     IFS="$s"
>
>     ip link del wg0
>     ip link add wg0 type wireguard
>     wg set wg0 peer "$(wg genkey)" allowed-ips "$a"
>     wg set wg0 peer "$(wg genkey)" allowed-ips "$a"
>     wg
>
> I'll have this fixed shortly and will ping back on this thread.
>
> Jason
>

[-- Attachment #2: Type: text/html, Size: 1079 bytes --]

<div dir="auto">Excellent! That&#39;s much more concise.<div dir="auto"><br></div><div dir="auto">- Matt</div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Aug 8, 2018, 8:11 PM Jason A. Donenfeld &lt;<a href="mailto:Jason@zx2c4.com">Jason@zx2c4.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks for letting me know. I&#39;ve got a simpler reproducer now:<br>
<br>
    #!/bin/bash<br>
<br>
    a=( )<br>
    for i in {1..197}; do<br>
            a+=( abcd::$i )<br>
    done<br>
<br>
    s=&quot;$IFS&quot;<br>
    IFS=,<br>
    a=&quot;${a[*]}&quot;<br>
    IFS=&quot;$s&quot;<br>
<br>
    ip link del wg0<br>
    ip link add wg0 type wireguard<br>
    wg set wg0 peer &quot;$(wg genkey)&quot; allowed-ips &quot;$a&quot;<br>
    wg set wg0 peer &quot;$(wg genkey)&quot; allowed-ips &quot;$a&quot;<br>
    wg<br>
<br>
I&#39;ll have this fixed shortly and will ping back on this thread.<br>
<br>
Jason<br>
</blockquote></div>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Kernel module sends infinite netlink messages on v0.0.20180802
  2018-08-09  0:30   ` Matt Layher
@ 2018-08-09  2:20     ` Jason A. Donenfeld
  2018-08-09  3:54       ` Matt Layher
  0 siblings, 1 reply; 5+ messages in thread
From: Jason A. Donenfeld @ 2018-08-09  2:20 UTC (permalink / raw)
  To: Matt Layher; +Cc: WireGuard mailing list

On Wed, Aug 8, 2018 at 5:30 PM Matt Layher <mdlayher@gmail.com> wrote:
>
> Excellent! That's much more concise.

Let me know if this fixes it for you, and please do keep messing with
weird cases to see if you can find more bugs. I really appreciate you
finding this.

https://git.zx2c4.com/WireGuard/commit/?id=fd60e07ba3e294b94985a42d11afebf55f1d8829

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Kernel module sends infinite netlink messages on v0.0.20180802
  2018-08-09  2:20     ` Jason A. Donenfeld
@ 2018-08-09  3:54       ` Matt Layher
  0 siblings, 0 replies; 5+ messages in thread
From: Matt Layher @ 2018-08-09  3:54 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

I can confirm that this is fixed for me as of latest master:

$ dmesg | grep wireguard
[   50.396241] wireguard: module verification failed: signature and/or 
required key missing - tainting kernel
[   50.396675] wireguard: WireGuard 0.0.20180802-11-gc6505ee loaded. See 
www.wireguard.com for information.
[   50.396675] wireguard: Copyright (C) 2015-2018 Jason A. Donenfeld 
<Jason@zx2c4.com>. All Rights Reserved.

$ sudo ip link add dev wg0 type wireguard

$ sudo ./wgnlbug -n 2
before: wg0
  after: wg0
- peer: ZoJIpwr1iel/9emt2bNlnHhvasjZdmUD6v92Ry8z1Ro=: 0 IPs
- peer: y84s8m/91ryGV8tTQbycauYcukCjrAG1B8vx44BsxWM=: 511 IPs

$ sudo wg show
interface: wg0

peer: ZoJIpwr1iel/9emt2bNlnHhvasjZdmUD6v92Ry8z1Ro=
   allowed ips: (none)

peer: y84s8m/91ryGV8tTQbycauYcukCjrAG1B8vx44BsxWM=
   allowed ips: 2001:db8::1ff/128, 2001:db8::1fe/128, ...

Thanks for the quick patch.  I started with a pretty naive approach on 
my netlink message chunking implementation, but I'm glad I was able to 
help find a problem that way.

I'll be sure to report anything else I find, but at this point, I think 
I'm feature-complete for both userspace and kernel APIs.

- Matt


On 08/08/2018 10:20 PM, Jason A. Donenfeld wrote:
> On Wed, Aug 8, 2018 at 5:30 PM Matt Layher <mdlayher@gmail.com> wrote:
>> Excellent! That's much more concise.
> Let me know if this fixes it for you, and please do keep messing with
> weird cases to see if you can find more bugs. I really appreciate you
> finding this.
>
> https://git.zx2c4.com/WireGuard/commit/?id=fd60e07ba3e294b94985a42d11afebf55f1d8829

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, back to index

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-08 19:36 Kernel module sends infinite netlink messages on v0.0.20180802 Matt Layher
2018-08-09  0:11 ` Jason A. Donenfeld
2018-08-09  0:30   ` Matt Layher
2018-08-09  2:20     ` Jason A. Donenfeld
2018-08-09  3:54       ` Matt Layher

WireGuard Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/wireguard/0 wireguard/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 wireguard wireguard/ https://lore.kernel.org/wireguard \
		wireguard@lists.zx2c4.com zx2c4-wireguard@archiver.kernel.org
	public-inbox-index wireguard


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/com.zx2c4.lists.wireguard


AGPL code for this site: git clone https://public-inbox.org/ public-inbox