Kernel module sends infinite netlink messages on v0.0.20180802

Kernel module sends infinite netlink messages on v0.0.20180802

[Matt Layher]

Hi all,

While working on wireguardctrl, I found what I believe to be a bug with 
the kernel module today.  I'm using v0.0.20180802.  At first I assumed 
that my code was doing something wrong, but I'm able to make "wg show" 
hang forever as well, so I believe this to be a problem with the kernel 
module itself.

System information:

matt@nerr-2:~$ dmesg | grep wireguard
[ 1075.085912] wireguard: module verification failed: signature and/or 
required key missing - tainting kernel
[ 1075.086235] wireguard: WireGuard 0.0.20180802 loaded. See 
www.wireguard.com for information.
[ 1075.086235] wireguard: Copyright (C) 2015-2018 Jason A. Donenfeld 
<Jason@zx2c4.com>. All Rights Reserved.

matt@nerr-2:~$ uname -a
Linux nerr-2 4.15.0-30-generic #32-Ubuntu SMP Thu Jul 26 17:42:43 UTC 
2018 x86_64 x86_64 x86_64 GNU/Linux

Here are the steps to reproduce the issue:

Grab my "wgnlbug" Go source program and build it: 
https://github.com/mdlayher/wireguardctrl/blob/master/cmd/wgnlbug/main.go

$ go install github.com/mdlayher/wireguardctrl/cmd/wgnlbug

Reset wg0 to a clean state:

$ sudo ip link del dev wg0 && sudo ip link add dev wg0 type wireguard

Attempt to add multiple peers with 511 addresses each (the actual CIDR 
is hard-coded for both and doesn't seem to matter).  Note that you have 
to Ctrl+C the program or it'll hang forever.

$ sudo time ./bin/wgnlbug -n 2
before: wg0
^CCommand terminated by signal 2
1.29user 2.62system 0:02.74elapsed 142%CPU (0avgtext+0avgdata 
385236maxresident)k
0inputs+0outputs (0major+98292minor)pagefaults 0swaps

At this point, "wg show" appears to hang forever until something sends 
it a KILL (kernel maybe?) as well:

$ sudo time wg show
Command terminated by signal 9
20.88user 40.39system 1:03.31elapsed 96%CPU (0avgtext+0avgdata 
12233204maxresident)k
16128inputs+0outputs (92major+3058349minor)pagefaults 0swaps

A look at strace reveals what appears to be an infinite stream of 
multi-part netlink messages with identical sequence numbers:

$ sudo strace wg show
...
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4072, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\x06\x00\x06\x00\x00\x00\x00\x00\x08\x00\x07\x00\x00\x00\x00\x00\x08\x00\x01\x00\x81\x00\x00\x00\x08\x00\x02\x00"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4072
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4068, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\xd0\x0f\x08\x00\xcc\x0f\x00\x00\x24\x00\x01\x00\xc6\x24\x8a\x34\xcc\x3c\x4a\x23\x00\xd4\x94\x8d\xec\x58\xc6\x7c"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4068
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4072, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\x06\x00\x06\x00\x00\x00\x00\x00\x08\x00\x07\x00\x00\x00\x00\x00\x08\x00\x01\x00\x81\x00\x00\x00\x08\x00\x02\x00"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4072
recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4068, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\xd0\x0f\x08\x00\xcc\x0f\x00\x00\x24\x00\x01\x00\xc6\x24\x8a\x34\xcc\x3c\x4a\x23\x00\xd4\x94\x8d\xec\x58\xc6\x7c"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4068
recvmsg(3, ^C{msg_name={sa_family=AF_NETLINK, nl_pid=0, 
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=4072, 
type=wireguard, flags=NLM_F_MULTI, seq=1533756618, pid=946}, 
"\x00\x01\x00\x00\x06\x00\x06\x00\x00\x00\x00\x00\x08\x00\x07\x00\x00\x00\x00\x00\x08\x00\x01\x00\x81\x00\x00\x00\x08\x00\x02\x00"...}, 
iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 4072
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
strace: Process 946 detached

Hope this is helpful.  If it isn't a kernel module problem, I'd be 
curious to see what both my code and "wg" are doing that causes this.  
It seems to be reproducible 100% of the time on my system.

- Matt Layher

Re: Kernel module sends infinite netlink messages on v0.0.20180802

[Jason A. Donenfeld]

Thanks for letting me know. I've got a simpler reproducer now:

    #!/bin/bash

    a=( )
    for i in {1..197}; do
            a+=( abcd::$i )
    done

    s="$IFS"
    IFS=,
    a="${a[*]}"
    IFS="$s"

    ip link del wg0
    ip link add wg0 type wireguard
    wg set wg0 peer "$(wg genkey)" allowed-ips "$a"
    wg set wg0 peer "$(wg genkey)" allowed-ips "$a"
    wg

I'll have this fixed shortly and will ping back on this thread.

Jason

Re: Kernel module sends infinite netlink messages on v0.0.20180802

[Matt Layher]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

Excellent! That's much more concise.

- Matt

On Wed, Aug 8, 2018, 8:11 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:

> Thanks for letting me know. I've got a simpler reproducer now:
>
>     #!/bin/bash
>
>     a=( )
>     for i in {1..197}; do
>             a+=( abcd::$i )
>     done
>
>     s="$IFS"
>     IFS=,
>     a="${a[*]}"
>     IFS="$s"
>
>     ip link del wg0
>     ip link add wg0 type wireguard
>     wg set wg0 peer "$(wg genkey)" allowed-ips "$a"
>     wg set wg0 peer "$(wg genkey)" allowed-ips "$a"
>     wg
>
> I'll have this fixed shortly and will ping back on this thread.
>
> Jason
>

[-- Attachment #2: Type: text/html, Size: 1079 bytes --]

Re: Kernel module sends infinite netlink messages on v0.0.20180802

[Jason A. Donenfeld]

On Wed, Aug 8, 2018 at 5:30 PM Matt Layher <mdlayher@gmail.com> wrote:
>
> Excellent! That's much more concise.

Let me know if this fixes it for you, and please do keep messing with
weird cases to see if you can find more bugs. I really appreciate you
finding this.

https://git.zx2c4.com/WireGuard/commit/?id=fd60e07ba3e294b94985a42d11afebf55f1d8829

Re: Kernel module sends infinite netlink messages on v0.0.20180802

[Matt Layher]

I can confirm that this is fixed for me as of latest master:

$ dmesg | grep wireguard
[   50.396241] wireguard: module verification failed: signature and/or 
required key missing - tainting kernel
[   50.396675] wireguard: WireGuard 0.0.20180802-11-gc6505ee loaded. See 
www.wireguard.com for information.
[   50.396675] wireguard: Copyright (C) 2015-2018 Jason A. Donenfeld 
<Jason@zx2c4.com>. All Rights Reserved.

$ sudo ip link add dev wg0 type wireguard

$ sudo ./wgnlbug -n 2
before: wg0
  after: wg0
- peer: ZoJIpwr1iel/9emt2bNlnHhvasjZdmUD6v92Ry8z1Ro=: 0 IPs
- peer: y84s8m/91ryGV8tTQbycauYcukCjrAG1B8vx44BsxWM=: 511 IPs

$ sudo wg show
interface: wg0

peer: ZoJIpwr1iel/9emt2bNlnHhvasjZdmUD6v92Ry8z1Ro=
   allowed ips: (none)

peer: y84s8m/91ryGV8tTQbycauYcukCjrAG1B8vx44BsxWM=
   allowed ips: 2001:db8::1ff/128, 2001:db8::1fe/128, ...

Thanks for the quick patch.  I started with a pretty naive approach on 
my netlink message chunking implementation, but I'm glad I was able to 
help find a problem that way.

I'll be sure to report anything else I find, but at this point, I think 
I'm feature-complete for both userspace and kernel APIs.

- Matt


On 08/08/2018 10:20 PM, Jason A. Donenfeld wrote:
> On Wed, Aug 8, 2018 at 5:30 PM Matt Layher <mdlayher@gmail.com> wrote:
>> Excellent! That's much more concise.
> Let me know if this fixes it for you, and please do keep messing with
> weird cases to see if you can find more bugs. I really appreciate you
> finding this.
>
> https://git.zx2c4.com/WireGuard/commit/?id=fd60e07ba3e294b94985a42d11afebf55f1d8829