Problems on linux-image-4.15.0-1047-aws

Problems on linux-image-4.15.0-1047-aws

[Johnny Rose Carlsen]

I am running Ubuntu 18.04 on Amazon AWS, and I just ran a dist-upgrade
that upgraded my kernel from linux-image-4.15.0-1041-aws to
linux-image-4.15.0-1047-aws, and Wireguard is now failing with the
following error:

[   50.640515] BUG: unable to handle kernel NULL pointer dereference
at 000000000000054a
[   50.647808] IP: skb_set_owner_w+0x30/0xb0
[   50.651708] PGD 0 P4D 0
[   50.654476] Oops: 0000 [#1] SMP PTI
[   50.657963] Modules linked in: wireguard(OE) ip6_udp_tunnel
udp_tunnel serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core
iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables
x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1
raid0 multipath linear crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd
glue_helper cryptd
[   50.691945] CPU: 0 PID: 19 Comm: kworker/0:1 Tainted: G
OE    4.15.0-1047-aws #49-Ubuntu
[   50.698795] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[   50.702783] Workqueue: wg-crypt-wg21 wg_packet_decrypt_worker [wireguard]
[   50.706986] RIP: 0010:skb_set_owner_w+0x30/0xb0
[   50.709950] RSP: 0018:ffffaf96c0223a70 EFLAGS: 00010246
[   50.713515] RAX: 0000000000000000 RBX: ffff979d1ebeba00 RCX: 0000000000000000
[   50.717860] RDX: ffff979d19646000 RSI: 0000000000000538 RDI: ffff979d1ebeba00
[   50.723405] RBP: ffffaf96c0223a80 R08: 00000000000006c0 R09: ffff979d1cc02d80
[   50.729688] R10: ffffaf96c0223970 R11: ffff979d1cc02d80 R12: 0000000000000538
[   50.736138] R13: 0000000000000000 R14: ffff979d1ebeba00 R15: ffff979d184fe100
[   50.743111] FS:  0000000000000000(0000) GS:ffff979d1f200000(0000)
knlGS:0000000000000000
[   50.750900] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   50.756707] CR2: 000000000000054a CR3: 0000000015e0a006 CR4: 00000000001606f0
[   50.763817] Call Trace:
[   50.766859]  skb_cow_data+0x9e/0x340
[   50.770908]  decrypt_packet+0xd1/0x1d0 [wireguard]
[   50.775834]  ? ip_finish_output+0x198/0x270
[   50.780265]  ? ip_finish_output+0x198/0x270
[   50.784982]  ? ip_output+0x70/0xe0
[   50.788772]  ? __update_load_avg_se.isra.37+0x1c0/0x1d0
[   50.794426]  ? prandom_u32+0x18/0x20
[   50.798727]  ? ip_local_out+0x3b/0x50
[   50.803031]  ? iptunnel_xmit+0x154/0x1f0
[   50.807233]  ? udp_tunnel_xmit_skb+0xef/0x110 [udp_tunnel]
[   50.812613]  ? send4+0x129/0x380 [wireguard]
[   50.817185]  ? send4+0x13e/0x380 [wireguard]
[   50.821825]  ? _cond_resched+0x19/0x40
[   50.826004]  ? __kmalloc_node+0x21b/0x2c0
[   50.830431]  ? kvmalloc_node+0x75/0x80
[   50.834761]  ? _cond_resched+0x19/0x40
[   50.838897]  ? kvmalloc_node+0x75/0x80
[   50.842989]  ? alloc_bucket_locks.isra.14+0x89/0xf0
[   50.847874]  ? call_rcu_sched+0x17/0x20
[   50.851553]  ? rht_deferred_worker+0x31c/0x400
[   50.855225]  wg_packet_decrypt_worker+0x90/0x180 [wireguard]
[   50.861068]  process_one_work+0x1de/0x420
[   50.865485]  worker_thread+0x32/0x410
[   50.869620]  kthread+0x121/0x140
[   50.873241]  ? process_one_work+0x420/0x420
[   50.877528]  ? kthread_create_worker_on_cpu+0x70/0x70
[   50.882591]  ret_from_fork+0x35/0x40
[   50.886446] Code: 48 8b 47 60 55 48 89 e5 41 54 53 48 85 c0 48 89
fb 49 89 f4 0f 84 83 00 00 00 e8 2c c3 41 00 48 c7 43 60 00 00 00 00
4c 89 63 18 <41> 0f b6 4c 24 12 b8 01 00 00 00 d3 e0 a9 bf ef ff ff 74
45 48
[   50.903567] RIP: skb_set_owner_w+0x30/0xb0 RSP: ffffaf96c0223a70
[   50.909341] CR2: 000000000000054a
[   50.913177] ---[ end trace e139c84255b1aea4 ]---

The Wireguard version is:
0.0.20190702-wg1~bionic

Please let me know what other information could be helpful to troubleshoot this.

-- 
Best Regards,
Johnny Carlsen
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

Re: Problems on linux-image-4.15.0-1047-aws

[Jason A. Donenfeld]

Hi Johnny,

Thanks for the report. Can you confirm whether or not you're running
the latest kernel offered by `apt update && apt upgrade`? If you are,
then perhaps Sultan (CC'd) will be be interested to learn about Ubuntu
4.15 breakage.

Jason

On Fri, Sep 6, 2019 at 1:09 PM Johnny Rose Carlsen <jrcarlsen@gmail.com> wrote:
>
> I am running Ubuntu 18.04 on Amazon AWS, and I just ran a dist-upgrade
> that upgraded my kernel from linux-image-4.15.0-1041-aws to
> linux-image-4.15.0-1047-aws, and Wireguard is now failing with the
> following error:
>
> [   50.640515] BUG: unable to handle kernel NULL pointer dereference
> at 000000000000054a
> [   50.647808] IP: skb_set_owner_w+0x30/0xb0
> [   50.651708] PGD 0 P4D 0
> [   50.654476] Oops: 0000 [#1] SMP PTI
> [   50.657963] Modules linked in: wireguard(OE) ip6_udp_tunnel
> udp_tunnel serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core
> iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables
> x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1
> raid0 multipath linear crct10dif_pclmul crc32_pclmul
> ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd
> glue_helper cryptd
> [   50.691945] CPU: 0 PID: 19 Comm: kworker/0:1 Tainted: G
> OE    4.15.0-1047-aws #49-Ubuntu
> [   50.698795] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
> [   50.702783] Workqueue: wg-crypt-wg21 wg_packet_decrypt_worker [wireguard]
> [   50.706986] RIP: 0010:skb_set_owner_w+0x30/0xb0
> [   50.709950] RSP: 0018:ffffaf96c0223a70 EFLAGS: 00010246
> [   50.713515] RAX: 0000000000000000 RBX: ffff979d1ebeba00 RCX: 0000000000000000
> [   50.717860] RDX: ffff979d19646000 RSI: 0000000000000538 RDI: ffff979d1ebeba00
> [   50.723405] RBP: ffffaf96c0223a80 R08: 00000000000006c0 R09: ffff979d1cc02d80
> [   50.729688] R10: ffffaf96c0223970 R11: ffff979d1cc02d80 R12: 0000000000000538
> [   50.736138] R13: 0000000000000000 R14: ffff979d1ebeba00 R15: ffff979d184fe100
> [   50.743111] FS:  0000000000000000(0000) GS:ffff979d1f200000(0000)
> knlGS:0000000000000000
> [   50.750900] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   50.756707] CR2: 000000000000054a CR3: 0000000015e0a006 CR4: 00000000001606f0
> [   50.763817] Call Trace:
> [   50.766859]  skb_cow_data+0x9e/0x340
> [   50.770908]  decrypt_packet+0xd1/0x1d0 [wireguard]
> [   50.775834]  ? ip_finish_output+0x198/0x270
> [   50.780265]  ? ip_finish_output+0x198/0x270
> [   50.784982]  ? ip_output+0x70/0xe0
> [   50.788772]  ? __update_load_avg_se.isra.37+0x1c0/0x1d0
> [   50.794426]  ? prandom_u32+0x18/0x20
> [   50.798727]  ? ip_local_out+0x3b/0x50
> [   50.803031]  ? iptunnel_xmit+0x154/0x1f0
> [   50.807233]  ? udp_tunnel_xmit_skb+0xef/0x110 [udp_tunnel]
> [   50.812613]  ? send4+0x129/0x380 [wireguard]
> [   50.817185]  ? send4+0x13e/0x380 [wireguard]
> [   50.821825]  ? _cond_resched+0x19/0x40
> [   50.826004]  ? __kmalloc_node+0x21b/0x2c0
> [   50.830431]  ? kvmalloc_node+0x75/0x80
> [   50.834761]  ? _cond_resched+0x19/0x40
> [   50.838897]  ? kvmalloc_node+0x75/0x80
> [   50.842989]  ? alloc_bucket_locks.isra.14+0x89/0xf0
> [   50.847874]  ? call_rcu_sched+0x17/0x20
> [   50.851553]  ? rht_deferred_worker+0x31c/0x400
> [   50.855225]  wg_packet_decrypt_worker+0x90/0x180 [wireguard]
> [   50.861068]  process_one_work+0x1de/0x420
> [   50.865485]  worker_thread+0x32/0x410
> [   50.869620]  kthread+0x121/0x140
> [   50.873241]  ? process_one_work+0x420/0x420
> [   50.877528]  ? kthread_create_worker_on_cpu+0x70/0x70
> [   50.882591]  ret_from_fork+0x35/0x40
> [   50.886446] Code: 48 8b 47 60 55 48 89 e5 41 54 53 48 85 c0 48 89
> fb 49 89 f4 0f 84 83 00 00 00 e8 2c c3 41 00 48 c7 43 60 00 00 00 00
> 4c 89 63 18 <41> 0f b6 4c 24 12 b8 01 00 00 00 d3 e0 a9 bf ef ff ff 74
> 45 48
> [   50.903567] RIP: skb_set_owner_w+0x30/0xb0 RSP: ffffaf96c0223a70
> [   50.909341] CR2: 000000000000054a
> [   50.913177] ---[ end trace e139c84255b1aea4 ]---
>
> The Wireguard version is:
> 0.0.20190702-wg1~bionic
>
> Please let me know what other information could be helpful to troubleshoot this.
>
> --
> Best Regards,
> Johnny Carlsen
> _______________________________________________
> WireGuard mailing list
> WireGuard@lists.zx2c4.com
> https://lists.zx2c4.com/mailman/listinfo/wireguard
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

Re: Problems on linux-image-4.15.0-1047-aws

[Jason A. Donenfeld]

Hi again,

It sounds like this has already been fixed:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447 And a
kernel is available in some "proposed" tree. However, it isn't yet
available to the general population because their kernel team doesn't
want to release on a Friday or something. Notably this also causes
Docker users to crash, so it's not just us.

I'm not sure what to recommend. There's not a super clear workaround
from a WireGuard perspective, but maybe some disgusting and totally
untested hack like this sort of maybe works?
https://xn--4db.cc/hh05UCL3

If not, then you might just consider using the last-working kernel
instead, and hope that next week Ubuntu gets their stuff settled.

Alternatively you could stop using Ubuntu.

Jason
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

Re: Problems on linux-image-4.15.0-1047-aws

[Jason A. Donenfeld]

On Fri, Sep 6, 2019 at 3:18 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> available to the general population

https://wiki.ubuntu.com/Testing/EnableProposed

This might work.
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

Re: Problems on linux-image-4.15.0-1047-aws

[Sultan Alsawaf]

> On Fri, Sep 6, 2019 at 3:18 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> > available to the general population
> 
> https://wiki.ubuntu.com/Testing/EnableProposed
> 
> This might work.

Hi,

In the case of the 4.15 bug, there is a fixed kernel present in the Proposed
release channel which will be available in the normal Updates channel next week.

If needed, the fixed 4.15 kernel can be installed from Proposed now. To do so,
first enable access to Proposed updates by following the instructions in the
quoted link above, then install the desired kernel with the following command:

If you're using the generic kernel, replace FLAVOR with generic, i.e.,

$ sudo apt-get install linux-image-generic/bionic-proposed

Similarly, if you're using the AWS kernel, adjust the command as so:

$ sudo apt-get install linux-image-aws/bionic-proposed

Then disable access to the Proposed channel and reboot. Disabling access to
Proposed is important since it contains untested packages that are not
release-worthy.

Sultan
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard