Re: Instability during large transfers

[Samuel Holland <samuel@sholland.org>]

[-- Attachment #1: Type: text/plain, Size: 1780 bytes --]

Hello,

On 03/01/17 16:44, Samuel Holland wrote:
> Is there anything else I can do to debug this? Enable some kernel
> debugging option? Try a vanilla kernel? Try a newer kernel?

I've upgraded to vanilla (kernel.org) Linux 4.9.15, and while I've been
unable to trigger the same panic so far, I'm getting a similar linked
list corruption warning. This time, instead of failing to pass data once
the warning hits, some packets are coming in out of order with 3000-5000
ms latency. Attached is the dmesg output, kernel config, and ping logs
for inside the VPN. The actual link latency between peers is around 10ms.

As I still haven't been able to reproduce this issue on another machine,
I cannot rule out hardware issues, but this exact machine worked fine
for a year straight (2x 6 months uptime) using OpenVPN. So I'm looking
at either WireGuard or padata as the cause.

The kernel has linked list debugging turned on.

--------

I upgraded again to 4.9.16, and enabled every debug and self-test option
that I could find that seemed relevant. The stack traces and symptoms
still look similar. Except now when it panics (instead of just a
warning), I don't get a log; the netconsole just suddenly cuts over to
the next boot.

You can also see how it stops sending keepalive packets once the
warnings hit.

At this point, unless you want me to try something, I'm going to stop
sending logs. I'm not very familiar with the kernel's
timer/workqueue/crypto subsystems, so there's not much debugging I can
do on my own until I have time to do research (and I have no idea when
that will be). But if you can't reproduce the issue, then it will be
rather difficult for you to debug as well.

I'd still like to get this fixed, and I'll gladly help any way I can.

Thanks,
Samuel Holland

[-- Attachment #2: warning_4.9.15.config.xz --]
[-- Type: application/x-xz, Size: 23520 bytes --]

[-- Attachment #3: warning_4.9.15.dmesg.xz --]
[-- Type: application/x-xz, Size: 18196 bytes --]

[-- Attachment #4: warning_4.9.15.dmesg_secondboot.xz --]
[-- Type: application/x-xz, Size: 3424 bytes --]

[-- Attachment #5: warning_4.9.15.vpn_ping.xz --]
[-- Type: application/x-xz, Size: 1708 bytes --]

[-- Attachment #6: warning_4.9.16.dmesg.xz --]
[-- Type: application/x-xz, Size: 8904 bytes --]