* Wireguard & hw flow offload incompatibility @ 2018-05-29 8:03 Jaap Buurman 2018-05-29 11:54 ` Jason A. Donenfeld 0 siblings, 1 reply; 7+ messages in thread From: Jaap Buurman @ 2018-05-29 8:03 UTC (permalink / raw) To: wireguard Dear all, When running a wireguard interface on the latest Lede master branch, the router will crash as soon as traffic hits the wireguard interface while (hw) flow offloading is enabled. I am not sure whether this is a bug with wireguard, hw flow offload, both or neither, so I am reporting the bug to both mailinglists. A more detailed description plus a properly formatted stack trace can be found on Lede's bug tracker: https://bugs.openwrt.org/index.php?do=details&task_id=1539 If you require any additional information, please do not hesitate to contact me. Thank you very much in advance. Yours sincerely, Jaap Buurman ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Wireguard & hw flow offload incompatibility 2018-05-29 8:03 Wireguard & hw flow offload incompatibility Jaap Buurman @ 2018-05-29 11:54 ` Jason A. Donenfeld 2018-05-29 11:59 ` Jaap Buurman 0 siblings, 1 reply; 7+ messages in thread From: Jason A. Donenfeld @ 2018-05-29 11:54 UTC (permalink / raw) To: Jaap Buurman; +Cc: WireGuard mailing list Hi Jaap, Thanks for the report. Is this a _new_ bug in _new_ version of WireGuard that wasn't there before. Or is this the first time you've observed this? Thanks, Jason ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Wireguard & hw flow offload incompatibility 2018-05-29 11:54 ` Jason A. Donenfeld @ 2018-05-29 11:59 ` Jaap Buurman 2018-05-29 12:14 ` Jason A. Donenfeld 0 siblings, 1 reply; 7+ messages in thread From: Jaap Buurman @ 2018-05-29 11:59 UTC (permalink / raw) To: Jason A. Donenfeld; +Cc: WireGuard mailing list Dear Jason, This isn't a regression. This is simply the first time this has been observed. (hw) flow offload is a new feature, and hence this interaction with wireguard is also new. Yours sincerely, Jaap On Tue, May 29, 2018 at 1:54 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > Hi Jaap, > > Thanks for the report. Is this a _new_ bug in _new_ version of > WireGuard that wasn't there before. Or is this the first time you've > observed this? > > Thanks, > Jason ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Wireguard & hw flow offload incompatibility 2018-05-29 11:59 ` Jaap Buurman @ 2018-05-29 12:14 ` Jason A. Donenfeld 2018-05-29 12:38 ` Jason A. Donenfeld 0 siblings, 1 reply; 7+ messages in thread From: Jason A. Donenfeld @ 2018-05-29 12:14 UTC (permalink / raw) To: Jaap Buurman; +Cc: openwrt-devel, WireGuard mailing list Hi Jaap, Thanks for the clarification. I downloaded the binary for that hardware and triaged where the bug occurs [1]. This patch [2] should probably fix it, but I'm rather surprised to see situations in which a skb is missing a dst entry in ndo_start_xmit; this might point to deeper kernel bugs in this hardware offloading feature, or some alternative mechanism for routing being used when hardware offloading is on. So I'm hesitant to merge this just yet, because perhaps this is better handled in the compat layer, if it is in fact vendor silliness. Do you have a link to the kernel source of these boxes? I'd like to see what exactly the vendor is doing. And if you could try [2] and see if that still crashes, this would be most appreciated. Thanks, Jason [1] https://data.zx2c4.com/openwrt-mips-offloading-bug.png [2] https://=D7=90.cc/Am4tZ0n8 On Tue, May 29, 2018 at 1:59 PM, Jaap Buurman <jaapbuurman@gmail.com> wrote= : > Dear Jason, > > This isn't a regression. This is simply the first time this has been > observed. (hw) flow offload is a new feature, and hence this > interaction with wireguard is also new. > > Yours sincerely, > > Jaap > > On Tue, May 29, 2018 at 1:54 PM, Jason A. Donenfeld <Jason@zx2c4.com> wro= te: >> Hi Jaap, >> >> Thanks for the report. Is this a _new_ bug in _new_ version of >> WireGuard that wasn't there before. Or is this the first time you've >> observed this? >> >> Thanks, >> Jason =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Original Mail =3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D > Dear all, > > When running a wireguard interface on the latest Lede master branch, > the router will crash as soon as traffic hits the wireguard interface > while (hw) flow offloading is enabled. I am not sure whether this is a > bug with wireguard, hw flow offload, both or neither, so I am > reporting the bug to both mailinglists. A more detailed description > plus a properly formatted stack trace can be found on Lede's bug > tracker: https://bugs.openwrt.org/index.php?do=3Ddetails&task_id=3D1539 > > If you require any additional information, please do not hesitate to > contact me. Thank you very much in advance. > > Yours sincerely, > > Jaap Buurman ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Wireguard & hw flow offload incompatibility 2018-05-29 12:14 ` Jason A. Donenfeld @ 2018-05-29 12:38 ` Jason A. Donenfeld 2018-05-29 13:26 ` Jaap Buurman 0 siblings, 1 reply; 7+ messages in thread From: Jason A. Donenfeld @ 2018-05-29 12:38 UTC (permalink / raw) To: Felix Fietkau; +Cc: Jaap Buurman, openwrt-devel, WireGuard mailing list Hey Felix, Per the below thread, I've been digging around trying to see what's going on. Apparently packets are hitting a virtual network interface's ndo_start_xmit with no dst when hardware offloading enabled. I assume that the path is something along the lines of a packet coming in on one of these hardware accelerated NICs and then being forwarded to the wireguard interface, which expects the dst. I found your ndo_flow_offload patchset, and I suspect that might have something to do with this. Any insights on dsts disappearing in skbs? Thanks, Jason On Tue, May 29, 2018 at 2:14 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote= : > Hi Jaap, > > Thanks for the clarification. I downloaded the binary for that > hardware and triaged where the bug occurs [1]. This patch [2] should > probably fix it, but I'm rather surprised to see situations in which a > skb is missing a dst entry in ndo_start_xmit; this might point to > deeper kernel bugs in this hardware offloading feature, or some > alternative mechanism for routing being used when hardware offloading > is on. So I'm hesitant to merge this just yet, because perhaps this is > better handled in the compat layer, if it is in fact vendor silliness. > Do you have a link to the kernel source of these boxes? I'd like to > see what exactly the vendor is doing. And if you could try [2] and see > if that still crashes, this would be most appreciated. > > Thanks, > Jason > > [1] https://data.zx2c4.com/openwrt-mips-offloading-bug.png > [2] https://=D7=90.cc/Am4tZ0n8 > > On Tue, May 29, 2018 at 1:59 PM, Jaap Buurman <jaapbuurman@gmail.com> wro= te: >> Dear Jason, >> >> This isn't a regression. This is simply the first time this has been >> observed. (hw) flow offload is a new feature, and hence this >> interaction with wireguard is also new. >> >> Yours sincerely, >> >> Jaap >> >> On Tue, May 29, 2018 at 1:54 PM, Jason A. Donenfeld <Jason@zx2c4.com> wr= ote: >>> Hi Jaap, >>> >>> Thanks for the report. Is this a _new_ bug in _new_ version of >>> WireGuard that wasn't there before. Or is this the first time you've >>> observed this? >>> >>> Thanks, >>> Jason > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Original Mail =3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > >> Dear all, >> >> When running a wireguard interface on the latest Lede master branch, >> the router will crash as soon as traffic hits the wireguard interface >> while (hw) flow offloading is enabled. I am not sure whether this is a >> bug with wireguard, hw flow offload, both or neither, so I am >> reporting the bug to both mailinglists. A more detailed description >> plus a properly formatted stack trace can be found on Lede's bug >> tracker: https://bugs.openwrt.org/index.php?do=3Ddetails&task_id=3D1539 >> >> If you require any additional information, please do not hesitate to >> contact me. Thank you very much in advance. >> >> Yours sincerely, >> >> Jaap Buurman ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Wireguard & hw flow offload incompatibility 2018-05-29 12:38 ` Jason A. Donenfeld @ 2018-05-29 13:26 ` Jaap Buurman 2018-05-31 1:14 ` Jason A. Donenfeld 0 siblings, 1 reply; 7+ messages in thread From: Jaap Buurman @ 2018-05-29 13:26 UTC (permalink / raw) To: Jason A. Donenfeld Cc: OpenWrt Development List, WireGuard mailing list, Felix Fietkau Dear Jason, The initial technical explanation unfortunately went over my head, since I am not that technical myself. But I will do my best to provide the required information. First of all, sorry for the confusion that I may have caused, but this happens with both the hardware version of the flow offload implementation and the software version, so it doesn't seem to be caused by any vendor specific (hardware) logic. So it is probably easier to focus on the software version for now. Also, in case that wasn't fully clear, both the flow offloading feature and the wireguard interface are both running on the router itself. What exactly do you mean with the kernel source of these boxes? As far as I understand, Lede/OpenWRT uses the upstream 4.14 kernel in this build (4.14.43 in the one I am running atm) with Lede/OpenWRT specific patches. The patches that enable flow offloading support for the 4.14 kernel can be found in these 2 folders: https://github.com/openwrt/openwrt/tree/openwrt-18.06/target/linux/generic/= backport-4.14 https://github.com/openwrt/openwrt/tree/openwrt-18.06/target/linux/generic/= pending-4.14 One important thing is that the upstream flow offloading code uses nftables, while Lede/OpenWRT uses iptables. Hence flow offload support has been backported to iptables, which also might be a contribution to this bug. I'm not even sure what dst entries are exactly, but I found one patch that is supposed to fix dst entries. Perhaps it is incomplete or contains a bug?: https://github.com/openwrt/openwrt/commit/c89e338fe68fd5af61b80ef37c55a6577= 21c6542 I will try to cross-compile wireguard with your suggested patch tomorrow or the day after tomorrow depending on my time schedule. I will report back whether it solves this issue. Thank you very much. Yours sincerely, Jaap Buurman On Tue, May 29, 2018 at 2:38 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote= : > Hey Felix, > > Per the below thread, I've been digging around trying to see what's > going on. Apparently packets are hitting a virtual network interface's > ndo_start_xmit with no dst when hardware offloading enabled. I assume > that the path is something along the lines of a packet coming in on > one of these hardware accelerated NICs and then being forwarded to the > wireguard interface, which expects the dst. I found your > ndo_flow_offload patchset, and I suspect that might have something to > do with this. Any insights on dsts disappearing in skbs? > > Thanks, > Jason > > On Tue, May 29, 2018 at 2:14 PM, Jason A. Donenfeld <Jason@zx2c4.com> wro= te: >> Hi Jaap, >> >> Thanks for the clarification. I downloaded the binary for that >> hardware and triaged where the bug occurs [1]. This patch [2] should >> probably fix it, but I'm rather surprised to see situations in which a >> skb is missing a dst entry in ndo_start_xmit; this might point to >> deeper kernel bugs in this hardware offloading feature, or some >> alternative mechanism for routing being used when hardware offloading >> is on. So I'm hesitant to merge this just yet, because perhaps this is >> better handled in the compat layer, if it is in fact vendor silliness. >> Do you have a link to the kernel source of these boxes? I'd like to >> see what exactly the vendor is doing. And if you could try [2] and see >> if that still crashes, this would be most appreciated. >> >> Thanks, >> Jason >> >> [1] https://data.zx2c4.com/openwrt-mips-offloading-bug.png >> [2] https://=D7=90.cc/Am4tZ0n8 >> >> On Tue, May 29, 2018 at 1:59 PM, Jaap Buurman <jaapbuurman@gmail.com> wr= ote: >>> Dear Jason, >>> >>> This isn't a regression. This is simply the first time this has been >>> observed. (hw) flow offload is a new feature, and hence this >>> interaction with wireguard is also new. >>> >>> Yours sincerely, >>> >>> Jaap >>> >>> On Tue, May 29, 2018 at 1:54 PM, Jason A. Donenfeld <Jason@zx2c4.com> w= rote: >>>> Hi Jaap, >>>> >>>> Thanks for the report. Is this a _new_ bug in _new_ version of >>>> WireGuard that wasn't there before. Or is this the first time you've >>>> observed this? >>>> >>>> Thanks, >>>> Jason >> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Original Mail =3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >> >>> Dear all, >>> >>> When running a wireguard interface on the latest Lede master branch, >>> the router will crash as soon as traffic hits the wireguard interface >>> while (hw) flow offloading is enabled. I am not sure whether this is a >>> bug with wireguard, hw flow offload, both or neither, so I am >>> reporting the bug to both mailinglists. A more detailed description >>> plus a properly formatted stack trace can be found on Lede's bug >>> tracker: https://bugs.openwrt.org/index.php?do=3Ddetails&task_id=3D1539 >>> >>> If you require any additional information, please do not hesitate to >>> contact me. Thank you very much in advance. >>> >>> Yours sincerely, >>> >>> Jaap Buurman ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Wireguard & hw flow offload incompatibility 2018-05-29 13:26 ` Jaap Buurman @ 2018-05-31 1:14 ` Jason A. Donenfeld 0 siblings, 0 replies; 7+ messages in thread From: Jason A. Donenfeld @ 2018-05-31 1:14 UTC (permalink / raw) To: Jaap Buurman Cc: OpenWrt Development List, WireGuard mailing list, Felix Fietkau Hi Jaap, This should now be taken care of by http://lists.infradead.org/pipermail/openwrt-devel/2018-May/012675.html . Thanks for the useful bug report. Jason ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-05-31 1:12 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-05-29 8:03 Wireguard & hw flow offload incompatibility Jaap Buurman 2018-05-29 11:54 ` Jason A. Donenfeld 2018-05-29 11:59 ` Jaap Buurman 2018-05-29 12:14 ` Jason A. Donenfeld 2018-05-29 12:38 ` Jason A. Donenfeld 2018-05-29 13:26 ` Jaap Buurman 2018-05-31 1:14 ` Jason A. Donenfeld
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).
