Very low throughput in *BSDs (but only as a router)

[Lee Yates <rainmakerraw@icloud.com>]

[-- Attachment #1: Type: text/plain, Size: 4216 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

This is my first time posting to this list, but I've followed along for
a while now. I've been happily using wg at home for months, and it's
been a revelation in terms of speed (practically no performance hit at
all on my 350/20 ISP line).

I recently decided to stop running wg on all my (capable)LAN devices,
and to 'just' run wg on my home-made x86_64 router instead. Since
pfSense and IPFire don't have wg packages (or the ability to add them),
I decided to roll my own environment using Linux or one of the BSDs. I
did very well with a quick virtualised Arch install (masquerade for LAN
to the wg interface) and throughput was perfect - 350/20! Not being a
huge fan of systemd or iptables, I really wanted to use BSD so I tried
out an OpenBSD install. Despite reading how performant it was (capable
of >10Gbps out of the box on appropriate hardware), I noticed throughput
on the virtual router crashed to 130Mbps (30% of full speed) when wg was
connected. I confirmed that my virtual LAN clients were also limited to
around 130Mbps if wg was connected on the OpenBSD 'router'.

Not being satisfied with this and wondering what I'd done wrong (or
whether OpenBSD was indeed capable), I span up a much more familiar (to
me) FreeBSD 11.2 install and set it up the same way. Gateway=yes, pf set
to NAT the virtual LAN traffic through wg, and away we go. Again, the
virtual router could run 350/20 easily on its own, but as soon as wg was
connected (AzireVPN 10Gb node, btw) the performance dropped to the same
130Mbps.

That just didn't seem right. I checked htop while connected to wg and
running iperf3 to a 10Gbps speedtest node in NL. Htop confirmed that the
wireguard process was only using a max of 7% CPU throughout the speed
test (the VMs have four cores from my i7 8700k at 5GHz each). So, it's
not a CPU bottleneck.

Weirdly, if I disconnect wg on the virtual router and run it from any of
the virtual LAN client machines instead, then throughput jumps back up
to 350/20 every single time. So, the virtual router seems capable of
routing 350/20 easily - provided the wg process is running on a client
machine and not itself. As soon as wg is connected on the router itself,
I'm down to 30% of my expected throughput no matter what.

To present it visually, in case it makes more sense for the visual
learners among us:

# Full speed
Virtual client OS [wg] > virtual router > real home router > WAN > [wg]
VPN server

# Crippled speed
Virtual client OS > virtual router [wg] > real router > WAN > [wg] VPN
server

I just can't make sense of it. I could literally run the iperf3 test on
the router+wg and get 130Mbps, but then fire up the exact same iperf3
test on any other machine on the network (connected via wg to the same
real external VPN server) and get full speed every single time.
Something seems to be hobbling wg when run on the router itself, but I'm
all out of ideas. I've tried tuning sysctl.conf etc on the virtual
routers (Open/Free BSD) but it made no difference at all.

Can anyone please offer any advice/help/tips or point out any glaring
omissions I may have made? I can upload my
rc.conf/sysctl.conf/pf.conf/dhcpd.conf/unbound.conf or other to pastebin
if anyone wishes to see them. Sorry if this would have been more
appropriate being sent to a BSD list, but unfortunately not many people
seem to be experienced with wg on BSDs yet so I'm finding help a little
thin on the ground.  Hence, posting to ask here where someone is more
likely to be experienced in the matter.

Many thanks in advance,

Lee Yates
-----BEGIN PGP SIGNATURE-----
Version: BCPG C# v1.8.1.0

iQFBBAEBCAArBQJbUkwhJBxMZWUgWWF0ZXMgPHJhaW5tYWtlcnJhd0BpY2xvdWQu
Y29tPgAKCRDvJcvMOyipkhAYB/9YfaXm5He7VmSTZMeJgYoICF0NDUcH7KmTkIwU
kLzflkzgEtM77mkN4xnA7xkvVMvWFq7F6osKuArJNiZNLoZPNfZPUfBm7ZPtVoXB
SBKbWco9vGqQdqFh3hrIwZYZQWFXoheWtAniOPp7Xv9RO3cFCOT9KcbN9ubLcqo9
NtjC2e3CQ9m17FNrxla5eRUzTT2lcrkMqBO+7ZgjEiQ6TWi/avw9jgErejAJpvoA
G2wlxZj0M5NxB2j6Mgn0ilzFeVzmP/GnprzcDyy6DANpi+rfIrZAKyTRhgpkWvnJ
531rCPK4HxnMKynsX+vH7sF9u0kxjPm6jYVFvTvkjqpLQ9DX
=/Rln
-----END PGP SIGNATURE-----

[-- Attachment #2: rainmakerraw@icloud.com.asc --]
[-- Type: application/pgp-keys, Size: 1677 bytes --]