WireGuard Archive on lore.kernel.org
 help / color / Atom feed
* Very low throughput in *BSDs (but only as a router)
@ 2018-07-20 20:54 Lee Yates
  2018-07-21 22:18 ` Jonathan Neuschäfer
  0 siblings, 1 reply; 4+ messages in thread
From: Lee Yates @ 2018-07-20 20:54 UTC (permalink / raw)
  To: wireguard

[-- Attachment #1: Type: text/plain, Size: 4216 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi all,

This is my first time posting to this list, but I've followed along for
a while now. I've been happily using wg at home for months, and it's
been a revelation in terms of speed (practically no performance hit at
all on my 350/20 ISP line).

I recently decided to stop running wg on all my (capable)LAN devices,
and to 'just' run wg on my home-made x86_64 router instead. Since
pfSense and IPFire don't have wg packages (or the ability to add them),
I decided to roll my own environment using Linux or one of the BSDs. I
did very well with a quick virtualised Arch install (masquerade for LAN
to the wg interface) and throughput was perfect - 350/20! Not being a
huge fan of systemd or iptables, I really wanted to use BSD so I tried
out an OpenBSD install. Despite reading how performant it was (capable
of >10Gbps out of the box on appropriate hardware), I noticed throughput
on the virtual router crashed to 130Mbps (30% of full speed) when wg was
connected. I confirmed that my virtual LAN clients were also limited to
around 130Mbps if wg was connected on the OpenBSD 'router'.

Not being satisfied with this and wondering what I'd done wrong (or
whether OpenBSD was indeed capable), I span up a much more familiar (to
me) FreeBSD 11.2 install and set it up the same way. Gateway=yes, pf set
to NAT the virtual LAN traffic through wg, and away we go. Again, the
virtual router could run 350/20 easily on its own, but as soon as wg was
connected (AzireVPN 10Gb node, btw) the performance dropped to the same
130Mbps.

That just didn't seem right. I checked htop while connected to wg and
running iperf3 to a 10Gbps speedtest node in NL. Htop confirmed that the
wireguard process was only using a max of 7% CPU throughout the speed
test (the VMs have four cores from my i7 8700k at 5GHz each). So, it's
not a CPU bottleneck.

Weirdly, if I disconnect wg on the virtual router and run it from any of
the virtual LAN client machines instead, then throughput jumps back up
to 350/20 every single time. So, the virtual router seems capable of
routing 350/20 easily - provided the wg process is running on a client
machine and not itself. As soon as wg is connected on the router itself,
I'm down to 30% of my expected throughput no matter what.

To present it visually, in case it makes more sense for the visual
learners among us:

# Full speed
Virtual client OS [wg] > virtual router > real home router > WAN > [wg]
VPN server

# Crippled speed
Virtual client OS > virtual router [wg] > real router > WAN > [wg] VPN
server

I just can't make sense of it. I could literally run the iperf3 test on
the router+wg and get 130Mbps, but then fire up the exact same iperf3
test on any other machine on the network (connected via wg to the same
real external VPN server) and get full speed every single time.
Something seems to be hobbling wg when run on the router itself, but I'm
all out of ideas. I've tried tuning sysctl.conf etc on the virtual
routers (Open/Free BSD) but it made no difference at all.

Can anyone please offer any advice/help/tips or point out any glaring
omissions I may have made? I can upload my
rc.conf/sysctl.conf/pf.conf/dhcpd.conf/unbound.conf or other to pastebin
if anyone wishes to see them. Sorry if this would have been more
appropriate being sent to a BSD list, but unfortunately not many people
seem to be experienced with wg on BSDs yet so I'm finding help a little
thin on the ground.  Hence, posting to ask here where someone is more
likely to be experienced in the matter.

Many thanks in advance,

Lee Yates
-----BEGIN PGP SIGNATURE-----
Version: BCPG C# v1.8.1.0

iQFBBAEBCAArBQJbUkwhJBxMZWUgWWF0ZXMgPHJhaW5tYWtlcnJhd0BpY2xvdWQu
Y29tPgAKCRDvJcvMOyipkhAYB/9YfaXm5He7VmSTZMeJgYoICF0NDUcH7KmTkIwU
kLzflkzgEtM77mkN4xnA7xkvVMvWFq7F6osKuArJNiZNLoZPNfZPUfBm7ZPtVoXB
SBKbWco9vGqQdqFh3hrIwZYZQWFXoheWtAniOPp7Xv9RO3cFCOT9KcbN9ubLcqo9
NtjC2e3CQ9m17FNrxla5eRUzTT2lcrkMqBO+7ZgjEiQ6TWi/avw9jgErejAJpvoA
G2wlxZj0M5NxB2j6Mgn0ilzFeVzmP/GnprzcDyy6DANpi+rfIrZAKyTRhgpkWvnJ
531rCPK4HxnMKynsX+vH7sF9u0kxjPm6jYVFvTvkjqpLQ9DX
=/Rln
-----END PGP SIGNATURE-----

[-- Attachment #2: rainmakerraw@icloud.com.asc --]
[-- Type: application/pgp-keys, Size: 1677 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Very low throughput in *BSDs (but only as a router)
  2018-07-20 20:54 Very low throughput in *BSDs (but only as a router) Lee Yates
@ 2018-07-21 22:18 ` Jonathan Neuschäfer
  2018-07-22 16:15   ` Re[2]: " Lee Yates
  0 siblings, 1 reply; 4+ messages in thread
From: Jonathan Neuschäfer @ 2018-07-21 22:18 UTC (permalink / raw)
  To: Lee Yates; +Cc: wireguard

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

Hi,

On Fri, Jul 20, 2018 at 08:54:48PM +0000, Lee Yates wrote:
[...]
> To present it visually, in case it makes more sense for the visual
> learners among us:
> 
> # Full speed
> Virtual client OS [wg] > virtual router > real home router > WAN > [wg]
> VPN server
> 
> # Crippled speed
> Virtual client OS > virtual router [wg] > real router > WAN > [wg] VPN
> server

As far as I understand it, the virtual router OS is based on BSD, right?
Are the virtual client OSes that you tested based on Linux?

If that's the case, then the result is quite expected: There is a fast,
in-kernel implementation for Linux[1], but no fast implementation for BSD.
The implementation for BSD is wireguard-go[2], which hasn't really been
optimized.


Jonathan Neuschäfer

[1]: https://git.zx2c4.com/WireGuard/tree/src
[2]: https://git.zx2c4.com/wireguard-go/about/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re[2]: Very low throughput in *BSDs (but only as a router)
  2018-07-21 22:18 ` Jonathan Neuschäfer
@ 2018-07-22 16:15   ` " Lee Yates
  2018-07-23 11:43     ` Jason A. Donenfeld
  0 siblings, 1 reply; 4+ messages in thread
From: Lee Yates @ 2018-07-22 16:15 UTC (permalink / raw)
  To: Jonathan Neuschäfer; +Cc: wireguard

[-- Attachment #1: Type: text/plain, Size: 2199 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

>As far as I understand it, the virtual router OS is based on BSD,
>right?
>Are the virtual client OSes that you tested based on Linux?
>
>If that's the case, then the result is quite expected: There is a fast,
>in-kernel implementation for Linux[1], but no fast implementation for
>BSD.
>The implementation for BSD is wireguard-go[2], which hasn't really been
>optimized.
>
>
>Jonathan Neuschäfer
>
>[1]: https://git.zx2c4.com/WireGuard/tree/src
>[2]: https://git.zx2c4.com/wireguard-go/about/

Thanks for your reply.

No, I can achieve (almost) full WAN line rate using *BSD as local
clients' OS too, not just Linux ones. The wireguard-go being in user
space doesn't really cause much damage on powerful hardware. For example
on GhostBSD (in a VM) I got >345Mbps down and 20Mbps up running wg-go on
the same machine. [1] The throughput problem only arises when I run wg
directly on the router instance.

I think I made some progress however. I need to do some testing on my
main workstation later, though, as I was working on a (lower powered)
laptop today. OpenBSD being limited to a single core for routing (I
believe its pf is now more multi-threaded however) could also be a
factor. I'm going to move my improved pf.conf over to FreeBSD on the
workstation and see if I get better throughput. I'm convinced I've made
a simple mistake in implementing NAT or one of the pf rules at this
point. It doesn't really make sense otherwise.

I'll get there in the end. :) Thanks again for your reply.
Kind regards,

Lee Yates
[1] https://i.imgur.com/XCFADnR.png
-----BEGIN PGP SIGNATURE-----
Version: BCPG C# v1.8.1.0

iQFBBAEBCAArBQJbVK2PJBxMZWUgWWF0ZXMgPHJhaW5tYWtlcnJhd0BpY2xvdWQu
Y29tPgAKCRDvJcvMOyipklBtB/90STajUjPPXF6F7hkfQdE3xVqNTjfaW4J93+MH
4CKC+wdGAS9riIycSTyEIT1VPjFm17dyUwAEO5hUNfF6anywjTEPWVnR2Mirvnkz
oKURCwEwMMQr1ZHEN/naiO9IfQm9OJKy/20RD0kYMT6Qdmejg7xtQWzkKUD745f/
sRzVxJe6484dHxLW/1bQc5ccWCe3rM6uq9Axo3RyOiWPvDey+pOBEnMvK3LtoGQg
EqddOo72dzjTlWwc2GP7wBxEWtlvMaIg0HYsxsbmh50zWSTuFYclBGDyiDSrHzfl
fe4iHqiRVa6sx7xVys903Dg83tTI/cdJbEGvH4lRu/VZguoe
=zIHq
-----END PGP SIGNATURE-----

[-- Attachment #2: rainmakerraw@icloud.com.asc --]
[-- Type: application/pgp-keys, Size: 1677 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Re[2]: Very low throughput in *BSDs (but only as a router)
  2018-07-22 16:15   ` Re[2]: " Lee Yates
@ 2018-07-23 11:43     ` Jason A. Donenfeld
  0 siblings, 0 replies; 4+ messages in thread
From: Jason A. Donenfeld @ 2018-07-23 11:43 UTC (permalink / raw)
  To: rainmakerraw; +Cc: WireGuard mailing list

Hey Lee,

That's surprising to hear. Indeed userspace is slower than kernel
space, but on sufficiently fast hardware you won't notice much of a
difference at those low speeds. Have you checked MTU issues? For
example, perhaps the router is trying to forward packets with 1500
bytes down a 1420 byte pipe?

Regards,
Jason

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-20 20:54 Very low throughput in *BSDs (but only as a router) Lee Yates
2018-07-21 22:18 ` Jonathan Neuschäfer
2018-07-22 16:15   ` Re[2]: " Lee Yates
2018-07-23 11:43     ` Jason A. Donenfeld

WireGuard Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/wireguard/0 wireguard/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 wireguard wireguard/ https://lore.kernel.org/wireguard \
		wireguard@lists.zx2c4.com zx2c4-wireguard@archiver.kernel.org
	public-inbox-index wireguard


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/com.zx2c4.lists.wireguard


AGPL code for this site: git clone https://public-inbox.org/ public-inbox