Using WG for transport security in a p2p network

Using WG for transport security in a p2p network

[Ximin Luo]

[-- Attachment #1: Type: text/plain, Size: 2987 bytes --]

Hi WireGuard mailing list,

As part of my day job we're building a p2p broadcast network for a new type
of
blockchain, where latency is very important and certain objects need to be
delivered with greater priority than other objects. So I've looked into both
QUIC and SCTP since they offer stream multiplexing within a single
connection.
To secure these connections, QUIC uses TLS 1.3 [1] which is in the process
of
being finalised, and there is an S-SCTP extension [2] with unclear delivery
date and no implementations.

[1] https://tools.ietf.org/html/draft-ietf-quic-tls-09
[2] https://tools.ietf.org/html/draft-hohendorf-secure-sctp-25

Another option would be to run insecure QUIC or SCTP on top of WireGuard,
ignoring TLS and S-SCTP completely. Noise is much simpler and we already
authenticate node public keys via other means, so have no use for the
certificate logic that both TLS and S-SCTP include. Since WireGuard runs as
a
network interface, it should be easy to transparently run QUIC or SCTP on
top
of it, allowing us to decouple the security mechanism from the transport
mechanism. Then there are other issues to consider hence my email:

Our network churn is not expected to be very heavy, perhaps on the order of
~30
new connections per node per week or so. So any extra latency in the initial
connection caused by this separation of layers, should not be significant.
However this churn is probably higher than what current typical WG usages
get
exposed to. For example in [1] Jason says:

> Secondly, I'm wondering if you tend to do, "anything strange". For
> example -- are you setting up and taking down the device often in an
> automated way? Or reconfiguring the interface (via wg(8), for example)
> often in an automated way?

[1] https://lists.zx2c4.com/pipermail/wireguard/2018-February/002370.html

Our usage would indeed involve setting up and tearing down interfaces ~30
times
a week in an automated fashion, which might be "strange" going by the above.

I'm also wondering how easy this would be to program. It would clearly be
much
more heavyweight than simply opening a socket, but I guess it can be done
via
invocations of the `wg` or `wg-quick` tools. Has anyone had any experience
with
this level of WG automation, could you share your thoughts? Would the
program
need any extra system-level privileges? Ideally we wouldn't need root, of
course - does that mean we're forced to wait for a userspace WG library
such as
wireguard-rs? I understand there is a performance penalty here, but I'd
have to
run benchmarks to know if this affects our use-case significantly.

Once the network is live, we'd need the transport protocol to be relatively
stable, or at least be easily upgradeable - perhaps using the noise
negotiation
subprotocol to support two protocols during network upgrade times. This is
an
extra requirement that seems beyond WG's current main use-case so I was also
wondering if that is something that you guys plan to cover.

Best,

Ximin

[-- Attachment #2: Type: text/html, Size: 3469 bytes --]

Re: Using WG for transport security in a p2p network

[Matthias Urlichs]

[-- Attachment #1: Type: text/plain, Size: 1172 bytes --]

Hi,<https://tools.ietf.org/html/draft-hohendorf-secure-sctp-25>
>
> Another option would be to run insecure QUIC or SCTP on top of WireGuard,
You cannot run SCTP on the Internet anyway. Too many routers block
anything that's not TCP/UDP/ICMP.

> I'm also wondering how easy this would be to program. It would clearly
> be much
> more heavyweight than simply opening a socket, but I guess it can be
> done via
> invocations of the `wg` or `wg-quick` tools.
Don't use the tools. There's a library around that you can use to do all
of the heavy lifting via netlink sockets. You'll also need the privilege
to assign addresses and routes to the WG interfaces.
> Ideally we wouldn't need root
If you go the netlink route, you do need one process that has the
appropriate privilege, which means root at install time (but not runtime).
>
> Once the network is live, we'd need the transport protocol to be
> relatively
> stable, or at least be easily upgradeable
Well, the WG wire protocol is supposed to be stable by now. Switching
away from it would require new code on your side anyway, so you can
implement the exact method of switching at that time.

-- 
-- Matthias Urlichs


[-- Attachment #2: Type: text/html, Size: 2197 bytes --]

Re: Using WG for transport security in a p2p network

[Kalin KOZHUHAROV]

Hello Ximin,


On Thu, Apr 5, 2018 at 5:22 AM, Ximin Luo <ximin@dfinity.org> wrote:
> Our network churn is not expected to be very heavy, perhaps on the order of
> ~30 new connections per node per week or so. So any extra latency in the initial
> connection caused by this separation of layers, should not be significant.
> However this churn is probably higher than what current typical WG usages
> get exposed to.
>
Few times a day, I would even say few times per hour is a very normal use and
should not be strange, AFAIK.

> I'm also wondering how easy this would be to program. It would clearly be
> much more heavyweight than simply opening a socket, but I guess it can be done
> via invocations of the `wg` or `wg-quick` tools. Has anyone had any experience
> with this level of WG automation, could you share your thoughts?
>
Definitely not "hard", it will depend more on what you are trying to
achieve exactly.

> Would the program need any extra system-level privileges?
>
Yes for sure ;-D Adding interfaces is a admin task, using sudo or
similar should be trivial.

> Ideally we wouldn't need root, of course - does that mean we're forced to wait for a userspace WG library such
> as wireguard-rs? I understand there is a performance penalty here, but I'd have
> to run benchmarks to know if this affects our use-case significantly.
>
I don't think performance matters in your case, as it will be only
during setup; once setup,
all data goes to a socket/kernel and it doesn't matter how it was set up.

> Once the network is live, we'd need the transport protocol to be relatively
> stable, or at least be easily upgradeable - perhaps using the noise
> negotiation subprotocol to support two protocols during network upgrade times. This is
> an extra requirement that seems beyond WG's current main use-case so I was also
> wondering if that is something that you guys plan to cover.
>
Making it "support 2 protocols" in the design phase is a good practice
for availability.
It will introduce complexity, maintainability issues and thus possible
security issues.
Working out a "maintenance mode" might be easier.

Cheers,
Kalin.

Re: Using WG for transport security in a p2p network

[Tim Sedlmeyer]

On Thu, Apr 5, 2018 at 3:13 AM, Matthias Urlichs <matthias@urlichs.de> wrote:
> Hi,
>
>
> Another option would be to run insecure QUIC or SCTP on top of WireGuard,
>
> You cannot run SCTP on the Internet anyway. Too many routers block anything
> that's not TCP/UDP/ICMP.
>
> I'm also wondering how easy this would be to program. It would clearly be
> much
> more heavyweight than simply opening a socket, but I guess it can be done
> via
> invocations of the `wg` or `wg-quick` tools.
>
> Don't use the tools. There's a library around that you can use to do all of
> the heavy lifting via netlink sockets. You'll also need the privilege to
> assign addresses and routes to the WG interfaces.
>
> Ideally we wouldn't need root
>
> If you go the netlink route, you do need one process that has the
> appropriate privilege, which means root at install time (but not runtime).

The process doesn't need full root permissions even at install time.
Whatever process is going to create and manage
the interfaces needs the CAP_NET_ADMIN capability.

>
>
> Once the network is live, we'd need the transport protocol to be relatively
> stable, or at least be easily upgradeable
>
> Well, the WG wire protocol is supposed to be stable by now. Switching away
> from it would require new code on your side anyway, so you can implement the
> exact method of switching at that time.
>
> --
> -- Matthias Urlichs
>
>
> _______________________________________________
> WireGuard mailing list
> WireGuard@lists.zx2c4.com
> https://lists.zx2c4.com/mailman/listinfo/wireguard
>

Re: Using WG for transport security in a p2p network

[Ximin Luo]

[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]

(reposting to the list)

On Thu, Apr 5, 2018 at 12:13 AM, Matthias Urlichs <matthias@urlichs.de>
wrote:

> Another option would be to run insecure QUIC or SCTP on top of WireGuard,
>
> You cannot run SCTP on the Internet anyway. Too many routers block
> anything that's not TCP/UDP/ICMP.
>

Well, that's another advantage of running on top of WG. AIUI the content
should be encrypted and authenticated so to all routers the traffic would
look like opaque UDP traffic. The endpoints will decrypt and pass it to
their SCTP/QUIC stack.

> I'm also wondering how easy this would be to program. It would clearly be
> much
> more heavyweight than simply opening a socket, but I guess it can be done
> via
> invocations of the `wg` or `wg-quick` tools.
>
> Don't use the tools. There's a library around that you can use to do all
> of the heavy lifting via netlink sockets. You'll also need the privilege to
> assign addresses and routes to the WG interfaces.
>
> Ideally we wouldn't need root
>
> If you go the netlink route, you do need one process that has the
> appropriate privilege, which means root at install time (but not runtime).
>

OK, thanks for the pointer to the netlink library I'll check that out.
Someone else mentioned CAP_NET_ADMIN, I'll follow that up in a separate
email.

>
> Once the network is live, we'd need the transport protocol to be relatively
> stable, or at least be easily upgradeable
>
> Well, the WG wire protocol is supposed to be stable by now. Switching away
> from it would require new code on your side anyway, so you can implement
> the exact method of switching at that time.
>

What does "stable" mean exactly here though? At some point the WG protocol
might need a new version, e.g. to reselect ciphers to e.g. add post-quantum
crypto. In the typical WG use-case this is not an issue because the network
admin controls both endpoints and can upgrade both simultaneously, but this
wouldn't be the case for our p2p network.

X

[-- Attachment #2: Type: text/html, Size: 3465 bytes --]

Re: Using WG for transport security in a p2p network

[Ximin Luo]

[-- Attachment #1: Type: text/plain, Size: 3179 bytes --]

On Thu, Apr 5, 2018 at 8:32 AM, Kalin KOZHUHAROV <me.kalin@gmail.com> wrote:

> On Thu, Apr 5, 2018 at 5:22 AM, Ximin Luo <ximin@dfinity.org> wrote:
> > Our network churn is not expected to be very heavy, perhaps on the order
> of
> > ~30 new connections per node per week or so. So any extra latency in the
> initial
> > connection caused by this separation of layers, should not be
> significant.
> > However this churn is probably higher than what current typical WG usages
> > get exposed to.
> >
> Few times a day, I would even say few times per hour is a very normal use
> and
> should not be strange, AFAIK.
>

OK great, thanks for clarifying.


> > I'm also wondering how easy this would be to program. It would clearly be
> > much more heavyweight than simply opening a socket, but I guess it can
> be done
> > via invocations of the `wg` or `wg-quick` tools. Has anyone had any
> experience
> > with this level of WG automation, could you share your thoughts?
> >
> Definitely not "hard", it will depend more on what you are trying to
> achieve exactly.
>
> > Would the program need any extra system-level privileges?
> >
> Yes for sure ;-D Adding interfaces is a admin task, using sudo or
> similar should be trivial.
>
> > Ideally we wouldn't need root, of course - does that mean we're forced
> to wait for a userspace WG library such
> > as wireguard-rs? I understand there is a performance penalty here, but
> I'd have
> > to run benchmarks to know if this affects our use-case significantly.
> >
> I don't think performance matters in your case, as it will be only
> during setup; once setup,
> all data goes to a socket/kernel and it doesn't matter how it was set up.
>

Application-level data goes to a socket, but AIUI adding/removing WG
protocol wrapping is done either in the kernel (as in the main
implementation) or in userspace (as in wireguard-rs). In the latter case
there is apparently a performance penalty in terms of throughput (i.e. not
only for the setup phase), judging by Jason's comments in various places.
Did I understand wrong / could you explain in more detail if so?


> > Once the network is live, we'd need the transport protocol to be
> relatively
> > stable, or at least be easily upgradeable - perhaps using the noise
> > negotiation subprotocol to support two protocols during network upgrade
> times. This is
> > an extra requirement that seems beyond WG's current main use-case so I
> was also
> > wondering if that is something that you guys plan to cover.
> >
> Making it "support 2 protocols" in the design phase is a good practice
> for availability.
> It will introduce complexity, maintainability issues and thus possible
> security issues.
> Working out a "maintenance mode" might be easier.
>

Could you elaborate what is meant by "maintenance mode"?

I suppose in the worst case we could do something like: add logic "change
from protocol X to protocol Y at future round N" to software version V and
expect that everyone upgrades to software version V before round N. That
should hopefully work even if protocol X doesn't explicitly define a smooth
upgrade path to protocol Y (e.g. X = WG version 3, Y = WG version 4).

X

[-- Attachment #2: Type: text/html, Size: 4354 bytes --]

Re: Using WG for transport security in a p2p network

[Ximin Luo]

[-- Attachment #1: Type: text/plain, Size: 1616 bytes --]

On Thu, Apr 5, 2018 at 9:06 AM, Tim Sedlmeyer <tim@sedlmeyer.org> wrote:

> On Thu, Apr 5, 2018 at 3:13 AM, Matthias Urlichs <matthias@urlichs.de>
> wrote:
> >
> > Ideally we wouldn't need root
> >
> > If you go the netlink route, you do need one process that has the
> > appropriate privilege, which means root at install time (but not
> runtime).
>
> The process doesn't need full root permissions even at install time.
> Whatever process is going to create and manage
> the interfaces needs the CAP_NET_ADMIN capability.


Thanks, that's good to know. Though CAP_NET_ADMIN is "almost root" (e.g.
see [1]) so it would be more comfortable to not even require that. But I
guess a "next best thing" would be to put minimal logic inside a small
program and give only this program CAP_NET_ADMIN. Possibly wireguard-rs's
`wgrs` has both sufficient and not-too-much functionality that we could
directly `setcap` that, I'll have to look into it in more detail.
Unfortunately `wg-quick` is a shell script which is more annoying to try to
`setcap` on.

Another approach I was thinking of, was to extend wireguard to expose a
SOCKS5-UDP interface, or even simpler just a socket-wrapper API like
Python's `ssl.wrap_socket()`, and not use tunnel interfaces at all. Of
course this would run in userspace, but wireguard-rs already does that so I
hope there would be no "extra" performance penalty. Then one could run QUIC
through this, and I know that some SCTP implementations can run on top of
UDP too. Any further comments on this idea would be much appreciated.

X

[1] https://forums.grsecurity.net/viewtopic.php?f=7&t=2522

[-- Attachment #2: Type: text/html, Size: 2232 bytes --]

Re: Using WG for transport security in a p2p network

[Matthias Urlichs]

On 05.04.2018 20:07, Ximin Luo wrote:
> In the typical WG use-case this is not an issue because the network
> admin controls both endpoints and can upgrade both simultaneously, but
> this wouldn't be the case for our p2p network.

Your p2p network would need to exchange v2 keys before upgrading. Thus,
as soon as you know that an endpoint supports v2 you also try to connect
to it that way. When the v2 connection succeeds, the other system knows
that you have its new key, thus it can disable v1 connections from you
(and vice versa).

You'll need to pre-define a private IPv6 address range (with enough
random bits in it that a collision with anybody's local address usage is
unlikely) for your wireguard network. To facilitate upgrades, you can
simply define two bits of that address range to be a rolling version
number and you're basically done: you just route each subnet to the
appropriate wireguard interface.

-- 
-- Matthias Urlichs

Re: Using WG for transport security in a p2p network

[Jason A. Donenfeld]

Hi Ximin,

On Thu, Apr 5, 2018 at 5:22 AM, Ximin Luo <ximin@dfinity.org> wrote:
> Our usage would indeed involve setting up and tearing down interfaces ~30
> times
> a week in an automated fashion, which might be "strange" going by the above.

No certainly not strange. Actually, there's no amount of setting up
and tearing down that should be considered a bad thing. I have a
script I run during development that sets up thousands of interfaces,
each with hundreds of thousands of peers, connects them to each other,
and then toggles everything up and down over and over. I have another
script that just adds and removes interfaces really fast. The purpose
is to stress test WireGuard to ensure it's resilient in these types of
configurations. So... I think you should be good with a mere 30 a week
;-).

>
> I'm also wondering how easy this would be to program. It would clearly be
> much
> more heavyweight than simply opening a socket, but I guess it can be done
> via
> invocations of the `wg` or `wg-quick` tools. Has anyone had any experience
> with
> this level of WG automation, could you share your thoughts? Would the
> program
> need any extra system-level privileges? Ideally we wouldn't need root, of
> course - does that mean we're forced to wait for a userspace WG library such
> as
> wireguard-rs? I understand there is a performance penalty here, but I'd have
> to
> run benchmarks to know if this affects our use-case significantly.

The overhead and performance penalty is minimal, and you should easily
be able to get away with doing this. You can script it pretty easily
using wg (it needs only CAP_NET_ADMIN), or if you want closer
integration and more sophisticated priv separation, you could open the
netlink socket, then drop privs, and use something like the
embeddable-wg-library to automate everything:
https://git.zx2c4.com/WireGuard/tree/contrib/examples/embeddable-wg-library


Let me know if you have any more questions or ways in which I can help
you guys out with the p2p protocol.

Regards,
Jason

Re: Using WG for transport security in a p2p network

[Bruno Wolff III]

On Thu, Apr 05, 2018 at 09:13:03 +0200,
  Matthias Urlichs <matthias@urlichs.de> wrote:
>Hi,<https://tools.ietf.org/html/draft-hohendorf-secure-sctp-25>
>>
>> Another option would be to run insecure QUIC or SCTP on top of WireGuard,
>You cannot run SCTP on the Internet anyway. Too many routers block
>anything that's not TCP/UDP/ICMP.

The tunnelled traffic is going to be UDP. If he controls everything from 
the tunnel endpoints to the traffic endpoints he can make sure SCTP isn't 
blocked.

Re: Using WG for transport security in a p2p network

[Matthias Urlichs]

On 14.04.2018 18:01, Bruno Wolff III wrote:
> On Thu, Apr 05, 2018 at 09:13:03 +0200,
>  Matthias Urlichs <matthias@urlichs.de> wrote:
>> Hi,<https://tools.ietf.org/html/draft-hohendorf-secure-sctp-25>
>>>
>>> Another option would be to run insecure QUIC or SCTP on top of
>>> WireGuard,
>> You cannot run SCTP on the Internet anyway. Too many routers block
>> anything that's not TCP/UDP/ICMP.
>
> The tunnelled traffic is going to be UDP. If he controls everything
> from the tunnel endpoints to the traffic endpoints he can make sure
> SCTP isn't blocked.

I know. That was kindof my point. Sorry if I didn't write that more clearly.

-- 
-- Matthias Urlichs

Re: Using WG for transport security in a p2p network

[Ximin Luo]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

(reposting to the list, I'll learn one of these days..)

On Fri, Apr 6, 2018 at 7:59 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:

> [..]
>
> Let me know if you have any more questions or ways in which I can help
> you guys out with the p2p protocol.
>
>
Hey, thanks for the reply. Another issue came up while I've been looking
into this:

At present, one has to manually add specific peers in order for WG to
authenticate them. I was wondering what the options are for dynamic
authentication, of peer keys that one doesn't know beforehand. The typical
example would be a PAKE but for us it would be an alternative
zero-knowledge proof that the initiator's key belongs to some allowed-set
of peers wrt the responder's key, as defined by the rest of the protocol
(I'm being vague because the details are TBD, actually).

It would be nice to keep WG's current property of being able to
authenticate a client on the first packet without requiring further
communication. To reduce the DoS-potential of having to verify a complex zk
proof, we can probably also include a proof-of-work linked to a recent
global shared source of randomness (we have that in our protocol). So one
way would be for WG to hook into a custom function that reads a custom
certificate from the first incoming packet and say whether it passes the
test.

Alternatively we can listen on another socket, perform the custom check on
incoming packets here, and then forward passing packets with our custom
portion stripped out onto the local WG socket. Hopefully this would "just
work" if the "from" address on the UDP packet is correct. However,
initiating these would be tricky, we'd have to intercept the WG initial
outgoing packet and rewrite it.

Other suggestions would be much appreciated.

X

[-- Attachment #2: Type: text/html, Size: 2477 bytes --]

Re: Using WG for transport security in a p2p network

[Ximin Luo]

[-- Attachment #1: Type: text/plain, Size: 2594 bytes --]

On Fri, Apr 20, 2018 at 5:20 PM, Ximin Luo <ximin@dfinity.org> wrote:

> (reposting to the list, I'll learn one of these days..)
>
> On Fri, Apr 6, 2018 at 7:59 PM, Jason A. Donenfeld <Jason@zx2c4.com>
> wrote:
>
>> [..]
>>
>> Let me know if you have any more questions or ways in which I can help
>> you guys out with the p2p protocol.
>>
>>
> Hey, thanks for the reply. Another issue came up while I've been looking
> into this:
>
> At present, one has to manually add specific peers in order for WG to
> authenticate them. I was wondering what the options are for dynamic
> authentication, of peer keys that one doesn't know beforehand. The typical
> example would be a PAKE but for us it would be an alternative
> zero-knowledge proof that the initiator's key belongs to some allowed-set
> of peers wrt the responder's key, as defined by the rest of the protocol
> (I'm being vague because the details are TBD, actually).
>
> It would be nice to keep WG's current property of being able to
> authenticate a client on the first packet without requiring further
> communication. To reduce the DoS-potential of having to verify a complex zk
> proof, we can probably also include a proof-of-work linked to a recent
> global shared source of randomness (we have that in our protocol). So one
> way would be for WG to hook into a custom function that reads a custom
> certificate from the first incoming packet and say whether it passes the
> test.
>
> Alternatively we can listen on another socket, perform the custom check on
> incoming packets here, and then forward passing packets with our custom
> portion stripped out onto the local WG socket. Hopefully this would "just
> work" if the "from" address on the UDP packet is correct. However,
> initiating these would be tricky, we'd have to intercept the WG initial
> outgoing packet and rewrite it.
>
> Other suggestions would be much appreciated.
>
>
Here is another option that adds a half-round but is much simpler than
either of my suggestions above and doesn't involve modifying WG, so I think
I'll go with that. Might be useful for other people looking to do dynamic
auth on top of WG.

Peer A first authenticates and locates B via the parent protocol, adds B as
a WG peer, then:
 -> zk-proof "I am A, I am allowed to connect to you B"
Peer B verifies this proof and adds A as a WG peer, triggering the standard
WG protocol flow
 <- WG initiation, Noise_IKpsk2, etc
 ->
 <-, etc

Since it authorises the keys and WG stores these, it shouldn't be necessary
to re-run this after e.g a disconnection, WG should "just work" by itself.

X

[-- Attachment #2: Type: text/html, Size: 3624 bytes --]

Re: Using WG for transport security in a p2p network

[Jason A. Donenfeld]

Hey Ximin,

Nice to see that you're thinking through these issues in a concrete setting.

For a while I had some notions about allowing WireGuard to pop up
initiaton messages from unauthenticated peers to userspace for userspace to
then validate in one way or another asynchornously, followed by userspace
adding that peer's key to the list of peers (or not), and initiating the
handshake back the other way reusing the initial UDP port. This wouldn't add
that much complexity and I think it'd be fairly reliable. However, I've held
off on implementing it because I'm skepical that people would actually use it
in a way that makes sense. For example, in your case, you already have some
other aspect of the protocol which seeks to exchange this information; in
that case, doing the exchange there makes most sense, since you can morph and
change that for your particular requirements. In other words, I like the idea
you presented in your follow up email. Does that seem like a okay solution for
you? Or do you think you do have a compelling reason for adding the above
semantics?

Jason