what to do when the peers use different IPs to transmit and receive

what to do when the peers use different IPs to transmit and receive

[Raffaele Spazzoli]

[-- Attachment #1: Type: text/plain, Size: 2244 bytes --]

Hi,

I am trying to build an encrypted tunnel between two Kubernetes clusters.
The distribution of Kubernetes that I use is OpenShift, so I'll make my
examples in OpenShift although the problem that I'm seeing is really more
general.

The nodes that comprise the cluster in OpenShift have an IP in what we'll
call the enterprise network. Also they establish between each other an SDN.
The SDN will have a separate CIDR from the enterprise network and the pods
that OpenShift starts receive an SDN IP. Each node manages a subnet of the
SDN CIDR.

What I'd like to do is to make the IPs of two different OpenShift SDNs
routable over and encrypted tunnel.

In my design each node of cluster A sees as its VPN peers the nodes of
cluster B. So this creates a sort of a meshed VPN.

Wireguard fits very well this series of requirements, but I have an issue.

Normally the nodes of a cluster are not directly exposed to the internet.
This is for security reasons. Whether the cluster is in the cloud or on
premise, normally what customers do is to use private internal addresses.
To access the cluster one can use VIPs. Especially in the cloud Kubernetes
makes it easy to create VIPs in an automated fashion. VIP are public IPs
that can be used to loadbalanced backed services. UDP VIPS are supported.

So if we assume that the two clusters that we want to connect using
wireguard are in two different geographies and can only be talk over the
internet and through VIPs, then the IP that a  node uses for its outbound
connection is not the same that its peer need to use for its inbound
connections.

Each node can have a dedicated VIP that peers need to use for its inbound
connection and then it will use the node's private IP for its outbound
connection.

In this situation wiregaurd assumes that the peer has changed IP (built-in
roaming feature) and it updates the peer endpoint value. This doesn't work
for my setup.

What can I do to fix this issue?
Or alternatively would it be reasonable to add a flag to make a peer
configuration immutable?



Thanks,
Raffaele

Raffaele Spazzoli
Senior Architect - OpenShift <https://www.openshift.com>, Containers
and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
Tel: +1 216-258-7717

[-- Attachment #2: Type: text/html, Size: 3043 bytes --]

Re: what to do when the peers use different IPs to transmit and receive

[Ivan Labáth]

Hi,

On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:
> ... then the IP that a  node uses for its outbound
> connection is not the same that its peer need to use for its inbound
> connections.

Who uses what for whose connection? You lost me here.
Looks like a broken network to me. Does TCP even work?

Anyway, SNAT/DNAT should be able to fix things up, if you want to go
that route.

Regards,
Ivan

Re: what to do when the peers use different IPs to transmit and receive

[Raffaele Spazzoli]

[-- Attachment #1: Type: text/plain, Size: 1712 bytes --]

I'll try to make an example
cluster 1 node 1 has private IP1 and VIP1
cluster 2 node 2 has private IP2 and VIP2

each node uses it's private ip for outbound connections.
each node can receive inbound connection on its VIP.

so the wireguard config file for node1 is going to look like:

[peer]
endpoint: VIP2:port

and for node 2:
[peer]
endpoint: VIP1: port

the problem is that after the handshake, wireguard updates the config to
the following (for example for node2):
[peer]
endpoint: IP1:port

but IP2 cannot route to IP1...

I think a well configured SNAT rule may work, although is not elegant
because it forces the cluster to exchange information about their private
IPs.
This should not be needed and in the cloud private IPs are ephemeral....

anyway thanks for the advice, I am going to try to use it  in my prototype.

I still think there is need for a better technical approach for a long term
solution.


Thanks,
Raffaele

Raffaele Spazzoli
Senior Architect - OpenShift <https://www.openshift.com>, Containers
and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
Tel: +1 216-258-7717



On Sun, Sep 16, 2018 at 12:54 PM, Ivan Labáth <labawi-wg@matrix-dream.net>
wrote:

> Hi,
>
> On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:
> > ... then the IP that a  node uses for its outbound
> > connection is not the same that its peer need to use for its inbound
> > connections.
>
> Who uses what for whose connection? You lost me here.
> Looks like a broken network to me. Does TCP even work?
>
> Anyway, SNAT/DNAT should be able to fix things up, if you want to go
> that route.
>
> Regards,
> Ivan
>

[-- Attachment #2: Type: text/html, Size: 3333 bytes --]

Re: what to do when the peers use different IPs to transmit and receive

[Raffaele Spazzoli]

[-- Attachment #1: Type: text/plain, Size: 8078 bytes --]

Ivan,

I tried the SNAT idea, and still have issue.
here is an example configuration of one of the nodes:

[Interface]
ListenPort = 5555
PrivateKey = ---

[Peer]
PublicKey = H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=
AllowedIPs = 10.128.2.0/23
Endpoint = 192.168.99.12:31112
PersistentKeepalive = 25

[Peer]
PublicKey = 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=
AllowedIPs = 10.128.0.0/23
Endpoint = 192.168.99.14:32188
PersistentKeepalive = 25

[Peer]
PublicKey = MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=
AllowedIPs = 10.130.0.0/23
Endpoint = 192.168.99.13:31992
PersistentKeepalive = 25

[Peer]
PublicKey = s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=
AllowedIPs = 10.129.0.0/23
Endpoint = 192.168.99.15:30305
PersistentKeepalive = 25

[Peer]
PublicKey = SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=
AllowedIPs = 10.131.0.0/23
Endpoint = 192.168.99.7:31714
PersistentKeepalive = 25

[Peer]
PublicKey = a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=
AllowedIPs = 10.129.2.0/23
Endpoint = 192.168.99.6:31165
PersistentKeepalive = 25

the private IP to VIP map for the peers of this node is:

10.128.2.10-192.168.99.12 10.128.1.94-192.168.99.14
10.130.0.136-192.168.99.13 10.129.1.158-192.168.99.15
10.131.0.199-192.168.99.7 10.129.2.217-192.168.99.6

I create the following iptables rules:

sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target prot
opt source destination Chain INPUT (policy ACCEPT) target prot opt source
destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to:
192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to:
192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to:
192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to:
192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to:
192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to:
192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source
destination Chain POSTROUTING (policy ACCEPT) target prot opt source
destination


after the handshake the configuration changes to:

sh-4.2# wg
interface: sdn-tunnel
  public key: gCFgNjOpObU71Vmjub/R9KIn3MHgzXnKtrh9Tf+W628=
  private key: (hidden)
  listening port: 5555

peer: s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=
  endpoint: 10.134.0.1:1033
  allowed ips: 10.129.0.0/23
  latest handshake: 31 seconds ago
  transfer: 180 B received, 452 B sent
  persistent keepalive: every 25 seconds

peer: MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=
  endpoint: 10.134.0.1:1032
  allowed ips: 10.130.0.0/23
  latest handshake: 37 seconds ago
  transfer: 212 B received, 272 B sent
  persistent keepalive: every 25 seconds

peer: 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=
  endpoint: 10.134.0.1:1031
  allowed ips: 10.128.0.0/23
  latest handshake: 39 seconds ago
  transfer: 180 B received, 304 B sent
  persistent keepalive: every 25 seconds

peer: a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=
  endpoint: 192.168.99.6:31165
  allowed ips: 10.129.2.0/23
  latest handshake: 41 seconds ago
  transfer: 156 B received, 180 B sent
  persistent keepalive: every 25 seconds

peer: H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=
  endpoint: 192.168.99.12:31112
  allowed ips: 10.128.2.0/23
  latest handshake: 41 seconds ago
  transfer: 156 B received, 180 B sent
  persistent keepalive: every 25 seconds

peer: SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=
  endpoint: 192.168.99.7:31714
  allowed ips: 10.131.0.0/23
  latest handshake: 41 seconds ago
  transfer: 156 B received, 180 B sent
  persistent keepalive: every 25 seconds

as you can see some of the endpoint's addresses have changed.
the first three are not correct anymore.
After the introduction of the iptables rules they change to an IP that
makes no sense to me 10.134.0.1

finally here are a few seconds of tcpdump in case it helps:

sh-4.2# tcpdump -i eth0 -nn -v tcpdump: listening on eth0, link-type EN10MB
(Ethernet), capture size 262144 bytes 23:07:01.045331 IP (tos 0x0, ttl 64,
id 27711, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 192.168.99.6.31165: UDP, length 32 23:07:01.045363 IP
(tos 0x0, ttl 64, id 53835, offset 0, flags [none], proto UDP (17), length
60) 10.134.0.72.5555 > 192.168.99.7.31714: UDP, length 32 23:07:01.045411
IP (tos 0x0, ttl 64, id 27009, offset 0, flags [none], proto UDP (17),
length 60) 10.134.0.72.5555 > 192.168.99.12.31112: UDP, length 32
23:07:02.758694 IP (tos 0x0, ttl 61, id 19309, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.1.1031 > 10.134.0.72.5555: UDP, length
32 23:07:04.053339 IP (tos 0x0, ttl 64, id 36786, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.72.5555 > 10.134.0.1.1032: UDP, length
32 23:07:07.765375 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
10.134.0.72 tell 10.134.0.1, length 28 23:07:07.765394 ARP, Ethernet (len
6), IPv4 (len 4), Reply 10.134.0.72 is-at 0a:58:0a:86:00:48, length 28
23:07:10.938921 IP (tos 0x0, ttl 61, id 33093, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.1.1033 > 10.134.0.72.5555: UDP, length
32 23:07:26.069271 IP (tos 0x0, ttl 64, id 37778, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.6.31165: UDP,
length 32 23:07:26.069271 IP (tos 0x0, ttl 64, id 59175, offset 0, flags
[none], proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.7.31714:
UDP, length 32 23:07:26.069303 IP (tos 0x0, ttl 64, id 49067, offset 0,
flags [none], proto UDP (17), length 60) 10.134.0.72.5555 >
192.168.99.12.31112: UDP, length 32 23:07:27.797284 IP (tos 0x0, ttl 64, id
57007, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555
> 10.134.0.1.1031: UDP, length 32 23:07:29.079935 IP (tos 0x0, ttl 61, id
18743, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.1.1032 >
10.134.0.72.5555: UDP, length 32

Thanks,
Raffaele

Raffaele Spazzoli
Senior Architect - OpenShift <https://www.openshift.com>, Containers
and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
Tel: +1 216-258-7717



On Sun, Sep 16, 2018 at 2:56 PM, Raffaele Spazzoli <rspazzol@redhat.com>
wrote:

> I'll try to make an example
> cluster 1 node 1 has private IP1 and VIP1
> cluster 2 node 2 has private IP2 and VIP2
>
> each node uses it's private ip for outbound connections.
> each node can receive inbound connection on its VIP.
>
> so the wireguard config file for node1 is going to look like:
>
> [peer]
> endpoint: VIP2:port
>
> and for node 2:
> [peer]
> endpoint: VIP1: port
>
> the problem is that after the handshake, wireguard updates the config to
> the following (for example for node2):
> [peer]
> endpoint: IP1:port
>
> but IP2 cannot route to IP1...
>
> I think a well configured SNAT rule may work, although is not elegant
> because it forces the cluster to exchange information about their private
> IPs.
> This should not be needed and in the cloud private IPs are ephemeral....
>
> anyway thanks for the advice, I am going to try to use it  in my
> prototype.
>
> I still think there is need for a better technical approach for a long
> term solution.
>
>
> Thanks,
> Raffaele
>
> Raffaele Spazzoli
> Senior Architect - OpenShift <https://www.openshift.com>, Containers
> and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
> Tel: +1 216-258-7717
>
>
>
> On Sun, Sep 16, 2018 at 12:54 PM, Ivan Labáth <labawi-wg@matrix-dream.net>
> wrote:
>
>> Hi,
>>
>> On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:
>> > ... then the IP that a  node uses for its outbound
>> > connection is not the same that its peer need to use for its inbound
>> > connections.
>>
>> Who uses what for whose connection? You lost me here.
>> Looks like a broken network to me. Does TCP even work?
>>
>> Anyway, SNAT/DNAT should be able to fix things up, if you want to go
>> that route.
>>
>> Regards,
>> Ivan
>>
>
>

[-- Attachment #2: Type: text/html, Size: 21109 bytes --]

Re: what to do when the peers use different IPs to transmit and receive

[Ivan Labáth]

On Sun, Sep 16, 2018 at 07:08:58PM -0400, Raffaele Spazzoli wrote:
> sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target prot
> opt source destination Chain INPUT (policy ACCEPT) target prot opt source
> destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source
> destination Chain POSTROUTING (policy ACCEPT) target prot opt source
> destination

Please try to have no or reasonable line wrapping.

If you are applying SNAT on your source node, you are setting
the source address, which should be set to the reachable address
for the replies to come to. In your case VIP.
If you are setting it on the destination, you are IMO doing it wrong.

Same thing applies to TCP and most typical protocol, nothing special
about wireguard here.

If you have a middlebox doing DNAT, it would normaly be expected
for it or something else to do SNAT in the reverse direction.
Or, if your node has both adresses assigned, then it might be
a case of improperly set source address on outgoing packets
(e.g. your routing might need tuning).

Regards,
Ivan

Re: what to do when the peers use different IPs to transmit and receive

[Raffaele Spazzoli]

[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]

Ivan,

sorry for the formatting, it seemed right on my email editor (gmail).
I cannot do SNAT at the source because the packet would be dropped if it
didn't come from the actual IP of the VM.
So I am doing SNAT at the destination. why do you say I am doing it wrong?
I know it would be ideal to do it at the source, but should it work when
done at the destination?

Thanks,
Raffaele

Raffaele Spazzoli
Senior Architect - OpenShift <https://www.openshift.com>, Containers
and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
Tel: +1 216-258-7717



On Mon, Sep 17, 2018 at 5:16 AM, Ivan Labáth <labawi-wg@matrix-dream.net>
wrote:

> On Sun, Sep 16, 2018 at 07:08:58PM -0400, Raffaele Spazzoli wrote:
> > sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target
> prot
> > opt source destination Chain INPUT (policy ACCEPT) target prot opt source
> > destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source
> > destination Chain POSTROUTING (policy ACCEPT) target prot opt source
> > destination
>
> Please try to have no or reasonable line wrapping.
>
> If you are applying SNAT on your source node, you are setting
> the source address, which should be set to the reachable address
> for the replies to come to. In your case VIP.
> If you are setting it on the destination, you are IMO doing it wrong.
>
> Same thing applies to TCP and most typical protocol, nothing special
> about wireguard here.
>
> If you have a middlebox doing DNAT, it would normaly be expected
> for it or something else to do SNAT in the reverse direction.
> Or, if your node has both adresses assigned, then it might be
> a case of improperly set source address on outgoing packets
> (e.g. your routing might need tuning).
>
> Regards,
> Ivan
> _______________________________________________
> WireGuard mailing list
> WireGuard@lists.zx2c4.com
> https://lists.zx2c4.com/mailman/listinfo/wireguard
>

[-- Attachment #2: Type: text/html, Size: 4349 bytes --]

Re: what to do when the peers use different IPs to transmit and receive

[Ivan Labáth]

Hi,

I guess SNAT-ing on the destination node should work.

Still, without a compelling use case, it seems
like a broken network to me. You will have packets
in transit with an unroutable/incorrect source address,
among other possible issues. ICMP signalling won't work.
I think TCP wouldn't work with such a setup, or even
most udp protocols.

Do you use other services over this type of network?

Also, please note that iptables rules can affect network performance,
especially if there are many of them and haphazardly placed.
For debugging iptables rules I would suggest adding -v, so you can see
the number of matched packets (caveats apply in NAT).

Regards,
Ivan Labáth


On Mon, Sep 17, 2018 at 07:10:05AM -0400, Raffaele Spazzoli wrote:
> Ivan,
> 
> sorry for the formatting, it seemed right on my email editor (gmail).
> I cannot do SNAT at the source because the packet would be dropped if it
> didn't come from the actual IP of the VM.
> So I am doing SNAT at the destination. why do you say I am doing it wrong?
> I know it would be ideal to do it at the source, but should it work when
> done at the destination?
> 
> Thanks,
> Raffaele
> 
> Raffaele Spazzoli
> Senior Architect - OpenShift <https://www.openshift.com>, Containers
> and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
> Tel: +1 216-258-7717
> 
> 
> 
> On Mon, Sep 17, 2018 at 5:16 AM, Ivan Labáth <labawi-wg@matrix-dream.net>
> wrote:
> 
> > On Sun, Sep 16, 2018 at 07:08:58PM -0400, Raffaele Spazzoli wrote:
> > > sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target
> > prot
> > > opt source destination Chain INPUT (policy ACCEPT) target prot opt source
> > > destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source
> > > destination Chain POSTROUTING (policy ACCEPT) target prot opt source
> > > destination
> >
> > Please try to have no or reasonable line wrapping.
> >
> > If you are applying SNAT on your source node, you are setting
> > the source address, which should be set to the reachable address
> > for the replies to come to. In your case VIP.
> > If you are setting it on the destination, you are IMO doing it wrong.
> >
> > Same thing applies to TCP and most typical protocol, nothing special
> > about wireguard here.
> >
> > If you have a middlebox doing DNAT, it would normaly be expected
> > for it or something else to do SNAT in the reverse direction.
> > Or, if your node has both adresses assigned, then it might be
> > a case of improperly set source address on outgoing packets
> > (e.g. your routing might need tuning).
> >
> > Regards,
> > Ivan
> > _______________________________________________
> > WireGuard mailing list
> > WireGuard@lists.zx2c4.com
> > https://lists.zx2c4.com/mailman/listinfo/wireguard
> >
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard