Re: what to do when the peers use different IPs to transmit and receive

[Raffaele Spazzoli <rspazzol@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 8078 bytes --]

Ivan,

I tried the SNAT idea, and still have issue.
here is an example configuration of one of the nodes:

[Interface]
ListenPort = 5555
PrivateKey = ---

[Peer]
PublicKey = H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=
AllowedIPs = 10.128.2.0/23
Endpoint = 192.168.99.12:31112
PersistentKeepalive = 25

[Peer]
PublicKey = 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=
AllowedIPs = 10.128.0.0/23
Endpoint = 192.168.99.14:32188
PersistentKeepalive = 25

[Peer]
PublicKey = MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=
AllowedIPs = 10.130.0.0/23
Endpoint = 192.168.99.13:31992
PersistentKeepalive = 25

[Peer]
PublicKey = s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=
AllowedIPs = 10.129.0.0/23
Endpoint = 192.168.99.15:30305
PersistentKeepalive = 25

[Peer]
PublicKey = SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=
AllowedIPs = 10.131.0.0/23
Endpoint = 192.168.99.7:31714
PersistentKeepalive = 25

[Peer]
PublicKey = a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=
AllowedIPs = 10.129.2.0/23
Endpoint = 192.168.99.6:31165
PersistentKeepalive = 25

the private IP to VIP map for the peers of this node is:

10.128.2.10-192.168.99.12 10.128.1.94-192.168.99.14
10.130.0.136-192.168.99.13 10.129.1.158-192.168.99.15
10.131.0.199-192.168.99.7 10.129.2.217-192.168.99.6

I create the following iptables rules:

sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target prot
opt source destination Chain INPUT (policy ACCEPT) target prot opt source
destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to:
192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to:
192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to:
192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to:
192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to:
192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to:
192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source
destination Chain POSTROUTING (policy ACCEPT) target prot opt source
destination


after the handshake the configuration changes to:

sh-4.2# wg
interface: sdn-tunnel
  public key: gCFgNjOpObU71Vmjub/R9KIn3MHgzXnKtrh9Tf+W628=
  private key: (hidden)
  listening port: 5555

peer: s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=
  endpoint: 10.134.0.1:1033
  allowed ips: 10.129.0.0/23
  latest handshake: 31 seconds ago
  transfer: 180 B received, 452 B sent
  persistent keepalive: every 25 seconds

peer: MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=
  endpoint: 10.134.0.1:1032
  allowed ips: 10.130.0.0/23
  latest handshake: 37 seconds ago
  transfer: 212 B received, 272 B sent
  persistent keepalive: every 25 seconds

peer: 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=
  endpoint: 10.134.0.1:1031
  allowed ips: 10.128.0.0/23
  latest handshake: 39 seconds ago
  transfer: 180 B received, 304 B sent
  persistent keepalive: every 25 seconds

peer: a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=
  endpoint: 192.168.99.6:31165
  allowed ips: 10.129.2.0/23
  latest handshake: 41 seconds ago
  transfer: 156 B received, 180 B sent
  persistent keepalive: every 25 seconds

peer: H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=
  endpoint: 192.168.99.12:31112
  allowed ips: 10.128.2.0/23
  latest handshake: 41 seconds ago
  transfer: 156 B received, 180 B sent
  persistent keepalive: every 25 seconds

peer: SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=
  endpoint: 192.168.99.7:31714
  allowed ips: 10.131.0.0/23
  latest handshake: 41 seconds ago
  transfer: 156 B received, 180 B sent
  persistent keepalive: every 25 seconds

as you can see some of the endpoint's addresses have changed.
the first three are not correct anymore.
After the introduction of the iptables rules they change to an IP that
makes no sense to me 10.134.0.1

finally here are a few seconds of tcpdump in case it helps:

sh-4.2# tcpdump -i eth0 -nn -v tcpdump: listening on eth0, link-type EN10MB
(Ethernet), capture size 262144 bytes 23:07:01.045331 IP (tos 0x0, ttl 64,
id 27711, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 192.168.99.6.31165: UDP, length 32 23:07:01.045363 IP
(tos 0x0, ttl 64, id 53835, offset 0, flags [none], proto UDP (17), length
60) 10.134.0.72.5555 > 192.168.99.7.31714: UDP, length 32 23:07:01.045411
IP (tos 0x0, ttl 64, id 27009, offset 0, flags [none], proto UDP (17),
length 60) 10.134.0.72.5555 > 192.168.99.12.31112: UDP, length 32
23:07:02.758694 IP (tos 0x0, ttl 61, id 19309, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.1.1031 > 10.134.0.72.5555: UDP, length
32 23:07:04.053339 IP (tos 0x0, ttl 64, id 36786, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.72.5555 > 10.134.0.1.1032: UDP, length
32 23:07:07.765375 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
10.134.0.72 tell 10.134.0.1, length 28 23:07:07.765394 ARP, Ethernet (len
6), IPv4 (len 4), Reply 10.134.0.72 is-at 0a:58:0a:86:00:48, length 28
23:07:10.938921 IP (tos 0x0, ttl 61, id 33093, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.1.1033 > 10.134.0.72.5555: UDP, length
32 23:07:26.069271 IP (tos 0x0, ttl 64, id 37778, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.6.31165: UDP,
length 32 23:07:26.069271 IP (tos 0x0, ttl 64, id 59175, offset 0, flags
[none], proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.7.31714:
UDP, length 32 23:07:26.069303 IP (tos 0x0, ttl 64, id 49067, offset 0,
flags [none], proto UDP (17), length 60) 10.134.0.72.5555 >
192.168.99.12.31112: UDP, length 32 23:07:27.797284 IP (tos 0x0, ttl 64, id
57007, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555
> 10.134.0.1.1031: UDP, length 32 23:07:29.079935 IP (tos 0x0, ttl 61, id
18743, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.1.1032 >
10.134.0.72.5555: UDP, length 32

Thanks,
Raffaele

Raffaele Spazzoli
Senior Architect - OpenShift <https://www.openshift.com>, Containers
and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
Tel: +1 216-258-7717



On Sun, Sep 16, 2018 at 2:56 PM, Raffaele Spazzoli <rspazzol@redhat.com>
wrote:

> I'll try to make an example
> cluster 1 node 1 has private IP1 and VIP1
> cluster 2 node 2 has private IP2 and VIP2
>
> each node uses it's private ip for outbound connections.
> each node can receive inbound connection on its VIP.
>
> so the wireguard config file for node1 is going to look like:
>
> [peer]
> endpoint: VIP2:port
>
> and for node 2:
> [peer]
> endpoint: VIP1: port
>
> the problem is that after the handshake, wireguard updates the config to
> the following (for example for node2):
> [peer]
> endpoint: IP1:port
>
> but IP2 cannot route to IP1...
>
> I think a well configured SNAT rule may work, although is not elegant
> because it forces the cluster to exchange information about their private
> IPs.
> This should not be needed and in the cloud private IPs are ephemeral....
>
> anyway thanks for the advice, I am going to try to use it  in my
> prototype.
>
> I still think there is need for a better technical approach for a long
> term solution.
>
>
> Thanks,
> Raffaele
>
> Raffaele Spazzoli
> Senior Architect - OpenShift <https://www.openshift.com>, Containers
> and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
> Tel: +1 216-258-7717
>
>
>
> On Sun, Sep 16, 2018 at 12:54 PM, Ivan Labáth <labawi-wg@matrix-dream.net>
> wrote:
>
>> Hi,
>>
>> On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:
>> > ... then the IP that a  node uses for its outbound
>> > connection is not the same that its peer need to use for its inbound
>> > connections.
>>
>> Who uses what for whose connection? You lost me here.
>> Looks like a broken network to me. Does TCP even work?
>>
>> Anyway, SNAT/DNAT should be able to fix things up, if you want to go
>> that route.
>>
>> Regards,
>> Ivan
>>
>
>

[-- Attachment #2: Type: text/html, Size: 21109 bytes --]