WireGuard Archive on lore.kernel.org
 help / color / Atom feed
* what to do when the peers use different IPs to transmit and receive
@ 2018-09-16 12:21 Raffaele Spazzoli
  2018-09-16 16:54 ` Ivan Labáth
  0 siblings, 1 reply; 7+ messages in thread
From: Raffaele Spazzoli @ 2018-09-16 12:21 UTC (permalink / raw)
  To: wireguard

[-- Attachment #1: Type: text/plain, Size: 2244 bytes --]

Hi,

I am trying to build an encrypted tunnel between two Kubernetes clusters.
The distribution of Kubernetes that I use is OpenShift, so I'll make my
examples in OpenShift although the problem that I'm seeing is really more
general.

The nodes that comprise the cluster in OpenShift have an IP in what we'll
call the enterprise network. Also they establish between each other an SDN.
The SDN will have a separate CIDR from the enterprise network and the pods
that OpenShift starts receive an SDN IP. Each node manages a subnet of the
SDN CIDR.

What I'd like to do is to make the IPs of two different OpenShift SDNs
routable over and encrypted tunnel.

In my design each node of cluster A sees as its VPN peers the nodes of
cluster B. So this creates a sort of a meshed VPN.

Wireguard fits very well this series of requirements, but I have an issue.

Normally the nodes of a cluster are not directly exposed to the internet.
This is for security reasons. Whether the cluster is in the cloud or on
premise, normally what customers do is to use private internal addresses.
To access the cluster one can use VIPs. Especially in the cloud Kubernetes
makes it easy to create VIPs in an automated fashion. VIP are public IPs
that can be used to loadbalanced backed services. UDP VIPS are supported.

So if we assume that the two clusters that we want to connect using
wireguard are in two different geographies and can only be talk over the
internet and through VIPs, then the IP that a  node uses for its outbound
connection is not the same that its peer need to use for its inbound
connections.

Each node can have a dedicated VIP that peers need to use for its inbound
connection and then it will use the node's private IP for its outbound
connection.

In this situation wiregaurd assumes that the peer has changed IP (built-in
roaming feature) and it updates the peer endpoint value. This doesn't work
for my setup.

What can I do to fix this issue?
Or alternatively would it be reasonable to add a flag to make a peer
configuration immutable?



Thanks,
Raffaele

Raffaele Spazzoli
Senior Architect - OpenShift <https://www.openshift.com>, Containers
and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
Tel: +1 216-258-7717

[-- Attachment #2: Type: text/html, Size: 3043 bytes --]

<div dir="ltr"><div>Hi,</div><div><br></div><div>I am trying to build an encrypted tunnel between two Kubernetes clusters.</div><div>The distribution of Kubernetes that I use is OpenShift, so I&#39;ll make my examples in OpenShift although the problem that I&#39;m seeing is really more general.</div><div><br></div><div>The nodes that comprise the cluster in OpenShift have an IP in what we&#39;ll call the enterprise network. Also they establish between each other an SDN. The SDN will have a separate CIDR from the enterprise network and the pods that OpenShift starts receive an SDN IP. Each node manages a subnet of the SDN CIDR.</div><div><br></div><div>What I&#39;d like to do is to make the IPs of two different OpenShift SDNs routable over and encrypted tunnel.</div><div><br></div><div>In my design each node of cluster A sees as its VPN peers the nodes of cluster B. So this creates a sort of a meshed VPN. </div><div><br></div><div>Wireguard fits very well this series of requirements, but I have an issue.</div><div><br></div><div>Normally the nodes of a cluster are not directly exposed to the internet. This is for security reasons. Whether the cluster is in the cloud or on premise, normally what customers do is to use private internal addresses. To access the cluster one can use VIPs. Especially in the cloud Kubernetes makes it easy to create VIPs in an automated fashion. VIP are public IPs that can be used to loadbalanced backed services. UDP VIPS are supported.</div><div><br></div><div>So if we assume that the two clusters that we want to connect using wireguard are in two different geographies and can only be talk over the internet and through VIPs, then the IP that a  node uses for its outbound connection is not the same that its peer need to use for its inbound connections. </div><div><br></div><div>Each node can have a dedicated VIP that peers need to use for its inbound connection and then it will use the node&#39;s private IP for its outbound connection.</div><div><br></div><div>In this situation wiregaurd assumes that the peer has changed IP (built-in roaming feature) and it updates the peer endpoint value. This doesn&#39;t work for my setup.</div><div><br></div><div>What can I do to fix this issue?</div><div>Or alternatively would it be reasonable to add a flag to make a peer configuration immutable?</div><div><br></div><div><br></div><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Thanks,<div>Raffaele</div><div><br></div><div>Raffaele Spazzoli</div><div>Senior Architect - <a href="https://www.openshift.com" target="_blank">OpenShift</a>, <a href="https://www.redhat.com/en/services/consulting/paas" target="_blank">Containers and PaaS Practice</a></div><div>Tel: +1 216-258-7717</div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: what to do when the peers use different IPs to transmit and receive
  2018-09-16 12:21 what to do when the peers use different IPs to transmit and receive Raffaele Spazzoli
@ 2018-09-16 16:54 ` Ivan Labáth
  2018-09-16 18:56   ` Raffaele Spazzoli
  0 siblings, 1 reply; 7+ messages in thread
From: Ivan Labáth @ 2018-09-16 16:54 UTC (permalink / raw)
  To: Raffaele Spazzoli; +Cc: wireguard

Hi,

On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:
> ... then the IP that a  node uses for its outbound
> connection is not the same that its peer need to use for its inbound
> connections.

Who uses what for whose connection? You lost me here.
Looks like a broken network to me. Does TCP even work?

Anyway, SNAT/DNAT should be able to fix things up, if you want to go
that route.

Regards,
Ivan

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: what to do when the peers use different IPs to transmit and receive
  2018-09-16 16:54 ` Ivan Labáth
@ 2018-09-16 18:56   ` Raffaele Spazzoli
  2018-09-16 23:08     ` Raffaele Spazzoli
  0 siblings, 1 reply; 7+ messages in thread
From: Raffaele Spazzoli @ 2018-09-16 18:56 UTC (permalink / raw)
  To: Ivan Labáth; +Cc: wireguard

[-- Attachment #1: Type: text/plain, Size: 1712 bytes --]

I'll try to make an example
cluster 1 node 1 has private IP1 and VIP1
cluster 2 node 2 has private IP2 and VIP2

each node uses it's private ip for outbound connections.
each node can receive inbound connection on its VIP.

so the wireguard config file for node1 is going to look like:

[peer]
endpoint: VIP2:port

and for node 2:
[peer]
endpoint: VIP1: port

the problem is that after the handshake, wireguard updates the config to
the following (for example for node2):
[peer]
endpoint: IP1:port

but IP2 cannot route to IP1...

I think a well configured SNAT rule may work, although is not elegant
because it forces the cluster to exchange information about their private
IPs.
This should not be needed and in the cloud private IPs are ephemeral....

anyway thanks for the advice, I am going to try to use it  in my prototype.

I still think there is need for a better technical approach for a long term
solution.


Thanks,
Raffaele

Raffaele Spazzoli
Senior Architect - OpenShift <https://www.openshift.com>, Containers
and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
Tel: +1 216-258-7717



On Sun, Sep 16, 2018 at 12:54 PM, Ivan Labáth <labawi-wg@matrix-dream.net>
wrote:

> Hi,
>
> On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:
> > ... then the IP that a  node uses for its outbound
> > connection is not the same that its peer need to use for its inbound
> > connections.
>
> Who uses what for whose connection? You lost me here.
> Looks like a broken network to me. Does TCP even work?
>
> Anyway, SNAT/DNAT should be able to fix things up, if you want to go
> that route.
>
> Regards,
> Ivan
>

[-- Attachment #2: Type: text/html, Size: 3333 bytes --]

<div dir="ltr">I&#39;ll try to make an example<div>cluster 1 node 1 has private IP1 and VIP1 </div><div>cluster 2 node 2 has private IP2 and VIP2</div><div class="gmail_extra"><br></div><div class="gmail_extra">each node uses it&#39;s private ip for outbound connections. </div><div class="gmail_extra">each node can receive inbound connection on its VIP.</div><div class="gmail_extra"><br></div><div class="gmail_extra">so the wireguard config file for node1 is going to look like:</div><div class="gmail_extra"><br></div><div class="gmail_extra">[peer]</div><div class="gmail_extra">endpoint: VIP2:port</div><div class="gmail_extra"><br></div><div class="gmail_extra">and for node 2:</div><div class="gmail_extra">[peer] </div><div class="gmail_extra">endpoint: VIP1: port</div><div class="gmail_extra"><br></div><div class="gmail_extra">the problem is that after the handshake, wireguard updates the config to the following (for example for node2):</div><div class="gmail_extra">[peer]</div><div class="gmail_extra">endpoint: IP1:port</div><div class="gmail_extra"><br></div><div class="gmail_extra">but IP2 cannot route to IP1...</div><div class="gmail_extra"><br></div><div class="gmail_extra">I think a well configured SNAT rule may work, although is not elegant because it forces the cluster to exchange information about their private IPs.</div><div class="gmail_extra">This should not be needed and in the cloud private IPs are ephemeral....</div><div class="gmail_extra"><br></div><div class="gmail_extra">anyway thanks for the advice, I am going to try to use it  in my prototype. </div><div class="gmail_extra"><br></div><div class="gmail_extra">I still think there is need for a better technical approach for a long term solution.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Thanks,<div>Raffaele</div><div><br></div><div>Raffaele Spazzoli</div><div>Senior Architect - <a href="https://www.openshift.com" target="_blank">OpenShift</a>, <a href="https://www.redhat.com/en/services/consulting/paas" target="_blank">Containers and PaaS Practice</a></div><div>Tel: +1 216-258-7717</div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Sun, Sep 16, 2018 at 12:54 PM, Ivan Labáth <span dir="ltr">&lt;<a href="mailto:labawi-wg@matrix-dream.net" target="_blank">labawi-wg@matrix-dream.net</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:<br>
&gt; ... then the IP that a  node uses for its outbound<br>
<span class="">&gt; connection is not the same that its peer need to use for its inbound<br>
&gt; connections.<br>
<br>
</span>Who uses what for whose connection? You lost me here.<br>
Looks like a broken network to me. Does TCP even work?<br>
<br>
Anyway, SNAT/DNAT should be able to fix things up, if you want to go<br>
that route.<br>
<br>
Regards,<br>
Ivan<br>
</blockquote></div><br></div></div>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: what to do when the peers use different IPs to transmit and receive
  2018-09-16 18:56   ` Raffaele Spazzoli
@ 2018-09-16 23:08     ` Raffaele Spazzoli
  2018-09-17  9:16       ` Ivan Labáth
  0 siblings, 1 reply; 7+ messages in thread
From: Raffaele Spazzoli @ 2018-09-16 23:08 UTC (permalink / raw)
  To: Ivan Labáth; +Cc: wireguard

[-- Attachment #1: Type: text/plain, Size: 8078 bytes --]

Ivan,

I tried the SNAT idea, and still have issue.
here is an example configuration of one of the nodes:

[Interface]
ListenPort = 5555
PrivateKey = ---

[Peer]
PublicKey = H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=
AllowedIPs = 10.128.2.0/23
Endpoint = 192.168.99.12:31112
PersistentKeepalive = 25

[Peer]
PublicKey = 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=
AllowedIPs = 10.128.0.0/23
Endpoint = 192.168.99.14:32188
PersistentKeepalive = 25

[Peer]
PublicKey = MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=
AllowedIPs = 10.130.0.0/23
Endpoint = 192.168.99.13:31992
PersistentKeepalive = 25

[Peer]
PublicKey = s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=
AllowedIPs = 10.129.0.0/23
Endpoint = 192.168.99.15:30305
PersistentKeepalive = 25

[Peer]
PublicKey = SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=
AllowedIPs = 10.131.0.0/23
Endpoint = 192.168.99.7:31714
PersistentKeepalive = 25

[Peer]
PublicKey = a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=
AllowedIPs = 10.129.2.0/23
Endpoint = 192.168.99.6:31165
PersistentKeepalive = 25

the private IP to VIP map for the peers of this node is:

10.128.2.10-192.168.99.12 10.128.1.94-192.168.99.14
10.130.0.136-192.168.99.13 10.129.1.158-192.168.99.15
10.131.0.199-192.168.99.7 10.129.2.217-192.168.99.6

I create the following iptables rules:

sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target prot
opt source destination Chain INPUT (policy ACCEPT) target prot opt source
destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to:
192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to:
192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to:
192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to:
192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to:
192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to:
192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source
destination Chain POSTROUTING (policy ACCEPT) target prot opt source
destination


after the handshake the configuration changes to:

sh-4.2# wg
interface: sdn-tunnel
  public key: gCFgNjOpObU71Vmjub/R9KIn3MHgzXnKtrh9Tf+W628=
  private key: (hidden)
  listening port: 5555

peer: s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=
  endpoint: 10.134.0.1:1033
  allowed ips: 10.129.0.0/23
  latest handshake: 31 seconds ago
  transfer: 180 B received, 452 B sent
  persistent keepalive: every 25 seconds

peer: MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=
  endpoint: 10.134.0.1:1032
  allowed ips: 10.130.0.0/23
  latest handshake: 37 seconds ago
  transfer: 212 B received, 272 B sent
  persistent keepalive: every 25 seconds

peer: 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=
  endpoint: 10.134.0.1:1031
  allowed ips: 10.128.0.0/23
  latest handshake: 39 seconds ago
  transfer: 180 B received, 304 B sent
  persistent keepalive: every 25 seconds

peer: a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=
  endpoint: 192.168.99.6:31165
  allowed ips: 10.129.2.0/23
  latest handshake: 41 seconds ago
  transfer: 156 B received, 180 B sent
  persistent keepalive: every 25 seconds

peer: H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=
  endpoint: 192.168.99.12:31112
  allowed ips: 10.128.2.0/23
  latest handshake: 41 seconds ago
  transfer: 156 B received, 180 B sent
  persistent keepalive: every 25 seconds

peer: SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=
  endpoint: 192.168.99.7:31714
  allowed ips: 10.131.0.0/23
  latest handshake: 41 seconds ago
  transfer: 156 B received, 180 B sent
  persistent keepalive: every 25 seconds

as you can see some of the endpoint's addresses have changed.
the first three are not correct anymore.
After the introduction of the iptables rules they change to an IP that
makes no sense to me 10.134.0.1

finally here are a few seconds of tcpdump in case it helps:

sh-4.2# tcpdump -i eth0 -nn -v tcpdump: listening on eth0, link-type EN10MB
(Ethernet), capture size 262144 bytes 23:07:01.045331 IP (tos 0x0, ttl 64,
id 27711, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 192.168.99.6.31165: UDP, length 32 23:07:01.045363 IP
(tos 0x0, ttl 64, id 53835, offset 0, flags [none], proto UDP (17), length
60) 10.134.0.72.5555 > 192.168.99.7.31714: UDP, length 32 23:07:01.045411
IP (tos 0x0, ttl 64, id 27009, offset 0, flags [none], proto UDP (17),
length 60) 10.134.0.72.5555 > 192.168.99.12.31112: UDP, length 32
23:07:02.758694 IP (tos 0x0, ttl 61, id 19309, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.1.1031 > 10.134.0.72.5555: UDP, length
32 23:07:04.053339 IP (tos 0x0, ttl 64, id 36786, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.72.5555 > 10.134.0.1.1032: UDP, length
32 23:07:07.765375 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
10.134.0.72 tell 10.134.0.1, length 28 23:07:07.765394 ARP, Ethernet (len
6), IPv4 (len 4), Reply 10.134.0.72 is-at 0a:58:0a:86:00:48, length 28
23:07:10.938921 IP (tos 0x0, ttl 61, id 33093, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.1.1033 > 10.134.0.72.5555: UDP, length
32 23:07:26.069271 IP (tos 0x0, ttl 64, id 37778, offset 0, flags [none],
proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.6.31165: UDP,
length 32 23:07:26.069271 IP (tos 0x0, ttl 64, id 59175, offset 0, flags
[none], proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.7.31714:
UDP, length 32 23:07:26.069303 IP (tos 0x0, ttl 64, id 49067, offset 0,
flags [none], proto UDP (17), length 60) 10.134.0.72.5555 >
192.168.99.12.31112: UDP, length 32 23:07:27.797284 IP (tos 0x0, ttl 64, id
57007, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555
> 10.134.0.1.1031: UDP, length 32 23:07:29.079935 IP (tos 0x0, ttl 61, id
18743, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.1.1032 >
10.134.0.72.5555: UDP, length 32

Thanks,
Raffaele

Raffaele Spazzoli
Senior Architect - OpenShift <https://www.openshift.com>, Containers
and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
Tel: +1 216-258-7717



On Sun, Sep 16, 2018 at 2:56 PM, Raffaele Spazzoli <rspazzol@redhat.com>
wrote:

> I'll try to make an example
> cluster 1 node 1 has private IP1 and VIP1
> cluster 2 node 2 has private IP2 and VIP2
>
> each node uses it's private ip for outbound connections.
> each node can receive inbound connection on its VIP.
>
> so the wireguard config file for node1 is going to look like:
>
> [peer]
> endpoint: VIP2:port
>
> and for node 2:
> [peer]
> endpoint: VIP1: port
>
> the problem is that after the handshake, wireguard updates the config to
> the following (for example for node2):
> [peer]
> endpoint: IP1:port
>
> but IP2 cannot route to IP1...
>
> I think a well configured SNAT rule may work, although is not elegant
> because it forces the cluster to exchange information about their private
> IPs.
> This should not be needed and in the cloud private IPs are ephemeral....
>
> anyway thanks for the advice, I am going to try to use it  in my
> prototype.
>
> I still think there is need for a better technical approach for a long
> term solution.
>
>
> Thanks,
> Raffaele
>
> Raffaele Spazzoli
> Senior Architect - OpenShift <https://www.openshift.com>, Containers
> and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
> Tel: +1 216-258-7717
>
>
>
> On Sun, Sep 16, 2018 at 12:54 PM, Ivan Labáth <labawi-wg@matrix-dream.net>
> wrote:
>
>> Hi,
>>
>> On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:
>> > ... then the IP that a  node uses for its outbound
>> > connection is not the same that its peer need to use for its inbound
>> > connections.
>>
>> Who uses what for whose connection? You lost me here.
>> Looks like a broken network to me. Does TCP even work?
>>
>> Anyway, SNAT/DNAT should be able to fix things up, if you want to go
>> that route.
>>
>> Regards,
>> Ivan
>>
>
>

[-- Attachment #2: Type: text/html, Size: 21109 bytes --]

<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Ivan,<div><br></div><div>I tried the SNAT idea, and still have issue. </div><div>here is an example configuration of one of the nodes:</div><div><br></div><div><div>[Interface]</div><div>ListenPort = 5555 </div><div>PrivateKey = ---</div><div> </div><div>[Peer]</div><div>PublicKey = H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=</div><div>AllowedIPs = <a href="http://10.128.2.0/23">10.128.2.0/23</a></div><div>Endpoint = <a href="http://192.168.99.12:31112">192.168.99.12:31112</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=</div><div>AllowedIPs = <a href="http://10.128.0.0/23">10.128.0.0/23</a></div><div>Endpoint = <a href="http://192.168.99.14:32188">192.168.99.14:32188</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=</div><div>AllowedIPs = <a href="http://10.130.0.0/23">10.130.0.0/23</a></div><div>Endpoint = <a href="http://192.168.99.13:31992">192.168.99.13:31992</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=</div><div>AllowedIPs = <a href="http://10.129.0.0/23">10.129.0.0/23</a></div><div>Endpoint = <a href="http://192.168.99.15:30305">192.168.99.15:30305</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=</div><div>AllowedIPs = <a href="http://10.131.0.0/23">10.131.0.0/23</a></div><div>Endpoint = <a href="http://192.168.99.7:31714">192.168.99.7:31714</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=</div><div>AllowedIPs = <a href="http://10.129.2.0/23">10.129.2.0/23</a></div><div>Endpoint = <a href="http://192.168.99.6:31165">192.168.99.6:31165</a></div><div>PersistentKeepalive = 25</div></div><div><br></div><div><font face="arial, helvetica, sans-serif">the private IP to VIP map for the peers of this node is: </font></div><div><br></div><div><span style="color:rgb(54,54,54);font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">10.128.2.10-192.168.99.12   
10.128.1.94-192.168.99.14   
10.130.0.136-192.168.99.13   
10.129.1.158-192.168.99.15   
10.131.0.199-192.168.99.7   
10.129.2.217-192.168.99.6 </span><br></div><div><span style="color:rgb(54,54,54);font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(54,54,54);font-size:12px;white-space:pre;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font face="arial, helvetica, sans-serif">I create the following iptables rules:</font></span></div><div><span style="color:rgb(54,54,54);font-size:12px;white-space:pre;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font face="arial, helvetica, sans-serif"><br></font></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636" face="Menlo, Monaco, Consolas, monospace"><span style="font-size:12px;white-space:pre">sh-4.2# iptables -t nat -n -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
SNAT       udp  --  10.128.2.10          <a href="http://0.0.0.0/0">0.0.0.0/0</a>            udp dpt:5555 to:<a href="http://192.168.99.12:5555">192.168.99.12:5555</a>
SNAT       udp  --  10.128.1.94          <a href="http://0.0.0.0/0">0.0.0.0/0</a>            udp dpt:5555 to:<a href="http://192.168.99.14:5555">192.168.99.14:5555</a>
SNAT       udp  --  10.130.0.136         <a href="http://0.0.0.0/0">0.0.0.0/0</a>            udp dpt:5555 to:<a href="http://192.168.99.13:5555">192.168.99.13:5555</a>
SNAT       udp  --  10.129.1.158         <a href="http://0.0.0.0/0">0.0.0.0/0</a>            udp dpt:5555 to:<a href="http://192.168.99.15:5555">192.168.99.15:5555</a>
SNAT       udp  --  10.131.0.199         <a href="http://0.0.0.0/0">0.0.0.0/0</a>            udp dpt:5555 to:<a href="http://192.168.99.7:5555">192.168.99.7:5555</a>
SNAT       udp  --  10.129.2.217         <a href="http://0.0.0.0/0">0.0.0.0/0</a>            udp dpt:5555 to:<a href="http://192.168.99.6:5555">192.168.99.6:5555</a>

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination</span></font><br></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636" face="Menlo, Monaco, Consolas, monospace"><span style="font-size:12px;white-space:pre"><br></span></font></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636" face="Menlo, Monaco, Consolas, monospace"><span style="font-size:12px;white-space:pre"><br></span></font></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636" face="arial, helvetica, sans-serif"><span style="font-size:12px;white-space:pre">after the handshake the configuration changes to:</span></font></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636"><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace"><span style="font-size:12px;white-space:pre">sh-4.2# wg</span><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">interface: sdn-tunnel</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  public key: gCFgNjOpObU71Vmjub/R9KIn3MHgzXnKtrh9Tf+W628=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  private key: (hidden)</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  listening port: 5555</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  endpoint: <a href="http://10.134.0.1:1033">10.134.0.1:1033</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  allowed ips: <a href="http://10.129.0.0/23">10.129.0.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  latest handshake: 31 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  transfer: 180 B received, 452 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  endpoint: <a href="http://10.134.0.1:1032">10.134.0.1:1032</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  allowed ips: <a href="http://10.130.0.0/23">10.130.0.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  latest handshake: 37 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  transfer: 212 B received, 272 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  endpoint: <a href="http://10.134.0.1:1031">10.134.0.1:1031</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  allowed ips: <a href="http://10.128.0.0/23">10.128.0.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  latest handshake: 39 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  transfer: 180 B received, 304 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  endpoint: <a href="http://192.168.99.6:31165">192.168.99.6:31165</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  allowed ips: <a href="http://10.129.2.0/23">10.129.2.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  latest handshake: 41 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  transfer: 156 B received, 180 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  endpoint: <a href="http://192.168.99.12:31112">192.168.99.12:31112</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  allowed ips: <a href="http://10.128.2.0/23">10.128.2.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  latest handshake: 41 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  transfer: 156 B received, 180 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  endpoint: <a href="http://192.168.99.7:31714">192.168.99.7:31714</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  allowed ips: <a href="http://10.131.0.0/23">10.131.0.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  latest handshake: 41 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  transfer: 156 B received, 180 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">  persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif">as you can see some of the endpoint&#39;s addresses have changed. </font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif">the first three are not correct anymore.</font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif">After the introduction of the iptables rules they change to an IP that makes no sense to me <span style="font-family:Menlo,Monaco,Consolas,monospace;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">10.134.0.1</span></font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif"><span style="font-family:Menlo,Monaco,Consolas,monospace;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">finally here are a few seconds of tcpdump in case it helps:</span></font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif"><span style="font-family:Menlo,Monaco,Consolas,monospace;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></font></div><div><font><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font face="Menlo, Monaco, Consolas, monospace"><span style="font-size:12px;white-space:pre">sh-4.2# tcpdump -i eth0 -nn -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:07:01.045331 IP (tos 0x0, ttl 64, id 27711, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.72.5555 &gt; 192.168.99.6.31165: UDP, length 32
23:07:01.045363 IP (tos 0x0, ttl 64, id 53835, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.72.5555 &gt; 192.168.99.7.31714: UDP, length 32
23:07:01.045411 IP (tos 0x0, ttl 64, id 27009, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.72.5555 &gt; 192.168.99.12.31112: UDP, length 32
23:07:02.758694 IP (tos 0x0, ttl 61, id 19309, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.1.1031 &gt; 10.134.0.72.5555: UDP, length 32
23:07:04.053339 IP (tos 0x0, ttl 64, id 36786, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.72.5555 &gt; 10.134.0.1.1032: UDP, length 32
23:07:07.765375 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.134.0.72 tell 10.134.0.1, length 28
23:07:07.765394 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.134.0.72 is-at 0a:58:0a:86:00:48, length 28
23:07:10.938921 IP (tos 0x0, ttl 61, id 33093, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.1.1033 &gt; 10.134.0.72.5555: UDP, length 32
23:07:26.069271 IP (tos 0x0, ttl 64, id 37778, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.72.5555 &gt; 192.168.99.6.31165: UDP, length 32
23:07:26.069271 IP (tos 0x0, ttl 64, id 59175, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.72.5555 &gt; 192.168.99.7.31714: UDP, length 32
23:07:26.069303 IP (tos 0x0, ttl 64, id 49067, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.72.5555 &gt; 192.168.99.12.31112: UDP, length 32
23:07:27.797284 IP (tos 0x0, ttl 64, id 57007, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.72.5555 &gt; 10.134.0.1.1031: UDP, length 32
23:07:29.079935 IP (tos 0x0, ttl 61, id 18743, offset 0, flags [none], proto UDP (17), length 60)
    10.134.0.1.1032 &gt; 10.134.0.72.5555: UDP, length 32</span></font><br></span></font></div></font></span></div></div></div></div></div></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Thanks,<div>Raffaele</div><div><br></div><div>Raffaele Spazzoli</div><div>Senior Architect - <a href="https://www.openshift.com" target="_blank">OpenShift</a>, <a href="https://www.redhat.com/en/services/consulting/paas" target="_blank">Containers and PaaS Practice</a></div><div>Tel: +1 216-258-7717</div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Sun, Sep 16, 2018 at 2:56 PM, Raffaele Spazzoli <span dir="ltr">&lt;<a href="mailto:rspazzol@redhat.com" target="_blank">rspazzol@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I&#39;ll try to make an example<div>cluster 1 node 1 has private IP1 and VIP1 </div><div>cluster 2 node 2 has private IP2 and VIP2</div><div class="gmail_extra"><br></div><div class="gmail_extra">each node uses it&#39;s private ip for outbound connections. </div><div class="gmail_extra">each node can receive inbound connection on its VIP.</div><div class="gmail_extra"><br></div><div class="gmail_extra">so the wireguard config file for node1 is going to look like:</div><div class="gmail_extra"><br></div><div class="gmail_extra">[peer]</div><div class="gmail_extra">endpoint: VIP2:port</div><div class="gmail_extra"><br></div><div class="gmail_extra">and for node 2:</div><div class="gmail_extra">[peer] </div><div class="gmail_extra">endpoint: VIP1: port</div><div class="gmail_extra"><br></div><div class="gmail_extra">the problem is that after the handshake, wireguard updates the config to the following (for example for node2):</div><div class="gmail_extra">[peer]</div><div class="gmail_extra">endpoint: IP1:port</div><div class="gmail_extra"><br></div><div class="gmail_extra">but IP2 cannot route to IP1...</div><div class="gmail_extra"><br></div><div class="gmail_extra">I think a well configured SNAT rule may work, although is not elegant because it forces the cluster to exchange information about their private IPs.</div><div class="gmail_extra">This should not be needed and in the cloud private IPs are ephemeral....</div><div class="gmail_extra"><br></div><div class="gmail_extra">anyway thanks for the advice, I am going to try to use it  in my prototype. </div><div class="gmail_extra"><br></div><div class="gmail_extra">I still think there is need for a better technical approach for a long term solution.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><span class=""><br clear="all"><div><div class="m_4340499724176942754gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Thanks,<div>Raffaele</div><div><br></div><div>Raffaele Spazzoli</div><div>Senior Architect - <a href="https://www.openshift.com" target="_blank">OpenShift</a>, <a href="https://www.redhat.com/en/services/consulting/paas" target="_blank">Containers and PaaS Practice</a></div><div>Tel: +1 216-258-7717</div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br></span><div><div class="h5"><div class="gmail_quote">On Sun, Sep 16, 2018 at 12:54 PM, Ivan Labáth <span dir="ltr">&lt;<a href="mailto:labawi-wg@matrix-dream.net" target="_blank">labawi-wg@matrix-dream.net</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:<br>
&gt; ... then the IP that a  node uses for its outbound<br>
<span>&gt; connection is not the same that its peer need to use for its inbound<br>
&gt; connections.<br>
<br>
</span>Who uses what for whose connection? You lost me here.<br>
Looks like a broken network to me. Does TCP even work?<br>
<br>
Anyway, SNAT/DNAT should be able to fix things up, if you want to go<br>
that route.<br>
<br>
Regards,<br>
Ivan<br>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: what to do when the peers use different IPs to transmit and receive
  2018-09-16 23:08     ` Raffaele Spazzoli
@ 2018-09-17  9:16       ` Ivan Labáth
  2018-09-17 11:10         ` Raffaele Spazzoli
  0 siblings, 1 reply; 7+ messages in thread
From: Ivan Labáth @ 2018-09-17  9:16 UTC (permalink / raw)
  Cc: wireguard

On Sun, Sep 16, 2018 at 07:08:58PM -0400, Raffaele Spazzoli wrote:
> sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target prot
> opt source destination Chain INPUT (policy ACCEPT) target prot opt source
> destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to:
> 192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source
> destination Chain POSTROUTING (policy ACCEPT) target prot opt source
> destination

Please try to have no or reasonable line wrapping.

If you are applying SNAT on your source node, you are setting
the source address, which should be set to the reachable address
for the replies to come to. In your case VIP.
If you are setting it on the destination, you are IMO doing it wrong.

Same thing applies to TCP and most typical protocol, nothing special
about wireguard here.

If you have a middlebox doing DNAT, it would normaly be expected
for it or something else to do SNAT in the reverse direction.
Or, if your node has both adresses assigned, then it might be
a case of improperly set source address on outgoing packets
(e.g. your routing might need tuning).

Regards,
Ivan

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: what to do when the peers use different IPs to transmit and receive
  2018-09-17  9:16       ` Ivan Labáth
@ 2018-09-17 11:10         ` Raffaele Spazzoli
  2018-09-25 21:16           ` Ivan Labáth
  0 siblings, 1 reply; 7+ messages in thread
From: Raffaele Spazzoli @ 2018-09-17 11:10 UTC (permalink / raw)
  To: Ivan Labáth; +Cc: wireguard

[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]

Ivan,

sorry for the formatting, it seemed right on my email editor (gmail).
I cannot do SNAT at the source because the packet would be dropped if it
didn't come from the actual IP of the VM.
So I am doing SNAT at the destination. why do you say I am doing it wrong?
I know it would be ideal to do it at the source, but should it work when
done at the destination?

Thanks,
Raffaele

Raffaele Spazzoli
Senior Architect - OpenShift <https://www.openshift.com>, Containers
and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
Tel: +1 216-258-7717



On Mon, Sep 17, 2018 at 5:16 AM, Ivan Labáth <labawi-wg@matrix-dream.net>
wrote:

> On Sun, Sep 16, 2018 at 07:08:58PM -0400, Raffaele Spazzoli wrote:
> > sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target
> prot
> > opt source destination Chain INPUT (policy ACCEPT) target prot opt source
> > destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to:
> > 192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source
> > destination Chain POSTROUTING (policy ACCEPT) target prot opt source
> > destination
>
> Please try to have no or reasonable line wrapping.
>
> If you are applying SNAT on your source node, you are setting
> the source address, which should be set to the reachable address
> for the replies to come to. In your case VIP.
> If you are setting it on the destination, you are IMO doing it wrong.
>
> Same thing applies to TCP and most typical protocol, nothing special
> about wireguard here.
>
> If you have a middlebox doing DNAT, it would normaly be expected
> for it or something else to do SNAT in the reverse direction.
> Or, if your node has both adresses assigned, then it might be
> a case of improperly set source address on outgoing packets
> (e.g. your routing might need tuning).
>
> Regards,
> Ivan
> _______________________________________________
> WireGuard mailing list
> WireGuard@lists.zx2c4.com
> https://lists.zx2c4.com/mailman/listinfo/wireguard
>

[-- Attachment #2: Type: text/html, Size: 4349 bytes --]

<div dir="ltr">Ivan,<div><br></div><div>sorry for the formatting, it seemed right on my email editor (gmail).</div><div>I cannot do SNAT at the source because the packet would be dropped if it didn&#39;t come from the actual IP of the VM.</div><div>So I am doing SNAT at the destination. why do you say I am doing it wrong?</div><div>I know it would be ideal to do it at the source, but should it work when done at the destination?</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Thanks,<div>Raffaele</div><div><br></div><div>Raffaele Spazzoli</div><div>Senior Architect - <a href="https://www.openshift.com" target="_blank">OpenShift</a>, <a href="https://www.redhat.com/en/services/consulting/paas" target="_blank">Containers and PaaS Practice</a></div><div>Tel: +1 216-258-7717</div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Mon, Sep 17, 2018 at 5:16 AM, Ivan Labáth <span dir="ltr">&lt;<a href="mailto:labawi-wg@matrix-dream.net" target="_blank">labawi-wg@matrix-dream.net</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Sun, Sep 16, 2018 at 07:08:58PM -0400, Raffaele Spazzoli wrote:<br>
</span><span class="">&gt; sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target prot<br>
&gt; opt source destination Chain INPUT (policy ACCEPT) target prot opt source<br>
&gt; destination SNAT udp -- 10.128.2.10 <a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a> udp dpt:5555 to:<br>
&gt; <a href="http://192.168.99.12:5555" rel="noreferrer" target="_blank">192.168.99.12:5555</a> SNAT udp -- 10.128.1.94 <a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a> udp dpt:5555 to:<br>
&gt; <a href="http://192.168.99.14:5555" rel="noreferrer" target="_blank">192.168.99.14:5555</a> SNAT udp -- 10.130.0.136 <a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a> udp dpt:5555 to:<br>
&gt; <a href="http://192.168.99.13:5555" rel="noreferrer" target="_blank">192.168.99.13:5555</a> SNAT udp -- 10.129.1.158 <a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a> udp dpt:5555 to:<br>
&gt; <a href="http://192.168.99.15:5555" rel="noreferrer" target="_blank">192.168.99.15:5555</a> SNAT udp -- 10.131.0.199 <a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a> udp dpt:5555 to:<br>
&gt; <a href="http://192.168.99.7:5555" rel="noreferrer" target="_blank">192.168.99.7:5555</a> SNAT udp -- 10.129.2.217 <a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a> udp dpt:5555 to:<br>
&gt; <a href="http://192.168.99.6:5555" rel="noreferrer" target="_blank">192.168.99.6:5555</a> Chain OUTPUT (policy ACCEPT) target prot opt source<br>
&gt; destination Chain POSTROUTING (policy ACCEPT) target prot opt source<br>
&gt; destination<br>
<br>
</span><span class="">Please try to have no or reasonable line wrapping.<br>
<br>
If you are applying SNAT on your source node, you are setting<br>
the source address, which should be set to the reachable address<br>
for the replies to come to. In your case VIP.<br>
If you are setting it on the destination, you are IMO doing it wrong.<br>
<br>
Same thing applies to TCP and most typical protocol, nothing special<br>
about wireguard here.<br>
<br>
If you have a middlebox doing DNAT, it would normaly be expected<br>
for it or something else to do SNAT in the reverse direction.<br>
Or, if your node has both adresses assigned, then it might be<br>
a case of improperly set source address on outgoing packets<br>
(e.g. your routing might need tuning).<br>
<br>
Regards,<br>
Ivan<br>
</span>______________________________<wbr>_________________<br>
WireGuard mailing list<br>
<a href="mailto:WireGuard@lists.zx2c4.com">WireGuard@lists.zx2c4.com</a><br>
<a href="https://lists.zx2c4.com/mailman/listinfo/wireguard" rel="noreferrer" target="_blank">https://lists.zx2c4.com/<wbr>mailman/listinfo/wireguard</a><br>
</blockquote></div><br></div>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: what to do when the peers use different IPs to transmit and receive
  2018-09-17 11:10         ` Raffaele Spazzoli
@ 2018-09-25 21:16           ` Ivan Labáth
  0 siblings, 0 replies; 7+ messages in thread
From: Ivan Labáth @ 2018-09-25 21:16 UTC (permalink / raw)
  To: Raffaele Spazzoli; +Cc: wireguard

Hi,

I guess SNAT-ing on the destination node should work.

Still, without a compelling use case, it seems
like a broken network to me. You will have packets
in transit with an unroutable/incorrect source address,
among other possible issues. ICMP signalling won't work.
I think TCP wouldn't work with such a setup, or even
most udp protocols.

Do you use other services over this type of network?

Also, please note that iptables rules can affect network performance,
especially if there are many of them and haphazardly placed.
For debugging iptables rules I would suggest adding -v, so you can see
the number of matched packets (caveats apply in NAT).

Regards,
Ivan Labáth


On Mon, Sep 17, 2018 at 07:10:05AM -0400, Raffaele Spazzoli wrote:
> Ivan,
> 
> sorry for the formatting, it seemed right on my email editor (gmail).
> I cannot do SNAT at the source because the packet would be dropped if it
> didn't come from the actual IP of the VM.
> So I am doing SNAT at the destination. why do you say I am doing it wrong?
> I know it would be ideal to do it at the source, but should it work when
> done at the destination?
> 
> Thanks,
> Raffaele
> 
> Raffaele Spazzoli
> Senior Architect - OpenShift <https://www.openshift.com>, Containers
> and PaaS Practice <https://www.redhat.com/en/services/consulting/paas>
> Tel: +1 216-258-7717
> 
> 
> 
> On Mon, Sep 17, 2018 at 5:16 AM, Ivan Labáth <labawi-wg@matrix-dream.net>
> wrote:
> 
> > On Sun, Sep 16, 2018 at 07:08:58PM -0400, Raffaele Spazzoli wrote:
> > > sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target
> > prot
> > > opt source destination Chain INPUT (policy ACCEPT) target prot opt source
> > > destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to:
> > > 192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source
> > > destination Chain POSTROUTING (policy ACCEPT) target prot opt source
> > > destination
> >
> > Please try to have no or reasonable line wrapping.
> >
> > If you are applying SNAT on your source node, you are setting
> > the source address, which should be set to the reachable address
> > for the replies to come to. In your case VIP.
> > If you are setting it on the destination, you are IMO doing it wrong.
> >
> > Same thing applies to TCP and most typical protocol, nothing special
> > about wireguard here.
> >
> > If you have a middlebox doing DNAT, it would normaly be expected
> > for it or something else to do SNAT in the reverse direction.
> > Or, if your node has both adresses assigned, then it might be
> > a case of improperly set source address on outgoing packets
> > (e.g. your routing might need tuning).
> >
> > Regards,
> > Ivan
> > _______________________________________________
> > WireGuard mailing list
> > WireGuard@lists.zx2c4.com
> > https://lists.zx2c4.com/mailman/listinfo/wireguard
> >
_______________________________________________
WireGuard mailing list
WireGuard@lists.zx2c4.com
https://lists.zx2c4.com/mailman/listinfo/wireguard

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, back to index

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-09-16 12:21 what to do when the peers use different IPs to transmit and receive Raffaele Spazzoli
2018-09-16 16:54 ` Ivan Labáth
2018-09-16 18:56   ` Raffaele Spazzoli
2018-09-16 23:08     ` Raffaele Spazzoli
2018-09-17  9:16       ` Ivan Labáth
2018-09-17 11:10         ` Raffaele Spazzoli
2018-09-25 21:16           ` Ivan Labáth

WireGuard Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/wireguard/0 wireguard/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 wireguard wireguard/ https://lore.kernel.org/wireguard \
		wireguard@lists.zx2c4.com zx2c4-wireguard@archiver.kernel.org
	public-inbox-index wireguard


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/com.zx2c4.lists.wireguard


AGPL code for this site: git clone https://public-inbox.org/ public-inbox