multi-home difficulty

multi-home difficulty

[d tbsky]

Hi:
   I tested wireguard and the speed is amazing. but when I try to
deploy it to our real linux firewall, I found it is hard to make it
work.

   our current linux firewall have multiple interface and multiple
routing tables. local program will get lan ip address and nat to
correct wan ip address when goto internet.

  since wireguard can not bind to specific ip address, it sometimes
use wrong ip address to reply and the vpn communication can not be
established.

for example:

config for client site: (assume wan ip is 2.2.2.2)
interface: wg0
  public key: ****
  private key: (hidden)
  listening port: 51820
peer: ****
  endpoint: 1.1.1.1:51820
  allowed ips: 0.0.0.0/0

config for server site: (assume wan ip is 1.1.1.1)
interface: wg0
  public key: ****
  private key: (hidden)
  listening port: 51820
peer: ****
  allowed ips: 0.0.0.0/0

when client initial connect to server, at server site I saw  flow like below:
"cat /proc/net/nf_conntrack | grep 51820"

ipv4     2 udp      17 23 src=172.18.1.254 dst=2.2.2.2 sport=51820
dport=51820 packets=1 bytes=120 [UNREPLIED] src=2.2.2.2 dst=1.1.1.1
sport=51820 dport=1085 packets=0 bytes=0 mark=1 zone=0 use=2
ipv4     2 udp      17 23 src=2.2.2.2 dst=1.1.1.1 sport=51820
dport=51820 packets=1 bytes=176 [UNREPLIED] src=1.1.1.1 dst=2.2.2.2
sport=51820 dport=51820 packets=0 bytes=0 mark=1 zone=0 use=2

so at first client  2.2.2.2:51820 connect to server 1.1.1.1:51820
but then server use 172.18.1.254(lan ip address) to reply and 51820
port is nat to 1085 so the communication is broken.

if wireguard can bind to specific ip address then there will be no problem.
or if wireguard can reply with the correct ip address.( eg: if client
connect to wireguard ip 1.1.1.1, then wiregurad should reply via ip
address 1.1.1.1) then maybe there will be no problem.

Regards,
tbskyd

Re: multi-home difficulty

[Tomas Herceg]

[-- Attachment #1.1: Type: text/plain, Size: 2235 bytes --]

+1 for binding only on specific IP

On 11/21/2017 02:21 PM, d tbsky wrote:
> Hi:
>    I tested wireguard and the speed is amazing. but when I try to
> deploy it to our real linux firewall, I found it is hard to make it
> work.
> 
>    our current linux firewall have multiple interface and multiple
> routing tables. local program will get lan ip address and nat to
> correct wan ip address when goto internet.
> 
>   since wireguard can not bind to specific ip address, it sometimes
> use wrong ip address to reply and the vpn communication can not be
> established.
> 
> for example:
> 
> config for client site: (assume wan ip is 2.2.2.2)
> interface: wg0
>   public key: ****
>   private key: (hidden)
>   listening port: 51820
> peer: ****
>   endpoint: 1.1.1.1:51820
>   allowed ips: 0.0.0.0/0
> 
> config for server site: (assume wan ip is 1.1.1.1)
> interface: wg0
>   public key: ****
>   private key: (hidden)
>   listening port: 51820
> peer: ****
>   allowed ips: 0.0.0.0/0
> 
> when client initial connect to server, at server site I saw  flow like below:
> "cat /proc/net/nf_conntrack | grep 51820"
> 
> ipv4     2 udp      17 23 src=172.18.1.254 dst=2.2.2.2 sport=51820
> dport=51820 packets=1 bytes=120 [UNREPLIED] src=2.2.2.2 dst=1.1.1.1
> sport=51820 dport=1085 packets=0 bytes=0 mark=1 zone=0 use=2
> ipv4     2 udp      17 23 src=2.2.2.2 dst=1.1.1.1 sport=51820
> dport=51820 packets=1 bytes=176 [UNREPLIED] src=1.1.1.1 dst=2.2.2.2
> sport=51820 dport=51820 packets=0 bytes=0 mark=1 zone=0 use=2
> 
> so at first client  2.2.2.2:51820 connect to server 1.1.1.1:51820
> but then server use 172.18.1.254(lan ip address) to reply and 51820
> port is nat to 1085 so the communication is broken.
> 
> if wireguard can bind to specific ip address then there will be no problem.
> or if wireguard can reply with the correct ip address.( eg: if client
> connect to wireguard ip 1.1.1.1, then wiregurad should reply via ip
> address 1.1.1.1) then maybe there will be no problem.
> 
> Regards,
> tbskyd
> _______________________________________________
> WireGuard mailing list
> WireGuard@lists.zx2c4.com
> https://lists.zx2c4.com/mailman/listinfo/wireguard
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: multi-home difficulty

[Jason A. Donenfeld]

On Tue, Nov 21, 2017 at 2:21 PM, d tbsky <tbskyd@gmail.com> wrote:
> so at first client  2.2.2.2:51820 connect to server 1.1.1.1:51820
> but then server use 172.18.1.254(lan ip address) to reply and 51820
> port is nat to 1085 so the communication is broken.

The server should use 1.1.1.1 to reply. If it's not, that's a bug that
I should fix. Can you give me a minimal configuration for reproducing
this setup, so that I can fix whatever issue is occurring?

Thanks,
Jason

Re: multi-home difficulty

[d tbsky]

2017-11-21 22:15 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
> On Tue, Nov 21, 2017 at 2:21 PM, d tbsky <tbskyd@gmail.com> wrote:
>> so at first client  2.2.2.2:51820 connect to server 1.1.1.1:51820
>> but then server use 172.18.1.254(lan ip address) to reply and 51820
>> port is nat to 1085 so the communication is broken.
>
> The server should use 1.1.1.1 to reply. If it's not, that's a bug that
> I should fix. Can you give me a minimal configuration for reproducing
> this setup, so that I can fix whatever issue is occurring?
>
> Thanks,
> Jason

thanks for the quick reply. my wireguard configuration is in the
previous mail, so I think the linux firewall part is what you want.
there is only one thing special in our firewall config. normally when
you use "ip route get 8.8.8.8", you will get a wan ip address through
main routing table(eg 1.1.1.1 in above example) . but since we have
multiple routing tables and there is little entries in main routing
table,  "ip route get 8.8.8.8" will get 172.18.1.254 (lan ip address)
in our firewall.

I don't know how wireguard decide its replying ip address, but it
seems wrong under the situation. maybe it decide it through main
routing table?

our linux firewall environment is RHEL 7.4 and wireguard version is
0.0.20171111 from official repository.

thanks a lot  for help!

Regards,
tbskyd

Re: multi-home difficulty

[Jason A. Donenfeld]

On Tue, Nov 21, 2017 at 3:35 PM, d tbsky <tbskyd@gmail.com> wrote:
> thanks for the quick reply. my wireguard configuration is in the
> previous mail, so I think the linux firewall part is what you want.

Right. So if you can give me minimal instructions on how to set up a
box that exhibits the buggy behavior you're seeing, I can try to fix
it.

Jason

Re: multi-home difficulty

[d tbsky]

2017-11-23 7:35 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
> On Tue, Nov 21, 2017 at 3:35 PM, d tbsky <tbskyd@gmail.com> wrote:
>> thanks for the quick reply. my wireguard configuration is in the
>> previous mail, so I think the linux firewall part is what you want.
>
> Right. So if you can give me minimal instructions on how to set up a
> box that exhibits the buggy behavior you're seeing, I can try to fix
> it.
>
> Jason

sorry for the delay.  I try to make a minimal config to reproduce the
problem in our firewall, but it's not easy. the communication
sometimes works, sometimes failed. suddenly I remember many years ago
I got similar problems with openvpn. according the manual pages of
openvpn:

 --multihome
              Configure a multi-homed UDP server.  This option needs
to be used when a server has more  than  one  IP
              address  (e.g. multiple interfaces, or secondary IP
addresses), and is not using --local to force bind=E2=80=90
              ing to one specific address only.  This option will add
some extra lookups to the packet path to ensure
              that  the UDP reply packets are always sent from the
address that the client is talking to. This is not
              supported on all platforms, and it adds more processing,
so it's not enabled by default.

              Note: this option is only relevant for UDP servers.

              Note 2: if you do an IPv6+IPv4 dual-stack bind on a
Linux machine with multiple IPv4  address,  connec=E2=80=90
              tions  to  IPv4 addresses will not work right on kernels
before 3.15, due to missing kernel support for
              the IPv4-mapped case (some distributions have ported
this to earlier kernel versions, though).

  I forgot these. many strange things happen if you didn't bind
specific ip, even with "--multihome"

  finally I made a environment for you to test. my OS is rehl 7.4,
kernel version 3.10.0-693.5.2

  1. build a virtual rhel 7.4 box, bind 2 virtio nic to it. (single
nic won't show the problem, I don't now why).
  2. stop NetworkManager
  3. setup network environment like below(skip eth0, setup eth1 with
two ip addresses):

ip addr flush dev eth1
ip addr add 10.99.1.99/24 dev eth1
ip addr add 10.99.1.100/24 dev eth1
ip link set eth1 up
ip route add default via 10.99.1.254

ip link add wg0 type wireguard
ip addr add 172.31.21.1 peer 172.31.21.2 dev wg0
wg setconf wg0 /root/server.conf
ip link set wg0 up

/root/server.conf like below:
[Interface]
PrivateKey =3D ****
ListenPort =3D 51820
[Peer]
PublicKey =3D ****
AllowedIPs =3D 0.0.0.0/0

    4. setup wireguard at client site. client.conf like below:

[Interface]
PrivateKey =3D ****
ListenPort =3D 51820
[Peer]
PublicKey =3D ****
Endpoint =3D  10.99.1.100:51820
AllowedIPs =3D 0.0.0.0/0

    5. at client site, "ping 172.31.21.1".

    6. at server site, "modprobe nf_conntrack_ipv4;cat
/proc/net/nf_conntrack | grep 51820":

ipv4     2 udp      17 29 src=3D10.99.1.99 dst=3D10.99.20.254 sport=3D51820
dport=3D51820 [UNREPLIED] src=3D10.99.20.254 dst=3D10.99.1.99 sport=3D51820
dport=3D51820 mark=3D0 zone=3D0 use=3D2
ipv4     2 udp      17 29 src=3D10.99.20.254 dst=3D10.99.1.100 sport=3D5182=
0
dport=3D51820 [UNREPLIED] src=3D10.99.1.100 dst=3D10.99.20.254 sport=3D5182=
0
dport=3D51820 mark=3D0 zone=3D0 use=3D2

   I don't know if you can reproduce in your environment.
   hope  wireguard can bind to specific ip in the future..

Regards,
tbskyd

Re: multi-home difficulty

[d tbsky]

2017-11-23 7:35 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
> On Tue, Nov 21, 2017 at 3:35 PM, d tbsky <tbskyd@gmail.com> wrote:
>> thanks for the quick reply. my wireguard configuration is in the
>> previous mail, so I think the linux firewall part is what you want.
>
> Right. So if you can give me minimal instructions on how to set up a
> box that exhibits the buggy behavior you're seeing, I can try to fix
> it.
>
> Jason

Hi jason:

    are you still interested with the problem? today I try to use
multi-home client to connect server. and I found not only server, but
client suffered.
the problem seems at rhel linux kernel side, but I am not sure. since
wireguard was the only victim I met.

   I can create a virtual machine with ssh access  if you want to test
these strange problems.

  btw, is it possible that wireguard bind to specific ip in the
future? I think it will solve all the problems, but maybe you have
technical concerns.

  thanks a lot for your help.

Regards,
tbskyd

Re: multi-home difficulty

[Jason A. Donenfeld]

On Wed, Nov 29, 2017 at 12:05 PM, d tbsky <tbskyd@gmail.com> wrote:
>     are you still interested with the problem?

Yes, patience please. Lots of things in motion at the moment.

Re: multi-home difficulty

[Jason A. Donenfeld]

Hi,

I made a small script in order to reproduce this issue, but I was not
able to replicate the results. Would you spend some time with the below
code tweaking it so that it exhibits the broken behavior you're seeing?

Jason

==== script (please mind the use of literal \t) ====

#!/bin/bash
set -e

exec 3>&1
export WG_HIDE_KEYS=never
netns1="wg-test-$$-1"
netns2="wg-test-$$-2"
pretty() { echo -e "\x1b[32m\x1b[1m[+] ${1:+NS$1: }${2}\x1b[0m" >&3; }
pp() { pretty "" "$*"; "$@"; }
maybe_exec() { if [[ $BASHPID -eq $$ ]]; then "$@"; else exec "$@"; fi; }
n1() { pretty 1 "$*"; maybe_exec ip netns exec $netns1 "$@"; }
n2() { pretty 2 "$*"; maybe_exec ip netns exec $netns2 "$@"; }
ip1() { pretty 1 "ip $*"; ip -n $netns1 "$@"; }
ip2() { pretty 2 "ip $*"; ip -n $netns2 "$@"; }
sleep() { read -t "$1" -N 0 || true; }
waitiface() { pretty "${1//*-}" "wait for $2 to come up"; ip netns exec "$1" bash -c "while [[ \$(< \"/sys/class/net/$2/operstate\") != up ]]; do read -t .1 -N 0 || true; done;"; }

cleanup() {
	set +e
	exec 2>/dev/null
	ip1 link del dev wg0
	ip2 link del dev wg0
	local to_kill="$(ip netns pids $netns1) $(ip netns pids $netns2)"
	[[ -n $to_kill ]] && kill $to_kill
	pp ip netns del $netns1
	pp ip netns del $netns2
	exit
}

trap cleanup EXIT

ip netns del $netns1 2>/dev/null || true
ip netns del $netns2 2>/dev/null || true
pp ip netns add $netns1
pp ip netns add $netns2

key1="$(pp wg genkey)"
key2="$(pp wg genkey)"
pub1="$(pp wg pubkey <<<"$key1")"
pub2="$(pp wg pubkey <<<"$key2")"
psk="$(pp wg genpsk)"
[[ -n $key1 && -n $key2 && -n $psk ]]

configure_peers() {
	ip1 addr add 192.168.241.1/24 dev wg0
	ip2 addr add 192.168.241.2/24 dev wg0

	n1 wg set wg0 \
		private-key <(echo "$key1") \
		listen-port 1 \
		peer "$pub2" \
			preshared-key <(echo "$psk") \
			allowed-ips 192.168.241.2/32,fd00::2/128
	n2 wg set wg0 \
		private-key <(echo "$key2") \
		listen-port 2 \
		peer "$pub1" \
			preshared-key <(echo "$psk") \
			allowed-ips 192.168.241.1/32,fd00::1/128

	ip1 link set up dev wg0
	ip2 link set up dev wg0
}

n1 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
n2 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
n1 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6'
n2 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6'

ip1 link add dev wg0 type wireguard
ip2 link add dev wg0 type wireguard
configure_peers

ip1 link add veth1 type veth peer name veth2
ip1 link set veth2 netns $netns2

ip1 addr add 10.0.0.1/24 dev veth1
ip1 addr add 10.0.0.2/24 dev veth1
ip2 addr add 10.0.0.3/24 dev veth2

ip1 link set veth1 up
ip2 link set veth2 up
waitiface $netns1 veth1
waitiface $netns2 veth2

n1 iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
n2 iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

n2 wg set wg0 peer "$pub1" endpoint 10.0.0.1:1
n2 ping -W 1 -c 5 -f 192.168.241.1
[[ $(n2 wg show wg0 endpoints) == "$pub1	10.0.0.1:1" ]]

n1 conntrack -L
n2 conntrack -L

n2 wg set wg0 peer "$pub1" endpoint 10.0.0.2:1
n2 ping -W 1 -c 5 -f 192.168.241.1
[[ $(n2 wg show wg0 endpoints) == "$pub1	10.0.0.2:1" ]]

n1 conntrack -L
n2 conntrack -L

==== output ====

[+] ip netns add wg-test-32269-1
[+] ip netns add wg-test-32269-2
[+] wg genkey
[+] wg genkey
[+] wg pubkey
[+] wg pubkey
[+] wg genpsk
[+] NS1: bash -c echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
[+] NS2: bash -c echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
[+] NS1: bash -c echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
[+] NS2: bash -c echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
[+] NS1: ip link add dev wg0 type wireguard
[+] NS2: ip link add dev wg0 type wireguard
[+] NS1: ip addr add 192.168.241.1/24 dev wg0
[+] NS2: ip addr add 192.168.241.2/24 dev wg0
[+] NS1: wg set wg0 private-key /dev/fd/63 listen-port 1 peer NNBvFmhApGEcgy8erS6bCLUi3+nRmg2mzV/xvek9PG0= preshared-key /dev/fd/62 allowed-ips 192.168.241.2/32,fd00::2/128
[+] NS2: wg set wg0 private-key /dev/fd/63 listen-port 2 peer nkdJlCF8z2+MH7aZV0FN9iO6UM+MUbPebADldwJmNRc= preshared-key /dev/fd/62 allowed-ips 192.168.241.1/32,fd00::1/128
[+] NS1: ip link set up dev wg0
[+] NS2: ip link set up dev wg0
[+] NS1: ip link add veth1 type veth peer name veth2
[+] NS1: ip link set veth2 netns wg-test-32269-2
[+] NS1: ip addr add 10.0.0.1/24 dev veth1
[+] NS1: ip addr add 10.0.0.2/24 dev veth1
[+] NS2: ip addr add 10.0.0.3/24 dev veth2
[+] NS1: ip link set veth1 up
[+] NS2: ip link set veth2 up
[+] NS1: wait for veth1 to come up
[+] NS2: wait for veth2 to come up
[+] NS1: iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
[+] NS2: iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
[+] NS2: wg set wg0 peer nkdJlCF8z2+MH7aZV0FN9iO6UM+MUbPebADldwJmNRc= endpoint 10.0.0.1:1
[+] NS2: ping -W 1 -c 5 -f 192.168.241.1
PING 192.168.241.1 (192.168.241.1) 56(84) bytes of data.

--- 192.168.241.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 1ms
rtt min/avg/max/mdev = 0.073/0.256/0.915/0.329 ms, ipg/ewma 0.339/0.575 ms
[+] NS2: wg show wg0 endpoints
[+] NS1: conntrack -L
icmp     1 29 src=192.168.241.2 dst=192.168.241.1 type=8 code=0 id=32322 src=192.168.241.1 dst=192.168.241.2 type=0 code=0 id=32322 mark=0 use=1
udp      17 179 src=10.0.0.3 dst=10.0.0.1 sport=2 dport=1 src=10.0.0.1 dst=10.0.0.3 sport=1 dport=2 [ASSURED] mark=0 use=1
conntrack v1.4.4 (conntrack-tools): 2 flow entries have been shown.
[+] NS2: conntrack -L
udp      17 179 src=10.0.0.3 dst=10.0.0.1 sport=2 dport=1 src=10.0.0.1 dst=10.0.0.3 sport=1 dport=2 [ASSURED] mark=0 use=1
icmp     1 29 src=192.168.241.2 dst=192.168.241.1 type=8 code=0 id=32322 src=192.168.241.1 dst=192.168.241.2 type=0 code=0 id=32322 mark=0 use=1
conntrack v1.4.4 (conntrack-tools): 2 flow entries have been shown.
[+] NS2: wg set wg0 peer nkdJlCF8z2+MH7aZV0FN9iO6UM+MUbPebADldwJmNRc= endpoint 10.0.0.2:1
[+] NS2: ping -W 1 -c 5 -f 192.168.241.1
PING 192.168.241.1 (192.168.241.1) 56(84) bytes of data.

--- 192.168.241.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.067/0.153/0.320/0.097 ms, ipg/ewma 0.205/0.172 ms
[+] NS2: wg show wg0 endpoints
[+] NS1: conntrack -L
udp      17 179 src=10.0.0.3 dst=10.0.0.2 sport=2 dport=1 src=10.0.0.2 dst=10.0.0.3 sport=1 dport=2 [ASSURED] mark=0 use=1
icmp     1 29 src=192.168.241.2 dst=192.168.241.1 type=8 code=0 id=32327 src=192.168.241.1 dst=192.168.241.2 type=0 code=0 id=32327 mark=0 use=1
icmp     1 29 src=192.168.241.2 dst=192.168.241.1 type=8 code=0 id=32322 src=192.168.241.1 dst=192.168.241.2 type=0 code=0 id=32322 mark=0 use=1
udp      17 179 src=10.0.0.3 dst=10.0.0.1 sport=2 dport=1 src=10.0.0.1 dst=10.0.0.3 sport=1 dport=2 [ASSURED] mark=0 use=1
conntrack v1.4.4 (conntrack-tools): 4 flow entries have been shown.
[+] NS2: conntrack -L
icmp     1 29 src=192.168.241.2 dst=192.168.241.1 type=8 code=0 id=32327 src=192.168.241.1 dst=192.168.241.2 type=0 code=0 id=32327 mark=0 use=1
udp      17 179 src=10.0.0.3 dst=10.0.0.1 sport=2 dport=1 src=10.0.0.1 dst=10.0.0.3 sport=1 dport=2 [ASSURED] mark=0 use=1
icmp     1 29 src=192.168.241.2 dst=192.168.241.1 type=8 code=0 id=32322 src=192.168.241.1 dst=192.168.241.2 type=0 code=0 id=32322 mark=0 use=1
udp      17 179 src=10.0.0.3 dst=10.0.0.2 sport=2 dport=1 src=10.0.0.2 dst=10.0.0.3 sport=1 dport=2 [ASSURED] mark=0 use=1
conntrack v1.4.4 (conntrack-tools): 4 flow entries have been shown.
[+] NS1: ip link del dev wg0
[+] NS2: ip link del dev wg0
[+] ip netns del wg-test-32269-1
[+] ip netns del wg-test-32269-2

Re: multi-home difficulty

[d tbsky]

2017-11-29 21:51 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
> Hi,
>
> I made a small script in order to reproduce this issue, but I was not
> able to replicate the results. Would you spend some time with the below
> code tweaking it so that it exhibits the broken behavior you're seeing?
>
> Jason

Hi jason:

    may I ask what kernel/distribution are you using? since under my
testing, just add an unused nic shows the problem, so I think it's
some kind of racing or conflict. or simply RHEL7.4 kernel bug maybe.
I don't know how to modify the script to reveal the problem in your
environment. I will try to install mainline kernel to see if the
problem is the same. or I can build an RHEL 7.4 vm  and let you ssh in
for testing  if you don' t mind.

Regards,
tbskyd

Re: multi-home difficulty

[Jason A. Donenfeld]

Hi tbskyd,

This is on 4.14.2. Would you confirm that this is an issue on your
kernel by actually _running that script and sending the output to the
list_? It would also be helpful to have the output of uname -a.

Jason

Re: multi-home difficulty

[d tbsky]

2017-11-29 22:10 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
> Hi tbskyd,
>
> This is on 4.14.2. Would you confirm that this is an issue on your
> kernel by actually _running that script and sending the output to the
> list_? It would also be helpful to have the output of uname -a.
>
> Jason

  Hi jason:

     sorry I misunderstand you. you mean I modify the script and run
in my environment to reveal the problem?
ok I will try to do it.


Regards,
tbskyd

Re: multi-home difficulty

[Jason A. Donenfeld]

On Wed, Nov 29, 2017 at 3:16 PM, d tbsky <tbskyd@gmail.com> wrote:
>      sorry I misunderstand you. you mean I modify the script and run
> in my environment to reveal the problem?
> ok I will try to do it.

Take what I sent you. Run it. If it breaks, send me the output and
your kernel. If it doesn't break, mess with it until it breaks, and
then send it back to me.

Re: multi-home difficulty

[d tbsky]

2017-11-29 22:49 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
> On Wed, Nov 29, 2017 at 3:16 PM, d tbsky <tbskyd@gmail.com> wrote:
>>      sorry I misunderstand you. you mean I modify the script and run
>> in my environment to reveal the problem?
>> ok I will try to do it.
>
> Take what I sent you. Run it. If it breaks, send me the output and
> your kernel. If it doesn't break, mess with it until it breaks, and
> then send it back to me.

Hi jason:

 "uname -a" result:

 Linux localhost.localdomain 3.10.0-693.5.2.el7.x86_64 #1 SMP Thu Oct
19 10:13:14 CDT 2017 x86_64 x86_64 x86_64 GNU/Linux

 your original script runs fine under my environment.
 I add  three 3 lines before "ip1 link add veth1"  to reveal the problem:

ip1 link add dummy0 type dummy
ip1 addr add 10.0.0.10/24 dev dummy0
ip1 link set dummy0 up

===== whole script below ======
#!/bin/bash
set -e

exec 3>&1
export WG_HIDE_KEYS=never
netns1="wg-test-$$-1"
netns2="wg-test-$$-2"
pretty() { echo -e "\x1b[32m\x1b[1m[+] ${1:+NS$1: }${2}\x1b[0m" >&3; }
pp() { pretty "" "$*"; "$@"; }
maybe_exec() { if [[ $BASHPID -eq $$ ]]; then "$@"; else exec "$@"; fi; }
n1() { pretty 1 "$*"; maybe_exec ip netns exec $netns1 "$@"; }
n2() { pretty 2 "$*"; maybe_exec ip netns exec $netns2 "$@"; }
ip1() { pretty 1 "ip $*"; ip -n $netns1 "$@"; }
ip2() { pretty 2 "ip $*"; ip -n $netns2 "$@"; }
sleep() { read -t "$1" -N 0 || true; }
waitiface() { pretty "${1//*-}" "wait for $2 to come up"; ip netns
exec "$1" bash -c "while [[ \$(< \"/sys/class/net/$2/operstate\") !=
up ]]; do read -t .1 -N 0 || true; done;"; }

cleanup() {
        set +e
        exec 2>/dev/null
        ip1 link del dev wg0
        ip2 link del dev wg0
        local to_kill="$(ip netns pids $netns1) $(ip netns pids $netns2)"
        [[ -n $to_kill ]] && kill $to_kill
        pp ip netns del $netns1
        pp ip netns del $netns2
        exit
}

trap cleanup EXIT

ip netns del $netns1 2>/dev/null || true
ip netns del $netns2 2>/dev/null || true
pp ip netns add $netns1
pp ip netns add $netns2

key1="$(pp wg genkey)"
key2="$(pp wg genkey)"
pub1="$(pp wg pubkey <<<"$key1")"
pub2="$(pp wg pubkey <<<"$key2")"
psk="$(pp wg genpsk)"
[[ -n $key1 && -n $key2 && -n $psk ]]

configure_peers() {
        ip1 addr add 192.168.241.1/24 dev wg0
        ip2 addr add 192.168.241.2/24 dev wg0

        n1 wg set wg0 \
                private-key <(echo "$key1") \
                listen-port 1 \
                peer "$pub2" \
                        preshared-key <(echo "$psk") \
                        allowed-ips 192.168.241.2/32,fd00::2/128
        n2 wg set wg0 \
                private-key <(echo "$key2") \
                listen-port 2 \
                peer "$pub1" \
                        preshared-key <(echo "$psk") \
                        allowed-ips 192.168.241.1/32,fd00::1/128

        ip1 link set up dev wg0
        ip2 link set up dev wg0
}

n1 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
n2 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
n1 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6'
n2 bash -c 'echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6'

ip1 link add dev wg0 type wireguard
ip2 link add dev wg0 type wireguard
configure_peers

ip1 link add dummy0 type dummy
ip1 addr add 10.0.0.10/24 dev dummy0
ip1 link set dummy0 up

ip1 link add veth1 type veth peer name veth2
ip1 link set veth2 netns $netns2

ip1 addr add 10.0.0.1/24 dev veth1
ip1 addr add 10.0.0.2/24 dev veth1
ip2 addr add 10.0.0.3/24 dev veth2

ip1 link set veth1 up
ip2 link set veth2 up
waitiface $netns1 veth1
waitiface $netns2 veth2

n1 iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
n2 iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

n2 wg set wg0 peer "$pub1" endpoint 10.0.0.1:1
n2 ping -W 1 -c 5 -f 192.168.241.1
[[ $(n2 wg show wg0 endpoints) == "$pub1        10.0.0.1:1" ]]

n1 conntrack -L
n2 conntrack -L

n2 wg set wg0 peer "$pub1" endpoint 10.0.0.2:1
n2 ping -W 1 -c 5 -f 192.168.241.1
[[ $(n2 wg show wg0 endpoints) == "$pub1        10.0.0.2:1" ]]

n1 conntrack -L
n2 conntrack -L

Re: multi-home difficulty

[d tbsky]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

2017-11-30 14:15 GMT+08:00 d tbsky <tbskyd@gmail.com>:
> 2017-11-29 22:49 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
>> On Wed, Nov 29, 2017 at 3:16 PM, d tbsky <tbskyd@gmail.com> wrote:
>>>      sorry I misunderstand you. you mean I modify the script and run
>>> in my environment to reveal the problem?
>>> ok I will try to do it.
>>
>> Take what I sent you. Run it. If it breaks, send me the output and
>> your kernel. If it doesn't break, mess with it until it breaks, and
>> then send it back to me.
>
> Hi jason:
>
>  "uname -a" result:
>
>  Linux localhost.localdomain 3.10.0-693.5.2.el7.x86_64 #1 SMP Thu Oct
> 19 10:13:14 CDT 2017 x86_64 x86_64 x86_64 GNU/Linux
>
>  your original script runs fine under my environment.
>  I add  three 3 lines before "ip1 link add veth1"  to reveal the problem:
>
> ip1 link add dummy0 type dummy
> ip1 addr add 10.0.0.10/24 dev dummy0
> ip1 link set dummy0 up
>

sorry my fault. incorrect copy paste to email break the script. script
as attachment.

Regards,
tbskyd

[-- Attachment #2: final.sh --]
[-- Type: application/x-sh, Size: 3264 bytes --]

Re: multi-home difficulty

[d tbsky]

2017-11-30 14:22 GMT+08:00 d tbsky <tbskyd@gmail.com>:
> 2017-11-30 14:15 GMT+08:00 d tbsky <tbskyd@gmail.com>:
>> 2017-11-29 22:49 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
>>> On Wed, Nov 29, 2017 at 3:16 PM, d tbsky <tbskyd@gmail.com> wrote:
>>>>      sorry I misunderstand you. you mean I modify the script and run
>>>> in my environment to reveal the problem?
>>>> ok I will try to do it.
>>>
>>> Take what I sent you. Run it. If it breaks, send me the output and
>>> your kernel. If it doesn't break, mess with it until it breaks, and
>>> then send it back to me.
>>
>> Hi jason:
>>
>>  "uname -a" result:
>>
>>  Linux localhost.localdomain 3.10.0-693.5.2.el7.x86_64 #1 SMP Thu Oct
>> 19 10:13:14 CDT 2017 x86_64 x86_64 x86_64 GNU/Linux
>>
>>  your original script runs fine under my environment.
>>  I add  three 3 lines before "ip1 link add veth1"  to reveal the problem:
>>
>> ip1 link add dummy0 type dummy
>> ip1 addr add 10.0.0.10/24 dev dummy0
>> ip1 link set dummy0 up
>>
>
> sorry my fault. incorrect copy paste to email break the script. script
> as attachment.
>
> Regards,
> tbskyd

Hi jason:

   sorry the routing is not correct under the situation. I will add
more rules to see if I can make a normal environment to reveal the
problem.

Re: multi-home difficulty

[d tbsky]

2017-11-29 22:49 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
> On Wed, Nov 29, 2017 at 3:16 PM, d tbsky <tbskyd@gmail.com> wrote:
>>      sorry I misunderstand you. you mean I modify the script and run
>> in my environment to reveal the problem?
>> ok I will try to do it.
>
> Take what I sent you. Run it. If it breaks, send me the output and
> your kernel. If it doesn't break, mess with it until it breaks, and
> then send it back to me.

Hi jason:

      during test in netns environment, I saw something that I never
saw at real world. the steps below:

1. client try connect to multi-home-server.
2. wait for conntrack session timeout both for client and server.
3. server try connect to client. server will use source ip at step1 to connect.

it means at step1, wireguard not only remember client's ip address,
but also remember self source ip address. even though the source
address didn't show at "wg wg0"  user interface.
is the assumption true? I didn't see this behavior in real world.

Regards,
tbskyd

Re: multi-home difficulty

[d tbsky]

2017-12-01 15:44 GMT+08:00 d tbsky <tbskyd@gmail.com>:
> 2017-11-29 22:49 GMT+08:00 Jason A. Donenfeld <Jason@zx2c4.com>:
>> On Wed, Nov 29, 2017 at 3:16 PM, d tbsky <tbskyd@gmail.com> wrote:
>>>      sorry I misunderstand you. you mean I modify the script and run
>>> in my environment to reveal the problem?
>>> ok I will try to do it.
>>
>> Take what I sent you. Run it. If it breaks, send me the output and
>> your kernel. If it doesn't break, mess with it until it breaks, and
>> then send it back to me.

Hi Jason:

     sorry for bothering your again. I still can not find the key
point. my testing environment is  rhel 7.4,
I have  tried kernel 3.10,  4.4,  4.14. wireguard 20171111 and 20171127.

I have three things in mind.

1. when wireguard communication established, it will remember self
source ip(although "wg wg0" didn't show) forever until changed next
time. I don't know if the assumption true, could you tell me? I don't
know if this is wireguard feature or netns feature.

2. I build three netns environment, to emulate multi-home-client,
multi-home-server, and a router between client/server. wireguard works
perfect under netns environment.

3. in real world the situation is strange. as I said last time, build
a simple vm with two nic(in the same host bridge)  will reveal the
problem. my vm looks like below:

>ip addr show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
    link/ether 52:54:00:ff:29:75 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
    link/ether 52:54:00:31:d3:1a brd ff:ff:ff:ff:ff:ff
    inet 10.99.1.99/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.99.1.100/24 scope global secondary eth1
       valid_lft forever preferred_lft forever

it is the most simple config I could find to reveal the problem.
situation below won't show any problem:
1. single nic
2. two nic but ip bound to first nic
3. two nic but first nic state is "down", not "up".

the problem is the same under kernel 3.10, 4.4, 4.14. when client
connect to server ip "10.99.1.100", server will reply with ip
"10.99.1.99". it is really a puzzle to me. but maybe you can see why
immediately if you have the environment.

thanks a lot for your patience.

Regards,
tbskyd