Re: VRF Issue Since kernel 5

[Maximilian Bosch <maximilian@mbosch.me>]

[-- Attachment #1: Type: text/plain, Size: 4509 bytes --]

Hi!

I suspect I hit the same issue which is why I decided to respond to this
thread (if that's wrong please let me know).

> sudo sysctl -a | grep l3mdev
> 
> If not,
> sudo sysctl net.ipv4.raw_l3mdev_accept=1
> sudo sysctl net.ipv4.udp_l3mdev_accept=1
> sudo sysctl net.ipv4.tcp_l3mdev_accept=1

On my system (NixOS 20.03, Linux 5.5.8) those values are set to `1`, but
I experience the same issue.

> Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages)

I can reproduce this on 5.4.x and 5.5.x. To be more precise, I suspect
that only TCP traffic hangs in the VRF. When I try to `ssh` through the
VRF, I get a timeout, but UDP traffic e.g. from WireGuard works just fine.

However, TCP traffic through a VRF works fine as well on 4.x (just tested this on
4.19.108 and 4.14.172).

I use VRFs to enslave my physical uplink interfaces (enp0s31f6, wlp2s0).
My main routing table has a default route via my WireGuard Gateway and I
only route my WireGuard uplink through the VRF. With this approach I can
make sure that all of my traffic goes through the VPN and only the
UDP packets of WireGuard will be routed through the uplink network.

My routing table (with `wg0` being the WireGuard interface and `uplink`
being the VRF-interface) looks like this:

```
$ ip route show
default via 10.94.0.1 dev wg0 proto static
10.94.0.0/16 dev wg0 proto kernel scope link src 10.94.0.2
<vpn-uplink> dev uplink proto static metric 100
```

The VRF-interface is associated with the routing-table `1`. This routing
table contains all routes configured on `enp0s31f6` or `wlp2s0`.

As mentioned above, the WireGuard traffic works perfectly fine, but I
can't access `<vpn-uplink>` via SSH:

```
$ ssh root@<vpn-uplink> -vvvv
OpenSSH_8.2p1, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /home/ma27/.ssh/config
debug1: /home/ma27/.ssh/config line 5: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 5: Applying options for *
debug2: resolve_canonicalize: hostname <vpn-uplink> is address
debug1: Control socket "/home/ma27/.ssh/master-root@<vpn-uplink>:22" does not exist
debug2: ssh_connect_direct
debug1: Connecting to <vpn-uplink> [<vpn-uplink>] port 22.
# Hangs here for a while
```

I get the following output when debugging this with `tcpdump`:

```
$ tcpdump -ni uplink tcp
20:06:40.409006 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798273519 ecr 0,nop,wscale 7], length 0
20:06:40.439699 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3289740891, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100235016 ecr 3798273519,nop,wscale 7], length 0
20:06:40.439751 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0
20:06:41.451871 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798274562 ecr 0,nop,wscale 7], length 0
20:06:41.484498 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3306036877, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100236059 ecr 3798274562,nop,wscale 7], length 0
20:06:41.484528 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0
```

AFAICS every SYN will be terminated with an RST which is the reason why
the connection hangs.

I can work around the issue by using `ip vrf exec`. However I get the
following error (unless I run `ulimit -l 2048`):

```
Failed to load BPF prog: 'Operation not permitted'
```

In case you need more information about my setup or if I can help e.g.
with testing patches, please let me know.

Note: I'm not a subscriber of this ML, so I'd appreciate it if you could
CC me.

Thanks!

  Maximilian

On Wed, Sep 11, 2019 at 01:15:54PM +0100, Mike Manning wrote:
> Hi Gareth,
> Could you please also check that all the following are set to 1, I
> appreciate you've confirmed that the one for tcp is set to 1, and by
> default the one for raw is also set to 1:
> 
> sudo sysctl -a | grep l3mdev
> 
> If not,
> sudo sysctl net.ipv4.raw_l3mdev_accept=1
> sudo sysctl net.ipv4.udp_l3mdev_accept=1
> sudo sysctl net.ipv4.tcp_l3mdev_accept=1
> 
> 
> Thanks
> Mike
> 
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]