Re: VRF Issue Since kernel 5

[Maximilian Bosch <maximilian@mbosch.me>]

[-- Attachment #1: Type: text/plain, Size: 6027 bytes --]

Hi!

First of all, sorry for my delayed response!

> functional test script under tools/testing/selftests/net covers VRF
> tests and it ran clean for 5.4 last time I checked. There were a few
> changes that went into 4.20 or 5.0 that might be tripping up this use
> case, but I need a lot more information.

I recently started an attempt to get those tests running on my machine
(and a Fedora VM after that), however I had several issues with
timeouts (when running `sudo -E make -C tools/testing/selftests TARGETS="net"
run_tests`).

May I ask if there are further things I need to take care of to get
those tests successfully running?

> are you saying wireguard worked with VRF in the past but is not now?

No. WireGuard traffic is still working fine. The only issue is
TCP-traffic through a VRF (which worked with 4.19, but doesn't anymore
with 5.4 and 5.5).

> 'ip vrf exec' loads a bpf program and that requires locked memory, so
> yes, you need to increase it.

Thanks a lot for the explanation!

> Let's start with lookups:
> 
> perf record -e fib:* -a -g
> <run test that fails, ctrl-c>
> perf script

For the record, please note that I'm now on Linux 5.5.13.

I ran the following command:

```
sudo perf record -e fib:* -a -g -- ssh root@92.60.36.231 -o ConnectTimeout=10s
```

The full output can be found here:

https://gist.githubusercontent.com/Ma27/a6f83e05f6ffede21c2e27d5c7d27098/raw/4852d97ee4860f7887e16f94a8ede4b4406f07bc/perf-report.txt

Thanks!

  Maximilian

On Wed, Mar 11, 2020 at 07:06:54PM -0600, David Ahern wrote:
> On 3/10/20 2:47 PM, Maximilian Bosch wrote:
> > Hi!
> > 
> > I suspect I hit the same issue which is why I decided to respond to this
> > thread (if that's wrong please let me know).
> > 
> >> sudo sysctl -a | grep l3mdev
> >>
> >> If not,
> >> sudo sysctl net.ipv4.raw_l3mdev_accept=1
> >> sudo sysctl net.ipv4.udp_l3mdev_accept=1
> >> sudo sysctl net.ipv4.tcp_l3mdev_accept=1
> > 
> > On my system (NixOS 20.03, Linux 5.5.8) those values are set to `1`, but
> > I experience the same issue.
> > 
> >> Since Kernel 5 though I am no longer able to update – but the issue is quite a curious one as some traffic appears to be fine (DNS lookups use VRF correctly) but others don’t (updating/upgrading the packages)
> > 
> > I can reproduce this on 5.4.x and 5.5.x. To be more precise, I suspect
> > that only TCP traffic hangs in the VRF. When I try to `ssh` through the
> > VRF, I get a timeout, but UDP traffic e.g. from WireGuard works just fine.
> > 
> > However, TCP traffic through a VRF works fine as well on 4.x (just tested this on
> > 4.19.108 and 4.14.172).
> 
> functional test script under tools/testing/selftests/net covers VRF
> tests and it ran clean for 5.4 last time I checked. There were a few
> changes that went into 4.20 or 5.0 that might be tripping up this use
> case, but I need a lot more information.
> 
> > 
> > I use VRFs to enslave my physical uplink interfaces (enp0s31f6, wlp2s0).
> > My main routing table has a default route via my WireGuard Gateway and I
> > only route my WireGuard uplink through the VRF. With this approach I can
> > make sure that all of my traffic goes through the VPN and only the
> > UDP packets of WireGuard will be routed through the uplink network.
> 
> are you saying wireguard worked with VRF in the past but is not now?
> 
> 
> > 
> > As mentioned above, the WireGuard traffic works perfectly fine, but I
> > can't access `<vpn-uplink>` via SSH:
> > 
> > ```
> > $ ssh root@<vpn-uplink> -vvvv
> > OpenSSH_8.2p1, OpenSSL 1.1.1d  10 Sep 2019
> > debug1: Reading configuration data /home/ma27/.ssh/config
> > debug1: /home/ma27/.ssh/config line 5: Applying options for *
> > debug1: Reading configuration data /etc/ssh/ssh_config
> > debug1: /etc/ssh/ssh_config line 5: Applying options for *
> > debug2: resolve_canonicalize: hostname <vpn-uplink> is address
> > debug1: Control socket "/home/ma27/.ssh/master-root@<vpn-uplink>:22" does not exist
> > debug2: ssh_connect_direct
> > debug1: Connecting to <vpn-uplink> [<vpn-uplink>] port 22.
> > # Hangs here for a while
> > ```
> > 
> > I get the following output when debugging this with `tcpdump`:
> > 
> > ```
> > $ tcpdump -ni uplink tcp
> > 20:06:40.409006 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798273519 ecr 0,nop,wscale 7], length 0
> > 20:06:40.439699 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3289740891, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100235016 ecr 3798273519,nop,wscale 7], length 0
> > 20:06:40.439751 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0
> 
> that suggests not finding a matching socket, so sending a reset.
> 
> > 20:06:41.451871 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [S], seq 4123706560, win 65495, options [mss 65495,sackOK,TS val 3798274562 ecr 0,nop,wscale 7], length 0
> > 20:06:41.484498 IP <vpn-uplink>.22 > 10.214.40.237.58928: Flags [S.], seq 3306036877, ack 4123706561, win 65160, options [mss 1460,sackOK,TS val 1100236059 ecr 3798274562,nop,wscale 7], length 0
> > 20:06:41.484528 IP 10.214.40.237.58928 > <vpn-uplink>.22: Flags [R], seq 4123706561, win 0, length 0
> > ```
> > 
> > AFAICS every SYN will be terminated with an RST which is the reason why
> > the connection hangs.
> > 
> > I can work around the issue by using `ip vrf exec`. However I get the
> > following error (unless I run `ulimit -l 2048`):
> > 
> > ```
> > Failed to load BPF prog: 'Operation not permitted'
> > ```
> 
> 'ip vrf exec' loads a bpf program and that requires locked memory, so
> yes, you need to increase it.
> 
> Let's start with lookups:
> 
> perf record -e fib:* -a -g
> <run test that fails, ctrl-c>
> perf script
> 
> That shows the lookups (inputs, table id, result) and context (stack
> trace). That might give some context.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]