XFRM programming with VRF enslaved interfaces

[Rob Dover <Rob.Dover@metaswitch.com>]

Hi there,

I'm working on an application that's programming IPSec connections via XFRM on VRFs. I'm seeing some strange behaviour in cases where there is an enslaved interface on the VRF - was wondering if anyone has seen something like this before or perhaps knows how this is supposed to work? 

In our setup, we have a VRF and an enslaved (sidebar: should I be using a different term for this? I would prefer to use something with fewer negative historic connotations if possible!) interface like so:

```
# ip link show vrf-1
33: vrf-1: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 2a:71:ba:bd:33:4d brd ff:ff:ff:ff:ff:ff 

# ip link show master vrf-1
32: serv1: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vrf-1 state UP mode DEFAULT group default qlen 1000
    link/ether 00:13:3e:00:16:68 brd ff:ff:ff:ff:ff:ff
```

The serv1 interface has some associated IPs but the vrf-1 interface does not:

```
# ip addr show dev vrf-1
33: vrf-1: <NOARP,MASTER,UP,LOWER_UP> mtu 65536 qdisc noqueue state UP group default qlen 1000
    link/ether 2a:71:ba:bd:33:4d brd ff:ff:ff:ff:ff:ff 

# ip addr show dev serv1
32: serv1: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vrf-1 state UP group default qlen 1000
    link/ether 00:13:3e:00:16:68 brd ff:ff:ff:ff:ff:ff
    inet 10.248.0.191/16 brd 10.248.255.255 scope global serv1
       valid_lft forever preferred_lft forever
    inet 10.248.0.250/16 brd 10.248.255.255 scope global secondary serv1
       valid_lft forever preferred_lft forever
    inet6 fd5f:5d21:845:1401:213:3eff:fe00:1668/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::213:3eff:fe00:1668/64 scope link
       valid_lft forever preferred_lft forever
```

We're trying to program XFRM using these addresses to send and receive IPSec traffic in transport mode. The interesting question is which interface the XFRM state should be programmed on. I started off by programming the following policies and SAs on the VRF:

```
# ip xfrm policy show
src 10.254.13.16/32 dst 10.248.0.191/32 sport 37409 dport 5080 dev vrf-1
        dir in priority 2147483648 ptype main
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 0 mode transport 
src 10.248.0.191/32 dst 10.254.13.16/32 sport 16381 dport 37409 dev vrf-1
        dir out priority 2147483648 ptype main
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 0 mode transport 
# ip xfrm state show 
src 10.254.13.16 dst 10.248.0.191
        proto esp spi 0x03a0392c reqid 3892838400 mode transport
        replay-window 0
        auth-trunc hmac(md5) 0x00112233445566778899aabbccddeeff 96
        enc cbc(des3_ede) 0xfeefdccdbaab98897667544532231001feefdccdbaab9889
        anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
        sel src 0.0.0.0/0 dst 0.0.0.0/0 sport 37409 dport 5080 dev vrf-1 
src 10.248.0.191 dst 10.254.13.16
        proto esp spi 0x00124f80 reqid 0 mode transport
        replay-window 0
        auth-trunc hmac(md5) 0x00112233445566778899aabbccddeeff 96
        enc cbc(des3_ede) 0xfeefdccdbaab98897667544532231001feefdccdbaab9889
        anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
        sel src 0.0.0.0/0 dst 0.0.0.0/0 sport 16381 dport 37409 dev vrf-1
```

Having programmed these, I can then send ESP packets from 10.254.13.16:37409 -> 10.248.0.191:5080 and they are successfully decoded and passed up to my application. However, when I try to send UDP packets out again from 10.248.0.191:16381 -> 10.254.13.16:37409, the packets are not encrypted but sent out in the clear!

Now, I've done some experimentation and found that if I program the outbound XFRM policy (eg. 10.248.0.191->10.254.13.16) to be on serv1 rather than vrf-1, the packets are correctly encrypted. But if I program the inbound XFRM policy (eg. 10.254.13.16->10.248.0.191) to be on serv1 rather than vrf-1, the inbound packets are not passed up to my application! That leaves me in a situation where I need to program the inbound and outbound XFRM policies asymmetrically in order to get my traffic to be sent properly, like so:

```
# ip xfrm policy show
src 10.254.13.16/32 dst 10.248.0.191/32 sport 37409 dport 5080 dev vrf-1
        dir in priority 2147483648 ptype main
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 0 mode transport 
src 10.248.0.191/32 dst 10.254.13.16/32 sport 16381 dport 37409 dev serv1
        dir out priority 2147483648 ptype main
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 0 mode transport 
# ip xfrm state show 
src 10.254.13.16 dst 10.248.0.191
        proto esp spi 0x03a0392c reqid 3892838400 mode transport
        replay-window 0
        auth-trunc hmac(md5) 0x00112233445566778899aabbccddeeff 96
        enc cbc(des3_ede) 0xfeefdccdbaab98897667544532231001feefdccdbaab9889
        anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
        sel src 0.0.0.0/0 dst 0.0.0.0/0 sport 37409 dport 5080 dev vrf-1 
src 10.248.0.191 dst 10.254.13.16
        proto esp spi 0x00124f80 reqid 0 mode transport
        replay-window 0
        auth-trunc hmac(md5) 0x00112233445566778899aabbccddeeff 96
        enc cbc(des3_ede) 0xfeefdccdbaab98897667544532231001feefdccdbaab9889
        anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
        sel src 0.0.0.0/0 dst 0.0.0.0/0 sport 16381 dport 37409 dev serv1
```

It feels like I'm doing something wrong here - the asymmetrical programming of the interfaces doesn't seem like the 'correct' approach. Seems like there are three possibilities: 
(a) this is standard behaviour, I just have to live with it (although maybe there are some tweaks I can make to settings to change things?),
(b) somewhere along the line the way the application is passing packets down to the kernel is incorrect and that's what's causing the mismatch,
(c) this is a bug in how the kernel works and it's not attributing the packets to the appropriate interface.

Any idea which of these is the right answer?

I'm running with the kernel that ships with Centos8 (4.18.0-240.1.1), so I know I'm a bit out of date! But I've done a trawl through recent changes to the kernel code in this area and I can't see anything that would have obviously changed the behaviour I'm seeing (feel free to correct me if I've missed something!).

Thanks for your help,
Rob Dover