Hi there!

We experienced a major network outage today when upgrading kernels.

The affected servers run the VRF+conntrack+nftables combo. They are edge 
firewalls/NAT boxes, meaning most interesting traffic is not locally generated, 
but forwarded.

What we experienced is NATed traffic in the reply direction never being 
forwarded back to the original client.

Good kernel: 5.10.40 (debian 5.10.0-0.bpo.7-amd64)
Bad kernel: 5.10.70 (debian 5.10.0-0.bpo.9-amd64)

I suspect the problem may be related to this patch: 
https://x-lore.kernel.org/stable/20210824165908.709932-58-sashal@kernel.org/

Would it be possible to confirm the offending change, and to get some advice on 
how to workaround the problem? I could run more tests and give additional 
information on demand.

Some bits of our configuration follows. The setup is rather simple, two 
interfaces, one pointing to the internet (eno2.2120) and the other to the 
internal network (eno2.2107). Both interfaces are attached to a VRF device 
'vrf-cloudgw'. The VRF is used to isolate forwarded traffic from the host 
network (eno1). The nftables firewall is also split: a table 'basefirewall' for 
input/output chains, a table 'cloudgw' for forwarded traffic, to perform NAT.


Interfaces setup:

=== 8< ===
user@cloudgw2002-dev:~ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1000
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group 
default qlen 1000
     link/ether 2c:ea:7f:7b:e1:04 brd ff:ff:ff:ff:ff:ff
     inet 10.192.20.18/24 brd 10.192.20.255 scope global eno1
        valid_lft forever preferred_lft forever
     inet6 2620:0:860:118:10:192:20:18/64 scope global
        valid_lft 2591995sec preferred_lft 604795sec
     inet6 fe80::2eea:7fff:fe7b:e104/64 scope link
        valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group 
default qlen 1000
     link/ether 2c:ea:7f:7b:e1:05 brd ff:ff:ff:ff:ff:ff
     inet6 fe80::2eea:7fff:fe7b:e105/64 scope link
        valid_lft forever preferred_lft forever
4: vrf-cloudgw: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP 
group default qlen 1000
     link/ether 1e:04:99:69:3e:56 brd ff:ff:ff:ff:ff:ff
5: eno2.2107@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
master vrf-cloudgw state UP group default qlen 1000
     link/ether 2c:ea:7f:7b:e1:05 brd ff:ff:ff:ff:ff:ff
     inet 185.15.57.9/30 scope global eno2.2107
        valid_lft forever preferred_lft forever
     inet6 fe80::2eea:7fff:fe7b:e105/64 scope link
        valid_lft forever preferred_lft forever
6: eno2.2120@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
master vrf-cloudgw state UP group default qlen 1000
     link/ether 2c:ea:7f:7b:e1:05 brd ff:ff:ff:ff:ff:ff
     inet 208.80.153.189/29 brd 208.80.153.191 scope global eno2.2120
        valid_lft forever preferred_lft forever
     inet 208.80.153.190/29 scope global secondary eno2.2120
        valid_lft forever preferred_lft forever
     inet6 fe80::2eea:7fff:fe7b:e105/64 scope link
        valid_lft forever preferred_lft forever
=== 8< ===

VRF routing table:

=== 8< ===
user@cloudgw2002-dev:~ $ ip route list vrf vrf-cloudgw
default via 208.80.153.185 dev eno2.2120 onlink
172.16.128.0/24 via 185.15.57.10 dev eno2.2107 proto 112 onlink
185.15.57.0/29 via 185.15.57.10 dev eno2.2107 proto 112 onlink
185.15.57.8/30 dev eno2.2107 proto kernel scope link src 185.15.57.9
208.80.153.184/29 dev eno2.2120 proto kernel scope link src 208.80.153.189
=== 8< ===

Find attached nftables ruleset.