Re: ip rule iif oif and vrf

[David Ahern <dsahern@gmail.com>]

On 9/30/20 7:23 PM, Stephen Suryaputra wrote:
> On Thu, Sep 24, 2020 at 08:41:54AM -0600, David Ahern wrote:
>>> We have multiple options on the table right now. One that can be done
>>> without writing any code is to use an nft prerouting rule to mark
>>> the packet with iif equals the tunnel and use ip rule fwmark to lookup
>>> the right table.
>>>
>>> ip netns exec r0 nft add table ip c2t
>>> ip netns exec r0 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
>>> ip netns exec r0 nft rule ip c2t prerouting iif gre01 mark set 101 counter
>>> ip netns exec r0 ip rule add fwmark 101 table 10 pref 999
>>>
>>> ip netns exec r1 nft add table ip c2t
>>> ip netns exec r1 nft add chain ip c2t prerouting '{ type filter hook prerouting priority 0; policy accept; }'
>>> ip netns exec r1 nft rule ip c2t prerouting iif gre10 mark set 101 counter
>>> ip netns exec r1 ip rule add fwmark 101 table 10 pref 999
>>>
>>> But this doesn't seem to work on my Ubuntu VM with the namespaces
>>> script, i.e. pinging from h0 to h1. The packet doesn't egress r1_v11. It
>>> does work on our target, based on 4.14 kernel.
>>
>> add debugs to net/core/fib_rules.c, fib_rule_match() to see if
>> flowi_mark is getting set properly. There could easily be places that
>> are missed. Or if it works on one setup, but not another compare sysctl
>> settings for net.core and net.ipv4
> 
> Sorry, I got side-tracked. Coming back to this: I made a mistake in the
> ip rule fwmark pref in the script. I have fixed it and the script is
> attached (gre_setup_nft.sh). It has the nft and ip rule commands above.
> The ping between h0 and h1 works.
> 
>>> We also notice though in on the target platform that the ip rule fwmark
>>> doesn't seem to change the skb->dev to the vrf of the lookup table.
>>
>> not following that statement. fwmark should be marking the skb, not
>> changing the skb->dev.
> 
> Yes and it causes the ping between h0 and r1 r1_v11 to not work, e.g.:
> 
> ip netns exec h0 ping -c 1 11.0.0.1
> 
> Similarly, ping between r0_v00 and r1_v11 also does not work:
> 
> ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1
> 
>>> E.g., ping from 10.0.0.1 to 11.0.0.1. With net.ipv4.fwmark_reflect set,
>>> the reply is generated but the originating ping application doesn't get
>>> the packet.  I suspect it is caused by the socket is bound to the tenant
>>> vrf. I haven't been able to repro this because of the problem with the
>>> nft approach above.
> 
> To illustrate my statements above, this is what I did:
> ip netns exec r1 sysctl -w net.ipv4.fwmark_reflect=1
> ip netns exec h0 ping -c 1 11.0.0.1
> PING 11.0.0.1 (11.0.0.1) 56(84) bytes of data.
> 64 bytes from 11.0.0.1: icmp_seq=1 ttl=63 time=0.079 ms
> 
> The ping between h0 and r1 r1_v11 works, but it still doesn't work for this:
> ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1
> 
> eventhough the reply is received by gre01:
> ip netns exec r0 tcpdump -nexi gre01
> 22:10:57.173680 Out ethertype IPv4 (0x0800), length 100: 10.0.0.1 > 11.0.0.1: ICMP echo request, id 3803, seq 1, length 64
> 	0x0000:  4500 0054 1d2a 4000 4001 087e 0a00 0001
> 	0x0010:  0b00 0001 0800 a410 0edb 0001 b13a 755f
> 	0x0020:  0000 0000 5da6 0200 0000 0000 1011 1213
> 	0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
> 	0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233
> 	0x0050:  3435 3637
> 22:10:57.173724  In ethertype IPv4 (0x0800), length 100: 11.0.0.1 > 10.0.0.1: ICMP echo reply, id 3803, seq 1, length 64
> 	0x0000:  4500 0054 c709 0000 4001 9e9e 0b00 0001
> 	0x0010:  0a00 0001 0000 ac10 0edb 0001 b13a 755f
> 	0x0020:  0000 0000 5da6 0200 0000 0000 1011 1213
> 	0x0030:  1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
> 	0x0040:  2425 2627 2829 2a2b 2c2d 2e2f 3031 3233
> 	0x0050:  3435 3637
> 
> In summary the question is: should ip rule with action FR_ACT_TO_TBL
> also change the skb->dev to the right l3mdev?

'ip rule' does not change the skb; only the vrf driver does that (and
some IPv4/IPv6 code reverts it). The rule code only changes the
flowi_oif setting.