Re: [PATCH bpf-next V15 2/7] bpf: fix bpf_fib_lookup helper MTU check for SKB ctx

[Daniel Borkmann <daniel@iogearbox.net>]

On 2/2/21 5:26 PM, Jesper Dangaard Brouer wrote:
> BPF end-user on Cilium slack-channel (Carlo Carraro) wants to use
> bpf_fib_lookup for doing MTU-check, but *prior* to extending packet size,
> by adjusting fib_params 'tot_len' with the packet length plus the expected
> encap size. (Just like the bpf_check_mtu helper supports). He discovered
> that for SKB ctx the param->tot_len was not used, instead skb->len was used
> (via MTU check in is_skb_forwardable() that checks against netdev MTU).
> 
> Fix this by using fib_params 'tot_len' for MTU check. If not provided (e.g.
> zero) then keep existing TC behaviour intact. Notice that 'tot_len' for MTU
> check is done like XDP code-path, which checks against FIB-dst MTU.
> 
> V13:
> - Only do ifindex lookup one time, calling dev_get_by_index_rcu().
> 
> V10:
> - Use same method as XDP for 'tot_len' MTU check
> 
> Fixes: 4c79579b44b1 ("bpf: Change bpf_fib_lookup to return lookup status")
> Reported-by: Carlo Carraro <colrack@gmail.com>
> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
> Acked-by: John Fastabend <john.fastabend@gmail.com>
[...]

I was about to apply the series just now, but on a last double check there is
a subtle logic bug in here that still needs fixing unfortunately. :/ See below:

> @@ -5568,7 +5565,9 @@ BPF_CALL_4(bpf_skb_fib_lookup, struct sk_buff *, skb,
>   	   struct bpf_fib_lookup *, params, int, plen, u32, flags)
>   {
>   	struct net *net = dev_net(skb->dev);
> +	struct net_device *dev;
>   	int rc = -EAFNOSUPPORT;
> +	bool check_mtu = false;
>   
>   	if (plen < sizeof(*params))
>   		return -EINVAL;
> @@ -5576,23 +5575,30 @@ BPF_CALL_4(bpf_skb_fib_lookup, struct sk_buff *, skb,
>   	if (flags & ~(BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_OUTPUT))
>   		return -EINVAL;
>   
> +	dev = dev_get_by_index_rcu(net, params->ifindex);
> +	if (unlikely(!dev))
> +		return -ENODEV;

Based on your earlier idea, you try to avoid refetching the dev this way, so
here it's being looked up via params->ifindex provided from the BPF prog ...

> +	if (params->tot_len)
> +		check_mtu = true;
> +
>   	switch (params->family) {
>   #if IS_ENABLED(CONFIG_INET)
>   	case AF_INET:
> -		rc = bpf_ipv4_fib_lookup(net, params, flags, false);
> +		rc = bpf_ipv4_fib_lookup(net, dev, params, flags, check_mtu);

... however, bpf_ipv{4,6}_fib_lookup() might change params->ifindex here to
indicate nexthop output dev:

[...]
         dev = nhc->nhc_dev;

         params->rt_metric = res.fi->fib_priority;
         params->ifindex = dev->ifindex;
[...]

>   		break;
>   #endif
>   #if IS_ENABLED(CONFIG_IPV6)
>   	case AF_INET6:
> -		rc = bpf_ipv6_fib_lookup(net, params, flags, false);
> +		rc = bpf_ipv6_fib_lookup(net, dev, params, flags, check_mtu);
>   		break;
>   #endif
>   	}
>   
> -	if (!rc) {
> -		struct net_device *dev;
> -
> -		dev = dev_get_by_index_rcu(net, params->ifindex);
> +	if (rc == BPF_FIB_LKUP_RET_SUCCESS && !check_mtu) {
> +		/* When tot_len isn't provided by user,
> +		 * check skb against net_device MTU
> +		 */
>   		if (!is_skb_forwardable(dev, skb))
>   			rc = BPF_FIB_LKUP_RET_FRAG_NEEDED;

... so using old cached dev from above will result in wrong MTU check &
subsequent passing of wrong params->mtu_result = dev->mtu this way. So one
way to fix is that we would need to pass &dev to bpf_ipv{4,6}_fib_lookup().

>   	}
> 
> 

Thanks,
Daniel