[PATCH] nf_flowtable: ensure dst.dev is not blackhole

[PATCH] nf_flowtable: ensure dst.dev is not blackhole

[Ritaro Takenaka]

Fixes sporadic IPv6 packet loss when flow offloading is enabled.
IPv6 route GC calls dst_dev_put() which makes dst.dev blackhole_netdev
even if dst is cached in flow offload. If a packet passes through this
invalid flow, packet loss will occur.
This is from Commit 227e1e4d0d6c (netfilter: nf_flowtable: skip device
lookup from interface index), as outdev was cached independently before.
Packet loss is reported on OpenWrt with Linux 5.4 and later.

Signed-off-by: Ritaro Takenaka <ritarot634@gmail.com>
---
 net/netfilter/nf_flow_table_ip.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/net/netfilter/nf_flow_table_ip.c b/net/netfilter/nf_flow_table_ip.c
index 32c0eb1b4..12f81661d 100644
--- a/net/netfilter/nf_flow_table_ip.c
+++ b/net/netfilter/nf_flow_table_ip.c
@@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
 	if (nf_flow_state_check(flow, ip6h->nexthdr, skb, thoff))
 		return NF_ACCEPT;
 
+	if (unlikely(tuplehash->tuple.dst_cache->dev == blackhole_netdev)) {
+		flow_offload_teardown(flow);
+		return NF_ACCEPT;
+	}
+
 	if (skb_try_make_writable(skb, thoff + hdrsize))
 		return NF_DROP;
 
-- 
2.25.1

Re: [PATCH] nf_flowtable: ensure dst.dev is not blackhole

[Pablo Neira Ayuso]

Hi,

On Mon, Apr 25, 2022 at 05:08:38PM +0900, Ritaro Takenaka wrote:
> Fixes sporadic IPv6 packet loss when flow offloading is enabled.
> IPv6 route GC calls dst_dev_put() which makes dst.dev blackhole_netdev
> even if dst is cached in flow offload. If a packet passes through this
> invalid flow, packet loss will occur.
> This is from Commit 227e1e4d0d6c (netfilter: nf_flowtable: skip device
> lookup from interface index), as outdev was cached independently before.
> Packet loss is reported on OpenWrt with Linux 5.4 and later.

dst_check() should deal with this.

In 5.4, this check is only enabled for xfrm.

> Signed-off-by: Ritaro Takenaka <ritarot634@gmail.com>
> ---
>  net/netfilter/nf_flow_table_ip.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/net/netfilter/nf_flow_table_ip.c b/net/netfilter/nf_flow_table_ip.c
> index 32c0eb1b4..12f81661d 100644
> --- a/net/netfilter/nf_flow_table_ip.c
> +++ b/net/netfilter/nf_flow_table_ip.c
> @@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
>  	if (nf_flow_state_check(flow, ip6h->nexthdr, skb, thoff))
>  		return NF_ACCEPT;
>  
> +	if (unlikely(tuplehash->tuple.dst_cache->dev == blackhole_netdev)) {
> +		flow_offload_teardown(flow);
> +		return NF_ACCEPT;
> +	}
> +
>  	if (skb_try_make_writable(skb, thoff + hdrsize))
>  		return NF_DROP;
>  
> -- 
> 2.25.1
> 

Re: [PATCH] nf_flowtable: ensure dst.dev is not blackhole

[りたろう]

Thanks for your reply.

> In 5.4, this check is only enabled for xfrm.
Packet loss occurs with xmit_neigh (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runs dst_check() periodically.
Route GC and flowtable GC are not synchronized, so it is necessary to check
each packet.

> dst_check() should deal with this.
When dst_check() is used, the performance degradation is not negligible.
From 940 Mbps to 700 Mbps with QCA9563 simple firewall.

On Tue, Apr 26, 2022 at 8:21 PM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>
> Hi,
>
> On Mon, Apr 25, 2022 at 05:08:38PM +0900, Ritaro Takenaka wrote:
> > Fixes sporadic IPv6 packet loss when flow offloading is enabled.
> > IPv6 route GC calls dst_dev_put() which makes dst.dev blackhole_netdev
> > even if dst is cached in flow offload. If a packet passes through this
> > invalid flow, packet loss will occur.
> > This is from Commit 227e1e4d0d6c (netfilter: nf_flowtable: skip device
> > lookup from interface index), as outdev was cached independently before.
> > Packet loss is reported on OpenWrt with Linux 5.4 and later.
>
> dst_check() should deal with this.
>
> In 5.4, this check is only enabled for xfrm.
>
> > Signed-off-by: Ritaro Takenaka <ritarot634@gmail.com>
> > ---
> >  net/netfilter/nf_flow_table_ip.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> >
> > diff --git a/net/netfilter/nf_flow_table_ip.c b/net/netfilter/nf_flow_table_ip.c
> > index 32c0eb1b4..12f81661d 100644
> > --- a/net/netfilter/nf_flow_table_ip.c
> > +++ b/net/netfilter/nf_flow_table_ip.c
> > @@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
> >       if (nf_flow_state_check(flow, ip6h->nexthdr, skb, thoff))
> >               return NF_ACCEPT;
> >
> > +     if (unlikely(tuplehash->tuple.dst_cache->dev == blackhole_netdev)) {
> > +             flow_offload_teardown(flow);
> > +             return NF_ACCEPT;
> > +     }
> > +
> >       if (skb_try_make_writable(skb, thoff + hdrsize))
> >               return NF_DROP;
> >
> > --
> > 2.25.1
> >

Re: [PATCH] nf_flowtable: ensure dst.dev is not blackhole

[Ritaro Takenaka]

Thanks for your reply.

> In 5.4, this check is only enabled for xfrm.
Packet loss occurs with xmit (xfrm is not confirmed).
I also experienced packet loss with 5.10, which runs dst_check periodically.
Route GC and flowtable GC are not synchronized, so it is
necessary to check each packet.

> dst_check() should deal with this.
When dst_check is used, the performance degradation is not negligible.
From 900 Mbps to 700 Mbps with QCA9563 simple firewall.

I am sorry if you have received this mail twice.

On 2022/04/26 20:21, Pablo Neira Ayuso wrote:
> Hi,
> 
> On Mon, Apr 25, 2022 at 05:08:38PM +0900, Ritaro Takenaka wrote:
>> Fixes sporadic IPv6 packet loss when flow offloading is enabled.
>> IPv6 route GC calls dst_dev_put() which makes dst.dev blackhole_netdev
>> even if dst is cached in flow offload. If a packet passes through this
>> invalid flow, packet loss will occur.
>> This is from Commit 227e1e4d0d6c (netfilter: nf_flowtable: skip device
>> lookup from interface index), as outdev was cached independently before.
>> Packet loss is reported on OpenWrt with Linux 5.4 and later.
> 
> dst_check() should deal with this.
> 
> In 5.4, this check is only enabled for xfrm.
> 
>> Signed-off-by: Ritaro Takenaka <ritarot634@gmail.com>
>> ---
>>  net/netfilter/nf_flow_table_ip.c | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/net/netfilter/nf_flow_table_ip.c b/net/netfilter/nf_flow_table_ip.c
>> index 32c0eb1b4..12f81661d 100644
>> --- a/net/netfilter/nf_flow_table_ip.c
>> +++ b/net/netfilter/nf_flow_table_ip.c
>> @@ -624,6 +624,11 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
>>  	if (nf_flow_state_check(flow, ip6h->nexthdr, skb, thoff))
>>  		return NF_ACCEPT;
>>  
>> +	if (unlikely(tuplehash->tuple.dst_cache->dev == blackhole_netdev)) {
>> +		flow_offload_teardown(flow);
>> +		return NF_ACCEPT;
>> +	}
>> +
>>  	if (skb_try_make_writable(skb, thoff + hdrsize))
>>  		return NF_DROP;
>>  
>> -- 
>> 2.25.1
>>

Re: [PATCH] nf_flowtable: ensure dst.dev is not blackhole

[Pablo Neira Ayuso]

On Tue, Apr 26, 2022 at 09:28:13PM +0900, Ritaro Takenaka wrote:
> Thanks for your reply.
> 
> > In 5.4, this check is only enabled for xfrm.
> Packet loss occurs with xmit (xfrm is not confirmed).
> I also experienced packet loss with 5.10, which runs dst_check periodically.
> Route GC and flowtable GC are not synchronized, so it is
> necessary to check each packet.
> 
> > dst_check() should deal with this.
> When dst_check is used, the performance degradation is not negligible.
> From 900 Mbps to 700 Mbps with QCA9563 simple firewall.

You mention 5.10 above.

Starting 5.12, dst_check() uses INDIRECT_CALL_INET.

Is dst_check() still slow with >= 5.12?

Asking this because my understanding (at this stage) is that this
check for blackhole_netdev is a faster way to check for stale cached
routes.

Re: [PATCH] nf_flowtable: ensure dst.dev is not blackhole

[Ritaro Takenaka]

On 2022/04/28 0:10, Pablo Neira Ayuso wrote:> On Tue, Apr 26, 2022 at 09:28:13PM +0900, Ritaro Takenaka wrote:
>> Thanks for your reply.
>>
>>> In 5.4, this check is only enabled for xfrm.
>> Packet loss occurs with xmit (xfrm is not confirmed).
>> I also experienced packet loss with 5.10, which runs dst_check periodically.
>> Route GC and flowtable GC are not synchronized, so it is
>> necessary to check each packet.
>>
>>> dst_check() should deal with this.
>> When dst_check is used, the performance degradation is not negligible.
>> From 900 Mbps to 700 Mbps with QCA9563 simple firewall.
> 
> You mention 5.10 above.
> 
> Starting 5.12, dst_check() uses INDIRECT_CALL_INET.
> 
> Is dst_check() still slow with >= 5.12?
> 
> Asking this because my understanding (at this stage) is that this
> check for blackhole_netdev is a faster way to check for stale cached
> routes.

I did the performance tests with 5.15, confirmed dst_check() is not slower
than checking for blackhole_netdev.

Good, dst_check() can be used.

Then, stale routes check should be moved from nf_flow_offload_gc_step() to
nf_flow_offload(_ipv6)_hook(). Is it correct?

Re: [PATCH] nf_flowtable: ensure dst.dev is not blackhole

[Pablo Neira Ayuso]

On Sun, May 01, 2022 at 02:23:55AM +0900, Ritaro Takenaka wrote:
> On 2022/04/28 0:10, Pablo Neira Ayuso wrote:> On Tue, Apr 26, 2022 at 09:28:13PM +0900, Ritaro Takenaka wrote:
> >> Thanks for your reply.
> >>
> >>> In 5.4, this check is only enabled for xfrm.
> >> Packet loss occurs with xmit (xfrm is not confirmed).
> >> I also experienced packet loss with 5.10, which runs dst_check periodically.
> >> Route GC and flowtable GC are not synchronized, so it is
> >> necessary to check each packet.
> >>
> >>> dst_check() should deal with this.
> >> When dst_check is used, the performance degradation is not negligible.
> >> From 900 Mbps to 700 Mbps with QCA9563 simple firewall.
> > 
> > You mention 5.10 above.
> > 
> > Starting 5.12, dst_check() uses INDIRECT_CALL_INET.
> > 
> > Is dst_check() still slow with >= 5.12?
> > 
> > Asking this because my understanding (at this stage) is that this
> > check for blackhole_netdev is a faster way to check for stale cached
> > routes.
> 
> I did the performance tests with 5.15, confirmed dst_check() is not slower
> than checking for blackhole_netdev.
> 
> Good, dst_check() can be used.
> 
> Then, stale routes check should be moved from nf_flow_offload_gc_step() to
> nf_flow_offload(_ipv6)_hook(). Is it correct?

Then, the check from packet path needs to be restored.