ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

[hujunwei]

Hi all,

In my systems, running v4.19.y longterm kernels, ping6 packets high probability loss occur by the default firewalld rule(rpfilter invert) with low traffic generated by iperf.
# ip6tables -nvL -t raw | grep rpfilter
7043  627K DROP       all      *      *       ::/0                 ::/0                 rpfilte invert

Follow these steps:
1) server
# ip -6 a a 2001:db8:0:f101::30/64 dev eth0
# service firewalld stop
# ./iperf3 -s -6

2) client
# ip -6 a a 2001:db8:0:f101::29/64 dev eth0
# ./iperf3 -6 -c 2001:0db8:0:f101::30 -t 3600 -l 4 M -P 5
# ping6 -i 0.2 -I 2001:0db8:0:f101::29 2001:0db8:0:f101::30 -c 100

# sar -n DEV 1 | grep eth0
03:59:41 PM  eth_eth0  38627.00  39353.00   3531.77   5863.12      0.00      0.00      0.00      0.48
03:59:42 PM  eth_eth0  40017.00  40010.00   3648.53   5931.89      0.00      0.00      0.00      0.49
03:59:43 PM  eth_eth0  40025.00  40013.00   3648.95   5977.62      0.00      0.00      0.00      0.49
03:59:44 PM  eth_eth0  39671.00  39657.00   3619.22   5898.70      0.00      0.00      0.00      0.48
03:59:45 PM  eth_eth0  37857.00  40093.00   3352.11   5857.46      0.00      0.00      0.00      0.48

# ping6 Result:
--- 2001:0db8:0:f101::30 ping statistics ---
100 packets transmitted, 64 received, 36% packet loss, time 20379ms

Afer debugging, I find that packet drop because system can't alloc ipv6 dst. when dst->pcpuc_entries exceed gc_thresh in dst_alloc() which called by rpfilter_lookup_reverse6().
When I increase gc_thresh, packets loss decreased, but I think it is unreasonable to lose packets by default firewalld rule with low traffic.

I have a question that why not check the percpu route first in ip6_pol_route_lookup() when rt6_find_cached_rt() retrun NULL, just like ip6_pol_route().
Can anyone help to answer this?
Thanks.

Re: ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

[Eric Dumazet]

On 11/3/19 1:32 AM, hujunwei wrote:
> 
> Afer debugging, I find that packet drop because system can't alloc ipv6 dst. when dst->pcpuc_entries exceed gc_thresh in dst_alloc() which called by rpfilter_lookup_reverse6().
> When I increase gc_thresh, packets loss decreased, but I think it is unreasonable to lose packets by default firewalld rule with low traffic.
> 


By how much have you increased gc_thresh ?

I wonder if you are not hit by RCU being unable to drain its queue fast enough.

Could you try backporting :

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cfcdef5e30469f3f2d6786ad35fc3fdef2a3833f

But the default gc_thresh of 1024 might be too small for this patch
to make any difference, since (1024 >> 7) is 8 

Re: ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

[hujunwei]

On 2019/11/4 2:53, Eric Dumazet wrote:
> 
> By how much have you increased gc_thresh ?
> 
> I wonder if you are not hit by RCU being unable to drain its queue fast enough.
> 
> Could you try backporting :
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cfcdef5e30469f3f2d6786ad35fc3fdef2a3833f
> 
> But the default gc_thresh of 1024 might be too small for this patch
> to make any difference, since (1024 >> 7) is 8 
> 
> 

Hi Eric,
Thanks for your reply, I set gc_thresh to 10240 last time. I will backport and test the patch later.

Regards Junwei

Re: ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

[hujunwei]

On 2019/11/4 10:29, hujunwei wrote:
> 
> 
> On 2019/11/4 2:53, Eric Dumazet wrote:
>>
>> By how much have you increased gc_thresh ?
>>
>> I wonder if you are not hit by RCU being unable to drain its queue fast enough.
>>
>> Could you try backporting :
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cfcdef5e30469f3f2d6786ad35fc3fdef2a3833f
>>
>> But the default gc_thresh of 1024 might be too small for this patch
>> to make any difference, since (1024 >> 7) is 8 
>>
>>
> 
> Hi Eric,
> Thanks for your reply, I set gc_thresh to 10240 last time. I will backport and test the patch later.
> 
> Regards Junwei
> 

Hi Eric,
I applied the patch and find packetes still drop with default value of gc_thresh.
I will submit a patch later.

Regards Junwei

Re: ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

[David Ahern]

On 11/11/19 6:26 PM, hujunwei wrote:
> 
> On 2019/11/4 10:29, hujunwei wrote:
>>
>>
>> On 2019/11/4 2:53, Eric Dumazet wrote:
>>>
>>> By how much have you increased gc_thresh ?
>>>
>>> I wonder if you are not hit by RCU being unable to drain its queue fast enough.
>>>
>>> Could you try backporting :
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cfcdef5e30469f3f2d6786ad35fc3fdef2a3833f
>>>
>>> But the default gc_thresh of 1024 might be too small for this patch
>>> to make any difference, since (1024 >> 7) is 8 
>>>
>>>
>>
>> Hi Eric,
>> Thanks for your reply, I set gc_thresh to 10240 last time. I will backport and test the patch later.
>>
>> Regards Junwei
>>
> 
> Hi Eric,
> I applied the patch and find packetes still drop with default value of gc_thresh.
> I will submit a patch later.
> 
> Regards Junwei
> 

perhaps it is time to update IPv6's gc_thresh to match IPv4's - meaning
it stays but gets set to -1 and is not used. Is there a reason to keep a
limit these days?

Re: ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

[hujunwei]

Hi David,

On 2019/11/12 9:35, David Ahern wrote:
> On 11/11/19 6:26 PM, hujunwei wrote:
>>
>>
>> Hi Eric,
>> I applied the patch and find packetes still drop with default value of gc_thresh.
>> I will submit a patch later.
>>
>> Regards Junwei
>>
> 
> perhaps it is time to update IPv6's gc_thresh to match IPv4's - meaning
> it stays but gets set to -1 and is not used. Is there a reason to keep a
> limit these days?
> 
I think it is unnecessary to keep this limit, because the default gc_thresh will result in ipv6 packet loss in low ipv6 traffic.
There are two solutions:
1) Why not check the percpu route first in ip6_pol_route_lookup() when rt6_find_cached_rt() retrun NULL, just like ip6_pol_route(). Can anyone help to answer this?
2) like you said, set gc_thresh to -1.

Regards Junwei

Re: ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

[David Ahern]

On 11/14/19 6:04 AM, hujunwei wrote:
> Hi David,
> 
> On 2019/11/12 9:35, David Ahern wrote:
>> On 11/11/19 6:26 PM, hujunwei wrote:
>>>
>>>
>>> Hi Eric,
>>> I applied the patch and find packetes still drop with default value of gc_thresh.
>>> I will submit a patch later.
>>>
>>> Regards Junwei
>>>
>>
>> perhaps it is time to update IPv6's gc_thresh to match IPv4's - meaning
>> it stays but gets set to -1 and is not used. Is there a reason to keep a
>> limit these days?
>>
> I think it is unnecessary to keep this limit, because the default gc_thresh will result in ipv6 packet loss in low ipv6 traffic.
> There are two solutions:
> 1) Why not check the percpu route first in ip6_pol_route_lookup() when rt6_find_cached_rt() retrun NULL, just like ip6_pol_route(). Can anyone help to answer this?

AIUI it's different for no specific reason - just a different lookup
written for different use cases.

ip6_route_lookup for example (which is what netfilter code is using)
does not even need an rt6_info, it just wants to know the lookup is good
and check the egress device. Meaning, all it really needs is the
fib6_info from the lookup under rcu. That (or helper to validate the
lookup and device) is the right long term solution, but a much bigger
change that I have time for right now.


> 2) like you said, set gc_thresh to -1.
> 
> Regards Junwei
> 

Re: ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

[hujunwei]

On 2019/11/14 23:39, David Ahern wrote:
>>>>
>>>
>>> perhaps it is time to update IPv6's gc_thresh to match IPv4's - meaning
>>> it stays but gets set to -1 and is not used. Is there a reason to keep a
>>> limit these days?
>>>
>> I think it is unnecessary to keep this limit, because the default gc_thresh will result in ipv6 packet loss in low ipv6 traffic.
>> There are two solutions:
>> 1) Why not check the percpu route first in ip6_pol_route_lookup() when rt6_find_cached_rt() retrun NULL, just like ip6_pol_route(). Can anyone help to answer this?
> 
> AIUI it's different for no specific reason - just a different lookup
> written for different use cases.
> 
> ip6_route_lookup for example (which is what netfilter code is using)
> does not even need an rt6_info, it just wants to know the lookup is good
> and check the egress device. Meaning, all it really needs is the
> fib6_info from the lookup under rcu. That (or helper to validate the
> lookup and device) is the right long term solution, but a much bigger
> change that I have time for right now.
> 
Hi David,
Thank you so much for the reply, so when will you update it?
I really look forward to your patch.

Regards Junwei

Re: ping6 packets high probability loss occurs by the default firewalld rule(rpfilter invert) with low traffic generated by iperf

[David Ahern]

On 11/14/19 8:57 PM, hujunwei wrote:
> Thank you so much for the reply, so when will you update it?
> I really look forward to your patch.

I looked at what is needed this past summer and as I recall most of the
effort is writing tests to validate the change. As far as I know there
are no tests under tools/testing/selftests/net that exercise the code paths.