From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757563AbdAJH3J (ORCPT ); Tue, 10 Jan 2017 02:29:09 -0500 Received: from szxga02-in.huawei.com ([119.145.14.65]:1400 "EHLO szxga02-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751430AbdAJH3H (ORCPT ); Tue, 10 Jan 2017 02:29:07 -0500 Subject: Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic To: References: <809d327e-d4e2-51a5-bbfd-9ff143ee55da@huawei.com> <20161119082209.GC3612@linux.vnet.ibm.com> <20161121001347.GA27732@linux.vnet.ibm.com> <749be737-bbba-cf4d-0d97-7657e3b1b76b@huawei.com> <20161229001315.GW3742@linux.vnet.ibm.com> <20170104005746.GA10429@linux.vnet.ibm.com> <20170104134843.GK3742@linux.vnet.ibm.com> <9fc43387-4a23-f89f-168e-b46e7bf94e40@huawei.com> <20170110055153.GL3800@linux.vnet.ibm.com> CC: , Eric Dumazet , , , , , "linux-kernel@vger.kernel.org" From: Ding Tianhong Message-ID: <19ba128e-c4b2-ba17-4a60-0e801a348f01@huawei.com> Date: Tue, 10 Jan 2017 15:28:32 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20170110055153.GL3800@linux.vnet.ibm.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.177.23.32] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A0B0201.58748D26.0190,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0, ip=0.0.0.0, so=2013-06-18 04:22:30, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: dba4a590d89b6f631e35281ca20b8fb1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi David： The Patch "rcu: Fix soft lockup for rcu_nocb_kthread" has been added to several stable tree， it may introduced an issue in certain special scenarios, The Patch "softirq: Let ksoftirqd do its job" could fix this issue, so I hope you could add this patch to stable tree, thanks. Ding On 2017/1/10 13:51, Paul E. McKenney wrote: > On Tue, Jan 10, 2017 at 11:20:40AM +0800, Ding Tianhong wrote: >> >> >> On 2017/1/4 21:48, Paul E. McKenney wrote: >>> On Wed, Jan 04, 2017 at 03:02:30PM +0800, Ding Tianhong wrote: >>>> >>>> >>>> On 2017/1/4 8:57, Paul E. McKenney wrote: >>>>> On Wed, Dec 28, 2016 at 04:13:15PM -0800, Paul E. McKenney wrote: >>>>>> On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote: >>>>>>> Hi, Paul: >>>>>>> >>>>>>> I try to debug this problem and found this solution could work well for both problem scene. >>>>>>> >>>>>>> >>>>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h >>>>>>> index 85c5a88..dbc14a7 100644 >>>>>>> --- a/kernel/rcu/tree_plugin.h >>>>>>> +++ b/kernel/rcu/tree_plugin.h >>>>>>> @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg) >>>>>>> if (__rcu_reclaim(rdp->rsp->name, list)) >>>>>>> cl++; >>>>>>> c++; >>>>>>> - local_bh_enable(); >>>>>>> + _local_bh_enable(); >>>>>>> cond_resched_rcu_qs(); >>>>>>> list = next; >>>>>>> } >>>>>>> >>>>>>> >>>>>>> The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use >>>>>>> local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives, >>>>>>> what do you think about it? Please give me some suggestion. >>>>>> >>>>>> From what I can see, there is absolutely no guarantee that >>>>>> cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee >>>>>> that it will process any pending softirqs -- and that is not part of >>>>>> its job in any case. So I cannot recommend the above patch. >>>>>> >>>>>> On efficient handling of large invalid packets (that is still the issue, >>>>>> right?), I must defer to Dave and Eric. >>>>> >>>>> On the perhaps unlikely off-chance that there is a fix for this outside >>>>> of networking, what symptoms are you seeing without this fix in place? >>>>> Still RCU CPU stall warnings? Soft lockups? Something else? >>>>> >>>>> Thanx, Paul >>>>> >>>> >>>> Hi Paul: >>>> >>>> I was still try to test and fix this by another way, but could explain more about this problem. >>>> >>>> when the huge packets coming, the packets was abnormal and will be freed by dst_release->call_rcu(dst_destroy_rcu), >>>> so the rcuos kthread will handle the dst_destroy_rcu to free them, but when the rcuos was looping ,I fould the local_bh_enable() will >>>> call do_softirq to receive a certain number of packets which is abnormal and need to be free, but more packets is coming so when cond_resched_rcu_qs run, >>>> it will do the ksoftirqd and do softirq again, so rcuos kthread need free more, it looks more and more worse and lead to OOM because many more packets need to >>>> be freed. >>>> So I think the do_softirq in the local_bh_enable is not need here, the cond_resched_rcu_qs() will handle the do_softirq once, it is enough. >>>> >>>> and recently I found that the Eric has upstream a new patch named (softirq: Let ksoftirqd do its job) may fix this, and still test it, not get any results yet. >>> >>> OK, I don't see any reasonable way that the RCU callback-offload tasks >>> (rcuos) can figure out whether or not they should let softirqs happen -- >>> unconditionally suppressing them might help your workload, but would >>> break workloads needing low networking latency, of which there are many. >>> >>> So please let me know now things go with Eric's patch. >>> >> Hi Paul: >> >> Good news, the Eric's patch could fix this problem, it means that if the softirqd kthread is running, we should not take too much >> time in the softirq process, this behavior equivalent that we remove the do_softirq in the local_bh_enable(), but this solution looks more >> perfect, we need to inform the lts kernel maintainer to applied this patch which is not looks like a bugfix. > > Here is hoping! ;-) > > Thanx, Paul > > > . >