From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1764917AbdAJFwE (ORCPT ); Tue, 10 Jan 2017 00:52:04 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:44460 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1755958AbdAJFwA (ORCPT ); Tue, 10 Jan 2017 00:52:00 -0500 Date: Mon, 9 Jan 2017 21:51:53 -0800 From: "Paul E. McKenney" To: Ding Tianhong Cc: davem@davemloft.net, Eric Dumazet , josh@joshtriplett.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, "linux-kernel@vger.kernel.org" Subject: Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic Reply-To: paulmck@linux.vnet.ibm.com References: <809d327e-d4e2-51a5-bbfd-9ff143ee55da@huawei.com> <20161119082209.GC3612@linux.vnet.ibm.com> <20161121001347.GA27732@linux.vnet.ibm.com> <749be737-bbba-cf4d-0d97-7657e3b1b76b@huawei.com> <20161229001315.GW3742@linux.vnet.ibm.com> <20170104005746.GA10429@linux.vnet.ibm.com> <20170104134843.GK3742@linux.vnet.ibm.com> <9fc43387-4a23-f89f-168e-b46e7bf94e40@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9fc43387-4a23-f89f-168e-b46e7bf94e40@huawei.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 17011005-0012-0000-0000-000012E8E374 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006406; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000199; SDB=6.00805275; UDB=6.00391913; IPR=6.00582896; BA=6.00005039; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00013874; XFM=3.00000011; UTC=2017-01-10 05:51:57 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17011005-0013-0000-0000-000049FB40FA Message-Id: <20170110055153.GL3800@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-01-10_04:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1612050000 definitions=main-1701100081 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jan 10, 2017 at 11:20:40AM +0800, Ding Tianhong wrote: > > > On 2017/1/4 21:48, Paul E. McKenney wrote: > > On Wed, Jan 04, 2017 at 03:02:30PM +0800, Ding Tianhong wrote: > >> > >> > >> On 2017/1/4 8:57, Paul E. McKenney wrote: > >>> On Wed, Dec 28, 2016 at 04:13:15PM -0800, Paul E. McKenney wrote: > >>>> On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote: > >>>>> Hi, Paul: > >>>>> > >>>>> I try to debug this problem and found this solution could work well for both problem scene. > >>>>> > >>>>> > >>>>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h > >>>>> index 85c5a88..dbc14a7 100644 > >>>>> --- a/kernel/rcu/tree_plugin.h > >>>>> +++ b/kernel/rcu/tree_plugin.h > >>>>> @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg) > >>>>> if (__rcu_reclaim(rdp->rsp->name, list)) > >>>>> cl++; > >>>>> c++; > >>>>> - local_bh_enable(); > >>>>> + _local_bh_enable(); > >>>>> cond_resched_rcu_qs(); > >>>>> list = next; > >>>>> } > >>>>> > >>>>> > >>>>> The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use > >>>>> local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives, > >>>>> what do you think about it? Please give me some suggestion. > >>>> > >>>> From what I can see, there is absolutely no guarantee that > >>>> cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee > >>>> that it will process any pending softirqs -- and that is not part of > >>>> its job in any case. So I cannot recommend the above patch. > >>>> > >>>> On efficient handling of large invalid packets (that is still the issue, > >>>> right?), I must defer to Dave and Eric. > >>> > >>> On the perhaps unlikely off-chance that there is a fix for this outside > >>> of networking, what symptoms are you seeing without this fix in place? > >>> Still RCU CPU stall warnings? Soft lockups? Something else? > >>> > >>> Thanx, Paul > >>> > >> > >> Hi Paul: > >> > >> I was still try to test and fix this by another way, but could explain more about this problem. > >> > >> when the huge packets coming, the packets was abnormal and will be freed by dst_release->call_rcu(dst_destroy_rcu), > >> so the rcuos kthread will handle the dst_destroy_rcu to free them, but when the rcuos was looping ,I fould the local_bh_enable() will > >> call do_softirq to receive a certain number of packets which is abnormal and need to be free, but more packets is coming so when cond_resched_rcu_qs run, > >> it will do the ksoftirqd and do softirq again, so rcuos kthread need free more, it looks more and more worse and lead to OOM because many more packets need to > >> be freed. > >> So I think the do_softirq in the local_bh_enable is not need here, the cond_resched_rcu_qs() will handle the do_softirq once, it is enough. > >> > >> and recently I found that the Eric has upstream a new patch named (softirq: Let ksoftirqd do its job) may fix this, and still test it, not get any results yet. > > > > OK, I don't see any reasonable way that the RCU callback-offload tasks > > (rcuos) can figure out whether or not they should let softirqs happen -- > > unconditionally suppressing them might help your workload, but would > > break workloads needing low networking latency, of which there are many. > > > > So please let me know now things go with Eric's patch. > > > Hi Paul: > > Good news, the Eric's patch could fix this problem, it means that if the softirqd kthread is running, we should not take too much > time in the softirq process, this behavior equivalent that we remove the do_softirq in the local_bh_enable(), but this solution looks more > perfect, we need to inform the lts kernel maintainer to applied this patch which is not looks like a bugfix. Here is hoping! ;-) Thanx, Paul