From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753309AbdADNuO (ORCPT ); Wed, 4 Jan 2017 08:50:14 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:49522 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751761AbdADNuJ (ORCPT ); Wed, 4 Jan 2017 08:50:09 -0500 Date: Wed, 4 Jan 2017 05:48:43 -0800 From: "Paul E. McKenney" To: Ding Tianhong Cc: davem@davemloft.net, Eric Dumazet , josh@joshtriplett.org, rostedt@goodmis.org, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, "linux-kernel@vger.kernel.org" Subject: Re: [PATCH] rcu: fix the OOM problem of huge IP abnormal packet traffic Reply-To: paulmck@linux.vnet.ibm.com References: <635ca612-370c-b6e4-7f2a-cba702dd0c4a@huawei.com> <20161118130144.GO3612@linux.vnet.ibm.com> <809d327e-d4e2-51a5-bbfd-9ff143ee55da@huawei.com> <20161119082209.GC3612@linux.vnet.ibm.com> <20161121001347.GA27732@linux.vnet.ibm.com> <749be737-bbba-cf4d-0d97-7657e3b1b76b@huawei.com> <20161229001315.GW3742@linux.vnet.ibm.com> <20170104005746.GA10429@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 17010413-0020-0000-0000-00000AA5D742 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006372; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000199; SDB=6.00803415; UDB=6.00390849; IPR=6.00581266; BA=6.00005030; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00013821; XFM=3.00000011; UTC=2017-01-04 13:48:46 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17010413-0021-0000-0000-00005895F1DF Message-Id: <20170104134843.GK3742@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-01-04_10:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1612050000 definitions=main-1701040223 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jan 04, 2017 at 03:02:30PM +0800, Ding Tianhong wrote: > > > On 2017/1/4 8:57, Paul E. McKenney wrote: > > On Wed, Dec 28, 2016 at 04:13:15PM -0800, Paul E. McKenney wrote: > >> On Wed, Dec 28, 2016 at 01:58:06PM +0800, Ding Tianhong wrote: > >>> Hi, Paul: > >>> > >>> I try to debug this problem and found this solution could work well for both problem scene. > >>> > >>> > >>> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h > >>> index 85c5a88..dbc14a7 100644 > >>> --- a/kernel/rcu/tree_plugin.h > >>> +++ b/kernel/rcu/tree_plugin.h > >>> @@ -2172,7 +2172,7 @@ static int rcu_nocb_kthread(void *arg) > >>> if (__rcu_reclaim(rdp->rsp->name, list)) > >>> cl++; > >>> c++; > >>> - local_bh_enable(); > >>> + _local_bh_enable(); > >>> cond_resched_rcu_qs(); > >>> list = next; > >>> } > >>> > >>> > >>> The cond_resched_rcu_qs() would process the softirq if the softirq is pending, so no need to use > >>> local_bh_enable() to process the softirq twice here, and it will avoid OOM when huge packets arrives, > >>> what do you think about it? Please give me some suggestion. > >> > >> From what I can see, there is absolutely no guarantee that > >> cond_resched_rcu_qs() will do local_bh_enable(), and thus no guarantee > >> that it will process any pending softirqs -- and that is not part of > >> its job in any case. So I cannot recommend the above patch. > >> > >> On efficient handling of large invalid packets (that is still the issue, > >> right?), I must defer to Dave and Eric. > > > > On the perhaps unlikely off-chance that there is a fix for this outside > > of networking, what symptoms are you seeing without this fix in place? > > Still RCU CPU stall warnings? Soft lockups? Something else? > > > > Thanx, Paul > > > > Hi Paul: > > I was still try to test and fix this by another way, but could explain more about this problem. > > when the huge packets coming, the packets was abnormal and will be freed by dst_release->call_rcu(dst_destroy_rcu), > so the rcuos kthread will handle the dst_destroy_rcu to free them, but when the rcuos was looping ,I fould the local_bh_enable() will > call do_softirq to receive a certain number of packets which is abnormal and need to be free, but more packets is coming so when cond_resched_rcu_qs run, > it will do the ksoftirqd and do softirq again, so rcuos kthread need free more, it looks more and more worse and lead to OOM because many more packets need to > be freed. > So I think the do_softirq in the local_bh_enable is not need here, the cond_resched_rcu_qs() will handle the do_softirq once, it is enough. > > and recently I found that the Eric has upstream a new patch named (softirq: Let ksoftirqd do its job) may fix this, and still test it, not get any results yet. OK, I don't see any reasonable way that the RCU callback-offload tasks (rcuos) can figure out whether or not they should let softirqs happen -- unconditionally suppressing them might help your workload, but would break workloads needing low networking latency, of which there are many. So please let me know now things go with Eric's patch. Thanx, Paul