From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1173455AbdDXQYy (ORCPT ); Mon, 24 Apr 2017 12:24:54 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:37612 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1173411AbdDXQYq (ORCPT ); Mon, 24 Apr 2017 12:24:46 -0400 Date: Mon, 24 Apr 2017 09:24:42 -0700 From: "Paul E. McKenney" To: Mike Galbraith Cc: LKML Subject: Re: TREE_SRCU slows hotplug by factor ~16 Reply-To: paulmck@linux.vnet.ibm.com References: <1493002089.4145.7.camel@gmx.de> <20170424033250.GX3956@linux.vnet.ibm.com> <1493011484.4166.13.camel@gmx.de> <20170424062220.GZ3956@linux.vnet.ibm.com> <1493019303.4617.6.camel@gmx.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1493019303.4617.6.camel@gmx.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17042416-0008-0000-0000-000002021E88 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006966; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000208; SDB=6.00851837; UDB=6.00420999; IPR=6.00630674; BA=6.00005311; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015158; XFM=3.00000013; UTC=2017-04-24 16:24:44 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17042416-0009-0000-0000-000034E5248A Message-Id: <20170424162442.GI3956@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-24_13:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1704240282 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Apr 24, 2017 at 09:35:03AM +0200, Mike Galbraith wrote: > On Sun, 2017-04-23 at 23:22 -0700, Paul E. McKenney wrote: > > > Could you please collect an ftrace (or whatever) showing the timestamp > > sequence of calls to synchronize_srcu(), synchronize_srcu_expedited(), > > and call_srcu() during the execution of the stress script? If it is easy > > to do, also the timestamp sequence of returns from synchronize_srcu() > > and synchronize_srcu_expedited()? > > The first two minutes below config bits, trace_printk([enter/exit]) in > each function. If you (unlikely, but..) want the whole trace, holler. > > # RCU Subsystem > CONFIG_TREE_RCU=y > CONFIG_RCU_EXPERT=y > CONFIG_SRCU=y > # CONFIG_CLASSIC_SRCU is not set > CONFIG_TREE_SRCU=y > # CONFIG_TASKS_RCU is not set > CONFIG_RCU_STALL_COMMON=y > CONFIG_RCU_FANOUT=64 > CONFIG_RCU_FANOUT_LEAF=16 > # CONFIG_RCU_FAST_NO_HZ is not set > # CONFIG_TREE_RCU_TRACE is not set > CONFIG_RCU_KTHREAD_PRIO=0 > CONFIG_RCU_NOCB_CPU=y > CONFIG_RCU_NOCB_CPU_NONE=y > # CONFIG_RCU_NOCB_CPU_ZERO is not set > # CONFIG_RCU_NOCB_CPU_ALL is not set > # RCU Debugging > # CONFIG_PROVE_RCU is not set > # CONFIG_SPARSE_RCU_POINTER is not set > # CONFIG_RCU_PERF_TEST is not set > # CONFIG_RCU_TORTURE_TEST is not set > CONFIG_RCU_CPU_STALL_TIMEOUT=60 > # CONFIG_RCU_TRACE is not set > # CONFIG_RCU_EQS_DEBUG is not set Thank you for the data, extremely helpful! I now know that I must immediately fix this the hard way rather than trying several simpler possibilities that, according to your trace data, would be complete wastes of time. My initial calculation was actually optimistic. There was one exit-to-enter interval at about 80 milliseconds, but the second longest is 342 microseconds, and the vast majority are under 200 microseconds. Back to the drawing board! This will take some time, but I should have a patch for you in a few days, hopefully sooner. Thanx, Paul > # tracer: nop > # > # entries-in-buffer/entries-written: 229332/229332 #P:8 > # > # _-----=> irqs-off > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / delay > # TASK-PID CPU# |||| TIMESTAMP FUNCTION > # | | | |||| | | > stress-cpu-hotp-2443 [004] .... 144.123608: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.123610: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.123612: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.139528: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.139646: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.139646: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.139648: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.155813: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.155981: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.155982: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.155983: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.172078: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.172234: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.172235: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.172237: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.188319: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.188465: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.188466: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.188468: call_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.202528: synchronize_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.202650: synchronize_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.202651: call_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.202652: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.219081: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.219238: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.219238: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.219240: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.235319: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.235567: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.235568: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.235571: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.251548: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.251706: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.251706: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.251708: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.267865: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.268132: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.268133: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.268136: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.284078: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.284257: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.284258: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.284260: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.300314: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.300462: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.300463: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.300464: call_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.314533: synchronize_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.314657: synchronize_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.314658: call_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.314659: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.331056: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.331227: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.331227: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.331229: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.347334: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.347517: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.347518: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.347519: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.363567: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.363731: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.363732: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.363733: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.379804: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.379987: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.379988: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.379989: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.396077: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.396259: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.396260: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.396261: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.412385: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.412653: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.412654: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.412657: call_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.426647: synchronize_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.426888: synchronize_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.426889: call_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.426891: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.443134: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.443325: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.443325: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.443327: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.459327: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.459521: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.459522: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.459524: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.475553: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.475672: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.475673: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.475674: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.491798: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.491919: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.491920: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.491921: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.508035: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.508160: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.508161: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.508162: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.524301: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.524419: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.524419: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.524421: call_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.538545: synchronize_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.538665: synchronize_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.538666: call_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.538667: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.555051: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.555174: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.555174: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.555175: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.571290: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.571412: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.571412: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.571413: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.587554: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.587672: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.587673: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.587674: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.603843: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.604008: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.604008: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.604010: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.620087: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.620317: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.620318: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.620321: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.636342: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.636520: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.636521: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.636523: call_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.650579: synchronize_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.650737: synchronize_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.650738: call_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.650739: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.667084: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.667247: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.667247: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.667249: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.683340: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.683513: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.683514: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.683516: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.699556: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.699675: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.699675: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.699676: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.715841: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.716013: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.716014: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.716015: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.732096: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.732269: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.732270: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.732271: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.748393: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.748641: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.748642: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.748645: call_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.762639: synchronize_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.762854: synchronize_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.762855: call_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.762857: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.779100: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.779278: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.779279: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.779280: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.795348: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.795525: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.795526: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.795527: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.811583: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.811749: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.811750: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.811751: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.827831: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.827997: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.827998: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.827999: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.844104: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.844281: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.844281: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.844283: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.860339: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.860484: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.860485: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.860487: call_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.874584: synchronize_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.874737: synchronize_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.874737: call_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.874739: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.891117: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 144.891459: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.891460: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 144.891463: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.907376: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 144.907628: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.907629: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 144.907631: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.923590: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 144.923756: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.923756: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 144.923758: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.939837: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 144.940007: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.940008: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 144.940009: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.956110: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 144.956285: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.956286: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 144.956287: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.972344: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 144.972492: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.972493: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 144.972495: call_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.986566: synchronize_srcu: exit > stress-cpu-hotp-2443 [002] .... 144.986689: synchronize_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.986689: call_srcu: enter > stress-cpu-hotp-2443 [002] .... 144.986690: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.003114: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.003292: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.003293: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.003294: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.019350: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.019528: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.019529: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.019531: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.035659: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.035939: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.035940: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.035943: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.051891: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.052106: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.052107: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.052109: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.068125: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.068279: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.068279: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.068280: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.084332: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.084472: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.084473: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.084474: call_srcu: exit > stress-cpu-hotp-2443 [002] .... 145.098605: synchronize_srcu: exit > stress-cpu-hotp-2443 [002] .... 145.098743: synchronize_srcu: enter > stress-cpu-hotp-2443 [002] .... 145.098744: call_srcu: enter > stress-cpu-hotp-2443 [002] .... 145.098745: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.114590: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.114728: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.114728: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.114729: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.130591: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.130728: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.130728: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.130730: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.147857: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.228078: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.228079: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.228080: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.244025: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.244106: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.244106: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.244107: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.260286: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.260361: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.260361: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.260362: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.274522: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.274594: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.274594: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.274595: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.291289: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.291358: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.291358: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.291359: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.307529: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.307599: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.307599: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.307600: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.323901: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.324063: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.324063: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.324065: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.340122: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.340288: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.340289: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.340291: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.356378: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.356519: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.356520: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.356521: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.370650: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.370797: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.370798: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.370799: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.387385: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.387547: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.387548: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.387549: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.403636: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.403799: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.403800: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.403802: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.419885: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.420055: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.420056: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.420057: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.436154: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.436372: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.436372: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.436374: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.452413: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.452550: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.452551: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.452552: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.466606: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.466721: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.466721: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.466722: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.483395: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.483555: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.483555: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.483557: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.499628: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.499761: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .N.. 145.499774: call_srcu: enter > stress-cpu-hotp-2443 [005] .N.. 145.499775: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.515879: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.516009: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.516009: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.516011: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.532164: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.532488: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.532489: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.532492: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.548447: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.548659: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.548660: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.548663: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.562706: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.562897: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.562898: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.562900: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.579403: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.579562: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.579563: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.579564: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.595636: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.595766: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.595766: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.595768: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.611902: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.612059: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.612060: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.612061: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.628136: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.628266: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.628267: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.628268: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.644392: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.644518: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.644519: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.644521: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.658609: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.658698: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.658699: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.658700: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.675410: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.675570: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.675571: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.675572: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.691706: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.691950: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.691951: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.691953: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.707926: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.708096: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.708097: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.708099: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.724154: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.724333: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.724333: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.724335: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.740412: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.740559: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.740559: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.740561: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.754685: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.754835: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.754836: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.754838: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.771415: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.771591: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.771592: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.771594: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.787635: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.787747: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.787748: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.787749: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.803943: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.804168: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.804170: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.804172: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.820168: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.820320: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.820321: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.820322: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.836412: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.836543: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.836544: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.836545: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.850638: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.850757: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.850757: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.850758: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.867433: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.867597: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.867598: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.867599: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.883680: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.883843: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.883843: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.883845: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.899908: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.900073: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.900073: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.900075: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.916213: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 145.916405: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.916406: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 145.916408: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.932453: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 145.932653: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.932653: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 145.932656: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.946674: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 145.946818: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.946819: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 145.946820: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.963402: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 145.963539: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.963540: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 145.963542: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.979671: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 145.979790: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.979790: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 145.979792: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.995927: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 145.996046: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.996046: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 145.996048: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 146.012167: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 146.012282: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 146.012283: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 146.012284: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 146.028405: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 146.028522: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 146.028522: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 146.028523: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 146.042653: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 146.042774: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 146.042774: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 146.042775: call_srcu: exit > stress-cpu-hotp-2443 [004] .... 146.059407: synchronize_srcu: exit > stress-cpu-hotp-2443 [004] .... 146.059521: synchronize_srcu: enter > stress-cpu-hotp-2443 [004] .... 146.059522: call_srcu: enter > stress-cpu-hotp-2443 [004] .... 146.059523: call_srcu: exit > stress-cpu-hotp-2443 [005] .... 146.075658: synchronize_srcu: exit > stress-cpu-hotp-2443 [005] .... 146.075772: synchronize_srcu: enter > stress-cpu-hotp-2443 [005] .... 146.075773: call_srcu: enter > stress-cpu-hotp-2443 [005] .... 146.075774: call_srcu: exit > stress-cpu-hotp-2443 [006] .... 146.091906: synchronize_srcu: exit > stress-cpu-hotp-2443 [006] .... 146.092021: synchronize_srcu: enter > stress-cpu-hotp-2443 [006] .... 146.092022: call_srcu: enter > stress-cpu-hotp-2443 [006] .... 146.092023: call_srcu: exit > stress-cpu-hotp-2443 [007] .... 146.108187: synchronize_srcu: exit > stress-cpu-hotp-2443 [007] .... 146.108336: synchronize_srcu: enter > stress-cpu-hotp-2443 [007] .... 146.108337: call_srcu: enter > stress-cpu-hotp-2443 [007] .... 146.108338: call_srcu: exit > stress-cpu-hotp-2443 [000] .... 146.124439: synchronize_srcu: exit > stress-cpu-hotp-2443 [000] .... 146.124572: synchronize_srcu: enter > stress-cpu-hotp-2443 [000] .... 146.124572: call_srcu: enter > stress-cpu-hotp-2443 [000] .... 146.124574: call_srcu: exit > stress-cpu-hotp-2443 [003] .... 146.138656: synchronize_srcu: exit > stress-cpu-hotp-2443 [003] .... 146.138773: synchronize_srcu: enter > stress-cpu-hotp-2443 [003] .... 146.138773: call_srcu: enter > stress-cpu-hotp-2443 [003] .... 146.138774: call_srcu: exit > >