From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from e28smtp02.in.ibm.com (e28smtp02.in.ibm.com [122.248.162.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "e28smtp02.in.ibm.com", Issuer "GeoTrust SSL CA" (not verified)) by ozlabs.org (Postfix) with ESMTPS id B39332C1074 for ; Mon, 18 Feb 2013 21:36:33 +1100 (EST) Received: from /spool/local by e28smtp02.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 18 Feb 2013 16:03:24 +0530 Received: from d28relay03.in.ibm.com (d28relay03.in.ibm.com [9.184.220.60]) by d28dlp03.in.ibm.com (Postfix) with ESMTP id 793A2125804E for ; Mon, 18 Feb 2013 16:07:11 +0530 (IST) Received: from d28av03.in.ibm.com (d28av03.in.ibm.com [9.184.220.65]) by d28relay03.in.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r1IAaMjb31785018 for ; Mon, 18 Feb 2013 16:06:22 +0530 Received: from d28av03.in.ibm.com (loopback [127.0.0.1]) by d28av03.in.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r1IAaNHQ010716 for ; Mon, 18 Feb 2013 21:36:26 +1100 Message-ID: <512203B3.7090002@linux.vnet.ibm.com> Date: Mon, 18 Feb 2013 16:04:27 +0530 From: "Srivatsa S. Bhat" MIME-Version: 1.0 To: Vincent Guittot Subject: Re: [PATCH v5 00/45] CPU hotplug: stop_machine()-free CPU hotplug References: <20130122073210.13822.50434.stgit@srivatsabhat.in.ibm.com> <510FBC01.2030405@linux.vnet.ibm.com> <87haloiwv0.fsf@rustcorp.com.au> <51134596.4080106@linux.vnet.ibm.com> <20130208154113.GV17833@n2100.arm.linux.org.uk> <51152B81.2050501@linux.vnet.ibm.com> <51153F72.1060005@linux.vnet.ibm.com> <5118E2CD.90401@linux.vnet.ibm.com> <20130211190852.GA5695@linux.vnet.ibm.com> <5119BDFD.1000909@linux.vnet.ibm.com> <511E8F3C.2010406@linux.vnet.ibm.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Cc: linux-doc@vger.kernel.org, peterz@infradead.org, fweisbec@gmail.com, linux-kernel@vger.kernel.org, walken@google.com, mingo@kernel.org, linux-arch@vger.kernel.org, Russell King - ARM Linux , xiaoguangrong@linux.vnet.ibm.com, wangyun@linux.vnet.ibm.com, paulmck@linux.vnet.ibm.com, nikunj@linux.vnet.ibm.com, linux-pm@vger.kernel.org, Rusty Russell , rostedt@goodmis.org, rjw@sisk.pl, namhyung@kernel.org, tglx@linutronix.de, linux-arm-kernel@lists.infradead.org, netdev@vger.kernel.org, oleg@redhat.com, sbw@mit.edu, tj@kernel.org, akpm@linux-foundation.org, linuxppc-dev@lists.ozlabs.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 02/18/2013 03:54 PM, Vincent Guittot wrote: > On 15 February 2013 20:40, Srivatsa S. Bhat > wrote: >> Hi Vincent, >> >> On 02/15/2013 06:58 PM, Vincent Guittot wrote: >>> Hi Srivatsa, >>> >>> I have run some tests with you branch (thanks Paul for the git tree) >>> and you will find results below. >>> >> >> Thank you very much for testing this patchset! >> >>> The tests condition are: >>> - 5 CPUs system in 2 clusters >>> - The test plugs/unplugs CPU2 and it increases the system load each 20 >>> plug/unplug sequence with either more cyclictests threads >>> - The test is done with all CPUs online and with only CPU0 and CPU2 >>> >>> The main conclusion is that there is no differences with and without >>> your patches with my stress tests. I'm not sure that it was the >>> expected results but the cpu_down is already quite low : 4-5ms in >>> average >>> >> >> Atleast my patchset doesn't perform _worse_ than mainline, with respect >> to cpu_down duration :-) > > yes exactly and it has pass more than 400 consecutive plug/unplug on > an ARM platform > Great! However, did you turn on CPU_IDLE during your tests? In my tests, I had turned off cpu idle in the .config, like I had mentioned in the cover letter. I'm struggling to get it working with CPU_IDLE/INTEL_IDLE turned on, because it gets into a lockup almost immediately. It appears that the lock-holder of clockevents_lock never releases it, for some reason.. See below for the full log. Lockdep has not been useful in debugging this, unfortunately :-( >> >> So, here is the analysis: >> Stop-machine() doesn't really slow down CPU-down operation, if the rest >> of the CPUs are mostly running in userspace all the time. Because, the >> CPUs running userspace workloads cooperate very eagerly with the stop-machine >> dance - they receive the resched IPI, and allow the per-cpu cpu-stopper >> thread to monopolize the CPU, almost immediately. >> >> The scenario where stop-machine() takes longer to take effect is when >> most of the online CPUs are running in kernelspace, because, then the >> probability that they call preempt_disable() frequently (and hence inhibit >> stop-machine) is higher. That's why, in my tests, I ran genload from LTP >> which generated a lot of system-time (system-time in 'top' indicates activity >> in kernelspace). Hence my patchset showed significant improvement over >> mainline in my tests. >> > > ok, I hadn't noticed this important point for the test > >> However, your test is very useful too, if we measure a different parameter: >> the latency impact on the workloads running on the system (cyclic test). >> One other important aim of this patchset is to make hotplug as less intrusive >> as possible, for other workloads running on the system. So if you measure >> the cyclictest numbers, I would expect my patchset to show better numbers >> than mainline, when you do cpu-hotplug in parallel (same test that you did). >> Mainline would run stop-machine and hence interrupt the cyclic test tasks >> too often. My patchset wouldn't do that, and hence cyclic test should >> ideally show better numbers. > > In fact, I haven't looked at the results as i was more interested by > the load that was generated > >> >> I'd really appreciate if you could try that out and let me know how it >> goes.. :-) Thank you very much! > > ok, I'm going to try to run a test series > Great! Thank you :-) Regards, Srivatsa S. Bhat -------------------------------------------------------------------------------- Lockup observed while running this patchset, with CPU_IDLE and INTEL_IDLE turned on in the .config: smpboot: CPU 1 is now offline Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 11 Pid: 0, comm: swapper/11 Not tainted 3.8.0-rc7+stpmch13-1 #8 Call Trace: [] panic+0xc9/0x1ee [] watchdog_overflow_callback+0xb1/0xc0 [] __perf_event_overflow+0x9c/0x330 [] ? x86_perf_event_set_period+0xd8/0x160 [] perf_event_overflow+0x14/0x20 [] intel_pmu_handle_irq+0x1c4/0x360 [] perf_event_nmi_handler+0x21/0x30 [] nmi_handle+0xb6/0x200 [] ? oops_begin+0xd0/0xd0 [] default_do_nmi+0x68/0x220 [] do_nmi+0xc0/0x110 [] end_repeat_nmi+0x1e/0x2e [] ? delay_tsc+0x38/0xb0 [] ? delay_tsc+0x38/0xb0 [] ? delay_tsc+0x38/0xb0 <> [] __delay+0xf/0x20 [] do_raw_spin_lock+0x7e/0x150 [] _raw_spin_lock_irqsave+0x61/0x70 [] ? clockevents_notify+0x28/0x150 [] ? _raw_spin_unlock_irqrestore+0x77/0x80 [] clockevents_notify+0x28/0x150 [] intel_idle+0xaf/0xe0 [] ? disable_cpuidle+0x20/0x20 [] cpuidle_enter+0x19/0x20 [] cpuidle_wrap_enter+0x41/0xa0 [] cpuidle_enter_tk+0x10/0x20 [] cpuidle_enter_state+0x17/0x50 [] cpuidle_idle_call+0xd9/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xdd/0xdf BUG: spinlock lockup suspected on CPU#2, migration/2/19 lock: clockevents_lock+0x0/0x40, .magic: dead4ead, .owner: swapper/8/0, .owner_cpu: 8 Pid: 19, comm: migration/2 Not tainted 3.8.0-rc7+stpmch13-1 #8 Call Trace: [] spin_dump+0x78/0xc0 [] do_raw_spin_lock+0x126/0x150 [] _raw_spin_lock_irqsave+0x61/0x70 [] ? clockevents_notify+0x28/0x150 [] clockevents_notify+0x28/0x150 [] hrtimer_cpu_notify+0xe3/0x107 [] notifier_call_chain+0x5c/0x120 [] __raw_notifier_call_chain+0xe/0x10 [] __cpu_notify+0x20/0x40 [] take_cpu_down+0x53/0x80 [] cpu_stopper_thread+0xfa/0x1e0 [] ? enable_nonboot_cpus+0xf0/0xf0 [] ? __schedule+0x469/0x890 [] ? res_counter_init+0x60/0x60 [] ? res_counter_init+0x60/0x60 [] kthread+0xee/0x100 [] ? __init_kthread_worker+0x70/0x70 [] ret_from_fork+0x7c/0xb0 [] ? __init_kthread_worker+0x70/0x70 sending NMI to all CPUs: NMI backtrace for cpu 3 CPU 3 Pid: 0, comm: swapper/3 Not tainted 3.8.0-rc7+stpmch13-1 #8 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] intel_idle+0x99/0xe0 RSP: 0018:ffff8808db57bdd8 EFLAGS: 00000046 RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff8808db57bfd8 RDI: ffffffff815a6d37 RBP: ffff8808db57be08 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003 R13: 0000000000000020 R14: ffffffff81a9c000 R15: 0000000000000003 FS: 0000000000000000(0000) GS:ffff8808ffcc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fffbbb902f8 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/3 (pid: 0, threadinfo ffff8808db57a000, task ffff8808db578380) Stack: 0000000000000000 0000000300000001 ffff8808db57be18 ffff8808ffcda070 000000bd1ad2c5de ffffffff81472ee0 ffff8808db57be18 ffffffff81472ef9 ffff8808db57be78 ffffffff814734c1 ffff8808ffcc0000 0000000100000046 Call Trace: [] ? disable_cpuidle+0x20/0x20 [] cpuidle_enter+0x19/0x20 [] cpuidle_wrap_enter+0x41/0xa0 [] cpuidle_enter_tk+0x10/0x20 [] cpuidle_enter_state+0x17/0x50 [] cpuidle_idle_call+0xd9/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xdd/0xdf Code: ff 48 8d 86 38 e0 ff ff 83 e2 08 75 1e 31 d2 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 69 7d 79 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 91 c1 db NMI backtrace for cpu 5 CPU 5 Pid: 0, comm: swapper/5 Not tainted 3.8.0-rc7+stpmch13-1 #8 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] intel_idle+0x99/0xe0 RSP: 0018:ffff8808db583dd8 EFLAGS: 00000046 RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff8808db583fd8 RDI: ffffffff815a6d37 RBP: ffff8808db583e08 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003 R13: 0000000000000020 R14: ffffffff81a9c000 R15: 0000000000000003 FS: 0000000000000000(0000) GS:ffff88117fc40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/5 (pid: 0, threadinfo ffff8808db582000, task ffff8808db580400) Stack: 0000000000000000 0000000500000001 ffff8808db583e18 ffff88117fc5a070 000000bd1af68ce5 ffffffff81472ee0 ffff8808db583e18 ffffffff81472ef9 ffff8808db583e78 ffffffff814734c1 ffff88117fc40000 0000000100000046 Call Trace: [] ? disable_cpuidle+0x20/0x20 [] cpuidle_enter+0x19/0x20 [] cpuidle_wrap_enter+0x41/0xa0 [] cpuidle_enter_tk+0x10/0x20 [] cpuidle_enter_state+0x17/0x50 [] cpuidle_idle_call+0xd9/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xdd/0xdf Code: ff 48 8d 86 38 e0 ff ff 83 e2 08 75 1e 31 d2 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 69 7d 79 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 91 c1 db NMI backtrace for cpu 6 CPU 6 Pid: 0, comm: swapper/6 Not tainted 3.8.0-rc7+stpmch13-1 #8 IBM IBM System x -[7870C4Q]-/68Y8033 RIP: 0010:[] [] intel_idle+0x99/0xe0 RSP: 0018:ffff8808db589dd8 EFLAGS: 00000046 RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffff8808db589fd8 RDI: ffffffff815a6d37 RBP: ffff8808db589e08 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002 R13: 0000000000000010 R14: ffffffff81a9c000 R15: 0000000000000002 FS: 0000000000000000(0000) GS:ffff88117fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fde168df000 CR3: 0000000001a0c000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper/6 (pid: 0, threadinfo ffff8808db588000, task ffff8808db586440) Stack: 0000000000000000 0000000600000001 ffff8808db589e18 ffff88117fc9a070 000000bd1af4a55f ffffffff81472ee0 ffff8808db589e18 ffffffff81472ef9 ffff8808db589e78 ffffffff814734c1 ffff88117fc80000 0000000b00000046 Call Trace: [] ? disable_cpuidle+0x20/0x20 [] cpuidle_enter+0x19/0x20 [] cpuidle_wrap_enter+0x41/0xa0 [] cpuidle_enter_tk+0x10/0x20 [] cpuidle_enter_state+0x17/0x50 [] cpuidle_idle_call+0xd9/0x290 [] cpu_idle+0xe5/0x140 [] start_secondary+0xdd/0xdf Code: ff 48 8d 86 38 e0 ff ff 83 e2 08 75 1e 31 d2 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 69 7d 79 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 91 c1 db 3h