From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756835Ab3BKMZx (ORCPT ); Mon, 11 Feb 2013 07:25:53 -0500 Received: from e23smtp01.au.ibm.com ([202.81.31.143]:47344 "EHLO e23smtp01.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756674Ab3BKMZv (ORCPT ); Mon, 11 Feb 2013 07:25:51 -0500 Message-ID: <5118E2CD.90401@linux.vnet.ibm.com> Date: Mon, 11 Feb 2013 17:53:41 +0530 From: "Srivatsa S. Bhat" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120828 Thunderbird/15.0 MIME-Version: 1.0 To: Vincent Guittot CC: Russell King - ARM Linux , linux-doc@vger.kernel.org, peterz@infradead.org, fweisbec@gmail.com, linux-kernel@vger.kernel.org, walken@google.com, mingo@kernel.org, linux-arch@vger.kernel.org, xiaoguangrong@linux.vnet.ibm.com, wangyun@linux.vnet.ibm.com, paulmck@linux.vnet.ibm.com, nikunj@linux.vnet.ibm.com, linux-pm@vger.kernel.org, Rusty Russell , rostedt@goodmis.org, rjw@sisk.pl, namhyung@kernel.org, tglx@linutronix.de, linux-arm-kernel@lists.infradead.org, netdev@vger.kernel.org, oleg@redhat.com, sbw@mit.edu, tj@kernel.org, akpm@linux-foundation.org, linuxppc-dev@lists.ozlabs.org Subject: Re: [PATCH v5 00/45] CPU hotplug: stop_machine()-free CPU hotplug References: <20130122073210.13822.50434.stgit@srivatsabhat.in.ibm.com> <510FBC01.2030405@linux.vnet.ibm.com> <87haloiwv0.fsf@rustcorp.com.au> <51134596.4080106@linux.vnet.ibm.com> <20130208154113.GV17833@n2100.arm.linux.org.uk> <51152B81.2050501@linux.vnet.ibm.com> <51153F72.1060005@linux.vnet.ibm.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Scanned: Fidelis XPS MAILER x-cbid: 13021112-1618-0000-0000-000003511A73 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/11/2013 05:28 PM, Vincent Guittot wrote: > On 8 February 2013 19:09, Srivatsa S. Bhat > wrote: >> On 02/08/2013 10:14 PM, Srivatsa S. Bhat wrote: >>> On 02/08/2013 09:11 PM, Russell King - ARM Linux wrote: >>>> On Thu, Feb 07, 2013 at 11:41:34AM +0530, Srivatsa S. Bhat wrote: >>>>> On 02/07/2013 09:44 AM, Rusty Russell wrote: >>>>>> "Srivatsa S. Bhat" writes: >>>>>>> On 01/22/2013 01:03 PM, Srivatsa S. Bhat wrote: >>>>>>> Avg. latency of 1 CPU offline (ms) [stop-cpu/stop-m/c latency] >>>>>>> >>>>>>> # online CPUs Mainline (with stop-m/c) This patchset (no stop-m/c) >>>>>>> >>>>>>> 8 17.04 7.73 >>>>>>> >>>>>>> 16 18.05 6.44 >>>>>>> >>>>>>> 32 17.31 7.39 >>>>>>> >>>>>>> 64 32.40 9.28 >>>>>>> >>>>>>> 128 98.23 7.35 >>>>>> >>>>>> Nice! >>>>> >>>>> Thank you :-) >>>>> >>>>>> I wonder how the ARM guys feel with their quad-cpu systems... >>>>>> >>>>> >>>>> That would be definitely interesting to know :-) >>>> >>>> That depends what exactly you'd like tested (and how) and whether you'd >>>> like it to be a test-chip based quad core, or an OMAP dual-core SoC. >>>> >>> >>> The effect of stop_machine() doesn't really depend on the CPU architecture >>> used underneath or the platform. It depends only on the _number_ of >>> _logical_ CPUs used. >>> >>> And stop_machine() has 2 noticeable drawbacks: >>> 1. It makes the hotplug operation itself slow >>> 2. and it causes disruptions to the workloads running on the other >>> CPUs by hijacking the entire machine for significant amounts of time. >>> >>> In my experiments (mentioned above), I tried to measure how my patchset >>> improves (reduces) the duration of hotplug (CPU offline) itself. Which is >>> also slightly indicative of the impact it has on the rest of the system. >>> >>> But what would be nice to test, is a setup where the workloads running on >>> the rest of the system are latency-sensitive, and measure the impact of >>> CPU offline on them, with this patchset applied. That would tell us how >>> far is this useful in making CPU hotplug less disruptive on the system. >>> >>> Of course, it would be nice to also see whether we observe any reduction >>> in hotplug duration itself (point 1 above) on ARM platforms with lot >>> of CPUs. [This could potentially speed up suspend/resume, which is used >>> rather heavily on ARM platforms]. >>> >>> The benefits from this patchset over mainline (both in terms of points >>> 1 and 2 above) is expected to increase, with increasing number of CPUs in >>> the system. >>> >> >> Adding Vincent to CC, who had previously evaluated the performance and >> latency implications of CPU hotplug on ARM platforms, IIRC. >> > > Hi Srivatsa, > > I can try to run some of our stress tests on your patches. Great! > Have you > got a git tree that i can pull ? > Unfortunately, no, none at the moment.. :-( Regards, Srivatsa S. Bhat From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from e23smtp04.au.ibm.com (e23smtp04.au.ibm.com [202.81.31.146]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "e23smtp04.au.ibm.com", Issuer "GeoTrust SSL CA" (not verified)) by ozlabs.org (Postfix) with ESMTPS id 63D112C0091 for ; Mon, 11 Feb 2013 23:25:47 +1100 (EST) Received: from /spool/local by e23smtp04.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 11 Feb 2013 22:16:58 +1000 Received: from d23relay04.au.ibm.com (d23relay04.au.ibm.com [9.190.234.120]) by d23dlp02.au.ibm.com (Postfix) with ESMTP id AD04E2BB0051 for ; Mon, 11 Feb 2013 23:25:43 +1100 (EST) Received: from d23av02.au.ibm.com (d23av02.au.ibm.com [9.190.235.138]) by d23relay04.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r1BCDYMP65929228 for ; Mon, 11 Feb 2013 23:13:34 +1100 Received: from d23av02.au.ibm.com (loopback [127.0.0.1]) by d23av02.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r1BCPfcj009827 for ; Mon, 11 Feb 2013 23:25:43 +1100 Message-ID: <5118E2CD.90401@linux.vnet.ibm.com> Date: Mon, 11 Feb 2013 17:53:41 +0530 From: "Srivatsa S. Bhat" MIME-Version: 1.0 To: Vincent Guittot Subject: Re: [PATCH v5 00/45] CPU hotplug: stop_machine()-free CPU hotplug References: <20130122073210.13822.50434.stgit@srivatsabhat.in.ibm.com> <510FBC01.2030405@linux.vnet.ibm.com> <87haloiwv0.fsf@rustcorp.com.au> <51134596.4080106@linux.vnet.ibm.com> <20130208154113.GV17833@n2100.arm.linux.org.uk> <51152B81.2050501@linux.vnet.ibm.com> <51153F72.1060005@linux.vnet.ibm.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Cc: linux-doc@vger.kernel.org, peterz@infradead.org, fweisbec@gmail.com, linux-kernel@vger.kernel.org, walken@google.com, mingo@kernel.org, linux-arch@vger.kernel.org, Russell King - ARM Linux , xiaoguangrong@linux.vnet.ibm.com, wangyun@linux.vnet.ibm.com, paulmck@linux.vnet.ibm.com, nikunj@linux.vnet.ibm.com, linux-pm@vger.kernel.org, Rusty Russell , rostedt@goodmis.org, rjw@sisk.pl, namhyung@kernel.org, tglx@linutronix.de, linux-arm-kernel@lists.infradead.org, netdev@vger.kernel.org, oleg@redhat.com, sbw@mit.edu, tj@kernel.org, akpm@linux-foundation.org, linuxppc-dev@lists.ozlabs.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 02/11/2013 05:28 PM, Vincent Guittot wrote: > On 8 February 2013 19:09, Srivatsa S. Bhat > wrote: >> On 02/08/2013 10:14 PM, Srivatsa S. Bhat wrote: >>> On 02/08/2013 09:11 PM, Russell King - ARM Linux wrote: >>>> On Thu, Feb 07, 2013 at 11:41:34AM +0530, Srivatsa S. Bhat wrote: >>>>> On 02/07/2013 09:44 AM, Rusty Russell wrote: >>>>>> "Srivatsa S. Bhat" writes: >>>>>>> On 01/22/2013 01:03 PM, Srivatsa S. Bhat wrote: >>>>>>> Avg. latency of 1 CPU offline (ms) [stop-cpu/stop-m/c latency] >>>>>>> >>>>>>> # online CPUs Mainline (with stop-m/c) This patchset (no stop-m/c) >>>>>>> >>>>>>> 8 17.04 7.73 >>>>>>> >>>>>>> 16 18.05 6.44 >>>>>>> >>>>>>> 32 17.31 7.39 >>>>>>> >>>>>>> 64 32.40 9.28 >>>>>>> >>>>>>> 128 98.23 7.35 >>>>>> >>>>>> Nice! >>>>> >>>>> Thank you :-) >>>>> >>>>>> I wonder how the ARM guys feel with their quad-cpu systems... >>>>>> >>>>> >>>>> That would be definitely interesting to know :-) >>>> >>>> That depends what exactly you'd like tested (and how) and whether you'd >>>> like it to be a test-chip based quad core, or an OMAP dual-core SoC. >>>> >>> >>> The effect of stop_machine() doesn't really depend on the CPU architecture >>> used underneath or the platform. It depends only on the _number_ of >>> _logical_ CPUs used. >>> >>> And stop_machine() has 2 noticeable drawbacks: >>> 1. It makes the hotplug operation itself slow >>> 2. and it causes disruptions to the workloads running on the other >>> CPUs by hijacking the entire machine for significant amounts of time. >>> >>> In my experiments (mentioned above), I tried to measure how my patchset >>> improves (reduces) the duration of hotplug (CPU offline) itself. Which is >>> also slightly indicative of the impact it has on the rest of the system. >>> >>> But what would be nice to test, is a setup where the workloads running on >>> the rest of the system are latency-sensitive, and measure the impact of >>> CPU offline on them, with this patchset applied. That would tell us how >>> far is this useful in making CPU hotplug less disruptive on the system. >>> >>> Of course, it would be nice to also see whether we observe any reduction >>> in hotplug duration itself (point 1 above) on ARM platforms with lot >>> of CPUs. [This could potentially speed up suspend/resume, which is used >>> rather heavily on ARM platforms]. >>> >>> The benefits from this patchset over mainline (both in terms of points >>> 1 and 2 above) is expected to increase, with increasing number of CPUs in >>> the system. >>> >> >> Adding Vincent to CC, who had previously evaluated the performance and >> latency implications of CPU hotplug on ARM platforms, IIRC. >> > > Hi Srivatsa, > > I can try to run some of our stress tests on your patches. Great! > Have you > got a git tree that i can pull ? > Unfortunately, no, none at the moment.. :-( Regards, Srivatsa S. Bhat From mboxrd@z Thu Jan 1 00:00:00 1970 From: srivatsa.bhat@linux.vnet.ibm.com (Srivatsa S. Bhat) Date: Mon, 11 Feb 2013 17:53:41 +0530 Subject: [PATCH v5 00/45] CPU hotplug: stop_machine()-free CPU hotplug In-Reply-To: References: <20130122073210.13822.50434.stgit@srivatsabhat.in.ibm.com> <510FBC01.2030405@linux.vnet.ibm.com> <87haloiwv0.fsf@rustcorp.com.au> <51134596.4080106@linux.vnet.ibm.com> <20130208154113.GV17833@n2100.arm.linux.org.uk> <51152B81.2050501@linux.vnet.ibm.com> <51153F72.1060005@linux.vnet.ibm.com> Message-ID: <5118E2CD.90401@linux.vnet.ibm.com> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On 02/11/2013 05:28 PM, Vincent Guittot wrote: > On 8 February 2013 19:09, Srivatsa S. Bhat > wrote: >> On 02/08/2013 10:14 PM, Srivatsa S. Bhat wrote: >>> On 02/08/2013 09:11 PM, Russell King - ARM Linux wrote: >>>> On Thu, Feb 07, 2013 at 11:41:34AM +0530, Srivatsa S. Bhat wrote: >>>>> On 02/07/2013 09:44 AM, Rusty Russell wrote: >>>>>> "Srivatsa S. Bhat" writes: >>>>>>> On 01/22/2013 01:03 PM, Srivatsa S. Bhat wrote: >>>>>>> Avg. latency of 1 CPU offline (ms) [stop-cpu/stop-m/c latency] >>>>>>> >>>>>>> # online CPUs Mainline (with stop-m/c) This patchset (no stop-m/c) >>>>>>> >>>>>>> 8 17.04 7.73 >>>>>>> >>>>>>> 16 18.05 6.44 >>>>>>> >>>>>>> 32 17.31 7.39 >>>>>>> >>>>>>> 64 32.40 9.28 >>>>>>> >>>>>>> 128 98.23 7.35 >>>>>> >>>>>> Nice! >>>>> >>>>> Thank you :-) >>>>> >>>>>> I wonder how the ARM guys feel with their quad-cpu systems... >>>>>> >>>>> >>>>> That would be definitely interesting to know :-) >>>> >>>> That depends what exactly you'd like tested (and how) and whether you'd >>>> like it to be a test-chip based quad core, or an OMAP dual-core SoC. >>>> >>> >>> The effect of stop_machine() doesn't really depend on the CPU architecture >>> used underneath or the platform. It depends only on the _number_ of >>> _logical_ CPUs used. >>> >>> And stop_machine() has 2 noticeable drawbacks: >>> 1. It makes the hotplug operation itself slow >>> 2. and it causes disruptions to the workloads running on the other >>> CPUs by hijacking the entire machine for significant amounts of time. >>> >>> In my experiments (mentioned above), I tried to measure how my patchset >>> improves (reduces) the duration of hotplug (CPU offline) itself. Which is >>> also slightly indicative of the impact it has on the rest of the system. >>> >>> But what would be nice to test, is a setup where the workloads running on >>> the rest of the system are latency-sensitive, and measure the impact of >>> CPU offline on them, with this patchset applied. That would tell us how >>> far is this useful in making CPU hotplug less disruptive on the system. >>> >>> Of course, it would be nice to also see whether we observe any reduction >>> in hotplug duration itself (point 1 above) on ARM platforms with lot >>> of CPUs. [This could potentially speed up suspend/resume, which is used >>> rather heavily on ARM platforms]. >>> >>> The benefits from this patchset over mainline (both in terms of points >>> 1 and 2 above) is expected to increase, with increasing number of CPUs in >>> the system. >>> >> >> Adding Vincent to CC, who had previously evaluated the performance and >> latency implications of CPU hotplug on ARM platforms, IIRC. >> > > Hi Srivatsa, > > I can try to run some of our stress tests on your patches. Great! > Have you > got a git tree that i can pull ? > Unfortunately, no, none at the moment.. :-( Regards, Srivatsa S. Bhat