* [PATCH] drivers: base: update cpu offline info when do hotplug @ 2014-10-20 3:29 Neil Zhang 2014-10-20 4:43 ` Greg KH 2014-10-21 13:02 ` Dan Streetman 0 siblings, 2 replies; 22+ messages in thread From: Neil Zhang @ 2014-10-20 3:29 UTC (permalink / raw) To: linux-kernel, gregkh; +Cc: Neil Zhang The current per-cpu offline info won't be updated if it is hotplugged in/out by a kernel governer. Let's update it via cpu notifier. Signed-off-by: Neil Zhang <zhangwm@marvell.com> --- drivers/base/cpu.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c index 006b1bc..9d61824 100644 --- a/drivers/base/cpu.c +++ b/drivers/base/cpu.c @@ -418,10 +418,35 @@ static void __init cpu_dev_register_generic(void) #endif } +static int device_hotplug_notifier(struct notifier_block *nfb, + unsigned long action, void *hcpu) +{ + unsigned int cpu = (unsigned long)hcpu; + struct device *dev = get_cpu_device(cpu); + int ret; + + switch (action & ~CPU_TASKS_FROZEN) { + case CPU_ONLINE: + dev->offline = false; + ret = NOTIFY_OK; + break; + case CPU_DYING: + dev->offline = true; + ret = NOTIFY_OK; + break; + default: + ret = NOTIFY_DONE; + break; + } + + return ret; +} + void __init cpu_dev_init(void) { if (subsys_system_register(&cpu_subsys, cpu_root_attr_groups)) panic("Failed to register CPU subsystem"); cpu_dev_register_generic(); + cpu_notifier(device_hotplug_notifier, 0); } -- 1.7.9.5 ^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-20 3:29 [PATCH] drivers: base: update cpu offline info when do hotplug Neil Zhang @ 2014-10-20 4:43 ` Greg KH 2014-10-20 6:39 ` Neil Zhang 2014-10-21 13:02 ` Dan Streetman 1 sibling, 1 reply; 22+ messages in thread From: Greg KH @ 2014-10-20 4:43 UTC (permalink / raw) To: Neil Zhang; +Cc: linux-kernel On Mon, Oct 20, 2014 at 11:29:08AM +0800, Neil Zhang wrote: > The current per-cpu offline info won't be updated if it is > hotplugged in/out by a kernel governer. > Let's update it via cpu notifier. > > Signed-off-by: Neil Zhang <zhangwm@marvell.com> > --- > drivers/base/cpu.c | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > > diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c > index 006b1bc..9d61824 100644 > --- a/drivers/base/cpu.c > +++ b/drivers/base/cpu.c > @@ -418,10 +418,35 @@ static void __init cpu_dev_register_generic(void) > #endif > } > > +static int device_hotplug_notifier(struct notifier_block *nfb, > + unsigned long action, void *hcpu) > +{ > + unsigned int cpu = (unsigned long)hcpu; > + struct device *dev = get_cpu_device(cpu); > + int ret; > + > + switch (action & ~CPU_TASKS_FROZEN) { > + case CPU_ONLINE: > + dev->offline = false; > + ret = NOTIFY_OK; > + break; > + case CPU_DYING: > + dev->offline = true; > + ret = NOTIFY_OK; > + break; > + default: > + ret = NOTIFY_DONE; > + break; > + } > + > + return ret; > +} > + > void __init cpu_dev_init(void) > { > if (subsys_system_register(&cpu_subsys, cpu_root_attr_groups)) > panic("Failed to register CPU subsystem"); > > cpu_dev_register_generic(); > + cpu_notifier(device_hotplug_notifier, 0); > } How much noise is this going to cause on a big/little system that constantly hot unplug/plugs processors all of the time? greg k-h ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-20 4:43 ` Greg KH @ 2014-10-20 6:39 ` Neil Zhang 2014-10-20 6:48 ` Greg KH 0 siblings, 1 reply; 22+ messages in thread From: Neil Zhang @ 2014-10-20 6:39 UTC (permalink / raw) To: Greg KH; +Cc: linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="gb2312", Size: 2133 bytes --] Greg, -----Original Message----- From: Greg KH [mailto:gregkh@linuxfoundation.org] Sent: 2014Äê10ÔÂ20ÈÕ 12:44 To: Neil Zhang Cc: linux-kernel@vger.kernel.org Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug On Mon, Oct 20, 2014 at 11:29:08AM +0800, Neil Zhang wrote: > The current per-cpu offline info won't be updated if it is hotplugged > in/out by a kernel governer. > Let's update it via cpu notifier. > > Signed-off-by: Neil Zhang <zhangwm@marvell.com> > --- > drivers/base/cpu.c | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > > diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c index > 006b1bc..9d61824 100644 > --- a/drivers/base/cpu.c > +++ b/drivers/base/cpu.c > @@ -418,10 +418,35 @@ static void __init > cpu_dev_register_generic(void) #endif } > > +static int device_hotplug_notifier(struct notifier_block *nfb, > + unsigned long action, void *hcpu) { > + unsigned int cpu = (unsigned long)hcpu; > + struct device *dev = get_cpu_device(cpu); > + int ret; > + > + switch (action & ~CPU_TASKS_FROZEN) { > + case CPU_ONLINE: > + dev->offline = false; > + ret = NOTIFY_OK; > + break; > + case CPU_DYING: > + dev->offline = true; > + ret = NOTIFY_OK; > + break; > + default: > + ret = NOTIFY_DONE; > + break; > + } > + > + return ret; > +} > + > void __init cpu_dev_init(void) > { > if (subsys_system_register(&cpu_subsys, cpu_root_attr_groups)) > panic("Failed to register CPU subsystem"); > > cpu_dev_register_generic(); > + cpu_notifier(device_hotplug_notifier, 0); > } How much noise is this going to cause on a big/little system that constantly hot unplug/plugs processors all of the time? Can you explain more what kind of noise will be introduced on a big/little system? As I know IKS on arm will use cpu_suspend way to power down a core. But I don't know well about other architectures. Please give your suggestions. Thanks! greg k-h Best Regards, Neil Zhang ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-20 6:39 ` Neil Zhang @ 2014-10-20 6:48 ` Greg KH 2014-10-20 7:40 ` Neil Zhang 0 siblings, 1 reply; 22+ messages in thread From: Greg KH @ 2014-10-20 6:48 UTC (permalink / raw) To: Neil Zhang; +Cc: linux-kernel On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: >> How much noise is this going to cause on a big/little system that >> constantly hot unplug/plugs processors all of the time? > > Can you explain more what kind of noise will be introduced on a big/little system? Have you tested this on such a machine? > As I know IKS on arm will use cpu_suspend way to power down a core. Are you sure that it also doesn't use that same functionality to drop a processor to save power? Why do you need/want this notification? What are you going to do with this information that you don't already have? thanks, greg k-h ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-20 6:48 ` Greg KH @ 2014-10-20 7:40 ` Neil Zhang 2014-10-20 17:02 ` Dan Streetman 0 siblings, 1 reply; 22+ messages in thread From: Neil Zhang @ 2014-10-20 7:40 UTC (permalink / raw) To: Greg KH; +Cc: linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="gb2312", Size: 1404 bytes --] Greg, -----Original Message----- From: Greg KH [mailto:gregkh@linuxfoundation.org] Sent: 2014Äê10ÔÂ20ÈÕ 14:48 To: Neil Zhang Cc: linux-kernel@vger.kernel.org Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: >> How much noise is this going to cause on a big/little system that >> constantly hot unplug/plugs processors all of the time? > > Can you explain more what kind of noise will be introduced on a big/little system? Have you tested this on such a machine? I didn't have such kind of machine on hand. Can anyone has such machine to verify it? Thanks! > As I know IKS on arm will use cpu_suspend way to power down a core. Are you sure that it also doesn't use that same functionality to drop a processor to save power? As I know it use cpu_suspend to switch out a processor in IKS and there is no cpu hotplug notifier in this procedure. Why do you need/want this notification? What are you going to do with this information that you don't already have? The offline won't be updated if an in kernel hotplug governor plug in / out a core which cause the sysfs interface report a wrong status. thanks, greg k-h Best Regards, Neil Zhang ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-20 7:40 ` Neil Zhang @ 2014-10-20 17:02 ` Dan Streetman 2014-10-21 0:47 ` Neil Zhang ` (2 more replies) 0 siblings, 3 replies; 22+ messages in thread From: Dan Streetman @ 2014-10-20 17:02 UTC (permalink / raw) To: Neil Zhang; +Cc: Greg KH, linux-kernel On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: > Greg, > > > -----Original Message----- > From: Greg KH [mailto:gregkh@linuxfoundation.org] > Sent: 2014年10月20日 14:48 > To: Neil Zhang > Cc: linux-kernel@vger.kernel.org > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug > > On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: >>> How much noise is this going to cause on a big/little system that >>> constantly hot unplug/plugs processors all of the time? >> >> Can you explain more what kind of noise will be introduced on a big/little system? > > Have you tested this on such a machine? > > I didn't have such kind of machine on hand. > Can anyone has such machine to verify it? > Thanks! I tested this on a ppc PowerVM system, using dlpar operations to remove/add cpus. Without this patch the cpu online nodes get out of sync with the main online node (and the actual state of the cpus), because they aren't updated as the cpus are brought up/down: [root@br10p02 cpu]$ pwd /sys/devices/system/cpu [root@br10p02 cpu]$ cat online 0-39 [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; done ; echo "" 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 While with the patch, the cpu online nodes are kept up to date as the cpus are brought up/down: [root@br10p02 cpu]$ pwd /sys/devices/system/cpu [root@br10p02 cpu]$ cat online 0-39 [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; done ; echo "" 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 Feel free to add Tested-by: Dan Streetman <ddstreet@ieee.org> > >> As I know IKS on arm will use cpu_suspend way to power down a core. > > Are you sure that it also doesn't use that same functionality to drop a processor to save power? > > As I know it use cpu_suspend to switch out a processor in IKS and there is no cpu hotplug notifier in this procedure. > > > Why do you need/want this notification? What are you going to do with this information that you don't already have? > > The offline won't be updated if an in kernel hotplug governor plug in / out a core which cause the sysfs interface report a wrong status. > > > thanks, > > greg k-h > > > Best Regards, > Neil Zhang ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-20 17:02 ` Dan Streetman @ 2014-10-21 0:47 ` Neil Zhang 2014-10-21 2:57 ` Yasuaki Ishimatsu 2014-10-27 1:43 ` Neil Zhang 2 siblings, 0 replies; 22+ messages in thread From: Neil Zhang @ 2014-10-21 0:47 UTC (permalink / raw) To: Dan Streetman; +Cc: Greg KH, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="utf-8", Size: 3119 bytes --] Dan, > -----Original Message----- > From: ddstreet@gmail.com [mailto:ddstreet@gmail.com] On Behalf Of Dan > Streetman > Sent: 2014å¹´10月21æ—¥ 1:03 > To: Neil Zhang > Cc: Greg KH; linux-kernel@vger.kernel.org > Subject: Re: [PATCH] drivers: base: update cpu offline info when do > hotplug > > On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: > > Greg, > > > > > > -----Original Message----- > > From: Greg KH [mailto:gregkh@linuxfoundation.org] > > Sent: 2014å¹´10月20æ—¥ 14:48 > > To: Neil Zhang > > Cc: linux-kernel@vger.kernel.org > > Subject: Re: [PATCH] drivers: base: update cpu offline info when do > > hotplug > > > > On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: > >>> How much noise is this going to cause on a big/little system that > >>> constantly hot unplug/plugs processors all of the time? > >> > >> Can you explain more what kind of noise will be introduced on a > big/little system? > > > > Have you tested this on such a machine? > > > > I didn't have such kind of machine on hand. > > Can anyone has such machine to verify it? > > Thanks! > > I tested this on a ppc PowerVM system, using dlpar operations to > remove/add cpus. > > Without this patch the cpu online nodes get out of sync with the main > online node (and the actual state of the cpus), because they aren't > updated as the cpus are brought up/down: > > [root@br10p02 cpu]$ pwd > /sys/devices/system/cpu > [root@br10p02 cpu]$ cat online > 0-39 > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) - > eq 1 && echo -n "$n " ; done ; echo "" > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > > > While with the patch, the cpu online nodes are kept up to date as the > cpus are brought up/down: > > [root@br10p02 cpu]$ pwd > /sys/devices/system/cpu > [root@br10p02 cpu]$ cat online > 0-39 > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) - > eq 1 && echo -n "$n " ; done ; echo "" > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > 27 28 29 30 31 32 33 34 35 36 37 38 39 > > > Feel free to add Thanks for your kindly test! > > Tested-by: Dan Streetman <ddstreet@ieee.org> > > > > >> As I know IKS on arm will use cpu_suspend way to power down a core. > > > > Are you sure that it also doesn't use that same functionality to drop > a processor to save power? > > > > As I know it use cpu_suspend to switch out a processor in IKS and > there is no cpu hotplug notifier in this procedure. > > > > > > Why do you need/want this notification? What are you going to do > with this information that you don't already have? > > > > The offline won't be updated if an in kernel hotplug governor plug in > / out a core which cause the sysfs interface report a wrong status. > > > > > > thanks, > > > > greg k-h > > > > > > Best Regards, > > Neil Zhang Best Regards, Neil Zhang ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-20 17:02 ` Dan Streetman 2014-10-21 0:47 ` Neil Zhang @ 2014-10-21 2:57 ` Yasuaki Ishimatsu 2014-10-21 3:18 ` Neil Zhang 2014-10-21 12:58 ` Dan Streetman 2014-10-27 1:43 ` Neil Zhang 2 siblings, 2 replies; 22+ messages in thread From: Yasuaki Ishimatsu @ 2014-10-21 2:57 UTC (permalink / raw) To: Dan Streetman, Neil Zhang; +Cc: Greg KH, linux-kernel Hi Neil and Dan, (2014/10/21 2:02), Dan Streetman wrote: > On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: >> Greg, >> >> >> -----Original Message----- >> From: Greg KH [mailto:gregkh@linuxfoundation.org] >> Sent: 2014年10月20日 14:48 >> To: Neil Zhang >> Cc: linux-kernel@vger.kernel.org >> Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug >> >> On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: >>>> How much noise is this going to cause on a big/little system that >>>> constantly hot unplug/plugs processors all of the time? >>> >>> Can you explain more what kind of noise will be introduced on a big/little system? >> >> Have you tested this on such a machine? >> >> I didn't have such kind of machine on hand. >> Can anyone has such machine to verify it? >> Thanks! > > I tested this on a ppc PowerVM system, using dlpar operations to > remove/add cpus. > > Without this patch the cpu online nodes get out of sync with the main > online node (and the actual state of the cpus), because they aren't > updated as the cpus are brought up/down: > > [root@br10p02 cpu]$ pwd > /sys/devices/system/cpu > [root@br10p02 cpu]$ cat online > 0-39 > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > -eq 1 && echo -n "$n " ; done ; echo "" > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 How is the issue reproduced. Here is a result on my x86 box with linux-3.18-rc1. - before offline CPU # cd /sys/devices/system/cpu/ # cat online 0-59 # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; done ; echo "" 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 - after offline CPU{1..59} # for n in {1..59} ; do echo 0 > cpu$n/online; done # cat online 0 # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; done ; echo "" 0 It seems that dev->offline is set to correct valute. Thanks, Yasuaki Ishimatsu > > > While with the patch, the cpu online nodes are kept up to date as the > cpus are brought up/down: > > [root@br10p02 cpu]$ pwd > /sys/devices/system/cpu > [root@br10p02 cpu]$ cat online > 0-39 > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > -eq 1 && echo -n "$n " ; done ; echo "" > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > 27 28 29 30 31 32 33 34 35 36 37 38 39 > > > Feel free to add > > Tested-by: Dan Streetman <ddstreet@ieee.org> > >> >>> As I know IKS on arm will use cpu_suspend way to power down a core. >> >> Are you sure that it also doesn't use that same functionality to drop a processor to save power? >> >> As I know it use cpu_suspend to switch out a processor in IKS and there is no cpu hotplug notifier in this procedure. >> >> >> Why do you need/want this notification? What are you going to do with this information that you don't already have? >> >> The offline won't be updated if an in kernel hotplug governor plug in / out a core which cause the sysfs interface report a wrong status. >> >> >> thanks, >> >> greg k-h >> >> >> Best Regards, >> Neil Zhang > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-21 2:57 ` Yasuaki Ishimatsu @ 2014-10-21 3:18 ` Neil Zhang 2014-10-21 3:26 ` Yasuaki Ishimatsu 2014-10-21 12:58 ` Dan Streetman 1 sibling, 1 reply; 22+ messages in thread From: Neil Zhang @ 2014-10-21 3:18 UTC (permalink / raw) To: Yasuaki Ishimatsu, Dan Streetman; +Cc: Greg KH, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="utf-8", Size: 4352 bytes --] Yasuaki, > -----Original Message----- > From: Yasuaki Ishimatsu [mailto:isimatu.yasuaki@jp.fujitsu.com] > Sent: 2014å¹´10月21æ—¥ 10:57 > To: Dan Streetman; Neil Zhang > Cc: Greg KH; linux-kernel@vger.kernel.org > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug > > Hi Neil and Dan, > > (2014/10/21 2:02), Dan Streetman wrote: > > On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: > >> Greg, > >> > >> > >> -----Original Message----- > >> From: Greg KH [mailto:gregkh@linuxfoundation.org] > >> Sent: 2014å¹´10月20æ—¥ 14:48 > >> To: Neil Zhang > >> Cc: linux-kernel@vger.kernel.org > >> Subject: Re: [PATCH] drivers: base: update cpu offline info when do > >> hotplug > >> > >> On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: > >>>> How much noise is this going to cause on a big/little system that > >>>> constantly hot unplug/plugs processors all of the time? > >>> > >>> Can you explain more what kind of noise will be introduced on a big/little > system? > >> > >> Have you tested this on such a machine? > >> > >> I didn't have such kind of machine on hand. > >> Can anyone has such machine to verify it? > >> Thanks! > > > > I tested this on a ppc PowerVM system, using dlpar operations to > > remove/add cpus. > > > > Without this patch the cpu online nodes get out of sync with the main > > online node (and the actual state of the cpus), because they aren't > > updated as the cpus are brought up/down: > > > > > [root@br10p02 cpu]$ pwd > > /sys/devices/system/cpu > > [root@br10p02 cpu]$ cat online > > 0-39 > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > > -eq 1 && echo -n "$n " ; done ; echo "" > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > > How is the issue reproduced. > > Here is a result on my x86 box with linux-3.18-rc1. > > - before offline CPU > # cd /sys/devices/system/cpu/ > # cat online > 0-59 > # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; > done ; echo "" > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 > 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 > 55 > 56 57 58 59 > > - after offline CPU{1..59} > # for n in {1..59} ; do echo 0 > cpu$n/online; done # cat online > 0 > # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; > done ; echo "" > 0 > > It seems that dev->offline is set to correct valute. > Please use an in kernel governor to up / down a core instead of sysfs interface. > Thanks, > Yasuaki Ishimatsu > > > > > > > While with the patch, the cpu online nodes are kept up to date as the > > cpus are brought up/down: > > > > [root@br10p02 cpu]$ pwd > > /sys/devices/system/cpu > > [root@br10p02 cpu]$ cat online > > 0-39 > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > > -eq 1 && echo -n "$n " ; done ; echo "" > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > > 27 28 29 30 31 32 33 34 35 36 37 38 39 > > > > > > Feel free to add > > > > Tested-by: Dan Streetman <ddstreet@ieee.org> > > > >> > >>> As I know IKS on arm will use cpu_suspend way to power down a core. > >> > >> Are you sure that it also doesn't use that same functionality to drop a > processor to save power? > >> > >> As I know it use cpu_suspend to switch out a processor in IKS and there is > no cpu hotplug notifier in this procedure. > >> > >> > >> Why do you need/want this notification? What are you going to do with this > information that you don't already have? > >> > >> The offline won't be updated if an in kernel hotplug governor plug in / out > a core which cause the sysfs interface report a wrong status. > >> > >> > >> thanks, > >> > >> greg k-h > >> > >> > >> Best Regards, > >> Neil Zhang > > -- > > To unsubscribe from this list: send the line "unsubscribe > > linux-kernel" in the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > > > Best Regards, Neil Zhang ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-21 3:18 ` Neil Zhang @ 2014-10-21 3:26 ` Yasuaki Ishimatsu 2014-10-21 3:36 ` Neil Zhang 0 siblings, 1 reply; 22+ messages in thread From: Yasuaki Ishimatsu @ 2014-10-21 3:26 UTC (permalink / raw) To: Neil Zhang, Dan Streetman; +Cc: Greg KH, linux-kernel (2014/10/21 12:18), Neil Zhang wrote: > Yasuaki, > >> -----Original Message----- >> From: Yasuaki Ishimatsu [mailto:isimatu.yasuaki@jp.fujitsu.com] >> Sent: 2014年10月21日 10:57 >> To: Dan Streetman; Neil Zhang >> Cc: Greg KH; linux-kernel@vger.kernel.org >> Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug >> >> Hi Neil and Dan, >> >> (2014/10/21 2:02), Dan Streetman wrote: >>> On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: >>>> Greg, >>>> >>>> >>>> -----Original Message----- >>>> From: Greg KH [mailto:gregkh@linuxfoundation.org] >>>> Sent: 2014年10月20日 14:48 >>>> To: Neil Zhang >>>> Cc: linux-kernel@vger.kernel.org >>>> Subject: Re: [PATCH] drivers: base: update cpu offline info when do >>>> hotplug >>>> >>>> On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: >>>>>> How much noise is this going to cause on a big/little system that >>>>>> constantly hot unplug/plugs processors all of the time? >>>>> >>>>> Can you explain more what kind of noise will be introduced on a big/little >> system? >>>> >>>> Have you tested this on such a machine? >>>> >>>> I didn't have such kind of machine on hand. >>>> Can anyone has such machine to verify it? >>>> Thanks! >>> >>> I tested this on a ppc PowerVM system, using dlpar operations to >>> remove/add cpus. >>> >>> Without this patch the cpu online nodes get out of sync with the main >>> online node (and the actual state of the cpus), because they aren't >>> updated as the cpus are brought up/down: >>> >> >>> [root@br10p02 cpu]$ pwd >>> /sys/devices/system/cpu >>> [root@br10p02 cpu]$ cat online >>> 0-39 >>> [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) >>> -eq 1 && echo -n "$n " ; done ; echo "" >>> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 >>> 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 >> >> How is the issue reproduced. >> >> Here is a result on my x86 box with linux-3.18-rc1. >> >> - before offline CPU >> # cd /sys/devices/system/cpu/ >> # cat online >> 0-59 >> # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; >> done ; echo "" >> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 >> 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 >> 55 >> 56 57 58 59 >> >> - after offline CPU{1..59} >> # for n in {1..59} ; do echo 0 > cpu$n/online; done # cat online >> 0 >> # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; >> done ; echo "" >> 0 >> >> It seems that dev->offline is set to correct valute. >> > > Please use an in kernel governor to up / down a core instead of sysfs interface. Thank you for the information. But I don't know the in kernel governor? Could you point a documentation of it? Thanks, Yasuaki Ishimatsu > >> Thanks, >> Yasuaki Ishimatsu >> >>> >>> >>> While with the patch, the cpu online nodes are kept up to date as the >>> cpus are brought up/down: >>> >>> [root@br10p02 cpu]$ pwd >>> /sys/devices/system/cpu >>> [root@br10p02 cpu]$ cat online >>> 0-39 >>> [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) >>> -eq 1 && echo -n "$n " ; done ; echo "" >>> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 >>> 27 28 29 30 31 32 33 34 35 36 37 38 39 >>> >>> >>> Feel free to add >>> >>> Tested-by: Dan Streetman <ddstreet@ieee.org> >>> >>>> >>>>> As I know IKS on arm will use cpu_suspend way to power down a core. >>>> >>>> Are you sure that it also doesn't use that same functionality to drop a >> processor to save power? >>>> >>>> As I know it use cpu_suspend to switch out a processor in IKS and there is >> no cpu hotplug notifier in this procedure. >>>> >>>> >>>> Why do you need/want this notification? What are you going to do with this >> information that you don't already have? >>>> >>>> The offline won't be updated if an in kernel hotplug governor plug in / out >> a core which cause the sysfs interface report a wrong status. >>>> >>>> >>>> thanks, >>>> >>>> greg k-h >>>> >>>> >>>> Best Regards, >>>> Neil Zhang >>> -- >>> To unsubscribe from this list: send the line "unsubscribe >>> linux-kernel" in the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> Please read the FAQ at http://www.tux.org/lkml/ >>> >> > > > Best Regards, > Neil Zhang > \x04�{.n�+�������+%��lzwm��b�맲��r��zX��\x19\x1e�w��{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+�����ݢj"��!�iO��z��v�^\x14\x04\x1a�^[m����\vnƊ��Y&� > ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-21 3:26 ` Yasuaki Ishimatsu @ 2014-10-21 3:36 ` Neil Zhang 2014-10-21 4:47 ` Yasuaki Ishimatsu 0 siblings, 1 reply; 22+ messages in thread From: Neil Zhang @ 2014-10-21 3:36 UTC (permalink / raw) To: Yasuaki Ishimatsu, Dan Streetman; +Cc: Greg KH, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="utf-8", Size: 5498 bytes --] > -----Original Message----- > From: Yasuaki Ishimatsu [mailto:isimatu.yasuaki@jp.fujitsu.com] > Sent: 2014å¹´10月21æ—¥ 11:27 > To: Neil Zhang; Dan Streetman > Cc: Greg KH; linux-kernel@vger.kernel.org > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug > > (2014/10/21 12:18), Neil Zhang wrote: > > Yasuaki, > > > >> -----Original Message----- > >> From: Yasuaki Ishimatsu [mailto:isimatu.yasuaki@jp.fujitsu.com] > >> Sent: 2014å¹´10月21æ—¥ 10:57 > >> To: Dan Streetman; Neil Zhang > >> Cc: Greg KH; linux-kernel@vger.kernel.org > >> Subject: Re: [PATCH] drivers: base: update cpu offline info when do > >> hotplug > >> > >> Hi Neil and Dan, > >> > >> (2014/10/21 2:02), Dan Streetman wrote: > >>> On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: > >>>> Greg, > >>>> > >>>> > >>>> -----Original Message----- > >>>> From: Greg KH [mailto:gregkh@linuxfoundation.org] > >>>> Sent: 2014å¹´10月20æ—¥ 14:48 > >>>> To: Neil Zhang > >>>> Cc: linux-kernel@vger.kernel.org > >>>> Subject: Re: [PATCH] drivers: base: update cpu offline info when do > >>>> hotplug > >>>> > >>>> On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: > >>>>>> How much noise is this going to cause on a big/little system that > >>>>>> constantly hot unplug/plugs processors all of the time? > >>>>> > >>>>> Can you explain more what kind of noise will be introduced on a > >>>>> big/little > >> system? > >>>> > >>>> Have you tested this on such a machine? > >>>> > >>>> I didn't have such kind of machine on hand. > >>>> Can anyone has such machine to verify it? > >>>> Thanks! > >>> > >>> I tested this on a ppc PowerVM system, using dlpar operations to > >>> remove/add cpus. > >>> > >>> Without this patch the cpu online nodes get out of sync with the > >>> main online node (and the actual state of the cpus), because they > >>> aren't updated as the cpus are brought up/down: > >>> > >> > >>> [root@br10p02 cpu]$ pwd > >>> /sys/devices/system/cpu > >>> [root@br10p02 cpu]$ cat online > >>> 0-39 > >>> [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > >>> -eq 1 && echo -n "$n " ; done ; echo "" > >>> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 > >>> 26 > >>> 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > >> > >> How is the issue reproduced. > >> > >> Here is a result on my x86 box with linux-3.18-rc1. > >> > >> - before offline CPU > >> # cd /sys/devices/system/cpu/ > >> # cat online > >> 0-59 > >> # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n > >> "$n " ; done ; echo "" > >> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 > >> 26 27 28 > >> 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 > >> 52 53 54 > >> 55 > >> 56 57 58 59 > >> > >> - after offline CPU{1..59} > >> # for n in {1..59} ; do echo 0 > cpu$n/online; done # cat online > >> 0 > >> # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n > >> "$n " ; done ; echo "" > >> 0 > >> > >> It seems that dev->offline is set to correct valute. > >> > > > > > Please use an in kernel governor to up / down a core instead of sysfs > interface. > > Thank you for the information. But I don't know the in kernel governor? > Could you point a documentation of it? > Simply means that you call cpu_down / cpu_up directly in kernel base on the profiler. > Thanks, > Yasuaki Ishimatsu > > > > >> Thanks, > >> Yasuaki Ishimatsu > >> > >>> > >>> > >>> While with the patch, the cpu online nodes are kept up to date as > >>> the cpus are brought up/down: > >>> > >>> [root@br10p02 cpu]$ pwd > >>> /sys/devices/system/cpu > >>> [root@br10p02 cpu]$ cat online > >>> 0-39 > >>> [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > >>> -eq 1 && echo -n "$n " ; done ; echo "" > >>> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 > >>> 26 > >>> 27 28 29 30 31 32 33 34 35 36 37 38 39 > >>> > >>> > >>> Feel free to add > >>> > >>> Tested-by: Dan Streetman <ddstreet@ieee.org> > >>> > >>>> > >>>>> As I know IKS on arm will use cpu_suspend way to power down a core. > >>>> > >>>> Are you sure that it also doesn't use that same functionality to > >>>> drop a > >> processor to save power? > >>>> > >>>> As I know it use cpu_suspend to switch out a processor in IKS and > >>>> there is > >> no cpu hotplug notifier in this procedure. > >>>> > >>>> > >>>> Why do you need/want this notification? What are you going to do > >>>> with this > >> information that you don't already have? > >>>> > >>>> The offline won't be updated if an in kernel hotplug governor plug > >>>> in / out > >> a core which cause the sysfs interface report a wrong status. > >>>> > >>>> > >>>> thanks, > >>>> > >>>> greg k-h > >>>> > >>>> > >>>> Best Regards, > >>>> Neil Zhang > >>> -- > >>> To unsubscribe from this list: send the line "unsubscribe > >>> linux-kernel" in the body of a message to majordomo@vger.kernel.org > >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>> Please read the FAQ at http://www.tux.org/lkml/ > >>> > >> > > > > > > Best Regards, > > Neil Zhang > > \x04 {.n + +% lzwm b 맲 r zX \x19\x1e w {ay \x1dʇڙ ,j > > f h z \x1e w > > j:+v w j m zZ+ Ý¢j" ! iO z v ^\x14\x04\x1a ^[m > nÆŠ Y& > > > Best Regards, Neil Zhang ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-21 3:36 ` Neil Zhang @ 2014-10-21 4:47 ` Yasuaki Ishimatsu 0 siblings, 0 replies; 22+ messages in thread From: Yasuaki Ishimatsu @ 2014-10-21 4:47 UTC (permalink / raw) To: Neil Zhang, Dan Streetman; +Cc: Greg KH, linux-kernel (2014/10/21 12:36), Neil Zhang wrote: > > >> -----Original Message----- >> From: Yasuaki Ishimatsu [mailto:isimatu.yasuaki@jp.fujitsu.com] >> Sent: 2014年10月21日 11:27 >> To: Neil Zhang; Dan Streetman >> Cc: Greg KH; linux-kernel@vger.kernel.org >> Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug >> >> (2014/10/21 12:18), Neil Zhang wrote: >>> Yasuaki, >>> >>>> -----Original Message----- >>>> From: Yasuaki Ishimatsu [mailto:isimatu.yasuaki@jp.fujitsu.com] >>>> Sent: 2014年10月21日 10:57 >>>> To: Dan Streetman; Neil Zhang >>>> Cc: Greg KH; linux-kernel@vger.kernel.org >>>> Subject: Re: [PATCH] drivers: base: update cpu offline info when do >>>> hotplug >>>> >>>> Hi Neil and Dan, >>>> >>>> (2014/10/21 2:02), Dan Streetman wrote: >>>>> On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: >>>>>> Greg, >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Greg KH [mailto:gregkh@linuxfoundation.org] >>>>>> Sent: 2014年10月20日 14:48 >>>>>> To: Neil Zhang >>>>>> Cc: linux-kernel@vger.kernel.org >>>>>> Subject: Re: [PATCH] drivers: base: update cpu offline info when do >>>>>> hotplug >>>>>> >>>>>> On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: >>>>>>>> How much noise is this going to cause on a big/little system that >>>>>>>> constantly hot unplug/plugs processors all of the time? >>>>>>> >>>>>>> Can you explain more what kind of noise will be introduced on a >>>>>>> big/little >>>> system? >>>>>> >>>>>> Have you tested this on such a machine? >>>>>> >>>>>> I didn't have such kind of machine on hand. >>>>>> Can anyone has such machine to verify it? >>>>>> Thanks! >>>>> >>>>> I tested this on a ppc PowerVM system, using dlpar operations to >>>>> remove/add cpus. >>>>> >>>>> Without this patch the cpu online nodes get out of sync with the >>>>> main online node (and the actual state of the cpus), because they >>>>> aren't updated as the cpus are brought up/down: >>>>> >>>> >>>>> [root@br10p02 cpu]$ pwd >>>>> /sys/devices/system/cpu >>>>> [root@br10p02 cpu]$ cat online >>>>> 0-39 >>>>> [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) >>>>> -eq 1 && echo -n "$n " ; done ; echo "" >>>>> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 >>>>> 26 >>>>> 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 >>>> >>>> How is the issue reproduced. >>>> >>>> Here is a result on my x86 box with linux-3.18-rc1. >>>> >>>> - before offline CPU >>>> # cd /sys/devices/system/cpu/ >>>> # cat online >>>> 0-59 >>>> # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n >>>> "$n " ; done ; echo "" >>>> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 >>>> 26 27 28 >>>> 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 >>>> 52 53 54 >>>> 55 >>>> 56 57 58 59 >>>> >>>> - after offline CPU{1..59} >>>> # for n in {1..59} ; do echo 0 > cpu$n/online; done # cat online >>>> 0 >>>> # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n >>>> "$n " ; done ; echo "" >>>> 0 >>>> >>>> It seems that dev->offline is set to correct valute. >>>> >>> >> >>> Please use an in kernel governor to up / down a core instead of sysfs >> interface. >> >> Thank you for the information. But I don't know the in kernel governor? >> Could you point a documentation of it? >> > > Simply means that you call cpu_down / cpu_up directly in kernel base on the profiler. I understood it. When using sysfs, device_online/offline() changes dev->offline. Therefore, this problem did not occur in my previous test. Thanks, Yasuaki Ishimatsu > >> Thanks, >> Yasuaki Ishimatsu >> >>> >>>> Thanks, >>>> Yasuaki Ishimatsu >>>> >>>>> >>>>> >>>>> While with the patch, the cpu online nodes are kept up to date as >>>>> the cpus are brought up/down: >>>>> >>>>> [root@br10p02 cpu]$ pwd >>>>> /sys/devices/system/cpu >>>>> [root@br10p02 cpu]$ cat online >>>>> 0-39 >>>>> [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) >>>>> -eq 1 && echo -n "$n " ; done ; echo "" >>>>> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 >>>>> 26 >>>>> 27 28 29 30 31 32 33 34 35 36 37 38 39 >>>>> >>>>> >>>>> Feel free to add >>>>> >>>>> Tested-by: Dan Streetman <ddstreet@ieee.org> >>>>> >>>>>> >>>>>>> As I know IKS on arm will use cpu_suspend way to power down a core. >>>>>> >>>>>> Are you sure that it also doesn't use that same functionality to >>>>>> drop a >>>> processor to save power? >>>>>> >>>>>> As I know it use cpu_suspend to switch out a processor in IKS and >>>>>> there is >>>> no cpu hotplug notifier in this procedure. >>>>>> >>>>>> >>>>>> Why do you need/want this notification? What are you going to do >>>>>> with this >>>> information that you don't already have? >>>>>> >>>>>> The offline won't be updated if an in kernel hotplug governor plug >>>>>> in / out >>>> a core which cause the sysfs interface report a wrong status. >>>>>> >>>>>> >>>>>> thanks, >>>>>> >>>>>> greg k-h >>>>>> >>>>>> >>>>>> Best Regards, >>>>>> Neil Zhang >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> Please read the FAQ at http://www.tux.org/lkml/ >>>>> >>>> >>> >>> >>> Best Regards, >>> Neil Zhang >>> \x04 {.n + +% lzwm b 맲 r zX \x19\x1e w {ay \x1dʇڙ ,j >>> f h z \x1e w >> >> j:+v w j m zZ+ ݢj" ! iO z v ^\x14\x04\x1a ^[m >> nƊ Y& >>> >> > > > Best Regards, > Neil Zhang > \x13��칻\x1c�&�~�&�\x18��+-��ݶ\x17��w��˛���m�b��dz�ޖ)���w*\x1fjg���\x1e�����ݢj/���z�ޖ��2�ޙ���&�)ߡ�a��\x7f��\x1e�G���h�\x0f�j:+v���w�٥>W����i�axP\x10j�m����\f-�\x1a+��d�_ > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-21 2:57 ` Yasuaki Ishimatsu 2014-10-21 3:18 ` Neil Zhang @ 2014-10-21 12:58 ` Dan Streetman 1 sibling, 0 replies; 22+ messages in thread From: Dan Streetman @ 2014-10-21 12:58 UTC (permalink / raw) To: Yasuaki Ishimatsu; +Cc: Neil Zhang, Greg KH, linux-kernel On Mon, Oct 20, 2014 at 10:57 PM, Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com> wrote: > Hi Neil and Dan, > > > (2014/10/21 2:02), Dan Streetman wrote: >> >> On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: >>> >>> Greg, >>> >>> >>> -----Original Message----- >>> From: Greg KH [mailto:gregkh@linuxfoundation.org] >>> Sent: 2014年10月20日 14:48 >>> To: Neil Zhang >>> Cc: linux-kernel@vger.kernel.org >>> Subject: Re: [PATCH] drivers: base: update cpu offline info when do >>> hotplug >>> >>> On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: >>>>> >>>>> How much noise is this going to cause on a big/little system that >>>>> constantly hot unplug/plugs processors all of the time? >>>> >>>> >>>> Can you explain more what kind of noise will be introduced on a >>>> big/little system? >>> >>> >>> Have you tested this on such a machine? >>> >>> I didn't have such kind of machine on hand. >>> Can anyone has such machine to verify it? >>> Thanks! >> >> >> I tested this on a ppc PowerVM system, using dlpar operations to >> remove/add cpus. >> >> Without this patch the cpu online nodes get out of sync with the main >> online node (and the actual state of the cpus), because they aren't >> updated as the cpus are brought up/down: >> > >> [root@br10p02 cpu]$ pwd >> /sys/devices/system/cpu >> [root@br10p02 cpu]$ cat online >> 0-39 >> [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) >> -eq 1 && echo -n "$n " ; done ; echo "" >> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 >> 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > > > How is the issue reproduced. > > Here is a result on my x86 box with linux-3.18-rc1. This is not an issue when using the sysfs interface to bring cpus up or down. This is only an issue when using any other method to bring cpus up or down. I used dlpar operations, which invoke the arch/powerpc/platforms/pseries/dlpar.c code, which brings cpus up and down. The dlpar operations are invoked by a hypervisor on a "LPAR" (virtualized guest) system to tell it some of the (virtualized) cpus are being added/removed. > > - before offline CPU > # cd /sys/devices/system/cpu/ > # cat online > 0-59 > # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; > done ; echo "" > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 > 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 > 54 55 > 56 57 58 59 > > - after offline CPU{1..59} > # for n in {1..59} ; do echo 0 > cpu$n/online; done > # cat online > 0 > # for n in {0..59} ; do test $( cat cpu$n/online ) -eq 1 && echo -n "$n " ; > done ; echo "" > 0 > > It seems that dev->offline is set to correct valute. > > Thanks, > Yasuaki Ishimatsu > >> >> >> While with the patch, the cpu online nodes are kept up to date as the >> cpus are brought up/down: >> >> [root@br10p02 cpu]$ pwd >> /sys/devices/system/cpu >> [root@br10p02 cpu]$ cat online >> 0-39 >> [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) >> -eq 1 && echo -n "$n " ; done ; echo "" >> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 >> 27 28 29 30 31 32 33 34 35 36 37 38 39 >> >> >> Feel free to add >> >> Tested-by: Dan Streetman <ddstreet@ieee.org> >> >>> >>>> As I know IKS on arm will use cpu_suspend way to power down a core. >>> >>> >>> Are you sure that it also doesn't use that same functionality to drop a >>> processor to save power? >>> >>> As I know it use cpu_suspend to switch out a processor in IKS and there >>> is no cpu hotplug notifier in this procedure. >>> >>> >>> Why do you need/want this notification? What are you going to do with >>> this information that you don't already have? >>> >>> The offline won't be updated if an in kernel hotplug governor plug in / >>> out a core which cause the sysfs interface report a wrong status. >>> >>> >>> thanks, >>> >>> greg k-h >>> >>> >>> Best Regards, >>> Neil Zhang >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ >> > > ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-20 17:02 ` Dan Streetman 2014-10-21 0:47 ` Neil Zhang 2014-10-21 2:57 ` Yasuaki Ishimatsu @ 2014-10-27 1:43 ` Neil Zhang 2014-10-27 1:58 ` Greg KH 2 siblings, 1 reply; 22+ messages in thread From: Neil Zhang @ 2014-10-27 1:43 UTC (permalink / raw) To: Dan Streetman; +Cc: Greg KH, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="utf-8", Size: 3157 bytes --] Greg, > -----Original Message----- > From: ddstreet@gmail.com [mailto:ddstreet@gmail.com] On Behalf Of Dan > Streetman > Sent: 2014å¹´10月21æ—¥ 1:03 > To: Neil Zhang > Cc: Greg KH; linux-kernel@vger.kernel.org > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug > > On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: > > Greg, > > > > > > -----Original Message----- > > From: Greg KH [mailto:gregkh@linuxfoundation.org] > > Sent: 2014å¹´10月20æ—¥ 14:48 > > To: Neil Zhang > > Cc: linux-kernel@vger.kernel.org > > Subject: Re: [PATCH] drivers: base: update cpu offline info when do > > hotplug > > > > On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: > >>> How much noise is this going to cause on a big/little system that > >>> constantly hot unplug/plugs processors all of the time? > >> > >> Can you explain more what kind of noise will be introduced on a big/little > system? > > > > Have you tested this on such a machine? > > > > I didn't have such kind of machine on hand. > > Can anyone has such machine to verify it? > > Thanks! > > I tested this on a ppc PowerVM system, using dlpar operations to remove/add > cpus. > > Without this patch the cpu online nodes get out of sync with the main online > node (and the actual state of the cpus), because they aren't updated as the > cpus are brought up/down: > > [root@br10p02 cpu]$ pwd > /sys/devices/system/cpu > [root@br10p02 cpu]$ cat online > 0-39 > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) -eq 1 && > echo -n "$n " ; done ; echo "" > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > > > While with the patch, the cpu online nodes are kept up to date as the cpus are > brought up/down: > > [root@br10p02 cpu]$ pwd > /sys/devices/system/cpu > [root@br10p02 cpu]$ cat online > 0-39 > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) -eq 1 && > echo -n "$n " ; done ; echo "" > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > 27 28 29 30 31 32 33 34 35 36 37 38 39 > > > Feel free to add > > Tested-by: Dan Streetman <ddstreet@ieee.org> > It's a real bug in the kernel. What's your comments about this patch? > > > >> As I know IKS on arm will use cpu_suspend way to power down a core. > > > > Are you sure that it also doesn't use that same functionality to drop a > processor to save power? > > > > As I know it use cpu_suspend to switch out a processor in IKS and there is > no cpu hotplug notifier in this procedure. > > > > > > Why do you need/want this notification? What are you going to do with this > information that you don't already have? > > > > The offline won't be updated if an in kernel hotplug governor plug in / out > a core which cause the sysfs interface report a wrong status. > > > > > > thanks, > > > > greg k-h > > > > > > Best Regards, > > Neil Zhang Best Regards, Neil Zhang ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-27 1:43 ` Neil Zhang @ 2014-10-27 1:58 ` Greg KH 2014-10-27 2:17 ` Neil Zhang 0 siblings, 1 reply; 22+ messages in thread From: Greg KH @ 2014-10-27 1:58 UTC (permalink / raw) To: Neil Zhang; +Cc: Dan Streetman, linux-kernel On Sun, Oct 26, 2014 at 06:43:11PM -0700, Neil Zhang wrote: > Greg, > > > > -----Original Message----- > > From: ddstreet@gmail.com [mailto:ddstreet@gmail.com] On Behalf Of Dan > > Streetman > > Sent: 2014年10月21日 1:03 > > To: Neil Zhang > > Cc: Greg KH; linux-kernel@vger.kernel.org > > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug > > > > On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: > > > Greg, > > > > > > > > > -----Original Message----- > > > From: Greg KH [mailto:gregkh@linuxfoundation.org] > > > Sent: 2014年10月20日 14:48 > > > To: Neil Zhang > > > Cc: linux-kernel@vger.kernel.org > > > Subject: Re: [PATCH] drivers: base: update cpu offline info when do > > > hotplug > > > > > > On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: > > >>> How much noise is this going to cause on a big/little system that > > >>> constantly hot unplug/plugs processors all of the time? > > >> > > >> Can you explain more what kind of noise will be introduced on a big/little > > system? > > > > > > Have you tested this on such a machine? > > > > > > I didn't have such kind of machine on hand. > > > Can anyone has such machine to verify it? > > > Thanks! > > > > I tested this on a ppc PowerVM system, using dlpar operations to remove/add > > cpus. > > > > Without this patch the cpu online nodes get out of sync with the main online > > node (and the actual state of the cpus), because they aren't updated as the > > cpus are brought up/down: > > > > [root@br10p02 cpu]$ pwd > > /sys/devices/system/cpu > > [root@br10p02 cpu]$ cat online > > 0-39 > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) -eq 1 && > > echo -n "$n " ; done ; echo "" > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > > > > > > While with the patch, the cpu online nodes are kept up to date as the cpus are > > brought up/down: > > > > [root@br10p02 cpu]$ pwd > > /sys/devices/system/cpu > > [root@br10p02 cpu]$ cat online > > 0-39 > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) -eq 1 && > > echo -n "$n " ; done ; echo "" > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 > > 27 28 29 30 31 32 33 34 35 36 37 38 39 > > > > > > Feel free to add > > > > Tested-by: Dan Streetman <ddstreet@ieee.org> > > > > It's a real bug in the kernel. As this has been this way for many years, I tend to think it's not all that important... > What's your comments about this patch? It's one of 1800+ patches that need my comments, please be patient. greg k-h ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-27 1:58 ` Greg KH @ 2014-10-27 2:17 ` Neil Zhang 2014-10-27 2:26 ` Greg KH 0 siblings, 1 reply; 22+ messages in thread From: Neil Zhang @ 2014-10-27 2:17 UTC (permalink / raw) To: Greg KH; +Cc: Dan Streetman, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="utf-8", Size: 3616 bytes --] > -----Original Message----- > From: Greg KH [mailto:gregkh@linuxfoundation.org] > Sent: 2014å¹´10月27æ—¥ 9:59 > To: Neil Zhang > Cc: Dan Streetman; linux-kernel@vger.kernel.org > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug > > On Sun, Oct 26, 2014 at 06:43:11PM -0700, Neil Zhang wrote: > > Greg, > > > > > > > -----Original Message----- > > > From: ddstreet@gmail.com [mailto:ddstreet@gmail.com] On Behalf Of > > > Dan Streetman > > > Sent: 2014å¹´10月21æ—¥ 1:03 > > > To: Neil Zhang > > > Cc: Greg KH; linux-kernel@vger.kernel.org > > > Subject: Re: [PATCH] drivers: base: update cpu offline info when do > > > hotplug > > > > > > On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: > > > > Greg, > > > > > > > > > > > > -----Original Message----- > > > > From: Greg KH [mailto:gregkh@linuxfoundation.org] > > > > Sent: 2014å¹´10月20æ—¥ 14:48 > > > > To: Neil Zhang > > > > Cc: linux-kernel@vger.kernel.org > > > > Subject: Re: [PATCH] drivers: base: update cpu offline info when > > > > do hotplug > > > > > > > > On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: > > > >>> How much noise is this going to cause on a big/little system > > > >>> that constantly hot unplug/plugs processors all of the time? > > > >> > > > >> Can you explain more what kind of noise will be introduced on a > > > >> big/little > > > system? > > > > > > > > Have you tested this on such a machine? > > > > > > > > I didn't have such kind of machine on hand. > > > > Can anyone has such machine to verify it? > > > > Thanks! > > > > > > I tested this on a ppc PowerVM system, using dlpar operations to > > > remove/add cpus. > > > > > > Without this patch the cpu online nodes get out of sync with the > > > main online node (and the actual state of the cpus), because they > > > aren't updated as the cpus are brought up/down: > > > > > > [root@br10p02 cpu]$ pwd > > > /sys/devices/system/cpu > > > [root@br10p02 cpu]$ cat online > > > 0-39 > > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > > > -eq 1 && echo -n "$n " ; done ; echo "" > > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 > > > 26 > > > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > > > > > > > > > While with the patch, the cpu online nodes are kept up to date as > > > the cpus are brought up/down: > > > > > > [root@br10p02 cpu]$ pwd > > > /sys/devices/system/cpu > > > [root@br10p02 cpu]$ cat online > > > 0-39 > > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > > > -eq 1 && echo -n "$n " ; done ; echo "" > > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 > > > 26 > > > 27 28 29 30 31 32 33 34 35 36 37 38 39 > > > > > > > > > Feel free to add > > > > > > Tested-by: Dan Streetman <ddstreet@ieee.org> > > > > > > > It's a real bug in the kernel. > > As this has been this way for many years, I tend to think it's not all that > important... Actually this bug was introduced by the following patch. commit 0902a9044fa5b7a0456ea4daacec2c2b3189ba8c Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com> Date: Fri May 3 00:25:49 2013 +0200 Driver core: Use generic offline/online for CPU offline/online So seems not that long :) > > > What's your comments about this patch? > > It's one of 1800+ patches that need my comments, please be patient. Got it, thanks! > > greg k-h Best Regards, Neil Zhang ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-27 2:17 ` Neil Zhang @ 2014-10-27 2:26 ` Greg KH 2014-10-27 2:30 ` Neil Zhang 2014-10-27 16:28 ` Dan Streetman 0 siblings, 2 replies; 22+ messages in thread From: Greg KH @ 2014-10-27 2:26 UTC (permalink / raw) To: Neil Zhang; +Cc: Dan Streetman, linux-kernel On Sun, Oct 26, 2014 at 07:17:14PM -0700, Neil Zhang wrote: > > > > -----Original Message----- > > From: Greg KH [mailto:gregkh@linuxfoundation.org] > > Sent: 2014年10月27日 9:59 > > To: Neil Zhang > > Cc: Dan Streetman; linux-kernel@vger.kernel.org > > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug > > > > On Sun, Oct 26, 2014 at 06:43:11PM -0700, Neil Zhang wrote: > > > Greg, > > > > > > > > > > -----Original Message----- > > > > From: ddstreet@gmail.com [mailto:ddstreet@gmail.com] On Behalf Of > > > > Dan Streetman > > > > Sent: 2014年10月21日 1:03 > > > > To: Neil Zhang > > > > Cc: Greg KH; linux-kernel@vger.kernel.org > > > > Subject: Re: [PATCH] drivers: base: update cpu offline info when do > > > > hotplug > > > > > > > > On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: > > > > > Greg, > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Greg KH [mailto:gregkh@linuxfoundation.org] > > > > > Sent: 2014年10月20日 14:48 > > > > > To: Neil Zhang > > > > > Cc: linux-kernel@vger.kernel.org > > > > > Subject: Re: [PATCH] drivers: base: update cpu offline info when > > > > > do hotplug > > > > > > > > > > On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: > > > > >>> How much noise is this going to cause on a big/little system > > > > >>> that constantly hot unplug/plugs processors all of the time? > > > > >> > > > > >> Can you explain more what kind of noise will be introduced on a > > > > >> big/little > > > > system? > > > > > > > > > > Have you tested this on such a machine? > > > > > > > > > > I didn't have such kind of machine on hand. > > > > > Can anyone has such machine to verify it? > > > > > Thanks! > > > > > > > > I tested this on a ppc PowerVM system, using dlpar operations to > > > > remove/add cpus. > > > > > > > > Without this patch the cpu online nodes get out of sync with the > > > > main online node (and the actual state of the cpus), because they > > > > aren't updated as the cpus are brought up/down: > > > > > > > > [root@br10p02 cpu]$ pwd > > > > /sys/devices/system/cpu > > > > [root@br10p02 cpu]$ cat online > > > > 0-39 > > > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > > > > -eq 1 && echo -n "$n " ; done ; echo "" > > > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 > > > > 26 > > > > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > > > > > > > > > > > > While with the patch, the cpu online nodes are kept up to date as > > > > the cpus are brought up/down: > > > > > > > > [root@br10p02 cpu]$ pwd > > > > /sys/devices/system/cpu > > > > [root@br10p02 cpu]$ cat online > > > > 0-39 > > > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) > > > > -eq 1 && echo -n "$n " ; done ; echo "" > > > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 > > > > 26 > > > > 27 28 29 30 31 32 33 34 35 36 37 38 39 > > > > > > > > > > > > Feel free to add > > > > > > > > Tested-by: Dan Streetman <ddstreet@ieee.org> > > > > > > > > > > It's a real bug in the kernel. > > > > As this has been this way for many years, I tend to think it's not all that > > important... > > Actually this bug was introduced by the following patch. > > commit 0902a9044fa5b7a0456ea4daacec2c2b3189ba8c > Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > Date: Fri May 3 00:25:49 2013 +0200 > > Driver core: Use generic offline/online for CPU offline/online > > So seems not that long :) Ok, over a year. Any reason why this information wasn't in this patch? Also, why not cc: the authors of that patch as well? Surely they would want to know about this, right? greg k-h ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-27 2:26 ` Greg KH @ 2014-10-27 2:30 ` Neil Zhang 2014-10-27 16:28 ` Dan Streetman 1 sibling, 0 replies; 22+ messages in thread From: Neil Zhang @ 2014-10-27 2:30 UTC (permalink / raw) To: Greg KH; +Cc: Dan Streetman, linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="utf-8", Size: 4486 bytes --] > -----Original Message----- > From: Greg KH [mailto:gregkh@linuxfoundation.org] > Sent: 2014å¹´10月27æ—¥ 10:27 > To: Neil Zhang > Cc: Dan Streetman; linux-kernel@vger.kernel.org > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug > > On Sun, Oct 26, 2014 at 07:17:14PM -0700, Neil Zhang wrote: > > > > > > > -----Original Message----- > > > From: Greg KH [mailto:gregkh@linuxfoundation.org] > > > Sent: 2014å¹´10月27æ—¥ 9:59 > > > To: Neil Zhang > > > Cc: Dan Streetman; linux-kernel@vger.kernel.org > > > Subject: Re: [PATCH] drivers: base: update cpu offline info when do > > > hotplug > > > > > > On Sun, Oct 26, 2014 at 06:43:11PM -0700, Neil Zhang wrote: > > > > Greg, > > > > > > > > > > > > > -----Original Message----- > > > > > From: ddstreet@gmail.com [mailto:ddstreet@gmail.com] On Behalf > > > > > Of Dan Streetman > > > > > Sent: 2014å¹´10月21æ—¥ 1:03 > > > > > To: Neil Zhang > > > > > Cc: Greg KH; linux-kernel@vger.kernel.org > > > > > Subject: Re: [PATCH] drivers: base: update cpu offline info when > > > > > do hotplug > > > > > > > > > > On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> > wrote: > > > > > > Greg, > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Greg KH [mailto:gregkh@linuxfoundation.org] > > > > > > Sent: 2014å¹´10月20æ—¥ 14:48 > > > > > > To: Neil Zhang > > > > > > Cc: linux-kernel@vger.kernel.org > > > > > > Subject: Re: [PATCH] drivers: base: update cpu offline info > > > > > > when do hotplug > > > > > > > > > > > > On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: > > > > > >>> How much noise is this going to cause on a big/little system > > > > > >>> that constantly hot unplug/plugs processors all of the time? > > > > > >> > > > > > >> Can you explain more what kind of noise will be introduced on > > > > > >> a big/little > > > > > system? > > > > > > > > > > > > Have you tested this on such a machine? > > > > > > > > > > > > I didn't have such kind of machine on hand. > > > > > > Can anyone has such machine to verify it? > > > > > > Thanks! > > > > > > > > > > I tested this on a ppc PowerVM system, using dlpar operations to > > > > > remove/add cpus. > > > > > > > > > > Without this patch the cpu online nodes get out of sync with the > > > > > main online node (and the actual state of the cpus), because > > > > > they aren't updated as the cpus are brought up/down: > > > > > > > > > > [root@br10p02 cpu]$ pwd > > > > > /sys/devices/system/cpu > > > > > [root@br10p02 cpu]$ cat online > > > > > 0-39 > > > > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat > > > > > cpu$n/online ) -eq 1 && echo -n "$n " ; done ; echo "" > > > > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 > > > > > 25 > > > > > 26 > > > > > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > > > > > > > > > > > > > > > While with the patch, the cpu online nodes are kept up to date > > > > > as the cpus are brought up/down: > > > > > > > > > > [root@br10p02 cpu]$ pwd > > > > > /sys/devices/system/cpu > > > > > [root@br10p02 cpu]$ cat online > > > > > 0-39 > > > > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat > > > > > cpu$n/online ) -eq 1 && echo -n "$n " ; done ; echo "" > > > > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 > > > > > 25 > > > > > 26 > > > > > 27 28 29 30 31 32 33 34 35 36 37 38 39 > > > > > > > > > > > > > > > Feel free to add > > > > > > > > > > Tested-by: Dan Streetman <ddstreet@ieee.org> > > > > > > > > > > > > > It's a real bug in the kernel. > > > > > > As this has been this way for many years, I tend to think it's not > > > all that important... > > > > Actually this bug was introduced by the following patch. > > > > commit 0902a9044fa5b7a0456ea4daacec2c2b3189ba8c > > Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > > Date: Fri May 3 00:25:49 2013 +0200 > > > > Driver core: Use generic offline/online for CPU offline/online > > > > So seems not that long :) > > Ok, over a year. > > Any reason why this information wasn't in this patch? Also, why not cc: > the authors of that patch as well? Surely they would want to know about this, > right? > > greg k-h Thanks for your remind. I will submit another patch to include these infos. Best Regards, Neil Zhang ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-27 2:26 ` Greg KH 2014-10-27 2:30 ` Neil Zhang @ 2014-10-27 16:28 ` Dan Streetman 1 sibling, 0 replies; 22+ messages in thread From: Dan Streetman @ 2014-10-27 16:28 UTC (permalink / raw) To: Greg KH; +Cc: Neil Zhang, linux-kernel, nfont On Sun, Oct 26, 2014 at 10:26 PM, Greg KH <gregkh@linuxfoundation.org> wrote: > On Sun, Oct 26, 2014 at 07:17:14PM -0700, Neil Zhang wrote: >> >> >> > -----Original Message----- >> > From: Greg KH [mailto:gregkh@linuxfoundation.org] >> > Sent: 2014年10月27日 9:59 >> > To: Neil Zhang >> > Cc: Dan Streetman; linux-kernel@vger.kernel.org >> > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug >> > >> > On Sun, Oct 26, 2014 at 06:43:11PM -0700, Neil Zhang wrote: >> > > Greg, >> > > >> > > >> > > > -----Original Message----- >> > > > From: ddstreet@gmail.com [mailto:ddstreet@gmail.com] On Behalf Of >> > > > Dan Streetman >> > > > Sent: 2014年10月21日 1:03 >> > > > To: Neil Zhang >> > > > Cc: Greg KH; linux-kernel@vger.kernel.org >> > > > Subject: Re: [PATCH] drivers: base: update cpu offline info when do >> > > > hotplug >> > > > >> > > > On Mon, Oct 20, 2014 at 3:40 AM, Neil Zhang <zhangwm@marvell.com> wrote: >> > > > > Greg, >> > > > > >> > > > > >> > > > > -----Original Message----- >> > > > > From: Greg KH [mailto:gregkh@linuxfoundation.org] >> > > > > Sent: 2014年10月20日 14:48 >> > > > > To: Neil Zhang >> > > > > Cc: linux-kernel@vger.kernel.org >> > > > > Subject: Re: [PATCH] drivers: base: update cpu offline info when >> > > > > do hotplug >> > > > > >> > > > > On Sun, Oct 19, 2014 at 11:39:23PM -0700, Neil Zhang wrote: >> > > > >>> How much noise is this going to cause on a big/little system >> > > > >>> that constantly hot unplug/plugs processors all of the time? >> > > > >> >> > > > >> Can you explain more what kind of noise will be introduced on a >> > > > >> big/little >> > > > system? >> > > > > >> > > > > Have you tested this on such a machine? >> > > > > >> > > > > I didn't have such kind of machine on hand. >> > > > > Can anyone has such machine to verify it? >> > > > > Thanks! >> > > > >> > > > I tested this on a ppc PowerVM system, using dlpar operations to >> > > > remove/add cpus. >> > > > >> > > > Without this patch the cpu online nodes get out of sync with the >> > > > main online node (and the actual state of the cpus), because they >> > > > aren't updated as the cpus are brought up/down: >> > > > >> > > > [root@br10p02 cpu]$ pwd >> > > > /sys/devices/system/cpu >> > > > [root@br10p02 cpu]$ cat online >> > > > 0-39 >> > > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) >> > > > -eq 1 && echo -n "$n " ; done ; echo "" >> > > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 >> > > > 26 >> > > > 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 >> > > > >> > > > >> > > > While with the patch, the cpu online nodes are kept up to date as >> > > > the cpus are brought up/down: >> > > > >> > > > [root@br10p02 cpu]$ pwd >> > > > /sys/devices/system/cpu >> > > > [root@br10p02 cpu]$ cat online >> > > > 0-39 >> > > > [root@br10p02 cpu]$ for n in {0..47} ; do test $( cat cpu$n/online ) >> > > > -eq 1 && echo -n "$n " ; done ; echo "" >> > > > 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 >> > > > 26 >> > > > 27 28 29 30 31 32 33 34 35 36 37 38 39 >> > > > >> > > > >> > > > Feel free to add >> > > > >> > > > Tested-by: Dan Streetman <ddstreet@ieee.org> >> > > > >> > > >> > > It's a real bug in the kernel. >> > >> > As this has been this way for many years, I tend to think it's not all that >> > important... >> >> Actually this bug was introduced by the following patch. >> >> commit 0902a9044fa5b7a0456ea4daacec2c2b3189ba8c >> Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >> Date: Fri May 3 00:25:49 2013 +0200 >> >> Driver core: Use generic offline/online for CPU offline/online >> >> So seems not that long :) > > Ok, over a year. > > Any reason why this information wasn't in this patch? Also, why not cc: > the authors of that patch as well? Surely they would want to know about > this, right? To add a bit more info to this, while the PPC (on PowerVM) method for cpu hotplug, in arch/powerpc/platform/pseries/dlpar.c dlpar_offline_cpu(), does require this patch because it only takes a cpu offline during hot remove, the x86/acpi code appears to be different as it fully unregisters a cpu during hot remove, in drivers/acpi/acpi_processor.c acpi_processor_remove() - so I believe the entire cpuN directory would be removed. I don't have a hw-hotpluggable x86 system though, and it doesn't look like qemu really supports cpu hotremove yet, so I can't test that. I don't know how other archs handle cpu hotplug. Also, the ppc pseries dlpar code may be changed in the future to unregister the cpu instead of only setting it offline; I've cc'ed Nathan who probably would be doing that. But regardless, after commit 0902a90, when a cpu is being taken up or down by anything other than generic offline/online code, the cpu's ->offline state does need to be updated. If not by a hotplug listener like this, then possibly by kernel/cpu.c set_cpu_online()... > > greg k-h ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-20 3:29 [PATCH] drivers: base: update cpu offline info when do hotplug Neil Zhang 2014-10-20 4:43 ` Greg KH @ 2014-10-21 13:02 ` Dan Streetman 2014-10-22 1:32 ` Neil Zhang 2014-10-23 14:05 ` Dan Streetman 1 sibling, 2 replies; 22+ messages in thread From: Dan Streetman @ 2014-10-21 13:02 UTC (permalink / raw) To: Neil Zhang; +Cc: linux-kernel, Greg Kroah-Hartman On Sun, Oct 19, 2014 at 11:29 PM, Neil Zhang <zhangwm@marvell.com> wrote: > The current per-cpu offline info won't be updated if it is > hotplugged in/out by a kernel governer. > Let's update it via cpu notifier. > > Signed-off-by: Neil Zhang <zhangwm@marvell.com> > --- > drivers/base/cpu.c | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > > diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c > index 006b1bc..9d61824 100644 > --- a/drivers/base/cpu.c > +++ b/drivers/base/cpu.c > @@ -418,10 +418,35 @@ static void __init cpu_dev_register_generic(void) > #endif > } > > +static int device_hotplug_notifier(struct notifier_block *nfb, > + unsigned long action, void *hcpu) > +{ > + unsigned int cpu = (unsigned long)hcpu; > + struct device *dev = get_cpu_device(cpu); > + int ret; > + > + switch (action & ~CPU_TASKS_FROZEN) { > + case CPU_ONLINE: > + dev->offline = false; > + ret = NOTIFY_OK; > + break; > + case CPU_DYING: > + dev->offline = true; one thing i just thought of here, since dev->offline is getting set here, can you remove it being set in drivers/base/core.c at device_online() and device_offline() ? That's probably redundant now right? > + ret = NOTIFY_OK; > + break; > + default: > + ret = NOTIFY_DONE; > + break; > + } > + > + return ret; > +} > + > void __init cpu_dev_init(void) > { > if (subsys_system_register(&cpu_subsys, cpu_root_attr_groups)) > panic("Failed to register CPU subsystem"); > > cpu_dev_register_generic(); > + cpu_notifier(device_hotplug_notifier, 0); > } > -- > 1.7.9.5 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 22+ messages in thread
* RE: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-21 13:02 ` Dan Streetman @ 2014-10-22 1:32 ` Neil Zhang 2014-10-23 14:05 ` Dan Streetman 1 sibling, 0 replies; 22+ messages in thread From: Neil Zhang @ 2014-10-22 1:32 UTC (permalink / raw) To: Dan Streetman; +Cc: linux-kernel, Greg Kroah-Hartman [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2820 bytes --] Dan, > -----Original Message----- > From: ddstreet@gmail.com [mailto:ddstreet@gmail.com] On Behalf Of Dan > Streetman > Sent: 2014å¹´10月21æ—¥ 21:03 > To: Neil Zhang > Cc: linux-kernel; Greg Kroah-Hartman > Subject: Re: [PATCH] drivers: base: update cpu offline info when do hotplug > > On Sun, Oct 19, 2014 at 11:29 PM, Neil Zhang <zhangwm@marvell.com> wrote: > > The current per-cpu offline info won't be updated if it is hotplugged > > in/out by a kernel governer. > > Let's update it via cpu notifier. > > > > Signed-off-by: Neil Zhang <zhangwm@marvell.com> > > --- > > drivers/base/cpu.c | 25 +++++++++++++++++++++++++ > > 1 file changed, 25 insertions(+) > > > > diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c index > > 006b1bc..9d61824 100644 > > --- a/drivers/base/cpu.c > > +++ b/drivers/base/cpu.c > > @@ -418,10 +418,35 @@ static void __init > > cpu_dev_register_generic(void) #endif } > > > > +static int device_hotplug_notifier(struct notifier_block *nfb, > > + unsigned long action, void > > +*hcpu) { > > + unsigned int cpu = (unsigned long)hcpu; > > + struct device *dev = get_cpu_device(cpu); > > + int ret; > > + > > + switch (action & ~CPU_TASKS_FROZEN) { > > + case CPU_ONLINE: > > + dev->offline = false; > > + ret = NOTIFY_OK; > > + break; > > + case CPU_DYING: > > + dev->offline = true; > > one thing i just thought of here, since dev->offline is getting set here, can > you remove it being set in drivers/base/core.c at > device_online() and device_offline() ? That's probably redundant now right? Yes, I ever thought about it. But I'm not sure whether some other devices (such as memory) will use it later too. So I still kept it there since seems won't cause any problem. > > > > + ret = NOTIFY_OK; > > + break; > > + default: > > + ret = NOTIFY_DONE; > > + break; > > + } > > + > > + return ret; > > +} > > + > > void __init cpu_dev_init(void) > > { > > if (subsys_system_register(&cpu_subsys, cpu_root_attr_groups)) > > panic("Failed to register CPU subsystem"); > > > > cpu_dev_register_generic(); > > + cpu_notifier(device_hotplug_notifier, 0); > > } > > -- > > 1.7.9.5 > > > > -- > > To unsubscribe from this list: send the line "unsubscribe > > linux-kernel" in the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ Best Regards, Neil Zhang ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] drivers: base: update cpu offline info when do hotplug 2014-10-21 13:02 ` Dan Streetman 2014-10-22 1:32 ` Neil Zhang @ 2014-10-23 14:05 ` Dan Streetman 1 sibling, 0 replies; 22+ messages in thread From: Dan Streetman @ 2014-10-23 14:05 UTC (permalink / raw) To: Neil Zhang; +Cc: linux-kernel, Greg Kroah-Hartman On Tue, Oct 21, 2014 at 9:02 AM, Dan Streetman <ddstreet@ieee.org> wrote: > On Sun, Oct 19, 2014 at 11:29 PM, Neil Zhang <zhangwm@marvell.com> wrote: >> The current per-cpu offline info won't be updated if it is >> hotplugged in/out by a kernel governer. >> Let's update it via cpu notifier. >> >> Signed-off-by: Neil Zhang <zhangwm@marvell.com> >> --- >> drivers/base/cpu.c | 25 +++++++++++++++++++++++++ >> 1 file changed, 25 insertions(+) >> >> diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c >> index 006b1bc..9d61824 100644 >> --- a/drivers/base/cpu.c >> +++ b/drivers/base/cpu.c >> @@ -418,10 +418,35 @@ static void __init cpu_dev_register_generic(void) >> #endif >> } >> >> +static int device_hotplug_notifier(struct notifier_block *nfb, >> + unsigned long action, void *hcpu) >> +{ >> + unsigned int cpu = (unsigned long)hcpu; >> + struct device *dev = get_cpu_device(cpu); >> + int ret; >> + >> + switch (action & ~CPU_TASKS_FROZEN) { >> + case CPU_ONLINE: >> + dev->offline = false; >> + ret = NOTIFY_OK; >> + break; >> + case CPU_DYING: >> + dev->offline = true; > > one thing i just thought of here, since dev->offline is getting set > here, can you remove it being set in drivers/base/core.c at > device_online() and device_offline() ? That's probably redundant now > right? sorry ignore that; device_on|offline() is common code, not cpu-specific. > > >> + ret = NOTIFY_OK; >> + break; >> + default: >> + ret = NOTIFY_DONE; >> + break; >> + } >> + >> + return ret; >> +} >> + >> void __init cpu_dev_init(void) >> { >> if (subsys_system_register(&cpu_subsys, cpu_root_attr_groups)) >> panic("Failed to register CPU subsystem"); >> >> cpu_dev_register_generic(); >> + cpu_notifier(device_hotplug_notifier, 0); >> } >> -- >> 1.7.9.5 >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2014-10-27 16:29 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-10-20 3:29 [PATCH] drivers: base: update cpu offline info when do hotplug Neil Zhang 2014-10-20 4:43 ` Greg KH 2014-10-20 6:39 ` Neil Zhang 2014-10-20 6:48 ` Greg KH 2014-10-20 7:40 ` Neil Zhang 2014-10-20 17:02 ` Dan Streetman 2014-10-21 0:47 ` Neil Zhang 2014-10-21 2:57 ` Yasuaki Ishimatsu 2014-10-21 3:18 ` Neil Zhang 2014-10-21 3:26 ` Yasuaki Ishimatsu 2014-10-21 3:36 ` Neil Zhang 2014-10-21 4:47 ` Yasuaki Ishimatsu 2014-10-21 12:58 ` Dan Streetman 2014-10-27 1:43 ` Neil Zhang 2014-10-27 1:58 ` Greg KH 2014-10-27 2:17 ` Neil Zhang 2014-10-27 2:26 ` Greg KH 2014-10-27 2:30 ` Neil Zhang 2014-10-27 16:28 ` Dan Streetman 2014-10-21 13:02 ` Dan Streetman 2014-10-22 1:32 ` Neil Zhang 2014-10-23 14:05 ` Dan Streetman
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).