RE: high power consumption in recent kernels

RE: high power consumption in recent kernels

[Shi, Alex]

CC to lkml since maybe other guys also have interesting on this. 
Norbert: 
In the powertop source code, the "extra timer interrupt" come from /proc/timer_stats. If my understanding is correct, so many "extra timer interrupt" and the nohz_ratemlimit checking make your system can not go to deep C states. but don't know what the effect for "load balancing tick" come from "extra timer interrupt". So maybe reduce the "extra timer interrupt" is  helpful on your system. Could you like to try this? 



Regards! 
Alex  

>-----Original Message-----
>From: Peter Zijlstra [mailto:peterz@infradead.org]
>Sent: Friday, September 10, 2010 4:25 PM
>To: Norbert Preining
>Cc: Chen, Tim C; Shi, Alex; arjan@infradead.org; efault@gmx.de; Li, Shaohua; tglx
>Subject: Re: high power consumption in recent kernels
>
>On Fri, 2010-09-10 at 05:51 +0900, Norbert Preining wrote:
>> On Do, 09 Sep 2010, Chen, Tim C wrote:
>> > Have you tried to run powertop to see the causes of the wakeups
>>
>> Of course, in the very first email of the thread I posted the results,
>> here again for your convenience:
>> Top causes for wakeups:
>>   34.2% (185.3)   [kernel scheduler] Load balancing tick
>>   23.9% (129.6)   [extra timer interrupt]
>>   10.8% ( 58.6)   firefox-bin
>>    9.2% ( 49.7)   [iwlagn] <interrupt>
>>    7.2% ( 39.1)   [kernel core] hrtimer_start (tick_sched_timer)
>>    3.9% ( 20.9)   PS/2 keyboard/mouse/touchpad interrupt
>>
>> The problem is the "Load balancing tick" that is far far to high.
>
>Right, which makes perfect sense, delaying going into nohz state simply
>costs power.
>
>One thing we could do is hand over nohz control to the power governor
>(I'm sure Arjan will be thrilled), it could compare the cost of nohz
>enter/exit against the projected sleep time and make an informed
>decision.
>
>Because from what I understand, the nohz enter/exit cycle is what is
>costing you performance on this workload, its simply very expensive to
>prod at timer hardware (which is of course totally intel's own fault for
>giving us crummy timers to begin with).
>

Re: high power consumption in recent kernels

[Norbert Preining]

On Fr, 10 Sep 2010, Shi, Alex wrote:
> In the powertop source code, the "extra timer interrupt" come from /proc/timer_stats. If my understanding is correct, so many "extra timer interrupt" and the nohz_ratemlimit checking make your system can not go to deep C states. but don't know what the effect for "load balancing tick" come from "extra timer interrupt". So maybe reduce the "extra timer interrupt" is  helpful on your system. Could you like to try this? 

Can you be more specific how to do that?

- revert the nohz patch (can you send me a patch for that)
- how to reduce the "extra timer interrupt"?

Thanks

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining            preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan                                 TeX Live & Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
CAIRNPAT (n.)
A large piece of dried dung found in mountainous terrain above the
cowline which leads the experienced tracker to believe that hikers
have recently passed.
			--- Douglas Adams, The Meaning of Liff

Re: high power consumption in recent kernels

[Alex,Shi]

On Fri, 2010-09-10 at 22:54 +0800, Norbert Preining wrote:
> On Fr, 10 Sep 2010, Shi, Alex wrote:
> > In the powertop source code, the "extra timer interrupt" come from /proc/timer_stats. If my understanding is correct, so many "extra timer interrupt" and the nohz_ratemlimit checking make your system can not go to deep C states. but don't know what the effect for "load balancing tick" come from "extra timer interrupt". So maybe reduce the "extra timer interrupt" is  helpful on your system. Could you like to try this? 
> 
> Can you be more specific how to do that?
> 
> - revert the nohz patch (can you send me a patch for that)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1e2a6db..e66c52e 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -274,8 +274,14 @@ extern cpumask_var_t nohz_cpu_mask;
 #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
 extern void select_nohz_load_balancer(int stop_tick);
 extern int get_nohz_timer_target(void);
+extern int nohz_ratelimit(int cpu);
 #else
 static inline void select_nohz_load_balancer(int stop_tick) { }
+static inline int nohz_ratelimit(int cpu)
+{
+       return 0;
+}
+
 #endif
 
 /*
diff --git a/kernel/sched.c b/kernel/sched.c
index ed09d4f..25399e7 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -1182,6 +1182,17 @@ static void resched_task(struct task_struct *p)
 		smp_send_reschedule(cpu);
 }
 
+int nohz_ratelimit(int cpu)
+{
+       struct rq *rq = cpu_rq(cpu);
+       u64 diff = rq->clock - rq->nohz_stamp;
+
+       rq->nohz_stamp = rq->clock;
+
+       return diff < (NSEC_PER_SEC / HZ) >> 1;
+}
+
+
 static void resched_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 3e216e0..d6bca17 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu)) {
+	    arch_needs_cpu(cpu)|| nohz_ratelimit(cpu)) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {

> - how to reduce the "extra timer interrupt"?
The timer was listed in /proc/timer_stats, you can post its contents,
and let's see what is abnormal here. 

> 
> Thanks
> 
> Best wishes
> 
> Norbert
> ------------------------------------------------------------------------
> Norbert Preining            preining@{jaist.ac.jp, logic.at, debian.org}
> JAIST, Japan                                 TeX Live & Debian Developer
> DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
> ------------------------------------------------------------------------
> CAIRNPAT (n.)
> A large piece of dried dung found in mountainous terrain above the
> cowline which leads the experienced tracker to believe that hikers
> have recently passed.
> 			--- Douglas Adams, The Meaning of Liff

Re: high power consumption in recent kernels

[Alex,Shi]

> > - how to reduce the "extra timer interrupt"?
> The timer was listed in /proc/timer_stats, you can post its contents,
> and let's see what is abnormal here. 
> 
Norbert, did you try the patch? 
We are still thinking the too many "extra timer interrupts" on your
system, maybe the following message can give some help. So, could you
like to check if your system is using pit timer? 
#cat /proc/timer_list
Or some APP/system services in your system is using lots of timer? 

===
http://www.linuxpowertop.org/faq.php 

Even with a tickless kernel, there are some timer ticks still happening:

1.) For userspace events (also shown in the lower "top 10 events" list).
2.) The kernel still has timer ticks when userspace is executing code,
to sample and get data for the "top" program.
3.) Hardware timers have a maximum time they can be set for, for the PIT
timer (the one used if you don't have HPET active). This is around 27
milliseconds. If you would have longer sleep times than the maximum, the
kernel has no choice but to set the timer to the maximum repeatedly
until the actual event is approaching.

The "extra timer interrupts" data is a combination fo 1 and 3.

Re: high power consumption in recent kernels

[Norbert Preining]

Hi Alex, hi all,

sorry for the long silence, I was on business travel and mountaineering
for some weeks.

On Mo, 13 Sep 2010, Alex,Shi wrote:
> > - revert the nohz patch (can you send me a patch for that)

I am now running 2.6.36-rc5 with this patch:

> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 1e2a6db..e66c52e 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -274,8 +274,14 @@ extern cpumask_var_t nohz_cpu_mask;
>  #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
>  extern void select_nohz_load_balancer(int stop_tick);
>  extern int get_nohz_timer_target(void);
> +extern int nohz_ratelimit(int cpu);
>  #else
>  static inline void select_nohz_load_balancer(int stop_tick) { }
> +static inline int nohz_ratelimit(int cpu)
> +{
> +       return 0;
> +}
> +
>  #endif
>  
>  /*
> diff --git a/kernel/sched.c b/kernel/sched.c
> index ed09d4f..25399e7 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
> @@ -1182,6 +1182,17 @@ static void resched_task(struct task_struct *p)
>  		smp_send_reschedule(cpu);
>  }
>  
> +int nohz_ratelimit(int cpu)
> +{
> +       struct rq *rq = cpu_rq(cpu);
> +       u64 diff = rq->clock - rq->nohz_stamp;
> +
> +       rq->nohz_stamp = rq->clock;
> +
> +       return diff < (NSEC_PER_SEC / HZ) >> 1;
> +}
> +
> +
>  static void resched_cpu(int cpu)
>  {
>  	struct rq *rq = cpu_rq(cpu);
> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> index 3e216e0..d6bca17 100644
> --- a/kernel/time/tick-sched.c
> +++ b/kernel/time/tick-sched.c
> @@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
>  	} while (read_seqretry(&xtime_lock, seq));
>  
>  	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
> -	    arch_needs_cpu(cpu)) {
> +	    arch_needs_cpu(cpu)|| nohz_ratelimit(cpu)) {
>  		next_jiffies = last_jiffies + 1;
>  		delta_jiffies = 1;
>  	} else {


and it seems that still works nicely. I am now running on battery power,
connected to the internet over bluetooth to my mobile, and get:
Wakeups-from-idle per second : 452.2    interval: 10.0s
Power usage (ACPI estimate): 12.4W (3.6 hours) (long term: 8.0W,/5.6h)

Top causes for wakeups:
  16.1% ( 79.5)   [yenta, ehci_hcd:usb2, uhci_hcd:usb6, uhci_hcd:usb7, uhci_hcd:
  14.3% ( 70.7)   [kernel scheduler] Load balancing tick
  14.1% ( 69.8)   [extra timer interrupt]
  13.1% ( 64.9)   kworker/0:0
  10.7% ( 53.0)   USB device  8-2 : BCM2046 Bluetooth Device (Broadcom Corp)
   8.4% ( 41.8)   [kernel core] hrtimer_start (tick_sched_timer)
   3.8% ( 19.0)   gnome-terminal
   2.9% ( 14.6)   icedove-bin
   2.2% ( 10.7)   PS/2 keyboard/mouse/touchpad interrupt

Which shows that the "Load balancing tick" and "extra timer interrupt" are
down to halfway normal levels. I guess when I shutdown the usb/bluetooth
connection both of them will drop even further.

> > - how to reduce the "extra timer interrupt"?
> The timer was listed in /proc/timer_stats, you can post its contents,
> and let's see what is abnormal here. 

Here is the output of /proc/timer_stats:

Timer Stats Version: v0.2
Sample period: 7.828 s
   66,  4378 firefox-bin      hrtimer_start_range_ns (hrtimer_wakeup)
   15, 12527 aptitude         hrtimer_start_range_ns (hrtimer_wakeup)
    8,  3629 cpufreq-applet   hrtimer_start_range_ns (hrtimer_wakeup)
  612,     0 swapper          hrtimer_start_range_ns (tick_sched_timer)
 483D,  6082 kworker/0:0      do_dbs_timer (delayed_work_timer_fn)
   17, 12527 aptitude         queue_delayed_work (delayed_work_timer_fn)
  376,     0 kworker/0:0      hrtimer_start_range_ns (tick_sched_timer)
 237D,  6810 kworker/1:1      do_dbs_timer (delayed_work_timer_fn)
   51,  7843 sakura           hrtimer_start_range_ns (hrtimer_wakeup)
    8,  2665 Xorg             queue_delayed_work (delayed_work_timer_fn)
  387,     0 swapper          hrtimer_start (tick_sched_timer)
   39,  3510 syndaemon        hrtimer_start_range_ns (hrtimer_wakeup)
    8,  2705 gpsd             hrtimer_start_range_ns (hrtimer_wakeup)
   32,     0 swapper          __mod_timer (rh_timer_func)
    8,  7572 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8, 12916 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8, 12949 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8,  2080 apache2          hrtimer_start_range_ns (hrtimer_wakeup)
    8,  2795 ntpd             hrtimer_start_range_ns (posix_timer_fn)
   26,  3555 yarssr           hrtimer_start_range_ns (hrtimer_wakeup)
   91, 12790 sakura           hrtimer_start_range_ns (hrtimer_wakeup)
   15,  3583 multiload-apple  hrtimer_start_range_ns (hrtimer_wakeup)
    8,  4376 firefox-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8,  7579 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
   8D,  6810 kworker/1:1      queue_delayed_work (delayed_work_timer_fn)
   4D,  6810 kworker/1:1      queue_delayed_work (delayed_work_timer_fn)
   8D,  6082 kworker/0:0      queue_delayed_work (delayed_work_timer_fn)
   4D,  6082 kworker/0:0      queue_delayed_work (delayed_work_timer_fn)
    8, 12895 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    4,  3516 udisks-daemon    hrtimer_start_range_ns (hrtimer_wakeup)
    4,  2642 hald-addon-stor  hrtimer_start_range_ns (hrtimer_wakeup)
    8,  7540 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8,  7777 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    9,     0 kworker/0:0      hrtimer_start (tick_sched_timer)
    6,  2665 Xorg             __mod_timer (i915_hangcheck_elapsed)
    8,  7550 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    8,  3531 gvfs-afc-volume  hrtimer_start_range_ns (hrtimer_wakeup)
    7,  7551 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    7,  7549 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    7,  7578 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    1,     0 swapper          __mod_timer (uhci_fsbr_timeout)
    7,  7527 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    7, 12890 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    7,  7536 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
   18, 12790 sakura           hrtimer_start_range_ns (hrtimer_wakeup)
    4,  3582 sensors-applet   hrtimer_start_range_ns (hrtimer_wakeup)
    4,     0 swapper          __mod_timer (laptop_mode_timer_fn)
    4,     0 swapper          __mod_timer (dev_watchdog)
   11,  2868 tor              __mod_timer (process_timeout)
    5,  7528 icedove-bin      hrtimer_start_range_ns (hrtimer_wakeup)
    4,  4807 dropbox          hrtimer_start_range_ns (hrtimer_wakeup)
    1, 12537 http             __mod_timer (tcp_write_timer)
    5,  2825 nmbd             hrtimer_start_range_ns (hrtimer_wakeup)
    1,  3439 gconfd-2         hrtimer_start_range_ns (hrtimer_wakeup)
    4,  7403 kbnepd bnep0     __mod_timer (tcp_delack_timer)
    1,  3422 ssh-agent        hrtimer_start_range_ns (hrtimer_wakeup)
    1,  2195 hald             hrtimer_start_range_ns (hrtimer_wakeup)
    1,  2195 hald             __mod_timer (process_timeout)
    1,     7 watchdog/0       hrtimer_start (watchdog_timer_fn)
    1,  3444 gnome-power-man  hrtimer_start_range_ns (hrtimer_wakeup)
    3,   816 flush-8:0        __mod_timer (blk_unplug_timeout)
    1,     1 init             hrtimer_start_range_ns (hrtimer_wakeup)
    1,     1 swapper          enqueue_task_rt (sched_rt_period_timer)
   16, 12790 sakura           queue_delayed_work (delayed_work_timer_fn)
   14, 12791 bash             queue_delayed_work (delayed_work_timer_fn)
    5,  2665 Xorg             hrtimer_start_range_ns (hrtimer_wakeup)
    2,  3582 sensors-applet   __mod_timer (process_timeout)
    1,  3569 gnome-volume-ma  hrtimer_start_range_ns (hrtimer_wakeup)
    1,  2247 NetworkManager   hrtimer_start_range_ns (hrtimer_wakeup)
   1D,  6082 kworker/0:0      queue_delayed_work (delayed_work_timer_fn)
    1,  7403 kbnepd bnep0     __mod_timer (tcp_delack_timer)
    1,  2642 hald-addon-stor  __mod_timer (blk_rq_timed_out_timer)
    1,  4804 dropbox          __mod_timer (tcp_write_timer)
    1,  3511 gnome-panel      hrtimer_start_range_ns (hrtimer_wakeup)
2772 total events, 354.113 events/sec

Hope you can do something with it!

Let me know if I should do more testing.

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining            preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan                                 TeX Live & Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
BERKHAMSTED
The massive three-course midmorning blow-out enjoyed by a dieter who
has already done his or her slimming duty by having a teaspoonful of
cottage cheese for breakfast.
			--- Douglas Adams, The Meaning of Liff

Re: high power consumption in recent kernels

[Alex,Shi]

> and it seems that still works nicely. I am now running on battery power,
> connected to the internet over bluetooth to my mobile, and get:
> Wakeups-from-idle per second : 452.2    interval: 10.0s
> Power usage (ACPI estimate): 12.4W (3.6 hours) (long term: 8.0W,/5.6h)
> 
> Top causes for wakeups:
>   16.1% ( 79.5)   [yenta, ehci_hcd:usb2, uhci_hcd:usb6, uhci_hcd:usb7, uhci_hcd:
>   14.3% ( 70.7)   [kernel scheduler] Load balancing tick
>   14.1% ( 69.8)   [extra timer interrupt]
>   13.1% ( 64.9)   kworker/0:0
>   10.7% ( 53.0)   USB device  8-2 : BCM2046 Bluetooth Device (Broadcom Corp)
>    8.4% ( 41.8)   [kernel core] hrtimer_start (tick_sched_timer)
>    3.8% ( 19.0)   gnome-terminal
>    2.9% ( 14.6)   icedove-bin
>    2.2% ( 10.7)   PS/2 keyboard/mouse/touchpad interrupt
> 
> Which shows that the "Load balancing tick" and "extra timer interrupt" are
> down to halfway normal levels. I guess when I shutdown the usb/bluetooth
> connection both of them will drop even further.

> Hope you can do something with it!
> 
> Let me know if I should do more testing.
> 
Seems in .36 kernel the extra timer reduced. Guess the following patch
worked. http://lkml.org/lkml/2010/6/9/109

Tim,Yakui and I talked with the issue. We thought the nohz_ratelimit()
only hart the power when lots of interrupts with light load in system.
Since the tick_nohz_stop_sched_tick() only be checked in cpu_idle() and
in irq_exit(), we remove the nohz checking in irq_exit(). The TCP/UDP RR
loop back testing of netperf also benefit from this. But it have no
effect on real network testing. 

Could you like to try the following patch on your system and see how
many power it consumption now? 

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1e2a6db..a4dbb37 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -274,8 +274,13 @@ extern cpumask_var_t nohz_cpu_mask;
 #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
 extern void select_nohz_load_balancer(int stop_tick);
 extern int get_nohz_timer_target(void);
+extern int nohz_ratelimit(int cpu);
 #else
 static inline void select_nohz_load_balancer(int stop_tick) { }
+static inline int nohz_ratelimit(int cpu)
+{
+	return 0;
+}
 #endif
 
 /*
diff --git a/kernel/sched.c b/kernel/sched.c
index dc85ceb..132a21c 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -1182,6 +1182,16 @@ static void resched_task(struct task_struct *p)
 		smp_send_reschedule(cpu);
 }
 
+int nohz_ratelimit(int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+	u64 diff = rq->clock - rq->nohz_stamp;
+
+	rq->nohz_stamp = rq->clock;
+
+	return diff < (NSEC_PER_SEC / HZ) >> 1;
+}
+
 static void resched_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 3e216e0..0b5b186 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu)) {
+		arch_needs_cpu(cpu)|| (inidle && nohz_ratelimit(cpu))) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {

Re: high power consumption in recent kernels

[Norbert Preining]

Hi Alex,

> Could you like to try the following patch on your system and see how
> many power it consumption now?

But this patch is excately what you send me already and what I tried
and I answered in the last email?

Do I miss something here?

All the best

Norbert

RE: high power consumption in recent kernels

[Chen, Tim C]

Norbert,

Can you try with the no-hz ratelimit now code REMOVED, and see if 
that will reduce your system power and cpu wakeups further.
Your last testing has the no-hz ratelimit code and we will like
to compare the effect with and without the code.

Sorry for the confusion.

Tim

>-----Original Message-----
>From: Norbert Preining [mailto:preining@logic.at] 
>Sent: Wednesday, September 29, 2010 7:39 AM
>To: Shi, Alex
>Cc: Peter Zijlstra; Chen, Tim C; arjan@infradead.org; 
>efault@gmx.de; Li, Shaohua; tglx; 
>linux-kernel@vger.kernel.org; Zhao, Yakui
>Subject: Re: high power consumption in recent kernels
>
>Hi Alex,
>
>> Could you like to try the following patch on your system and see how
>> many power it consumption now?
>
>But this patch is excately what you send me already and what I tried
>and I answered in the last email?
>
>Do I miss something here?
>
>All the best
>
>Norbert
>

Re: high power consumption in recent kernels

[Alex,Shi]

On Wed, 2010-09-29 at 22:39 +0800, Norbert Preining wrote:
> Hi Alex,
> 
> > Could you like to try the following patch on your system and see how
> > many power it consumption now?
> 
> But this patch is excately what you send me already and what I tried
> and I answered in the last email?
> 
> Do I miss something here?
> 
> All the best
> 
> Norbert
The following line is different, the new code remove the nohz_ratelimit
from irq_exit(). That may resolve your system too many interrupts
issue. 

+               arch_needs_cpu(cpu)|| (inidle && nohz_ratelimit(cpu))) {

Re: high power consumption in recent kernels

[Norbert Preining]

On Do, 30 Sep 2010, Alex,Shi wrote:
> +               arch_needs_cpu(cpu)|| (inidle && nohz_ratelimit(cpu))) {

Sorry right, the 
	inidle
is new, missed that one. I will rebuild in a minute and test.

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining            preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan                                 TeX Live & Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
AITH (n.)
The single bristle that sticks out sideways on a cheap paintbrush.
			--- Douglas Adams, The Meaning of Liff

Re: high power consumption in recent kernels

[Norbert Preining]

> The following line is different, the new code remove the nohz_ratelimit
> from irq_exit(). That may resolve your system too many interrupts
> issue. 
> 
> +               arch_needs_cpu(cpu)|| (inidle && nohz_ratelimit(cpu))) {

That is a very nice one ..., first time I came down
with inactive laptop, lowest brightness, down to below 8W:
Cn                Avg residency       P-states (frequencies)
C0 (cpu running)        ( 1.1%)       Turbo Mode     0.0%
C0                0.0ms ( 0.0%)         2.54 Ghz     0.0%
C1 mwait          0.0ms ( 0.0%)         1.60 Ghz     0.0%
C2 mwait          0.3ms ( 0.8%)          800 Mhz    99.9%
C6 mwait         16.7ms (98.2%)

Wakeups-from-idle per second : 81.3     interval: 15.0s
Power usage (ACPI estimate): 7.9W (11.0 hours)

Top causes for wakeups:
  23.8% ( 18.1)   kworker/0:0
  14.0% ( 10.7)   [ahci] <interrupt>
  13.4% ( 10.2)   [kernel scheduler] Load balancing tick
   6.6% (  5.0)   syndaemon
   5.5% (  4.2)   Xorg
   5.5% (  4.2)   [extra timer interrupt]
   4.6% (  3.5)   [acpi] <interrupt>

Very nice, with activity and bluetooth and internet over bluetooth
I am back at 10.3W, but that is also fine.

Anything else you need to know (kernel was latest git, -rc6 AFAIR).

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining            preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan                                 TeX Live & Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
The main reception foyer was almost empty but Ford
nevertheless weaved his way through it.
                 --- Ford making his way out of Milliways whilst under the
                 --- influence of enough alchol to make a rhino sing.
                 --- Douglas Adams, The Hitchhikers Guide to the Galaxy

Re: high power consumption in recent kernels

[Alex,Shi]

Thanks Norbert! 
Mike&Peter:
would you like to add signed-off for the following patch?

---
    sched: nohz_ratelimit function refresh
    
    The nohz_ratelimit() function that written by Mike Galbraith
    can bring about more than 10% throughput for netperf TCP/UDP RR
    when scheduling cross-cpu. It did this by reducing down to nohz
    mode chance.
    But the patch also reduce CPU chance to nohz mode after
    interrupt processed, that cause Norbert's system have 4 watts power
    increase(the system have about 100 int/sec and with a light load).
    That is not acceptable for a laptop.
    
    So, I remove the nohz_ratelimit from irq_exit(). and then the
    Norbert's system back to low power consumption.
    
    Tested-by: Norbert Preining <preining@logic.at>
    Signed-off-by: Alex Shi <alex.shi@intel.com>

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1e2a6db..a4dbb37 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -274,8 +274,13 @@ extern cpumask_var_t nohz_cpu_mask;
 #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
 extern void select_nohz_load_balancer(int stop_tick);
 extern int get_nohz_timer_target(void);
+extern int nohz_ratelimit(int cpu);
 #else
 static inline void select_nohz_load_balancer(int stop_tick) { }
+static inline int nohz_ratelimit(int cpu)
+{
+	return 0;
+}
 #endif
 
 /*
diff --git a/kernel/sched.c b/kernel/sched.c
index dc85ceb..132a21c 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -1182,6 +1182,16 @@ static void resched_task(struct task_struct *p)
 		smp_send_reschedule(cpu);
 }
 
+int nohz_ratelimit(int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+	u64 diff = rq->clock - rq->nohz_stamp;
+
+	rq->nohz_stamp = rq->clock;
+
+	return diff < (NSEC_PER_SEC / HZ) >> 1;
+}
+
 static void resched_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 3e216e0..19a7914 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu)) {
+		arch_needs_cpu(cpu) || (inidle && nohz_ratelimit(cpu))) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {

Re: high power consumption in recent kernels

[Peter Zijlstra]

On Thu, 2010-09-30 at 16:27 +0800, Alex,Shi wrote:
> Thanks Norbert! 
> Mike&Peter:
> would you like to add signed-off for the following patch?

I would really rather see the nohz decision move into the whole cpuidle
governor thing, which can make a much better cost vs benefit decision.

Thomas, Arjan?

> ---
>     sched: nohz_ratelimit function refresh
>     
>     The nohz_ratelimit() function that written by Mike Galbraith
>     can bring about more than 10% throughput for netperf TCP/UDP RR
>     when scheduling cross-cpu. It did this by reducing down to nohz
>     mode chance.
>     But the patch also reduce CPU chance to nohz mode after
>     interrupt processed, that cause Norbert's system have 4 watts power
>     increase(the system have about 100 int/sec and with a light load).
>     That is not acceptable for a laptop.
>     
>     So, I remove the nohz_ratelimit from irq_exit(). and then the
>     Norbert's system back to low power consumption.
>     
>     Tested-by: Norbert Preining <preining@logic.at>
>     Signed-off-by: Alex Shi <alex.shi@intel.com>
> 
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 1e2a6db..a4dbb37 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -274,8 +274,13 @@ extern cpumask_var_t nohz_cpu_mask;
>  #if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ)
>  extern void select_nohz_load_balancer(int stop_tick);
>  extern int get_nohz_timer_target(void);
> +extern int nohz_ratelimit(int cpu);
>  #else
>  static inline void select_nohz_load_balancer(int stop_tick) { }
> +static inline int nohz_ratelimit(int cpu)
> +{
> +	return 0;
> +}
>  #endif
>  
>  /*
> diff --git a/kernel/sched.c b/kernel/sched.c
> index dc85ceb..132a21c 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
> @@ -1182,6 +1182,16 @@ static void resched_task(struct task_struct *p)
>  		smp_send_reschedule(cpu);
>  }
>  
> +int nohz_ratelimit(int cpu)
> +{
> +	struct rq *rq = cpu_rq(cpu);
> +	u64 diff = rq->clock - rq->nohz_stamp;
> +
> +	rq->nohz_stamp = rq->clock;
> +
> +	return diff < (NSEC_PER_SEC / HZ) >> 1;
> +}
> +
>  static void resched_cpu(int cpu)
>  {
>  	struct rq *rq = cpu_rq(cpu);
> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> index 3e216e0..19a7914 100644
> --- a/kernel/time/tick-sched.c
> +++ b/kernel/time/tick-sched.c
> @@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
>  	} while (read_seqretry(&xtime_lock, seq));
>  
>  	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
> -	    arch_needs_cpu(cpu)) {
> +		arch_needs_cpu(cpu) || (inidle && nohz_ratelimit(cpu))) {
>  		next_jiffies = last_jiffies + 1;
>  		delta_jiffies = 1;
>  	} else {
> 
> 

RE: high power consumption in recent kernels

[Shi, Alex]

>> Thanks Norbert!
>> Mike&Peter:
>> would you like to add signed-off for the following patch?
>
>I would really rather see the nohz decision move into the whole cpuidle
>governor thing, which can make a much better cost vs benefit decision.
>
See! 

Re: high power consumption in recent kernels

[Pavel Machek]

On Thu 2010-07-08 11:06:32, Peter Zijlstra wrote:
> On Wed, 2010-07-07 at 01:45 +0900, Norbert Preining wrote:
> 
> > it seems that some of the (?)recent(?) changes have increased the
> > power consumption of my note book considerably.
> > 
> > First of all, running powertop with normal programs started, but 
> > doing nothing, I am still at 14W while I could go down to 9W before
> > (but the 9W was with dimmed display).
> > 
> > In the list of top causes for wakeup I have
> > Top causes for wakeups:
> >   34.2% (185.3)   [kernel scheduler] Load balancing tick
> >   23.9% (129.6)   [extra timer interrupt]
> >   10.8% ( 58.6)   firefox-bin
> >    9.2% ( 49.7)   [iwlagn] <interrupt>
> >    7.2% ( 39.1)   [kernel core] hrtimer_start (tick_sched_timer)
> >    3.9% ( 20.9)   PS/2 keyboard/mouse/touchpad interrupt
> > which show one new thing to me I haven't seen before, the Loa balancing tick.
>

14W vs 9W is very significant -- is some DMA keeping system from
entering C3? rmmod usb?
								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: high power consumption in recent kernels

[Mike Galbraith]

On Thu, 2010-07-08 at 22:44 +0200, Peter Zijlstra wrote:
> On Thu, 2010-07-08 at 21:40 +0200, Mike Galbraith wrote:
> > > Mike could you re-run your netperf tests that showed the 10% throughput
> > > gain? Hopefully the fixed governor will yield the same result and we can
> > > kill off this ratelimit thing.
> > 
> > The gain is (well was last time I checked), but as noted, I'd just call
> > it a misguided optimization and be done with it. 
> 
> It would still be good to know what your machine does, if you can still
> see a difference there might still be something to look at.

git .today.

marge:..git/linux-2.6 # netperf.sh 30
Starting netserver at port 12865     
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 (127.0.0.1) port 0 AF_INET
Local /Remote                                                                                          
Socket Size   Request  Resp.   Elapsed  Trans.                                                         
Send   Recv   Size     Size    Time     Rate                                                           
bytes  Bytes  bytes    bytes   secs.    per sec                                                        

16384  87380  1        1       30.00    102272.73   
16384  87380                                        
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 (127.0.0.1) port 0 AF_INET : cpu bind
Local /Remote                                                                                                     
Socket Size   Request  Resp.   Elapsed  Trans.                                                                    
Send   Recv   Size     Size    Time     Rate                                                                      
bytes  Bytes  bytes    bytes   secs.    per sec                                                                   

16384  87380  1        1       30.00    97638.99   
16384  87380  

turns ratelimiting off

marge:..git/linux-2.6 # netperf.sh 30
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 (127.0.0.1) port 0 AF_INET
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384  87380  1        1       30.00    92991.02
16384  87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 (127.0.0.1) port 0 AF_INET : cpu bind
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384  87380  1        1       30.00    97665.27
16384  87380

btw, if you don't set governor to performance, you can get crud
throughput like below, because the ondemand governor doesn't necessarily
notice that the CPUs really are busy when waking cross-cpu.

marge:..git/linux-2.6 # netperf.sh 10
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 (127.0.0.1) port 0 AF_INET
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384  87380  1        1       10.00    73341.95 <== blech
16384  87380
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 (127.0.0.1) port 0 AF_INET : cpu bind
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384  87380  1        1       10.00    97695.45
16384  87380

Re: high power consumption in recent kernels

[Arjan van de Ven]

On Thu, 08 Jul 2010 22:44:14 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> On Thu, 2010-07-08 at 21:40 +0200, Mike Galbraith wrote:
> > > Mike could you re-run your netperf tests that showed the 10%
> > > throughput gain? Hopefully the fixed governor will yield the same
> > > result and we can kill off this ratelimit thing.
> > 
> > The gain is (well was last time I checked), but as noted, I'd just
> > call it a misguided optimization and be done with it. 
> 
> It would still be good to know what your machine does, if you can
> still see a difference there might still be something to look at.
> 

yeah and if you give me a reason / workload to improve the idle
governor even more.....


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: high power consumption in recent kernels

[Peter Zijlstra]

On Thu, 2010-07-08 at 21:40 +0200, Mike Galbraith wrote:
> > Mike could you re-run your netperf tests that showed the 10% throughput
> > gain? Hopefully the fixed governor will yield the same result and we can
> > kill off this ratelimit thing.
> 
> The gain is (well was last time I checked), but as noted, I'd just call
> it a misguided optimization and be done with it. 

It would still be good to know what your machine does, if you can still
see a difference there might still be something to look at.

Re: high power consumption in recent kernels

[Mike Galbraith]

On Thu, 2010-07-08 at 17:59 +0200, Peter Zijlstra wrote:
> On Thu, 2010-07-08 at 15:23 +0200, Peter Zijlstra wrote:
> > On Thu, 2010-07-08 at 21:46 +0900, Norbert Preining wrote:
> > > Looks promising, reverting the old patch, adding that one, building,
> > > running, unplugging ppower, powertop runs now since some time,
> > > it seems that we are back to better situation: 
> > 
> > Hrmm, Mike seems you wrecked power usage.. 
> > 
> > So nohz_ratelimit() prevents us from entering NOHZ when the last attempt
> > was less than 1/2 a jiffy ago (fwiw: NSEC_PER_SEC/HZ == TICK_NSEC).
> > 
> > Its either entering idle or irq_exit trying to enter nohz state, if we
> > keep skipping it it means that we get enough interrupt activity to
> > render nohz useless anyway.. so not quite sure how this wrecks things..
> 
> OK, so Arjan said the gain could come from tricking the idle governor
> into not using deeper C states. He also said he significantly cured said
> governor in .35.
> 
> Mike could you re-run your netperf tests that showed the 10% throughput
> gain? Hopefully the fixed governor will yield the same result and we can
> kill off this ratelimit thing.

The gain is (well was last time I checked), but as noted, I'd just call
it a misguided optimization and be done with it.

	-Mike

Re: high power consumption in recent kernels

[Mike Galbraith]

On Thu, 2010-07-08 at 15:23 +0200, Peter Zijlstra wrote:
> On Thu, 2010-07-08 at 21:46 +0900, Norbert Preining wrote:
> > Looks promising, reverting the old patch, adding that one, building,
> > running, unplugging ppower, powertop runs now since some time,
> > it seems that we are back to better situation: 
> 
> Hrmm, Mike seems you wrecked power usage.. 
> 
> So nohz_ratelimit() prevents us from entering NOHZ when the last attempt
> was less than 1/2 a jiffy ago (fwiw: NSEC_PER_SEC/HZ == TICK_NSEC).
> 
> Its either entering idle or irq_exit trying to enter nohz state, if we
> keep skipping it it means that we get enough interrupt activity to
> render nohz useless anyway.. so not quite sure how this wrecks things.

You can't win at this game.

I really don't like giving up anything, but thinking about it, if I were
the maintainer, I'd just revert the damn thing as being more trouble
than it's worth.

It makes a large difference at the extreme end of the spectrum when
scheduling cross cpu (which we now actively try to do to maximize ramp
throughput), but ever less as frequency diminishes.  I've yet to see a
load I can respect at close to max ctx frequency, where optimization
earns it's beans and biscuits.

Ego: If thine eye offends thee, pluck it out.

	-Mike

Re: high power consumption in recent kernels

[Peter Zijlstra]

On Thu, 2010-07-08 at 17:59 +0200, Peter Zijlstra wrote:

> OK, so Arjan said the gain could come from tricking the idle governor
> into not using deeper C states. He also said he significantly cured said
> governor in .35.
> 
> Mike could you re-run your netperf tests that showed the 10% throughput
> gain? Hopefully the fixed governor will yield the same result and we can
> kill off this ratelimit thing.

FWIW, on my test-box, 35-rc4-tip, using ondemand:

with    nohz_ratelimit() : ~2119.54 MB/s
without nohz_ratelimit() : ~2353.03 MB/s

Seems to suggest we should simply kill the thing..

Re: high power consumption in recent kernels

[Peter Zijlstra]

On Thu, 2010-07-08 at 15:23 +0200, Peter Zijlstra wrote:
> On Thu, 2010-07-08 at 21:46 +0900, Norbert Preining wrote:
> > Looks promising, reverting the old patch, adding that one, building,
> > running, unplugging ppower, powertop runs now since some time,
> > it seems that we are back to better situation: 
> 
> Hrmm, Mike seems you wrecked power usage.. 
> 
> So nohz_ratelimit() prevents us from entering NOHZ when the last attempt
> was less than 1/2 a jiffy ago (fwiw: NSEC_PER_SEC/HZ == TICK_NSEC).
> 
> Its either entering idle or irq_exit trying to enter nohz state, if we
> keep skipping it it means that we get enough interrupt activity to
> render nohz useless anyway.. so not quite sure how this wrecks things..

OK, so Arjan said the gain could come from tricking the idle governor
into not using deeper C states. He also said he significantly cured said
governor in .35.

Mike could you re-run your netperf tests that showed the 10% throughput
gain? Hopefully the fixed governor will yield the same result and we can
kill off this ratelimit thing.

Re: high power consumption in recent kernels

[Arjan van de Ven]

> and for that matter, what is "[extra timer interrupt]", surely the
> timer hardware doesn't generate spurious interrupts?

extra timer interrupt are those cases where we see the hardware
interrupt fire, but we do not have software timers to account for them.
two cases this can happen
* a NO_HZ bug
* we are idle longer than the longest interval we can program the hw
  timer for. Without HPET this can happen.



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: high power consumption in recent kernels

[Peter Zijlstra]

On Thu, 2010-07-08 at 21:46 +0900, Norbert Preining wrote:
> Looks promising, reverting the old patch, adding that one, building,
> running, unplugging ppower, powertop runs now since some time,
> it seems that we are back to better situation: 

Hrmm, Mike seems you wrecked power usage.. 

So nohz_ratelimit() prevents us from entering NOHZ when the last attempt
was less than 1/2 a jiffy ago (fwiw: NSEC_PER_SEC/HZ == TICK_NSEC).

Its either entering idle or irq_exit trying to enter nohz state, if we
keep skipping it it means that we get enough interrupt activity to
render nohz useless anyway.. so not quite sure how this wrecks things..

Re: high power consumption in recent kernels

[Norbert Preining]

On Do, 08 Jul 2010, Peter Zijlstra wrote:
> Right, so that is a buggy patch, see the original discussion:
>   http://lkml.org/lkml/2010/4/26/249

Well, to me it wasn't so clear that this was buggy *for*my*system*
(core2)

> That said, we did frob something with the whole nohz thing, does the
> below cure anything:

Looks promising, reverting the old patch, adding that one, building,
running, unplugging ppower, powertop runs now since some time,
it seems that we are back to better situation:
Cn                Avg residency       P-states (frequencies)
C0 (cpu running)        ( 1.5%)       Turbo Mode     0.0%
C0                0.0ms ( 0.0%)         2.54 Ghz     0.0%
C1 mwait          0.0ms ( 0.0%)         1.60 Ghz     0.0%
C2 mwait          0.3ms ( 0.9%)          800 Mhz   100.0%
C6 mwait          8.5ms (97.6%)

Wakeups-from-idle per second : 139.9    interval: 15.0s
Power usage (ACPI estimate): 10.0W (8.8 hours) (long term: 1.7W,/50.9h)

Top causes for wakeups:
  32.2% ( 46.1)   [kernel scheduler] Load balancing tick
  20.4% ( 29.1)   [iwlagn] <interrupt>
  12.6% ( 18.0)   [extra timer interrupt]
   6.5% (  9.3)   [ahci] <interrupt>
   3.7% (  5.3)   [kernel core] hrtimer_start (tick_sched_timer)
   3.5% (  5.0)   syndaemon
   3.4% (  4.8)   [acpi] <interrupt>
   2.3% (  3.3)   yarssr

Power is going down to below 10W with brightness dimmed.

Thanks.

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining            preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan                                 TeX Live & Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
I'm going to have a look.'
He glanced round at the others.
`Is no one going to say, "No you can't possibly, let me go
instead"?'
They all shook their heads.
`Oh well.'
                 --- Ford attempting to be heroic whilst being seiged by
                 --- Shooty and Bangbang.
                 --- Douglas Adams, The Hitchhikers Guide to the Galaxy

Re: high power consumption in recent kernels

[Peter Zijlstra]

On Thu, 2010-07-08 at 21:04 +0900, Norbert Preining wrote:

> Just one more point, searching a bit more in the net I found the following
> patch (forgot who wrote it) which I merged into my current git:

> diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
> index a878b53..f26efba 100644
> --- a/kernel/sched_fair.c
> +++ b/kernel/sched_fair.c
> @@ -3248,6 +3248,9 @@ int select_nohz_load_balancer(int stop_tick)
>         if (stop_tick) {
>                 cpu_rq(cpu)->in_nohz_recently = 1;
>  
> +               if (!mc_capable())
> +                       return 0;
> +
>                 if (!cpu_active(cpu)) {
>                         if (atomic_read(&nohz.load_balancer) != cpu)
>                                 return 0;
> @@ -3297,6 +3300,9 @@ int select_nohz_load_balancer(int stop_tick)
>                 if (!cpumask_test_cpu(cpu, nohz.cpu_mask))
>                         return 0;
>  
> +               if (!mc_capable())
> +                       return 0;
> +
>                 cpumask_clear_cpu(cpu, nohz.cpu_mask);
>  
>                 if (atomic_read(&nohz.load_balancer) == cpu)

Right, so that is a buggy patch, see the original discussion:
  http://lkml.org/lkml/2010/4/26/249

> which looks better

The thing is, we didn't change that code recently, the patches that are
supposed to cure the nohz balancer are still pending (in -tip and
-next).

That said, we did frob something with the whole nohz thing, does the
below cure anything:

---
 kernel/time/tick-sched.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 813993b..9bc8029 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -325,7 +325,7 @@ void tick_nohz_stop_sched_tick(int inidle)
 	} while (read_seqretry(&xtime_lock, seq));
 
 	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
-	    arch_needs_cpu(cpu) || nohz_ratelimit(cpu)) {
+	    arch_needs_cpu(cpu) /* || nohz_ratelimit(cpu) */) {
 		next_jiffies = last_jiffies + 1;
 		delta_jiffies = 1;
 	} else {

Re: high power consumption in recent kernels

[Norbert Preining]

On Do, 08 Jul 2010, Peter Zijlstra wrote:
> then wth is "[kernel scheduler] load balancing tick"?
> and for that matter, what is "[extra timer interrupt]", surely the timer
> hardware doesn't generate spurious interrupts?

Just one more point, searching a bit more in the net I found the following
patch (forgot who wrote it) which I merged into my current git:
diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index a878b53..f26efba 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -3248,6 +3248,9 @@ int select_nohz_load_balancer(int stop_tick)
        if (stop_tick) {
                cpu_rq(cpu)->in_nohz_recently = 1;
 
+               if (!mc_capable())
+                       return 0;
+
                if (!cpu_active(cpu)) {
                        if (atomic_read(&nohz.load_balancer) != cpu)
                                return 0;
@@ -3297,6 +3300,9 @@ int select_nohz_load_balancer(int stop_tick)
                if (!cpumask_test_cpu(cpu, nohz.cpu_mask))
                        return 0;
 
+               if (!mc_capable())
+                       return 0;
+
                cpumask_clear_cpu(cpu, nohz.cpu_mask);
 
                if (atomic_read(&nohz.load_balancer) == cpu)

Now the output looks like:
Cn                Avg residency       P-states (frequencies)
C0 (cpu running)        ( 2.7%)       Turbo Mode     0.2%
C0                0.0ms ( 0.0%)         2.54 Ghz     0.0%
C1 mwait          0.8ms ( 0.0%)         1.60 Ghz     0.0%
C2 mwait          0.4ms ( 1.4%)          800 Mhz    99.8%
C6 mwait          4.9ms (95.9%)

Wakeups-from-idle per second : 228.4    interval: 10.0s
Power usage (ACPI estimate): 11.7W (7.3 hours)

Top causes for wakeups:
  22.4% ( 55.4)   [kernel scheduler] Load balancing tick
  16.6% ( 41.2)   [iwlagn] <interrupt>
  16.0% ( 39.7)   [extra timer interrupt]
  15.2% ( 37.7)   [kernel core] hrtimer_start (tick_sched_timer)
  13.7% ( 33.9)   firefox-bin
   2.4% (  5.9)   [ahci] <interrupt>
   2.0% (  5.0)   syndaemon


which looks better

Best wishes

Norbert
------------------------------------------------------------------------
Norbert Preining            preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan                                 TeX Live & Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
OSWALDTWISTLE (n. Old Norse)
Small brass wind instrument used for summoning Vikings to lunch when
they're off on their longships, playing.
			--- Douglas Adams, The Meaning of Liff

Re: high power consumption in recent kernels

[Peter Zijlstra]

On Thu, 2010-07-08 at 04:57 -0700, Arjan van de Ven wrote:
> On Thu, 08 Jul 2010 11:06:32 +0200
> Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > On Wed, 2010-07-07 at 01:45 +0900, Norbert Preining wrote:
> > 
> > > it seems that some of the (?)recent(?) changes have increased the
> > > power consumption of my note book considerably.
> > > 
> > > First of all, running powertop with normal programs started, but 
> > > doing nothing, I am still at 14W while I could go down to 9W before
> > > (but the 9W was with dimmed display).
> > > 
> > > In the list of top causes for wakeup I have
> > > Top causes for wakeups:
> > >   34.2% (185.3)   [kernel scheduler] Load balancing tick
> > >   23.9% (129.6)   [extra timer interrupt]
> > >   10.8% ( 58.6)   firefox-bin
> > >    9.2% ( 49.7)   [iwlagn] <interrupt>
> > >    7.2% ( 39.1)   [kernel core] hrtimer_start (tick_sched_timer)
> > >    3.9% ( 20.9)   PS/2 keyboard/mouse/touchpad interrupt
> > > which show one new thing to me I haven't seen before, the Loa
> > > balancing tick.
> > 
> > I think that is what powertop calls our regular tick (Arjan?), and as
> 
> it's "hrtimer_start_range_ns (tick_sched_timer)" if it's done by the
> idle thread.
> 

then wth is "[kernel scheduler] load balancing tick"?
and for that matter, what is "[extra timer interrupt]", surely the timer
hardware doesn't generate spurious interrupts?

Re: high power consumption in recent kernels

[Arjan van de Ven]

On Thu, 08 Jul 2010 11:06:32 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> On Wed, 2010-07-07 at 01:45 +0900, Norbert Preining wrote:
> 
> > it seems that some of the (?)recent(?) changes have increased the
> > power consumption of my note book considerably.
> > 
> > First of all, running powertop with normal programs started, but 
> > doing nothing, I am still at 14W while I could go down to 9W before
> > (but the 9W was with dimmed display).
> > 
> > In the list of top causes for wakeup I have
> > Top causes for wakeups:
> >   34.2% (185.3)   [kernel scheduler] Load balancing tick
> >   23.9% (129.6)   [extra timer interrupt]
> >   10.8% ( 58.6)   firefox-bin
> >    9.2% ( 49.7)   [iwlagn] <interrupt>
> >    7.2% ( 39.1)   [kernel core] hrtimer_start (tick_sched_timer)
> >    3.9% ( 20.9)   PS/2 keyboard/mouse/touchpad interrupt
> > which show one new thing to me I haven't seen before, the Loa
> > balancing tick.
> 
> I think that is what powertop calls our regular tick (Arjan?), and as

it's "hrtimer_start_range_ns (tick_sched_timer)" if it's done by the
idle thread.

Re: high power consumption in recent kernels

[Peter Zijlstra]

On Wed, 2010-07-07 at 01:45 +0900, Norbert Preining wrote:

> it seems that some of the (?)recent(?) changes have increased the
> power consumption of my note book considerably.
> 
> First of all, running powertop with normal programs started, but 
> doing nothing, I am still at 14W while I could go down to 9W before
> (but the 9W was with dimmed display).
> 
> In the list of top causes for wakeup I have
> Top causes for wakeups:
>   34.2% (185.3)   [kernel scheduler] Load balancing tick
>   23.9% (129.6)   [extra timer interrupt]
>   10.8% ( 58.6)   firefox-bin
>    9.2% ( 49.7)   [iwlagn] <interrupt>
>    7.2% ( 39.1)   [kernel core] hrtimer_start (tick_sched_timer)
>    3.9% ( 20.9)   PS/2 keyboard/mouse/touchpad interrupt
> which show one new thing to me I haven't seen before, the Loa balancing tick.

I think that is what powertop calls our regular tick (Arjan?), and as
long as you're not idle nohz can't kick in, I'd be looking at what keeps
nohz from happening,.. those extra timer interrupts could be
responsible..

> Furthermore, I also had the feeling that while being suspend to RAM 
> the battery is also emptied faster then before, but by now I have not
> concise data points for that.
> 
> I am running the latest kernel pulled from git.kernel.org, linux-2.6.

Right, I'm quite puzzled by that, looking through the result of:
  git log v2.6.34..origin/master kernel/sched*

Nothing really shouts me, me, me :-)

You could try and look at:

99bd5e2f245d8 (sched: Fix select_idle_sibling() logic in select_task_rq_fair())
3c93717cfa513 (sched: Fix over-scheduling bug)

And that latter only if you actually have group scheduling enabled.

Other than that, I'm afraid I'll have to ask you to bisect this.

> I have seen some patches from Venkatesh Pallipadi dealing twith this 
> issue (http://lkml.org/lkml/2010/6/9/109) but it does not apply to
> the current kernel sources at all.

Those patches are merged and waiting for the .36 merge window, try a
-tip kernel
(git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git) if
you want to give them a go.

high power consumption in recent kernels

[Norbert Preining]

Dear all,

(please CC)

it seems that some of the (?)recent(?) changes have increased the
power consumption of my note book considerably.

First of all, running powertop with normal programs started, but 
doing nothing, I am still at 14W while I could go down to 9W before
(but the 9W was with dimmed display).

In the list of top causes for wakeup I have
Top causes for wakeups:
  34.2% (185.3)   [kernel scheduler] Load balancing tick
  23.9% (129.6)   [extra timer interrupt]
  10.8% ( 58.6)   firefox-bin
   9.2% ( 49.7)   [iwlagn] <interrupt>
   7.2% ( 39.1)   [kernel core] hrtimer_start (tick_sched_timer)
   3.9% ( 20.9)   PS/2 keyboard/mouse/touchpad interrupt
which show one new thing to me I haven't seen before, the Loa balancing tick.

Furthermore, I also had the feeling that while being suspend to RAM 
the battery is also emptied faster then before, but by now I have not
concise data points for that.

I am running the latest kernel pulled from git.kernel.org, linux-2.6.

I have seen some patches from Venkatesh Pallipadi dealing twith this 
issue (http://lkml.org/lkml/2010/6/9/109) but it does not apply to
the current kernel sources at all.

Please let me know if you have any suggestions of if I should test
something.

Thanks and all the best

Norbert

Please CC
------------------------------------------------------------------------
Norbert Preining            preining@{jaist.ac.jp, logic.at, debian.org}
JAIST, Japan                                 TeX Live & Debian Developer
DSA: 0x09C5B094   fp: 14DF 2E6C 0307 BE6D AD76  A9C0 D2BF 4AA3 09C5 B094
------------------------------------------------------------------------
HARBOTTLE (n.)
A particular kind of fly which lives inside double glazing.
			--- Douglas Adams, The Meaning of Liff