[GIT PULL] nohz: Full dynticks base interface

[GIT PULL] nohz: Full dynticks base interface

[Frederic Weisbecker]

Hi Ingo,

This settles the initial ground to start a special full dynticks tree in -tip
that we can iterate incrementally to accelerate the development.
It is based on tip:sched/core.

I tried to rearrange a bit the naming. We are probably not yet done with
that but I guess we can fix it along with the rest.

Please pull from:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
	full-dynticks-for-mingo

Changes on these commits since they were part of 3.9-rc1-nohz1:

* Force a timekeeping CPU over the full dynticks range
* Rename CONFIG_NO_HZ_FULL to CONFIG_NO_HZ_EXTENDED
* Following *_nohz_extended_* APIs renames
* Handle CPU hotplug for timekeeping
* Rename full_nohz= kernel parameter to nohz_extended=
* Improve some error handling
* Improve some documentation (comments, Kconfig, ...)

Many of these changes may have been suggested by several reviewers.
Sorry for the missing credits, I lost a bit track...

Thanks.
---
Frederic Weisbecker (3):
  nohz: Basic full dynticks interface
  nohz: Assign timekeeping duty to a CPU outside the full dynticks
    range
  nohz: Wake up full dynticks CPUs when a timer gets enqueued

 Documentation/kernel-parameters.txt |    6 ++
 include/linux/sched.h               |    4 +-
 include/linux/tick.h                |    7 ++
 kernel/sched/core.c                 |   20 ++++++-
 kernel/time/Kconfig                 |   19 ++++++
 kernel/time/tick-broadcast.c        |    3 +-
 kernel/time/tick-common.c           |    5 +-
 kernel/time/tick-sched.c            |  107 ++++++++++++++++++++++++++++++++++-
 kernel/timer.c                      |   12 ++--
 9 files changed, 171 insertions(+), 12 deletions(-)

-- 
1.7.5.4

[PATCH 1/3] nohz: Basic full dynticks interface

[Frederic Weisbecker]

For extreme usecases such as Real Time or HPC, having
the ability to shutdown the tick when a single task runs
on a CPU is a desired feature:

* Reducing the amount of interrupts improves throughput
for CPU-bound tasks. The CPU is less distracted from its
real job, from an execution time and from the cache point
of views.

* This also improve latency response as we have less critical
sections.

Start with introducing a very simple interface to define
full dynticks CPU: use a boot time option defined cpumask
through the "nohz_extended=" kernel parameter. CPUs that
are part of this range will have their tick shutdown
whenever possible: provided they run a single task and
they don't do kernel activity that require the periodic
tick. These details will be later documented in
Documentation/*

An online CPU must be kept outside this range to handle the
timekeeping.

Suggested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Geoff Levand <geoff@infradead.org>
Cc: Gilad Ben Yossef <gilad@benyossef.com>
Cc: Hakan Akkan <hakanakkan@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Kevin Hilman <khilman@linaro.org>
Cc: Li Zhong <zhong@linux.vnet.ibm.com>
Cc: Namhyung Kim <namhyung.kim@lge.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
---
 Documentation/kernel-parameters.txt |    6 +++
 include/linux/tick.h                |    7 ++++
 kernel/time/Kconfig                 |   19 +++++++++++
 kernel/time/tick-sched.c            |   62 +++++++++++++++++++++++++++++++++++
 4 files changed, 94 insertions(+), 0 deletions(-)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 4609e81..231698f 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1913,6 +1913,12 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			Valid arguments: on, off
 			Default: on
 
+	nohz_extended=  [KNL,BOOT]
+			In kernels built with CONFIG_NO_HZ_EXTENDED=y, set
+			the specified list of CPUs whose tick will be stopped
+			whenever possible. You need to keep at least one online
+			CPU outside the range to maintain the timekeeping.
+
 	noiotrap	[SH] Disables trapped I/O port accesses.
 
 	noirqdebug	[X86-32] Disables the code which attempts to detect and
diff --git a/include/linux/tick.h b/include/linux/tick.h
index 553272e..44bfa8a 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -157,6 +157,13 @@ static inline u64 get_cpu_idle_time_us(int cpu, u64 *unused) { return -1; }
 static inline u64 get_cpu_iowait_time_us(int cpu, u64 *unused) { return -1; }
 # endif /* !NO_HZ */
 
+#ifdef CONFIG_NO_HZ_EXTENDED
+extern int tick_nohz_extended_cpu(int cpu);
+#else
+static inline int tick_nohz_extended_cpu(int cpu) { return 0; }
+#endif
+
+
 # ifdef CONFIG_CPU_IDLE_GOV_MENU
 extern void menu_hrtimer_cancel(void);
 # else
diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig
index 24510d8..5a87c03 100644
--- a/kernel/time/Kconfig
+++ b/kernel/time/Kconfig
@@ -79,6 +79,25 @@ config NO_HZ
 	  only trigger on an as-needed basis both when the system is
 	  busy and when the system is idle.
 
+config NO_HZ_EXTENDED
+       bool "Full dynticks system"
+       depends on NO_HZ && RCU_USER_QS && VIRT_CPU_ACCOUNTING_GEN && RCU_NOCB_CPU && SMP
+       select CONTEXT_TRACKING_FORCE
+       help
+         Adaptively try to shutdown the tick whenever possible, even when
+	 the CPU is running tasks. Typically this requires running a single
+	 task on the CPU. Chances for running tickless are maximized when
+	 the task mostly runs in userspace and has few kernel activity.
+
+	 You need to fill up the nohz_extended boot parameter with the
+	 desired range of dynticks CPUs.
+
+	 This is implemented at the expense of some overhead in user <-> kernel
+	 transitions: syscalls, exceptions and interrupts. Even when it's
+	 dynamically off.
+
+	 Say N.
+
 config HIGH_RES_TIMERS
 	bool "High Resolution Timer Support"
 	depends on !ARCH_USES_GETTIMEOFFSET && GENERIC_CLOCKEVENTS
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index a19a399..79c275f0 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -142,6 +142,68 @@ static void tick_sched_handle(struct tick_sched *ts, struct pt_regs *regs)
 	profile_tick(CPU_PROFILING);
 }
 
+#ifdef CONFIG_NO_HZ_EXTENDED
+static cpumask_var_t nohz_extended_mask;
+bool have_nohz_extended_mask;
+
+int tick_nohz_extended_cpu(int cpu)
+{
+	if (!have_nohz_extended_mask)
+		return 0;
+
+	return cpumask_test_cpu(cpu, nohz_extended_mask);
+}
+
+/* Parse the boot-time nohz CPU list from the kernel parameters. */
+static int __init tick_nohz_extended_setup(char *str)
+{
+	alloc_bootmem_cpumask_var(&nohz_extended_mask);
+	if (cpulist_parse(str, nohz_extended_mask) < 0)
+		pr_warning("NOHZ: Incorrect nohz_extended cpumask\n");
+	else
+		have_nohz_extended_mask = true;
+	return 1;
+}
+__setup("nohz_extended=", tick_nohz_extended_setup);
+
+static int __init init_tick_nohz_extended(void)
+{
+	cpumask_var_t online_nohz;
+	int cpu;
+
+	if (!have_nohz_extended_mask)
+		return 0;
+
+	if (!zalloc_cpumask_var(&online_nohz, GFP_KERNEL)) {
+		pr_warning("NO_HZ: Not enough memory to check extended nohz mask\n");
+		return -ENOMEM;
+	}
+
+	/*
+	 * CPUs can probably not be concurrently offlined on initcall time.
+	 * But we are paranoid, aren't we?
+	 */
+	get_online_cpus();
+
+	/* Ensure we keep a CPU outside the dynticks range for timekeeping */
+	cpumask_and(online_nohz, cpu_online_mask, nohz_extended_mask);
+	if (cpumask_equal(online_nohz, cpu_online_mask)) {
+		cpu = cpumask_any(cpu_online_mask);
+		pr_warning("NO_HZ: Must keep at least one online CPU "
+			   "out of nohz_extended range\n");
+		pr_warning("NO_HZ: Clearing %d from nohz_extended range\n", cpu);
+		cpumask_clear_cpu(cpu, nohz_extended_mask);
+	}
+	put_online_cpus();
+	free_cpumask_var(online_nohz);
+
+	return 0;
+}
+core_initcall(init_tick_nohz_extended);
+#else
+#define have_nohz_extended_mask (0)
+#endif
+
 /*
  * NOHZ - aka dynamic tick functionality
  */
-- 
1.7.5.4

[PATCH 2/3] nohz: Assign timekeeping duty to a CPU outside the full dynticks range

[Frederic Weisbecker]

This way the full nohz CPUs can safely run with the tick
stopped with a guarantee that somebody else is taking
care of the jiffies and GTOD progression.

Once the duty is attributed to a CPU, it won't change. Also that
CPU can't enter into dyntick idle mode or be hot unplugged.

This may later be improved from a power consumption POV. At
least we should be able to share the duty amongst all CPUs
outside the full dynticks range. Then the duty could even be
shared with full dynticks CPUs when those can't stop their
tick for any reason.

But let's start with that very simple approach first.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Geoff Levand <geoff@infradead.org>
Cc: Gilad Ben Yossef <gilad@benyossef.com>
Cc: Hakan Akkan <hakanakkan@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Kevin Hilman <khilman@linaro.org>
Cc: Li Zhong <zhong@linux.vnet.ibm.com>
Cc: Namhyung Kim <namhyung.kim@lge.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
[fix have_nohz_full_mask offcase]
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
 kernel/time/tick-broadcast.c |    3 +-
 kernel/time/tick-common.c    |    5 +++-
 kernel/time/tick-sched.c     |   47 ++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 51 insertions(+), 4 deletions(-)

diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c
index 2fb8cb8..8a6875c 100644
--- a/kernel/time/tick-broadcast.c
+++ b/kernel/time/tick-broadcast.c
@@ -573,7 +573,8 @@ void tick_broadcast_setup_oneshot(struct clock_event_device *bc)
 		bc->event_handler = tick_handle_oneshot_broadcast;
 
 		/* Take the do_timer update */
-		tick_do_timer_cpu = cpu;
+		if (!tick_nohz_extended_cpu(cpu))
+			tick_do_timer_cpu = cpu;
 
 		/*
 		 * We must be careful here. There might be other CPUs
diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
index b1600a6..b7dc0cb 100644
--- a/kernel/time/tick-common.c
+++ b/kernel/time/tick-common.c
@@ -163,7 +163,10 @@ static void tick_setup_device(struct tick_device *td,
 		 * this cpu:
 		 */
 		if (tick_do_timer_cpu == TICK_DO_TIMER_BOOT) {
-			tick_do_timer_cpu = cpu;
+			if (!tick_nohz_extended_cpu(cpu))
+				tick_do_timer_cpu = cpu;
+			else
+				tick_do_timer_cpu = TICK_DO_TIMER_NONE;
 			tick_next_period = ktime_get();
 			tick_period = ktime_set(0, NSEC_PER_SEC / HZ);
 		}
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 79c275f0..57bb3fe 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -112,7 +112,8 @@ static void tick_sched_do_timer(ktime_t now)
 	 * this duty, then the jiffies update is still serialized by
 	 * jiffies_lock.
 	 */
-	if (unlikely(tick_do_timer_cpu == TICK_DO_TIMER_NONE))
+	if (unlikely(tick_do_timer_cpu == TICK_DO_TIMER_NONE)
+	    && !tick_nohz_extended_cpu(cpu))
 		tick_do_timer_cpu = cpu;
 #endif
 
@@ -166,6 +167,25 @@ static int __init tick_nohz_extended_setup(char *str)
 }
 __setup("nohz_extended=", tick_nohz_extended_setup);
 
+static int __cpuinit tick_nohz_cpu_down_callback(struct notifier_block *nfb,
+						 unsigned long action,
+						 void *hcpu)
+{
+	unsigned int cpu = (unsigned long)hcpu;
+
+	switch (action & ~CPU_TASKS_FROZEN) {
+	case CPU_DOWN_PREPARE:
+		/*
+		 * If we handle the timekeeping duty for full dynticks CPUs,
+		 * we can't safely shutdown that CPU.
+		 */
+		if (have_nohz_extended_mask && tick_do_timer_cpu == cpu)
+			return -EINVAL;
+		break;
+	}
+	return NOTIFY_OK;
+}
+
 static int __init init_tick_nohz_extended(void)
 {
 	cpumask_var_t online_nohz;
@@ -174,6 +194,8 @@ static int __init init_tick_nohz_extended(void)
 	if (!have_nohz_extended_mask)
 		return 0;
 
+	cpu_notifier(tick_nohz_cpu_down_callback, 0);
+
 	if (!zalloc_cpumask_var(&online_nohz, GFP_KERNEL)) {
 		pr_warning("NO_HZ: Not enough memory to check extended nohz mask\n");
 		return -ENOMEM;
@@ -188,11 +210,17 @@ static int __init init_tick_nohz_extended(void)
 	/* Ensure we keep a CPU outside the dynticks range for timekeeping */
 	cpumask_and(online_nohz, cpu_online_mask, nohz_extended_mask);
 	if (cpumask_equal(online_nohz, cpu_online_mask)) {
-		cpu = cpumask_any(cpu_online_mask);
 		pr_warning("NO_HZ: Must keep at least one online CPU "
 			   "out of nohz_extended range\n");
+		/*
+		 * We know the current CPU doesn't have its tick stopped.
+		 * Let's use it for the timekeeping duty.
+		 */
+		preempt_disable();
+		cpu = smp_processor_id();
 		pr_warning("NO_HZ: Clearing %d from nohz_extended range\n", cpu);
 		cpumask_clear_cpu(cpu, nohz_extended_mask);
+		preempt_enable();
 	}
 	put_online_cpus();
 	free_cpumask_var(online_nohz);
@@ -551,6 +579,21 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 		return false;
 	}
 
+	if (have_nohz_extended_mask) {
+		/*
+		 * Keep the tick alive to guarantee timekeeping progression
+		 * if there are full dynticks CPUs around
+		 */
+		if (tick_do_timer_cpu == cpu)
+			return false;
+		/*
+		 * Boot safety: make sure the timekeeping duty has been
+		 * assigned before entering dyntick-idle mode,
+		 */
+		if (tick_do_timer_cpu == TICK_DO_TIMER_NONE)
+			return false;
+	}
+
 	return true;
 }
 
-- 
1.7.5.4

[PATCH 3/3] nohz: Wake up full dynticks CPUs when a timer gets enqueued

[Frederic Weisbecker]

Wake up a CPU when a timer list timer is enqueued there and
the target is part of the full dynticks range. Sending an IPI
to it makes it reconsidering the next timer to program on top
of recent updates.

This may later be improved by checking if the tick is really
stopped on the target. This would need some careful
synchronization though. So deal with such optimization later
and start simple.

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Geoff Levand <geoff@infradead.org>
Cc: Gilad Ben Yossef <gilad@benyossef.com>
Cc: Hakan Akkan <hakanakkan@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Kevin Hilman <khilman@linaro.org>
Cc: Li Zhong <zhong@linux.vnet.ibm.com>
Cc: Namhyung Kim <namhyung.kim@lge.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
---
 include/linux/sched.h |    4 ++--
 kernel/sched/core.c   |   20 +++++++++++++++++++-
 kernel/timer.c        |   12 ++++++------
 3 files changed, 27 insertions(+), 9 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 9004f6e..10626e2e 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1851,9 +1851,9 @@ static inline void idle_task_exit(void) {}
 #endif
 
 #if defined(CONFIG_NO_HZ) && defined(CONFIG_SMP)
-extern void wake_up_idle_cpu(int cpu);
+extern void wake_up_nohz_cpu(int cpu);
 #else
-static inline void wake_up_idle_cpu(int cpu) { }
+static inline void wake_up_nohz_cpu(int cpu) { }
 #endif
 
 #ifdef CONFIG_SCHED_AUTOGROUP
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 849deb9..e91ee58 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -587,7 +587,7 @@ unlock:
  * account when the CPU goes back to idle and evaluates the timer
  * wheel for the next timer event.
  */
-void wake_up_idle_cpu(int cpu)
+static void wake_up_idle_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
 
@@ -617,6 +617,24 @@ void wake_up_idle_cpu(int cpu)
 		smp_send_reschedule(cpu);
 }
 
+static bool wake_up_extended_nohz_cpu(int cpu)
+{
+	if (tick_nohz_extended_cpu(cpu)) {
+		if (cpu != smp_processor_id() ||
+		    tick_nohz_tick_stopped())
+			smp_send_reschedule(cpu);
+		return true;
+	}
+
+	return false;
+}
+
+void wake_up_nohz_cpu(int cpu)
+{
+	if (!wake_up_extended_nohz_cpu(cpu))
+		wake_up_idle_cpu(cpu);
+}
+
 static inline bool got_nohz_idle_kick(void)
 {
 	int cpu = smp_processor_id();
diff --git a/kernel/timer.c b/kernel/timer.c
index dbf7a78..4e3040b 100644
--- a/kernel/timer.c
+++ b/kernel/timer.c
@@ -930,14 +930,14 @@ void add_timer_on(struct timer_list *timer, int cpu)
 	debug_activate(timer, timer->expires);
 	internal_add_timer(base, timer);
 	/*
-	 * Check whether the other CPU is idle and needs to be
-	 * triggered to reevaluate the timer wheel when nohz is
-	 * active. We are protected against the other CPU fiddling
+	 * Check whether the other CPU is in dynticks mode and needs
+	 * to be triggered to reevaluate the timer wheel.
+	 * We are protected against the other CPU fiddling
 	 * with the timer by holding the timer base lock. This also
-	 * makes sure that a CPU on the way to idle can not evaluate
-	 * the timer wheel.
+	 * makes sure that a CPU on the way to stop its tick can not
+	 * evaluate the timer wheel.
 	 */
-	wake_up_idle_cpu(cpu);
+	wake_up_nohz_cpu(cpu);
 	spin_unlock_irqrestore(&base->lock, flags);
 }
 EXPORT_SYMBOL_GPL(add_timer_on);
-- 
1.7.5.4

Re: [GIT PULL] nohz: Full dynticks base interface

[Ingo Molnar]

* Frederic Weisbecker <fweisbec@gmail.com> wrote:

> Hi Ingo,
> 
> This settles the initial ground to start a special full dynticks tree in -tip
> that we can iterate incrementally to accelerate the development.
> It is based on tip:sched/core.
> 
> I tried to rearrange a bit the naming. We are probably not yet done with
> that but I guess we can fix it along with the rest.
> 
> Please pull from:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> 	full-dynticks-for-mingo
> 
> Changes on these commits since they were part of 3.9-rc1-nohz1:
> 
> * Force a timekeeping CPU over the full dynticks range
> * Rename CONFIG_NO_HZ_FULL to CONFIG_NO_HZ_EXTENDED
> * Following *_nohz_extended_* APIs renames
> * Handle CPU hotplug for timekeeping
> * Rename full_nohz= kernel parameter to nohz_extended=

Note that boot parameters suck for pretty much any purpose but quirks - 
please also add a (default off!) Kconfig option to easily enable 
nohz_extended for all CPUs.

That way I will be able to test it automatically via randconfig and such.

My next question/request after that would be: could we make sure that 
enabling this option does not break any applications or kernel 
functionality, ASAP? Once that is offered, it becomes pushable to v3.10 I 
think.

Meanwhile I have pulled your tree into tip:sched/nohz to stage and test 
it, in the hope that it becomes mergable quickly! :-)

Thanks,

	Ingo

Re: [GIT PULL] nohz: Full dynticks base interface

[Frederic Weisbecker]

2013/3/24 Ingo Molnar <mingo@kernel.org>:
>
> * Frederic Weisbecker <fweisbec@gmail.com> wrote:
>
>> Hi Ingo,
>>
>> This settles the initial ground to start a special full dynticks tree in -tip
>> that we can iterate incrementally to accelerate the development.
>> It is based on tip:sched/core.
>>
>> I tried to rearrange a bit the naming. We are probably not yet done with
>> that but I guess we can fix it along with the rest.
>>
>> Please pull from:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
>>       full-dynticks-for-mingo
>>
>> Changes on these commits since they were part of 3.9-rc1-nohz1:
>>
>> * Force a timekeeping CPU over the full dynticks range
>> * Rename CONFIG_NO_HZ_FULL to CONFIG_NO_HZ_EXTENDED
>> * Following *_nohz_extended_* APIs renames
>> * Handle CPU hotplug for timekeeping
>> * Rename full_nohz= kernel parameter to nohz_extended=
>
> Note that boot parameters suck for pretty much any purpose but quirks -
> please also add a (default off!) Kconfig option to easily enable
> nohz_extended for all CPUs.
>
> That way I will be able to test it automatically via randconfig and such.

Sure, I'm adding such an option.

>
> My next question/request after that would be: could we make sure that
> enabling this option does not break any applications or kernel
> functionality, ASAP? Once that is offered, it becomes pushable to v3.10 I
> think.

So that's already the case. I'm careful that, as we iterate, the
changes aren't supposed to break something in the middle. the final
code that eventually shuts the tick down should be the last one in the
series: once we know it is safe to do so.

>
> Meanwhile I have pulled your tree into tip:sched/nohz to stage and test
> it, in the hope that it becomes mergable quickly! :-)

Thanks! :)

Re: [GIT PULL] nohz: Full dynticks base interface

[Paul E. McKenney]

On Sun, Mar 24, 2013 at 03:46:40PM +0100, Frederic Weisbecker wrote:
> 2013/3/24 Ingo Molnar <mingo@kernel.org>:
> >
> > * Frederic Weisbecker <fweisbec@gmail.com> wrote:
> >
> >> Hi Ingo,
> >>
> >> This settles the initial ground to start a special full dynticks tree in -tip
> >> that we can iterate incrementally to accelerate the development.
> >> It is based on tip:sched/core.
> >>
> >> I tried to rearrange a bit the naming. We are probably not yet done with
> >> that but I guess we can fix it along with the rest.
> >>
> >> Please pull from:
> >>
> >> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> >>       full-dynticks-for-mingo
> >>
> >> Changes on these commits since they were part of 3.9-rc1-nohz1:
> >>
> >> * Force a timekeeping CPU over the full dynticks range
> >> * Rename CONFIG_NO_HZ_FULL to CONFIG_NO_HZ_EXTENDED
> >> * Following *_nohz_extended_* APIs renames
> >> * Handle CPU hotplug for timekeeping
> >> * Rename full_nohz= kernel parameter to nohz_extended=
> >
> > Note that boot parameters suck for pretty much any purpose but quirks -
> > please also add a (default off!) Kconfig option to easily enable
> > nohz_extended for all CPUs.
> >
> > That way I will be able to test it automatically via randconfig and such.
> 
> Sure, I'm adding such an option.

Hmmm...  This would be an option to make all but one CPU an adaptive-ticks
CPU, right?  If so, this leads to the question of whether I should add a
matching no-CBs Kconfig option.  My guess is "no", because the existing
CONFIG_RCU_NOCB_CPU_ALL should work just fine -- there would be a CPU that
was not an adaptive-ticks CPU, but does have its RCU callbacks offloaded.

Or am I missing something here?

							Thanx, Paul

> > My next question/request after that would be: could we make sure that
> > enabling this option does not break any applications or kernel
> > functionality, ASAP? Once that is offered, it becomes pushable to v3.10 I
> > think.
> 
> So that's already the case. I'm careful that, as we iterate, the
> changes aren't supposed to break something in the middle. the final
> code that eventually shuts the tick down should be the last one in the
> series: once we know it is safe to do so.
> 
> >
> > Meanwhile I have pulled your tree into tip:sched/nohz to stage and test
> > it, in the hope that it becomes mergable quickly! :-)
> 
> Thanks! :)
> 

Re: [GIT PULL] nohz: Full dynticks base interface

[Frederic Weisbecker]

2013/3/25 Paul E. McKenney <paulmck@linux.vnet.ibm.com>:
> On Sun, Mar 24, 2013 at 03:46:40PM +0100, Frederic Weisbecker wrote:
>> 2013/3/24 Ingo Molnar <mingo@kernel.org>:
>> >
>> > * Frederic Weisbecker <fweisbec@gmail.com> wrote:
>> >
>> >> Hi Ingo,
>> >>
>> >> This settles the initial ground to start a special full dynticks tree in -tip
>> >> that we can iterate incrementally to accelerate the development.
>> >> It is based on tip:sched/core.
>> >>
>> >> I tried to rearrange a bit the naming. We are probably not yet done with
>> >> that but I guess we can fix it along with the rest.
>> >>
>> >> Please pull from:
>> >>
>> >> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
>> >>       full-dynticks-for-mingo
>> >>
>> >> Changes on these commits since they were part of 3.9-rc1-nohz1:
>> >>
>> >> * Force a timekeeping CPU over the full dynticks range
>> >> * Rename CONFIG_NO_HZ_FULL to CONFIG_NO_HZ_EXTENDED
>> >> * Following *_nohz_extended_* APIs renames
>> >> * Handle CPU hotplug for timekeeping
>> >> * Rename full_nohz= kernel parameter to nohz_extended=
>> >
>> > Note that boot parameters suck for pretty much any purpose but quirks -
>> > please also add a (default off!) Kconfig option to easily enable
>> > nohz_extended for all CPUs.
>> >
>> > That way I will be able to test it automatically via randconfig and such.
>>
>> Sure, I'm adding such an option.
>
> Hmmm...  This would be an option to make all but one CPU an adaptive-ticks
> CPU, right?  If so, this leads to the question of whether I should add a
> matching no-CBs Kconfig option.  My guess is "no", because the existing
> CONFIG_RCU_NOCB_CPU_ALL should work just fine -- there would be a CPU that
> was not an adaptive-ticks CPU, but does have its RCU callbacks offloaded.
>
> Or am I missing something here?

No that looks right. Now I wonder if I should select
CONFIG_RCU_NOCB_CPU_ALL at the same time. Probably.

Re: [GIT PULL] nohz: Full dynticks base interface

[Paul E. McKenney]

On Mon, Mar 25, 2013 at 06:12:12PM +0100, Frederic Weisbecker wrote:
> 2013/3/25 Paul E. McKenney <paulmck@linux.vnet.ibm.com>:
> > On Sun, Mar 24, 2013 at 03:46:40PM +0100, Frederic Weisbecker wrote:
> >> 2013/3/24 Ingo Molnar <mingo@kernel.org>:
> >> >
> >> > * Frederic Weisbecker <fweisbec@gmail.com> wrote:
> >> >
> >> >> Hi Ingo,
> >> >>
> >> >> This settles the initial ground to start a special full dynticks tree in -tip
> >> >> that we can iterate incrementally to accelerate the development.
> >> >> It is based on tip:sched/core.
> >> >>
> >> >> I tried to rearrange a bit the naming. We are probably not yet done with
> >> >> that but I guess we can fix it along with the rest.
> >> >>
> >> >> Please pull from:
> >> >>
> >> >> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> >> >>       full-dynticks-for-mingo
> >> >>
> >> >> Changes on these commits since they were part of 3.9-rc1-nohz1:
> >> >>
> >> >> * Force a timekeeping CPU over the full dynticks range
> >> >> * Rename CONFIG_NO_HZ_FULL to CONFIG_NO_HZ_EXTENDED
> >> >> * Following *_nohz_extended_* APIs renames
> >> >> * Handle CPU hotplug for timekeeping
> >> >> * Rename full_nohz= kernel parameter to nohz_extended=
> >> >
> >> > Note that boot parameters suck for pretty much any purpose but quirks -
> >> > please also add a (default off!) Kconfig option to easily enable
> >> > nohz_extended for all CPUs.
> >> >
> >> > That way I will be able to test it automatically via randconfig and such.
> >>
> >> Sure, I'm adding such an option.
> >
> > Hmmm...  This would be an option to make all but one CPU an adaptive-ticks
> > CPU, right?  If so, this leads to the question of whether I should add a
> > matching no-CBs Kconfig option.  My guess is "no", because the existing
> > CONFIG_RCU_NOCB_CPU_ALL should work just fine -- there would be a CPU that
> > was not an adaptive-ticks CPU, but does have its RCU callbacks offloaded.
> >
> > Or am I missing something here?
> 
> No that looks right. Now I wonder if I should select
> CONFIG_RCU_NOCB_CPU_ALL at the same time. Probably.

Sounds like a good initial position to me.  If it somehow causes problems,
we can always change it later.

							Thanx, Paul

Re: [GIT PULL] nohz: Full dynticks base interface

[Ingo Molnar]

* Frederic Weisbecker <fweisbec@gmail.com> wrote:

> > That way I will be able to test it automatically via randconfig and 
> > such.
> 
> Sure, I'm adding such an option.
> 
> > My next question/request after that would be: could we make sure that 
> > enabling this option does not break any applications or kernel 
> > functionality, ASAP? Once that is offered, it becomes pushable to 
> > v3.10 I think.
> 
> So that's already the case. I'm careful that, as we iterate, the changes 
> aren't supposed to break something in the middle. the final code that 
> eventually shuts the tick down should be the last one in the series: 
> once we know it is safe to do so.

I mean, with the .config (or boot) option enabled, with tip:timers/nohz 
as-is, will everything work? We cannot push a feature to Linus that 
doesn't work reliably when enabled ...

Thanks,

	Ingo

Re: [GIT PULL] nohz: Full dynticks base interface

[Frederic Weisbecker]

2013/3/26 Ingo Molnar <mingo@kernel.org>:
>
> * Frederic Weisbecker <fweisbec@gmail.com> wrote:
>
>> > That way I will be able to test it automatically via randconfig and
>> > such.
>>
>> Sure, I'm adding such an option.
>>
>> > My next question/request after that would be: could we make sure that
>> > enabling this option does not break any applications or kernel
>> > functionality, ASAP? Once that is offered, it becomes pushable to
>> > v3.10 I think.
>>
>> So that's already the case. I'm careful that, as we iterate, the changes
>> aren't supposed to break something in the middle. the final code that
>> eventually shuts the tick down should be the last one in the series:
>> once we know it is safe to do so.
>
> I mean, with the .config (or boot) option enabled, with tip:timers/nohz
> as-is, will everything work? We cannot push a feature to Linus that
> doesn't work reliably when enabled ...

Right. So indeed the feature is not implemented yet in
tip:timers/nohz, ie: the tick won't be stopped. But the current
halfway state is not supposed to break existing things.

But I agree.

Re: [GIT PULL] nohz: Full dynticks base interface

[Frederic Weisbecker]

2013/3/25 Paul E. McKenney <paulmck@linux.vnet.ibm.com>:
> On Mon, Mar 25, 2013 at 06:12:12PM +0100, Frederic Weisbecker wrote:
>> 2013/3/25 Paul E. McKenney <paulmck@linux.vnet.ibm.com>:
>> > On Sun, Mar 24, 2013 at 03:46:40PM +0100, Frederic Weisbecker wrote:
>> >> 2013/3/24 Ingo Molnar <mingo@kernel.org>:
>> >> >
>> >> > * Frederic Weisbecker <fweisbec@gmail.com> wrote:
>> >> >
>> >> >> Hi Ingo,
>> >> >>
>> >> >> This settles the initial ground to start a special full dynticks tree in -tip
>> >> >> that we can iterate incrementally to accelerate the development.
>> >> >> It is based on tip:sched/core.
>> >> >>
>> >> >> I tried to rearrange a bit the naming. We are probably not yet done with
>> >> >> that but I guess we can fix it along with the rest.
>> >> >>
>> >> >> Please pull from:
>> >> >>
>> >> >> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
>> >> >>       full-dynticks-for-mingo
>> >> >>
>> >> >> Changes on these commits since they were part of 3.9-rc1-nohz1:
>> >> >>
>> >> >> * Force a timekeeping CPU over the full dynticks range
>> >> >> * Rename CONFIG_NO_HZ_FULL to CONFIG_NO_HZ_EXTENDED
>> >> >> * Following *_nohz_extended_* APIs renames
>> >> >> * Handle CPU hotplug for timekeeping
>> >> >> * Rename full_nohz= kernel parameter to nohz_extended=
>> >> >
>> >> > Note that boot parameters suck for pretty much any purpose but quirks -
>> >> > please also add a (default off!) Kconfig option to easily enable
>> >> > nohz_extended for all CPUs.
>> >> >
>> >> > That way I will be able to test it automatically via randconfig and such.
>> >>
>> >> Sure, I'm adding such an option.
>> >
>> > Hmmm...  This would be an option to make all but one CPU an adaptive-ticks
>> > CPU, right?  If so, this leads to the question of whether I should add a
>> > matching no-CBs Kconfig option.  My guess is "no", because the existing
>> > CONFIG_RCU_NOCB_CPU_ALL should work just fine -- there would be a CPU that
>> > was not an adaptive-ticks CPU, but does have its RCU callbacks offloaded.
>> >
>> > Or am I missing something here?
>>
>> No that looks right. Now I wonder if I should select
>> CONFIG_RCU_NOCB_CPU_ALL at the same time. Probably.
>
> Sounds like a good initial position to me.  If it somehow causes problems,
> we can always change it later.

Ah "rcu: Provide compile-time control for no-CBs CPUs" is not yet in
-tip so I can't do that yet. Ok for now I'm going to add
CONFIG_NO_HZ_EXTENDED_ALL and will select the matching RCU config once
it's visible upstream.

Re: [GIT PULL] nohz: Full dynticks base interface

[Paul E. McKenney]

On Wed, Mar 27, 2013 at 12:48:20AM +0100, Frederic Weisbecker wrote:
> 2013/3/25 Paul E. McKenney <paulmck@linux.vnet.ibm.com>:
> > On Mon, Mar 25, 2013 at 06:12:12PM +0100, Frederic Weisbecker wrote:
> >> 2013/3/25 Paul E. McKenney <paulmck@linux.vnet.ibm.com>:
> >> > On Sun, Mar 24, 2013 at 03:46:40PM +0100, Frederic Weisbecker wrote:
> >> >> 2013/3/24 Ingo Molnar <mingo@kernel.org>:
> >> >> >
> >> >> > * Frederic Weisbecker <fweisbec@gmail.com> wrote:
> >> >> >
> >> >> >> Hi Ingo,
> >> >> >>
> >> >> >> This settles the initial ground to start a special full dynticks tree in -tip
> >> >> >> that we can iterate incrementally to accelerate the development.
> >> >> >> It is based on tip:sched/core.
> >> >> >>
> >> >> >> I tried to rearrange a bit the naming. We are probably not yet done with
> >> >> >> that but I guess we can fix it along with the rest.
> >> >> >>
> >> >> >> Please pull from:
> >> >> >>
> >> >> >> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> >> >> >>       full-dynticks-for-mingo
> >> >> >>
> >> >> >> Changes on these commits since they were part of 3.9-rc1-nohz1:
> >> >> >>
> >> >> >> * Force a timekeeping CPU over the full dynticks range
> >> >> >> * Rename CONFIG_NO_HZ_FULL to CONFIG_NO_HZ_EXTENDED
> >> >> >> * Following *_nohz_extended_* APIs renames
> >> >> >> * Handle CPU hotplug for timekeeping
> >> >> >> * Rename full_nohz= kernel parameter to nohz_extended=
> >> >> >
> >> >> > Note that boot parameters suck for pretty much any purpose but quirks -
> >> >> > please also add a (default off!) Kconfig option to easily enable
> >> >> > nohz_extended for all CPUs.
> >> >> >
> >> >> > That way I will be able to test it automatically via randconfig and such.
> >> >>
> >> >> Sure, I'm adding such an option.
> >> >
> >> > Hmmm...  This would be an option to make all but one CPU an adaptive-ticks
> >> > CPU, right?  If so, this leads to the question of whether I should add a
> >> > matching no-CBs Kconfig option.  My guess is "no", because the existing
> >> > CONFIG_RCU_NOCB_CPU_ALL should work just fine -- there would be a CPU that
> >> > was not an adaptive-ticks CPU, but does have its RCU callbacks offloaded.
> >> >
> >> > Or am I missing something here?
> >>
> >> No that looks right. Now I wonder if I should select
> >> CONFIG_RCU_NOCB_CPU_ALL at the same time. Probably.
> >
> > Sounds like a good initial position to me.  If it somehow causes problems,
> > we can always change it later.
> 
> Ah "rcu: Provide compile-time control for no-CBs CPUs" is not yet in
> -tip so I can't do that yet. Ok for now I'm going to add
> CONFIG_NO_HZ_EXTENDED_ALL and will select the matching RCU config once
> it's visible upstream.

Good point...  I expect to be sending a pull request in a day or two.

							Thanx, Paul