* [PATCH 0/2] sched/nohz: disallow non-existent cores from nohz-full
@ 2021-12-06 14:59 Paul Gortmaker
2021-12-06 14:59 ` [PATCH 1/2] sched/isolation: really align nohz_full with rcu_nocbs Paul Gortmaker
2021-12-06 14:59 ` [PATCH 2/2] tick/nohz: WARN_ON --> WARN_ON_ONCE to prevent console saturation Paul Gortmaker
0 siblings, 2 replies; 6+ messages in thread
From: Paul Gortmaker @ 2021-12-06 14:59 UTC (permalink / raw)
To: linux-kernel
Cc: Paul Gortmaker, Frederic Weisbecker, Ingo Molnar,
Nicholas Piggin, Paul E . McKenney, Thomas Gleixner
A couple months back I sent a fix to reconcile rcu_nocbs= input
restrictions with nohz_full= input restrictions; with the latter being
more restrictive than the former.
However, in relaxing the nohz_full restrictions, I made it possible to
boot with a nohz_full= parameter that contains nothing but nonexistent
and not-possible cores - which will trigger a WARN.
This fixes the original reconcile commit by explicitly coding our
allowed values just like RCU does, and changes the WARN_ON to a
WARN_ON_ONCE, since it needlessly rendered the machine unusable.
---
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Paul Gortmaker (2):
sched/isolation: really align nohz_full with rcu_nocbs
tick/nohz: WARN_ON --> WARN_ON_ONCE to prevent console saturation
kernel/sched/isolation.c | 12 ++++++++++++
kernel/time/tick-sched.c | 2 +-
2 files changed, 13 insertions(+), 1 deletion(-)
--
2.17.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/2] sched/isolation: really align nohz_full with rcu_nocbs
2021-12-06 14:59 [PATCH 0/2] sched/nohz: disallow non-existent cores from nohz-full Paul Gortmaker
@ 2021-12-06 14:59 ` Paul Gortmaker
2021-12-06 21:33 ` Paul E. McKenney
2021-12-06 14:59 ` [PATCH 2/2] tick/nohz: WARN_ON --> WARN_ON_ONCE to prevent console saturation Paul Gortmaker
1 sibling, 1 reply; 6+ messages in thread
From: Paul Gortmaker @ 2021-12-06 14:59 UTC (permalink / raw)
To: linux-kernel
Cc: Paul Gortmaker, Paul E . McKenney, Frederic Weisbecker,
Thomas Gleixner, Ingo Molnar
At the moment it is currently possible to sneak a core into nohz_full
that lies between nr_possible and NR_CPUS - but you won't "see" it
because cpumask_pr_args() implicitly hides anything above nr_cpu_ids.
This becomes a problem when the nohz_full CPU set doesn't contain at
least one other valid nohz CPU - in which case we end up with the
tick_nohz_full_running set and no tick core specified, which trips an
endless sequence of WARN() and renders the machine unusable.
I inadvertently opened the door for this when fixing an overly
restrictive nohz_full conditional in the below Fixes: commit - and then
courtesy of my optimistic ACPI reporting nr_possible of 64 (the default
Kconfig for NR_CPUS) and the not-so helpful implict filtering done by
cpumask_pr_args, I unfortunately did not spot it during my testing.
So here, I don't rely on what was printed anymore, but code exactly what
our restrictions should be in order to be aligned with rcu_nocbs - which
was the original goal. Since the checks lie in "__init" code it is largely
free for us to do this anyway.
Building with NOHZ_FULL and NR_CPUS=128 on an otherwise defconfig, and
booting with "rcu_nocbs=8-127 nohz_full=96-127" on the same 16 core T5500
Dell machine now results in the following (only relevant lines shown):
smpboot: Allowing 64 CPUs, 48 hotplug CPUs
setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:64 nr_node_ids:2
housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' contains nonexistent CPUs.
housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' has no valid CPUs.
rcu: RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=64.
rcu: Note: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.
rcu: Offload RCU callbacks from CPUs: 8-63.
One can see both new housekeeping checks are triggered in the above.
The same invalid boot arg combination would have previously resulted in
an infinitely scrolling mix of WARN from all cores per tick on this box.
Fixes: 915a2bc3c6b7 ("sched/isolation: Reconcile rcu_nocbs= and nohz_full=")
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
kernel/sched/isolation.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c
index 7f06eaf12818..01abc8400d6c 100644
--- a/kernel/sched/isolation.c
+++ b/kernel/sched/isolation.c
@@ -89,6 +89,18 @@ static int __init housekeeping_setup(char *str, enum hk_flags flags)
return 0;
}
+ if (!cpumask_subset(non_housekeeping_mask, cpu_possible_mask)) {
+ pr_info("housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' contains nonexistent CPUs.\n");
+ cpumask_and(non_housekeeping_mask, cpu_possible_mask,
+ non_housekeeping_mask);
+ }
+
+ if (cpumask_empty(non_housekeeping_mask)) {
+ pr_info("housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' has no valid CPUs.\n");
+ free_bootmem_cpumask_var(non_housekeeping_mask);
+ return 0;
+ }
+
alloc_bootmem_cpumask_var(&tmp);
if (!housekeeping_flags) {
alloc_bootmem_cpumask_var(&housekeeping_mask);
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/2] tick/nohz: WARN_ON --> WARN_ON_ONCE to prevent console saturation
2021-12-06 14:59 [PATCH 0/2] sched/nohz: disallow non-existent cores from nohz-full Paul Gortmaker
2021-12-06 14:59 ` [PATCH 1/2] sched/isolation: really align nohz_full with rcu_nocbs Paul Gortmaker
@ 2021-12-06 14:59 ` Paul Gortmaker
2022-04-10 10:33 ` [tip: timers/urgent] tick/nohz: Use WARN_ON_ONCE() " tip-bot2 for Paul Gortmaker
1 sibling, 1 reply; 6+ messages in thread
From: Paul Gortmaker @ 2021-12-06 14:59 UTC (permalink / raw)
To: linux-kernel
Cc: Paul Gortmaker, Nicholas Piggin, Frederic Weisbecker,
Thomas Gleixner, Ingo Molnar
While running some testing on code that happened to allow the variable
tick_nohz_full_running to get set but with no "possible" NOHZ cores to
back up that setting, I tripped this WARN:
if (unlikely(tick_do_timer_cpu == TICK_DO_TIMER_NONE))
WARN_ON(tick_nohz_full_running);
The console was overwhemled with an endless stream of one WARN per tick
per core and there was no way to even see what was going on w/o using a
serial console to capture it and then trace it back to this guy.
Changing it to ONCE reveals that we get the message we need in a
civilized fashion, and the system can limp along until rebooted.
Fixes: 08ae95f4fd3b ("nohz_full: Allow the boot CPU to be nohz_full")
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
---
kernel/time/tick-sched.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 6bffe5af8cb1..19e6b861de97 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -186,7 +186,7 @@ static void tick_sched_do_timer(struct tick_sched *ts, ktime_t now)
*/
if (unlikely(tick_do_timer_cpu == TICK_DO_TIMER_NONE)) {
#ifdef CONFIG_NO_HZ_FULL
- WARN_ON(tick_nohz_full_running);
+ WARN_ON_ONCE(tick_nohz_full_running);
#endif
tick_do_timer_cpu = cpu;
}
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] sched/isolation: really align nohz_full with rcu_nocbs
2021-12-06 14:59 ` [PATCH 1/2] sched/isolation: really align nohz_full with rcu_nocbs Paul Gortmaker
@ 2021-12-06 21:33 ` Paul E. McKenney
2021-12-08 5:32 ` Paul Gortmaker
0 siblings, 1 reply; 6+ messages in thread
From: Paul E. McKenney @ 2021-12-06 21:33 UTC (permalink / raw)
To: Paul Gortmaker
Cc: linux-kernel, Frederic Weisbecker, Thomas Gleixner, Ingo Molnar
On Mon, Dec 06, 2021 at 09:59:49AM -0500, Paul Gortmaker wrote:
> At the moment it is currently possible to sneak a core into nohz_full
> that lies between nr_possible and NR_CPUS - but you won't "see" it
> because cpumask_pr_args() implicitly hides anything above nr_cpu_ids.
>
> This becomes a problem when the nohz_full CPU set doesn't contain at
> least one other valid nohz CPU - in which case we end up with the
> tick_nohz_full_running set and no tick core specified, which trips an
> endless sequence of WARN() and renders the machine unusable.
>
> I inadvertently opened the door for this when fixing an overly
> restrictive nohz_full conditional in the below Fixes: commit - and then
> courtesy of my optimistic ACPI reporting nr_possible of 64 (the default
> Kconfig for NR_CPUS) and the not-so helpful implict filtering done by
> cpumask_pr_args, I unfortunately did not spot it during my testing.
>
> So here, I don't rely on what was printed anymore, but code exactly what
> our restrictions should be in order to be aligned with rcu_nocbs - which
> was the original goal. Since the checks lie in "__init" code it is largely
> free for us to do this anyway.
>
> Building with NOHZ_FULL and NR_CPUS=128 on an otherwise defconfig, and
> booting with "rcu_nocbs=8-127 nohz_full=96-127" on the same 16 core T5500
> Dell machine now results in the following (only relevant lines shown):
>
> smpboot: Allowing 64 CPUs, 48 hotplug CPUs
> setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:64 nr_node_ids:2
> housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' contains nonexistent CPUs.
> housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' has no valid CPUs.
> rcu: RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=64.
> rcu: Note: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.
> rcu: Offload RCU callbacks from CPUs: 8-63.
>
> One can see both new housekeeping checks are triggered in the above.
> The same invalid boot arg combination would have previously resulted in
> an infinitely scrolling mix of WARN from all cores per tick on this box.
>
> Fixes: 915a2bc3c6b7 ("sched/isolation: Reconcile rcu_nocbs= and nohz_full=")
> Cc: Paul E. McKenney <paulmck@kernel.org>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@kernel.org>
> Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> ---
> kernel/sched/isolation.c | 12 ++++++++++++
> 1 file changed, 12 insertions(+)
>
> diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c
> index 7f06eaf12818..01abc8400d6c 100644
> --- a/kernel/sched/isolation.c
> +++ b/kernel/sched/isolation.c
> @@ -89,6 +89,18 @@ static int __init housekeeping_setup(char *str, enum hk_flags flags)
> return 0;
> }
>
> + if (!cpumask_subset(non_housekeeping_mask, cpu_possible_mask)) {
> + pr_info("housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' contains nonexistent CPUs.\n");
> + cpumask_and(non_housekeeping_mask, cpu_possible_mask,
> + non_housekeeping_mask);
> + }
> +
> + if (cpumask_empty(non_housekeeping_mask)) {
> + pr_info("housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' has no valid CPUs.\n");
> + free_bootmem_cpumask_var(non_housekeeping_mask);
> + return 0;
If Frederic applies his rcu_nocbs work to nohz_full, it may some day be
valid to specify an empty nohz_full CPU mask. Of course, it might well
be that warning in the meantime is a good thing, but I figured that I
should call attention to the possibility.
Thanx, Paul
> + }
> +
> alloc_bootmem_cpumask_var(&tmp);
> if (!housekeeping_flags) {
> alloc_bootmem_cpumask_var(&housekeeping_mask);
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] sched/isolation: really align nohz_full with rcu_nocbs
2021-12-06 21:33 ` Paul E. McKenney
@ 2021-12-08 5:32 ` Paul Gortmaker
0 siblings, 0 replies; 6+ messages in thread
From: Paul Gortmaker @ 2021-12-08 5:32 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, Frederic Weisbecker, Thomas Gleixner, Ingo Molnar
[Re: [PATCH 1/2] sched/isolation: really align nohz_full with rcu_nocbs] On 06/12/2021 (Mon 13:33) Paul E. McKenney wrote:
> On Mon, Dec 06, 2021 at 09:59:49AM -0500, Paul Gortmaker wrote:
> > At the moment it is currently possible to sneak a core into nohz_full
> > that lies between nr_possible and NR_CPUS - but you won't "see" it
> > because cpumask_pr_args() implicitly hides anything above nr_cpu_ids.
> >
> > This becomes a problem when the nohz_full CPU set doesn't contain at
> > least one other valid nohz CPU - in which case we end up with the
> > tick_nohz_full_running set and no tick core specified, which trips an
> > endless sequence of WARN() and renders the machine unusable.
> >
> > I inadvertently opened the door for this when fixing an overly
> > restrictive nohz_full conditional in the below Fixes: commit - and then
> > courtesy of my optimistic ACPI reporting nr_possible of 64 (the default
> > Kconfig for NR_CPUS) and the not-so helpful implict filtering done by
> > cpumask_pr_args, I unfortunately did not spot it during my testing.
> >
> > So here, I don't rely on what was printed anymore, but code exactly what
> > our restrictions should be in order to be aligned with rcu_nocbs - which
> > was the original goal. Since the checks lie in "__init" code it is largely
> > free for us to do this anyway.
> >
> > Building with NOHZ_FULL and NR_CPUS=128 on an otherwise defconfig, and
> > booting with "rcu_nocbs=8-127 nohz_full=96-127" on the same 16 core T5500
> > Dell machine now results in the following (only relevant lines shown):
> >
> > smpboot: Allowing 64 CPUs, 48 hotplug CPUs
> > setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:64 nr_node_ids:2
> > housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' contains nonexistent CPUs.
> > housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' has no valid CPUs.
> > rcu: RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=64.
> > rcu: Note: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.
> > rcu: Offload RCU callbacks from CPUs: 8-63.
> >
> > One can see both new housekeeping checks are triggered in the above.
> > The same invalid boot arg combination would have previously resulted in
> > an infinitely scrolling mix of WARN from all cores per tick on this box.
> >
> > Fixes: 915a2bc3c6b7 ("sched/isolation: Reconcile rcu_nocbs= and nohz_full=")
> > Cc: Paul E. McKenney <paulmck@kernel.org>
> > Cc: Frederic Weisbecker <fweisbec@gmail.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Ingo Molnar <mingo@kernel.org>
> > Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
> > ---
> > kernel/sched/isolation.c | 12 ++++++++++++
> > 1 file changed, 12 insertions(+)
> >
> > diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c
> > index 7f06eaf12818..01abc8400d6c 100644
> > --- a/kernel/sched/isolation.c
> > +++ b/kernel/sched/isolation.c
> > @@ -89,6 +89,18 @@ static int __init housekeeping_setup(char *str, enum hk_flags flags)
> > return 0;
> > }
> >
> > + if (!cpumask_subset(non_housekeeping_mask, cpu_possible_mask)) {
> > + pr_info("housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' contains nonexistent CPUs.\n");
> > + cpumask_and(non_housekeeping_mask, cpu_possible_mask,
> > + non_housekeeping_mask);
> > + }
> > +
> > + if (cpumask_empty(non_housekeeping_mask)) {
> > + pr_info("housekeeping: kernel parameter 'nohz_full=' or 'isolcpus=' has no valid CPUs.\n");
> > + free_bootmem_cpumask_var(non_housekeeping_mask);
> > + return 0;
>
> If Frederic applies his rcu_nocbs work to nohz_full, it may some day be
> valid to specify an empty nohz_full CPU mask. Of course, it might well
> be that warning in the meantime is a good thing, but I figured that I
> should call attention to the possibility.
It isn't just a good thing ; it is required. Call chain is as this:
nohz_full= / isolcpus=
housekeeping_nohz_full_setup / housekeeping_isolcpus_setup
housekeeping_setup
tick_nohz_full_setup
tick_nohz_full_running = true;
So housekeeping setup is the "last chance" to validate inputs and
avoid calling tick_nohz_full_setup which unconditionally sets the
tick_nohz_full_running (as the crux of this problem).
At least that is as things stand today based on my understanding.
Paul.
--
>
> Thanx, Paul
>
> > + }
> > +
> > alloc_bootmem_cpumask_var(&tmp);
> > if (!housekeeping_flags) {
> > alloc_bootmem_cpumask_var(&housekeeping_mask);
> > --
> > 2.17.1
> >
^ permalink raw reply [flat|nested] 6+ messages in thread
* [tip: timers/urgent] tick/nohz: Use WARN_ON_ONCE() to prevent console saturation
2021-12-06 14:59 ` [PATCH 2/2] tick/nohz: WARN_ON --> WARN_ON_ONCE to prevent console saturation Paul Gortmaker
@ 2022-04-10 10:33 ` tip-bot2 for Paul Gortmaker
0 siblings, 0 replies; 6+ messages in thread
From: tip-bot2 for Paul Gortmaker @ 2022-04-10 10:33 UTC (permalink / raw)
To: linux-tip-commits
Cc: Paul Gortmaker, Thomas Gleixner, stable, x86, linux-kernel
The following commit has been merged into the timers/urgent branch of tip:
Commit-ID: 40e97e42961f8c6cc7bd5fe67cc18417e02d78f1
Gitweb: https://git.kernel.org/tip/40e97e42961f8c6cc7bd5fe67cc18417e02d78f1
Author: Paul Gortmaker <paul.gortmaker@windriver.com>
AuthorDate: Mon, 06 Dec 2021 09:59:50 -05:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Sun, 10 Apr 2022 12:23:34 +02:00
tick/nohz: Use WARN_ON_ONCE() to prevent console saturation
While running some testing on code that happened to allow the variable
tick_nohz_full_running to get set but with no "possible" NOHZ cores to
back up that setting, this warning triggered:
if (unlikely(tick_do_timer_cpu == TICK_DO_TIMER_NONE))
WARN_ON(tick_nohz_full_running);
The console was overwhemled with an endless stream of one WARN per tick
per core and there was no way to even see what was going on w/o using a
serial console to capture it and then trace it back to this.
Change it to WARN_ON_ONCE().
Fixes: 08ae95f4fd3b ("nohz_full: Allow the boot CPU to be nohz_full")
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/r/20211206145950.10927-3-paul.gortmaker@windriver.com
---
kernel/time/tick-sched.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 2d76c91..3506f6e 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -188,7 +188,7 @@ static void tick_sched_do_timer(struct tick_sched *ts, ktime_t now)
*/
if (unlikely(tick_do_timer_cpu == TICK_DO_TIMER_NONE)) {
#ifdef CONFIG_NO_HZ_FULL
- WARN_ON(tick_nohz_full_running);
+ WARN_ON_ONCE(tick_nohz_full_running);
#endif
tick_do_timer_cpu = cpu;
}
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-04-10 10:33 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-06 14:59 [PATCH 0/2] sched/nohz: disallow non-existent cores from nohz-full Paul Gortmaker
2021-12-06 14:59 ` [PATCH 1/2] sched/isolation: really align nohz_full with rcu_nocbs Paul Gortmaker
2021-12-06 21:33 ` Paul E. McKenney
2021-12-08 5:32 ` Paul Gortmaker
2021-12-06 14:59 ` [PATCH 2/2] tick/nohz: WARN_ON --> WARN_ON_ONCE to prevent console saturation Paul Gortmaker
2022-04-10 10:33 ` [tip: timers/urgent] tick/nohz: Use WARN_ON_ONCE() " tip-bot2 for Paul Gortmaker
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).