[PATCH v10 00/20] timers: Move from a push remote at enqueue to a pull at expiry model

[PATCH v10 00/20] timers: Move from a push remote at enqueue to a pull at expiry model

[Anna-Maria Behnsen]

Hi,

the cleanup patches are already applied and so the contains only two parts:

- Patches 1 - 4: timer base idle marking rework with two preparatory
  changes. See the section below for more details.

- Patches 5 - 20: Updated timer pull model on top of timer idle rework


The queue is available here:

  git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel timers/pushpull


Move marking timer bases as idle into tick_nohz_stop_tick()
-----------------------------------------------------------

The idle marking of timer bases is done in get_next_timer_interrupt()
whenever possible. The timer bases are idle, even if the tick will not be
stopped. This lead to an IPI when a new first timer is enqueued remote. To
prevent this, setting timer_base->in_idle flag is postponed to
tick_nohz_stop_tick().

Furthermore this synchronizes the states of timer base is_idle and
tick_stopped. With the timer pull model in place, also the idle state in
the hierarchy of a CPU is synchronized with the other idle related states.


Timer pull model
----------------

Placing timers at enqueue time on a target CPU based on dubious heuristics
does not make any sense:

 1) Most timer wheel timers are canceled or rearmed before they expire.

 2) The heuristics to predict which CPU will be busy when the timer expires
    are wrong by definition.

So placing the timers at enqueue wastes precious cycles.

The proper solution to this problem is to always queue the timers on the
local CPU and allow the non pinned timers to be pulled onto a busy CPU at
expiry time.

Therefore split the timer storage into local pinned and global timers:
Local pinned timers are always expired on the CPU on which they have been
queued. Global timers can be expired on any CPU.

As long as a CPU is busy it expires both local and global timers. When a
CPU goes idle it arms for the first expiring local timer. If the first
expiring pinned (local) timer is before the first expiring movable timer,
then no action is required because the CPU will wake up before the first
movable timer expires. If the first expiring movable timer is before the
first expiring pinned (local) timer, then this timer is queued into a idle
timerqueue and eventually expired by some other active CPU.

To avoid global locking the timerqueues are implemented as a hierarchy. The
lowest level of the hierarchy holds the CPUs. The CPUs are associated to
groups of 8, which are separated per node. If more than one CPU group
exist, then a second level in the hierarchy collects the groups. Depending
on the size of the system more than 2 levels are required. Each group has a
"migrator" which checks the timerqueue during the tick for remote timers to
be expired.

If the last CPU in a group goes idle it reports the first expiring event in
the group up to the next group(s) in the hierarchy. If the last CPU goes
idle it arms its timer for the first system wide expiring timer to ensure
that no timer event is missed.


Testing
~~~~~~~

Enqueue
^^^^^^^

The impact of wasting cycles during enqueue by using the heuristic in
contrast to always queuing the timer on the local CPU was measured with a
micro benchmark. Therefore a timer is enqueued and dequeued in a loop with
1000 repetitions on a isolated CPU. The time the loop takes is measured. A
quarter of the remaining CPUs was kept busy. This measurement was repeated
several times. With the patch queue the average duration was reduced by
approximately 25%.

	145ns	plain v6
	109ns	v6 with patch queue


Furthermore the impact of residence in deep idle states of an idle system
was investigated. The patch queue doesn't downgrade this behavior.

dbench test
^^^^^^^^^^^

A dbench test starting X pairs of client servers are used to create load on
the system. The measurable value is the throughput. The tests were executed
on a zen3 machine. The base is the tip tree branch timers/core which is
based on a v6.6-rc1.

governor menu

NR	timers/core	pull-model	impact
----------------------------------------------
1	353.19 (0.19)	353.45 (0.30)	0.07%
2	700.10 (0.96)	687.00 (0.20)	-1.87%
4	1329.37 (0.63)	1282.91 (0.64)	-3.49%
8	2561.16 (1.28)	2493.56	(1.76)	-2.64%
16	4959.96 (0.80)	4914.59 (0.64)	-0.91%
32	9741.92 (3.44)	8979.83 (1.13)	-7.82%
64	16535.40 (2.84)	16388.47 (4.02)	-0.89%
128	22136.83 (2.42)	23174.50 (1.43)	4.69%
256	39256.77 (4.48)	38994.00 (0.39)	-0.67%
512	36799.03 (1.83)	38091.10 (0.63)	3.51%
1024	32903.03 (0.86)	35370.70 (0.89)	7.50%


governor teo

NR	timers/core	pull-model	impact
----------------------------------------------
1	350.83 (1.27)	352.45 (0.96)	0.46%
2	699.52 (0.85)	690.10 (0.54)	-1.35%
4	1339.53 (1.99)	1294.71 (2.71)	-3.35%
8	2574.10 (0.76)	2495.46 (1.97)	-3.06%
16	4898.50 (1.74)	4783.06 (1.64)	-2.36%
32	9115.50 (4.63)	9037.83 (1.58)	-0.85%
64	16663.90 (3.80)	16042.00 (1.72)	-3.73%
128	25044.93 (1.11)	23250.03 (1.08)	-7.17%
256	38059.53 (1.70)	39658.57 (2.98)	4.20%
512	36369.30 (0.39)	38890.13 (0.36)	6.93%
1024	33956.83 (1.14)	35514.83 (0.29)	4.59%



Ping Pong Oberservation
^^^^^^^^^^^^^^^^^^^^^^^

During testing on a mostly idle machine a ping pong game could be observed:
a process_timeout timer is expired remotely on a non idle CPU. Then the CPU
where the schedule_timeout() was executed to enqueue the timer comes out of
idle and restarts the timer using schedule_timeout() and goes back to idle
again. This is due to the fair scheduler which tries to keep the task on
the CPU which it previously executed on.




Possible Next Steps
~~~~~~~~~~~~~~~~~~~

Simple deferrable timers are no longer required as they can be converted to
global timers. If a CPU goes idle, a formerly deferrable timer will not
prevent the CPU to sleep as long as possible. Only the last migrator CPU
has to take care of them. Deferrable timers with timer pinned flags needs
to be expired on the specified CPU but must not prevent CPU from going
idle. They require their own timer base which is never taken into account
when calculating the next expiry time. This conversation and required
cleanup will be done in a follow up series.


v9..v10: https://lore.kernel.org/r/20231201092654.34614-1-anna-maria@linutronix.de/
  - Address review Feedback of Bigeasy


v8..v9: https://lore.kernel.org/r/20231004123454.15691-1-anna-maria@linutronix.de
  - Address review feedback
  - Add more minor cleanup fixes
  - fixes inconsistent idle related states


v7..v8: https://lore.kernel.org/r/20230524070629.6377-1-anna-maria@linutronix.de
  - Address review feedback
  - Move marking timer base idle into tick_nohz_stop_tick()
  - Look ahead function to determine possible sleep lenght


v6..v7:
  - Address review feedback of Frederic and bigeasy
  - Change lock, unlock fetch next timer interrupt logic after remote expiry
  - Move timer_expire_remote() into tick-internal.h
  - Add documentation section about "Required event and timerqueue update
    after remote expiry"
  - Fix fallout of kernel test robot


v5..v6:

  - Address review of Frederic Weisbecker and Peter Zijlstra (spelling,
    locking, race in tmigr_handle_remote_cpu())

  - unconditionally set TIMER_PINNED flag in add_timer_on(); introduce
    add_timer() variants which set/unset TIMER_PINNED flag; drop fixing
    add_timer_on() call sites, as TIMER_PINNED flag is set implicitly;
    Fixing workqueue to use add_timer_global() instead of simply
    add_timer() for unbound work.

  - Drop support for siblings to end up in the same level 0 group (could be
    added again in a better way as an improvement later on)

  - Do not send IPI for new first deferrable timers

v4..v5:
  - address review feedback of Frederic Weisbecker
  - fix issue with group timer update after remote expiry

v3..v4:
  - address review feedback of Frederic Weisbecker
  - address kernel test robot fallout
  - Move patch 16 "add_timer_on(): Make sure callers have TIMER_PINNED
    flag" at the begin of the queue to prevent timers to end up in global
    timer base when they were queued using add_timer_on()
  - Fix some comments and typos

v2..v3: https://lore.kernel.org/r/20170418111102.490432548@linutronix.de/
  - Minimize usage of locks by storing data using atomic_cmpxchg() for
    migrator information and information about active cpus.


Thanks,

	Anna-Maria




Anna-Maria Behnsen (18):
  timers: Restructure get_next_timer_interrupt()
  timers: Split out get next timer interrupt
  timers: Move marking timer bases idle into tick_nohz_stop_tick()
  timers: Optimization for timer_base_try_to_set_idle()
  timers: Introduce add_timer() variants which modify timer flags
  workqueue: Use global variant for add_timer()
  timers: add_timer_on(): Make sure TIMER_PINNED flag is set
  timers: Ease code in run_local_timers()
  timers: Split next timer interrupt logic
  timers: Keep the pinned timers separate from the others
  timers: Retrieve next expiry of pinned/non-pinned timers separately
  timers: Split out "get next timer interrupt" functionality
  timers: Add get next timer interrupt functionality for remote CPUs
  timers: Check if timers base is handled already
  timers: Introduce function to check timer base is_idle flag
  timers: Implement the hierarchical pull model
  timer_migration: Add tracepoints
  timers: Always queue timers on the local CPU

Richard Cochran (linutronix GmbH) (2):
  timers: Restructure internal locking
  tick/sched: Split out jiffies update helper function

 MAINTAINERS                            |    1 +
 include/linux/cpuhotplug.h             |    1 +
 include/linux/timer.h                  |   16 +-
 include/trace/events/timer_migration.h |  297 +++++
 kernel/time/Makefile                   |    3 +
 kernel/time/tick-internal.h            |   14 +
 kernel/time/tick-sched.c               |   65 +-
 kernel/time/timer.c                    |  505 +++++--
 kernel/time/timer_migration.c          | 1693 ++++++++++++++++++++++++
 kernel/time/timer_migration.h          |  147 ++
 kernel/workqueue.c                     |    2 +-
 11 files changed, 2629 insertions(+), 115 deletions(-)
 create mode 100644 include/trace/events/timer_migration.h
 create mode 100644 kernel/time/timer_migration.c
 create mode 100644 kernel/time/timer_migration.h

-- 
2.39.2

[PATCH v10 01/20] timers: Restructure get_next_timer_interrupt()

[Anna-Maria Behnsen]

get_next_timer_interrupt() contains two parts for the next timer interrupt
calculation. Those two parts are separated by forwarding the base
clock. But the second part does not depend on the forwarded base
clock.

Therefore restructure get_next_timer_interrupt() to keep things together
which belong together.

No functional change.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
 kernel/time/timer.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 352b161113cd..5f21db4fa3ca 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1937,12 +1937,6 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
 	if (base->next_expiry_recalc)
 		next_expiry_recalc(base);
 
-	/*
-	 * We have a fresh next event. Check whether we can forward the
-	 * base.
-	 */
-	__forward_timer_base(base, basej);
-
 	if (base->timers_pending) {
 		nextevt = base->next_expiry;
 
@@ -1960,6 +1954,12 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
 		base->next_expiry = nextevt;
 	}
 
+	/*
+	 * We have a fresh next event. Check whether we can forward the
+	 * base.
+	 */
+	__forward_timer_base(base, basej);
+
 	/*
 	 * Base is idle if the next event is more than a tick away.
 	 *
-- 
2.39.2

[PATCH v10 02/20] timers: Split out get next timer interrupt

[Anna-Maria Behnsen]

Split out get_next_timer_interrupt() to be able to extend it and make it
reusable for other call sites.

No functional change.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
 kernel/time/timer.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 5f21db4fa3ca..2aea55d53416 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1911,15 +1911,7 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires)
 	return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC;
 }
 
-/**
- * get_next_timer_interrupt - return the time (clock mono) of the next timer
- * @basej:	base time jiffies
- * @basem:	base time clock monotonic
- *
- * Returns the tick aligned clock monotonic time of the next pending
- * timer or KTIME_MAX if no timer is pending.
- */
-u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
+static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem)
 {
 	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
 	unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA;
@@ -1978,6 +1970,19 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
 	return cmp_next_hrtimer_event(basem, expires);
 }
 
+/**
+ * get_next_timer_interrupt() - return the time (clock mono) of the next timer
+ * @basej:	base time jiffies
+ * @basem:	base time clock monotonic
+ *
+ * Returns the tick aligned clock monotonic time of the next pending
+ * timer or KTIME_MAX if no timer is pending.
+ */
+u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
+{
+	return __get_next_timer_interrupt(basej, basem);
+}
+
 /**
  * timer_clear_idle - Clear the idle state of the timer base
  *
-- 
2.39.2

[PATCH v10 03/20] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Anna-Maria Behnsen]

 tick_nohz_stop_tick()

The timer base is marked idle when get_next_timer_interrupt() is
executed. But the decision whether the tick will be stopped and whether the
system is able to go idle is done later. When the timer bases is marked
idle and a new first timer is enqueued remote an IPI is raised. Even if it
is not required because the tick is not stopped and the timer base is
evaluated again at the next tick.

To prevent this, the timer base is marked idle in tick_nohz_stop_tick() and
get_next_timer_interrupt() is streamlined by only looking for the next
timer interrupt. All other work is postponed to
timer_base_try_to_set_idle() which is called by tick_nohz_stop_tick().

With this, tick_sched::tick_stopped and timer_base::is_idle is always in
sync. So there is no longer the need to execute timer_clear_idle() in
tick_nohz_idle_retain_tick(). This was required before, as
tick_nohz_next_event() set timer_base::is_idle even if the tick would not
be stopped. So timer_clear_idle() is only executed, when timer base is
idle. So the check whether timer base is idle, is now no longer required as
well.

While at it fix some nearby whitespace damage as well.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
 kernel/time/tick-internal.h |  1 +
 kernel/time/tick-sched.c    | 47 +++++++++++++++++++++--------
 kernel/time/timer.c         | 60 ++++++++++++++++++++++++++-----------
 3 files changed, 78 insertions(+), 30 deletions(-)

diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index 481b7ab65e2c..47df30b871e4 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -163,6 +163,7 @@ static inline void timers_update_nohz(void) { }
 DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases);
 
 extern u64 get_next_timer_interrupt(unsigned long basej, u64 basem);
+u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle);
 void timer_clear_idle(void);
 
 #define CLOCK_SET_WALL							\
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index a17d26002831..c6223afc801f 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -849,11 +849,6 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu)
 	 */
 	delta = next_tick - basemono;
 	if (delta <= (u64)TICK_NSEC) {
-		/*
-		 * Tell the timer code that the base is not idle, i.e. undo
-		 * the effect of get_next_timer_interrupt():
-		 */
-		timer_clear_idle();
 		/*
 		 * We've not stopped the tick yet, and there's a timer in the
 		 * next period, so no point in stopping it either, bail.
@@ -889,12 +884,41 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu)
 static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
 {
 	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
+	unsigned long basejiff = ts->last_jiffies;
 	u64 basemono = ts->timer_expires_base;
-	u64 expires = ts->timer_expires;
+	bool timer_idle;
+	u64 expires;
 
 	/* Make sure we won't be trying to stop it twice in a row. */
 	ts->timer_expires_base = 0;
 
+	/*
+	 * Now the tick should be stopped definitely - so the timer base needs
+	 * to be marked idle as well to not miss a newly queued timer.
+	 */
+	expires = timer_base_try_to_set_idle(basejiff, basemono, &timer_idle);
+	if (!timer_idle) {
+		/*
+		 * Do not clear tick_stopped here when it was already set - it
+		 * will be retained on the next idle iteration when the tick
+		 * expired earlier than expected.
+		 */
+		expires = basemono + TICK_NSEC;
+	} else if (expires > ts->timer_expires) {
+		/*
+		 * This path could only happen when the first timer was removed
+		 * between calculating the possible sleep length and now (when
+		 * high resolution mode is not active, timer could also be a
+		 * hrtimer).
+		 *
+		 * We have to stick to the original calculated expiry value to
+		 * not stop the tick for too long with a shallow C-state (which
+		 * was programmed by cpuidle because of an early next expiration
+		 * value).
+		 */
+		expires = ts->timer_expires;
+	}
+
 	/*
 	 * If this CPU is the one which updates jiffies, then give up
 	 * the assignment and let it be taken by the CPU which runs
@@ -930,6 +954,10 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
 	 * scheduler tick in tick_nohz_restart_sched_tick().
 	 */
 	if (!ts->tick_stopped) {
+		/* If the timer base is not idle, retain the tick. */
+		if (!timer_idle)
+			return;
+
 		calc_load_nohz_start();
 		quiet_vmstat();
 
@@ -991,7 +1019,7 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now)
 	touch_softlockup_watchdog_sched();
 
 	/* Cancel the scheduled timer and restore the tick: */
-	ts->tick_stopped  = 0;
+	ts->tick_stopped = 0;
 	tick_nohz_restart(ts, now);
 }
 
@@ -1147,11 +1175,6 @@ void tick_nohz_idle_stop_tick(void)
 void tick_nohz_idle_retain_tick(void)
 {
 	tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched));
-	/*
-	 * Undo the effect of get_next_timer_interrupt() called from
-	 * tick_nohz_next_event().
-	 */
-	timer_clear_idle();
 }
 
 /**
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 2aea55d53416..3a668060692e 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1911,19 +1911,22 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires)
 	return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC;
 }
 
-static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem)
+static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
+					     bool *idle)
 {
 	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
 	unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA;
 	u64 expires = KTIME_MAX;
-	bool was_idle;
 
 	/*
 	 * Pretend that there is no timer pending if the cpu is offline.
 	 * Possible pending timers will be migrated later to an active cpu.
 	 */
-	if (cpu_is_offline(smp_processor_id()))
+	if (cpu_is_offline(smp_processor_id())) {
+		if (idle)
+			*idle = true;
 		return expires;
+	}
 
 	raw_spin_lock(&base->lock);
 	if (base->next_expiry_recalc)
@@ -1953,17 +1956,26 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem)
 	__forward_timer_base(base, basej);
 
 	/*
-	 * Base is idle if the next event is more than a tick away.
-	 *
-	 * If the base is marked idle then any timer add operation must forward
-	 * the base clk itself to keep granularity small. This idle logic is
-	 * only maintained for the BASE_STD base, deferrable timers may still
-	 * see large granularity skew (by design).
+	 * Set base->is_idle only when caller is timer_base_try_to_set_idle()
 	 */
-	was_idle = base->is_idle;
-	base->is_idle = time_after(nextevt, basej + 1);
-	if (was_idle != base->is_idle)
-		trace_timer_base_idle(base->is_idle, base->cpu);
+	if (idle) {
+		/*
+		 * Base is idle if the next event is more than a tick away.
+		 *
+		 * If the base is marked idle then any timer add operation must
+		 * forward the base clk itself to keep granularity small. This
+		 * idle logic is only maintained for the BASE_STD base,
+		 * deferrable timers may still see large granularity skew (by
+		 * design).
+		 */
+		if (!base->is_idle) {
+			if (time_after(nextevt, basej + 1)) {
+				base->is_idle = true;
+				trace_timer_base_idle(true, base->cpu);
+			}
+		}
+		*idle = base->is_idle;
+	}
 
 	raw_spin_unlock(&base->lock);
 
@@ -1980,7 +1992,21 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem)
  */
 u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
 {
-	return __get_next_timer_interrupt(basej, basem);
+	return __get_next_timer_interrupt(basej, basem, NULL);
+}
+
+/**
+ * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases
+ * @basej:	base time jiffies
+ * @basem:	base time clock monotonic
+ * @idle:	pointer to store the value of timer_base->is_idle
+ *
+ * Returns the tick aligned clock monotonic time of the next pending
+ * timer or KTIME_MAX if no timer is pending.
+ */
+u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle)
+{
+	return __get_next_timer_interrupt(basej, basem, idle);
 }
 
 /**
@@ -1998,10 +2024,8 @@ void timer_clear_idle(void)
 	 * sending the IPI a few instructions smaller for the cost of taking
 	 * the lock in the exit from idle path.
 	 */
-	if (base->is_idle) {
-		base->is_idle = false;
-		trace_timer_base_idle(false, smp_processor_id());
-	}
+	base->is_idle = false;
+	trace_timer_base_idle(false, smp_processor_id());
 }
 #endif
 
-- 
2.39.2

[PATCH v10 04/20] timers: Optimization for timer_base_try_to_set_idle()

[Anna-Maria Behnsen]

When tick is stopped also the timer base is_idle flag is set. When
reentering the timer_base_try_to_set_idle() with the tick stopped, there is
no need to check whether the timer base needs to be set idle again. When a
timer was enqueued in the meantime, this is already handled by the
tick_nohz_next_event() call which was executed before
tick_nohz_stop_tick().

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
 kernel/time/tick-sched.c |  2 +-
 kernel/time/timer.c      | 11 ++++++++---
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index c6223afc801f..27f1a2ae7f39 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -886,7 +886,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
 	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
 	unsigned long basejiff = ts->last_jiffies;
 	u64 basemono = ts->timer_expires_base;
-	bool timer_idle;
+	bool timer_idle = ts->tick_stopped;
 	u64 expires;
 
 	/* Make sure we won't be trying to stop it twice in a row. */
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 3a668060692e..2f69a485a070 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1999,13 +1999,18 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
  * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases
  * @basej:	base time jiffies
  * @basem:	base time clock monotonic
- * @idle:	pointer to store the value of timer_base->is_idle
+ * @idle:	pointer to store the value of timer_base->is_idle on return;
+ *		*idle contains the information whether tick was already stopped
  *
- * Returns the tick aligned clock monotonic time of the next pending
- * timer or KTIME_MAX if no timer is pending.
+ * Returns the tick aligned clock monotonic time of the next pending timer or
+ * KTIME_MAX if no timer is pending. When tick was already stopped KTIME_MAX is
+ * returned as well.
  */
 u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle)
 {
+	if (*idle)
+		return KTIME_MAX;
+
 	return __get_next_timer_interrupt(basej, basem, idle);
 }
 
-- 
2.39.2

[PATCH v10 05/20] timers: Introduce add_timer() variants which modify timer flags

[Anna-Maria Behnsen]

 timer flags

A timer might be used as a pinned timer (using add_timer_on()) and later on
as non-pinned timer using add_timer(). When the "NOHZ timer pull at expiry
model" is in place, the TIMER_PINNED flag is required to be used whenever a
timer needs to expire on a dedicated CPU. Otherwise the flag must not be
set if expiration on a dedicated CPU is not required.

add_timer_on()'s behavior will be changed during the preparation patches
for the "NOHZ timer pull at expiry model" to unconditionally set
TIMER_PINNED flag. To be able to clear/ set the flag when queueing a
timer, two variants of add_timer() are introduced.

This is a preparatory patch and has no functional change.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
v10: Commit message reworded as suggested by bigeasy

v9: Update documentation to match kernel-doc style (missing brackets after
    function names)

New in v6
---
 include/linux/timer.h |  2 ++
 kernel/time/timer.c   | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)

diff --git a/include/linux/timer.h b/include/linux/timer.h
index 26a545bb0153..404bb31a95c7 100644
--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -179,6 +179,8 @@ extern int timer_reduce(struct timer_list *timer, unsigned long expires);
 #define NEXT_TIMER_MAX_DELTA	((1UL << 30) - 1)
 
 extern void add_timer(struct timer_list *timer);
+extern void add_timer_local(struct timer_list *timer);
+extern void add_timer_global(struct timer_list *timer);
 
 extern int try_to_del_timer_sync(struct timer_list *timer);
 extern int timer_delete_sync(struct timer_list *timer);
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 2f69a485a070..3cf016d6fa59 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1245,6 +1245,40 @@ void add_timer(struct timer_list *timer)
 }
 EXPORT_SYMBOL(add_timer);
 
+/**
+ * add_timer_local() - Start a timer on the local CPU
+ * @timer:	The timer to be started
+ *
+ * Same as add_timer() except that the timer flag TIMER_PINNED is set.
+ *
+ * See add_timer() for further details.
+ */
+void add_timer_local(struct timer_list *timer)
+{
+	if (WARN_ON_ONCE(timer_pending(timer)))
+		return;
+	timer->flags |= TIMER_PINNED;
+	__mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING);
+}
+EXPORT_SYMBOL(add_timer_local);
+
+/**
+ * add_timer_global() - Start a timer without TIMER_PINNED flag set
+ * @timer:	The timer to be started
+ *
+ * Same as add_timer() except that the timer flag TIMER_PINNED is unset.
+ *
+ * See add_timer() for further details.
+ */
+void add_timer_global(struct timer_list *timer)
+{
+	if (WARN_ON_ONCE(timer_pending(timer)))
+		return;
+	timer->flags &= ~TIMER_PINNED;
+	__mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING);
+}
+EXPORT_SYMBOL(add_timer_global);
+
 /**
  * add_timer_on - Start a timer on a particular CPU
  * @timer:	The timer to be started
-- 
2.39.2

[PATCH v10 06/20] workqueue: Use global variant for add_timer()

[Anna-Maria Behnsen]

The implementation of the NOHZ pull at expiry model will change the timer
bases per CPU. Timers, that have to expire on a specific CPU, require the
TIMER_PINNED flag. If the CPU doesn't matter, the TIMER_PINNED flag must be
dropped. This is required for call sites which use the timer alternately as
pinned and not pinned timer like workqueues do.

Therefore use add_timer_global() to make sure TIMER_PINNED flag is dropped.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Acked-by: Tejun Heo <tj@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
---
v6:
  - New patch: As v6 provides unconditially setting TIMER_PINNED flag in
    add_timer_on() workqueue requires new add_timer_global() variant.
---
 kernel/workqueue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 2989b57e154a..31a6b0831318 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -1955,7 +1955,7 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq,
 	if (unlikely(cpu != WORK_CPU_UNBOUND))
 		add_timer_on(timer, cpu);
 	else
-		add_timer(timer);
+		add_timer_global(timer);
 }
 
 /**
-- 
2.39.2

[PATCH v10 07/20] timers: add_timer_on(): Make sure TIMER_PINNED flag is set

[Anna-Maria Behnsen]

 set

When adding a timer to the timer wheel using add_timer_on(), it is an
implicitly pinned timer. With the timer pull at expiry time model in place,
TIMER_PINNED flag is required to make sure timers end up in proper base.

Add TIMER_PINNED flag unconditionally when add_timer_on() is executed.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
---
 kernel/time/timer.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 3cf016d6fa59..fc4c406c9ec7 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1284,7 +1284,10 @@ EXPORT_SYMBOL(add_timer_global);
  * @timer:	The timer to be started
  * @cpu:	The CPU to start it on
  *
- * Same as add_timer() except that it starts the timer on the given CPU.
+ * Same as add_timer() except that it starts the timer on the given CPU and
+ * the TIMER_PINNED flag is set. When timer shouldn't be a pinned timer in
+ * the next round, add_timer_global() should be used instead as it unsets
+ * the TIMER_PINNED flag.
  *
  * See add_timer() for further details.
  */
@@ -1298,6 +1301,9 @@ void add_timer_on(struct timer_list *timer, int cpu)
 	if (WARN_ON_ONCE(timer_pending(timer)))
 		return;
 
+	/* Make sure timer flags have TIMER_PINNED flag set */
+	timer->flags |= TIMER_PINNED;
+
 	new_base = get_timer_cpu_base(timer->flags, cpu);
 
 	/*
-- 
2.39.2

[PATCH v10 08/20] timers: Ease code in run_local_timers()

[Anna-Maria Behnsen]

The logic for raising a softirq the way it is implemented right now, is
readable for two timer bases. When increasing numbers of timer bases, code
gets harder to read. With the introduction of the timer migration
hierarchy, there will be three timer bases.

Therefore ease the code. No functional change.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
---
v5: New patch to decrease patch size of follow up patches
---
 kernel/time/timer.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index fc4c406c9ec7..793848167852 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -2135,16 +2135,14 @@ static void run_local_timers(void)
 	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
 
 	hrtimer_run_queues();
-	/* Raise the softirq only if required. */
-	if (time_before(jiffies, base->next_expiry)) {
-		if (!IS_ENABLED(CONFIG_NO_HZ_COMMON))
-			return;
-		/* CPU is awake, so check the deferrable base. */
-		base++;
-		if (time_before(jiffies, base->next_expiry))
+
+	for (int i = 0; i < NR_BASES; i++, base++) {
+		/* Raise the softirq only if required. */
+		if (time_after_eq(jiffies, base->next_expiry)) {
+			raise_softirq(TIMER_SOFTIRQ);
 			return;
+		}
 	}
-	raise_softirq(TIMER_SOFTIRQ);
 }
 
 /*
-- 
2.39.2

[PATCH v10 09/20] timers: Split next timer interrupt logic

[Anna-Maria Behnsen]

Split the logic for getting next timer interrupt (no matter of recalculated
or already stored in base->next_expiry) into a separate function named
next_timer_interrupt(). Make it available to local call sites only.

No functional change.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
v10: Reword commit message

v9: Adapt to the fix for empty timer bases.
---
 kernel/time/timer.c | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 793848167852..4d6cf49a2fd1 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1951,12 +1951,29 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires)
 	return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC;
 }
 
+static unsigned long next_timer_interrupt(struct timer_base *base,
+					  unsigned long basej)
+{
+	if (base->next_expiry_recalc)
+		next_expiry_recalc(base);
+
+	/*
+	 * Move next_expiry for the empty base into the future to prevent an
+	 * unnecessary raise of the timer softirq when the next_expiry value
+	 * will be reached even if there is no timer pending.
+	 */
+	if (!base->timers_pending)
+		base->next_expiry = basej + NEXT_TIMER_MAX_DELTA;
+
+	return base->next_expiry;
+}
+
 static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 					     bool *idle)
 {
 	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
-	unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA;
 	u64 expires = KTIME_MAX;
+	unsigned long nextevt;
 
 	/*
 	 * Pretend that there is no timer pending if the cpu is offline.
@@ -1969,24 +1986,13 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 	}
 
 	raw_spin_lock(&base->lock);
-	if (base->next_expiry_recalc)
-		next_expiry_recalc(base);
+	nextevt = next_timer_interrupt(base, basej);
 
 	if (base->timers_pending) {
-		nextevt = base->next_expiry;
-
 		/* If we missed a tick already, force 0 delta */
 		if (time_before(nextevt, basej))
 			nextevt = basej;
 		expires = basem + (u64)(nextevt - basej) * TICK_NSEC;
-	} else {
-		/*
-		 * Move next_expiry for the empty base into the future to
-		 * prevent a unnecessary raise of the timer softirq when the
-		 * next_expiry value will be reached even if there is no timer
-		 * pending.
-		 */
-		base->next_expiry = nextevt;
 	}
 
 	/*
-- 
2.39.2

[PATCH v10 10/20] timers: Keep the pinned timers separate from the others

[Anna-Maria Behnsen]

Separate the storage space for pinned timers. Deferrable timers (doesn't
matter if pinned or non pinned) are still enqueued into their own base.

This is preparatory work for changing the NOHZ timer placement from a push
at enqueue time to a pull at expiry time model.

Originally-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
---
v10:
  - Simplify local_first check in __get_next_timer_interrupt() due to
    updated next_expiry values of empty timer bases

v9:
  - Update was required (change of preceding patches)

v6:
  - Drop set TIMER_PINNED flag in add_timer_on() and drop related
    warning. add_timer_on() fix is splitted into a separate
    patch. Therefore also drop "Reviewed-by" of Frederic Weisbecker

v5:
  - Add WARN_ONCE() in add_timer_on()
  - Decrease patch size by splitting into three patches (this patch and the
    two before)

v4:
  - split out logic to forward base clock into a helper function
    forward_base_clk() (Frederic)
  - ease the code in run_local_timers() and timer_clear_idle() (Frederic)
---
 kernel/time/timer.c | 85 +++++++++++++++++++++++++++++----------------
 1 file changed, 56 insertions(+), 29 deletions(-)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 4d6cf49a2fd1..5ca831444954 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -187,12 +187,18 @@ EXPORT_SYMBOL(jiffies_64);
 #define WHEEL_SIZE	(LVL_SIZE * LVL_DEPTH)
 
 #ifdef CONFIG_NO_HZ_COMMON
-# define NR_BASES	2
-# define BASE_STD	0
-# define BASE_DEF	1
+/*
+ * If multiple bases need to be locked, use the base ordering for lock
+ * nesting, i.e. lowest number first.
+ */
+# define NR_BASES	3
+# define BASE_LOCAL	0
+# define BASE_GLOBAL	1
+# define BASE_DEF	2
 #else
 # define NR_BASES	1
-# define BASE_STD	0
+# define BASE_LOCAL	0
+# define BASE_GLOBAL	0
 # define BASE_DEF	0
 #endif
 
@@ -899,7 +905,10 @@ static int detach_if_pending(struct timer_list *timer, struct timer_base *base,
 
 static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu)
 {
-	struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_STD], cpu);
+	int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL;
+	struct timer_base *base;
+
+	base = per_cpu_ptr(&timer_bases[index], cpu);
 
 	/*
 	 * If the timer is deferrable and NO_HZ_COMMON is set then we need
@@ -912,7 +921,10 @@ static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu)
 
 static inline struct timer_base *get_timer_this_cpu_base(u32 tflags)
 {
-	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
+	int index = tflags & TIMER_PINNED ? BASE_LOCAL : BASE_GLOBAL;
+	struct timer_base *base;
+
+	base = this_cpu_ptr(&timer_bases[index]);
 
 	/*
 	 * If the timer is deferrable and NO_HZ_COMMON is set then we need
@@ -1961,6 +1973,9 @@ static unsigned long next_timer_interrupt(struct timer_base *base,
 	 * Move next_expiry for the empty base into the future to prevent an
 	 * unnecessary raise of the timer softirq when the next_expiry value
 	 * will be reached even if there is no timer pending.
+	 *
+	 * This update is also required to make timer_base::next_expiry values
+	 * easy comparable to find out which base holds the first pending timer.
 	 */
 	if (!base->timers_pending)
 		base->next_expiry = basej + NEXT_TIMER_MAX_DELTA;
@@ -1971,9 +1986,10 @@ static unsigned long next_timer_interrupt(struct timer_base *base,
 static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 					     bool *idle)
 {
-	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
+	unsigned long nextevt, nextevt_local, nextevt_global;
+	struct timer_base *base_local, *base_global;
 	u64 expires = KTIME_MAX;
-	unsigned long nextevt;
+	bool local_first;
 
 	/*
 	 * Pretend that there is no timer pending if the cpu is offline.
@@ -1985,10 +2001,20 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 		return expires;
 	}
 
-	raw_spin_lock(&base->lock);
-	nextevt = next_timer_interrupt(base, basej);
+	base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]);
+	base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]);
 
-	if (base->timers_pending) {
+	raw_spin_lock(&base_local->lock);
+	raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING);
+
+	nextevt_local = next_timer_interrupt(base_local, basej);
+	nextevt_global = next_timer_interrupt(base_global, basej);
+
+	local_first = time_before_eq(nextevt_local, nextevt_global);
+
+	nextevt = local_first ? nextevt_local : nextevt_global;
+
+	if (base_local->timers_pending || base_global->timers_pending) {
 		/* If we missed a tick already, force 0 delta */
 		if (time_before(nextevt, basej))
 			nextevt = basej;
@@ -1999,31 +2025,31 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 	 * We have a fresh next event. Check whether we can forward the
 	 * base.
 	 */
-	__forward_timer_base(base, basej);
+	__forward_timer_base(base_local, basej);
+	__forward_timer_base(base_global, basej);
 
 	/*
 	 * Set base->is_idle only when caller is timer_base_try_to_set_idle()
 	 */
 	if (idle) {
 		/*
-		 * Base is idle if the next event is more than a tick away.
+		 * Bases are idle if the next event is more than a tick away.
 		 *
 		 * If the base is marked idle then any timer add operation must
 		 * forward the base clk itself to keep granularity small. This
-		 * idle logic is only maintained for the BASE_STD base,
-		 * deferrable timers may still see large granularity skew (by
-		 * design).
+		 * idle logic is only maintained for the BASE_LOCAL and
+		 * BASE_GLOBAL base, deferrable timers may still see large
+		 * granularity skew (by design).
 		 */
-		if (!base->is_idle) {
-			if (time_after(nextevt, basej + 1)) {
-				base->is_idle = true;
-				trace_timer_base_idle(true, base->cpu);
-			}
+		if (!base_local->is_idle && time_after(nextevt, basej + 1)) {
+			base_local->is_idle = base_global->is_idle = true;
+			trace_timer_base_idle(true, base_local->cpu);
 		}
-		*idle = base->is_idle;
+		*idle = base_local->is_idle;
 	}
 
-	raw_spin_unlock(&base->lock);
+	raw_spin_unlock(&base_global->lock);
+	raw_spin_unlock(&base_local->lock);
 
 	return cmp_next_hrtimer_event(basem, expires);
 }
@@ -2067,15 +2093,14 @@ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle)
  */
 void timer_clear_idle(void)
 {
-	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
-
 	/*
 	 * We do this unlocked. The worst outcome is a remote enqueue sending
 	 * a pointless IPI, but taking the lock would just make the window for
 	 * sending the IPI a few instructions smaller for the cost of taking
 	 * the lock in the exit from idle path.
 	 */
-	base->is_idle = false;
+	__this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false);
+	__this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false);
 	trace_timer_base_idle(false, smp_processor_id());
 }
 #endif
@@ -2126,11 +2151,13 @@ static inline void __run_timers(struct timer_base *base)
  */
 static __latent_entropy void run_timer_softirq(struct softirq_action *h)
 {
-	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
+	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]);
 
 	__run_timers(base);
-	if (IS_ENABLED(CONFIG_NO_HZ_COMMON))
+	if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) {
+		__run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL]));
 		__run_timers(this_cpu_ptr(&timer_bases[BASE_DEF]));
+	}
 }
 
 /*
@@ -2138,7 +2165,7 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h)
  */
 static void run_local_timers(void)
 {
-	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
+	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]);
 
 	hrtimer_run_queues();
 
-- 
2.39.2

[PATCH v10 11/20] timers: Retrieve next expiry of pinned/non-pinned timers separately

[Anna-Maria Behnsen]

 timers separately

For the conversion of the NOHZ timer placement to a pull at expiry time
model it's required to have separate expiry times for the pinned and the
non-pinned (movable) timers. Therefore struct timer_events is introduced.

No functional change

Originally-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
---
v10: Fix no functional change message

v9: Update was required (change of preceding patches)
---
 kernel/time/timer.c | 35 +++++++++++++++++++++++++++++++----
 1 file changed, 31 insertions(+), 4 deletions(-)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 5ca831444954..f119b44e44e0 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -221,6 +221,11 @@ struct timer_base {
 
 static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]);
 
+struct timer_events {
+	u64	local;
+	u64	global;
+};
+
 #ifdef CONFIG_NO_HZ_COMMON
 
 static DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
@@ -1986,10 +1991,11 @@ static unsigned long next_timer_interrupt(struct timer_base *base,
 static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 					     bool *idle)
 {
+	struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX };
 	unsigned long nextevt, nextevt_local, nextevt_global;
 	struct timer_base *base_local, *base_global;
-	u64 expires = KTIME_MAX;
 	bool local_first;
+	u64 expires;
 
 	/*
 	 * Pretend that there is no timer pending if the cpu is offline.
@@ -1998,7 +2004,7 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 	if (cpu_is_offline(smp_processor_id())) {
 		if (idle)
 			*idle = true;
-		return expires;
+		return tevt.local;
 	}
 
 	base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]);
@@ -2014,13 +2020,32 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 
 	nextevt = local_first ? nextevt_local : nextevt_global;
 
-	if (base_local->timers_pending || base_global->timers_pending) {
+	/*
+	 * If the @nextevt is at max. one tick away, use @nextevt and store
+	 * it in the local expiry value. The next global event is irrelevant in
+	 * this case and can be left as KTIME_MAX.
+	 */
+	if (time_before_eq(nextevt, basej + 1)) {
 		/* If we missed a tick already, force 0 delta */
 		if (time_before(nextevt, basej))
 			nextevt = basej;
-		expires = basem + (u64)(nextevt - basej) * TICK_NSEC;
+		tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC;
+		goto forward;
 	}
 
+	/*
+	 * Update tevt.* values:
+	 *
+	 * If the local queue expires first, then the global event can be
+	 * ignored. If the global queue is empty, nothing to do either.
+	 */
+	if (!local_first && base_global->timers_pending)
+		tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC;
+
+	if (base_local->timers_pending)
+		tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC;
+
+forward:
 	/*
 	 * We have a fresh next event. Check whether we can forward the
 	 * base.
@@ -2051,6 +2076,8 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 	raw_spin_unlock(&base_global->lock);
 	raw_spin_unlock(&base_local->lock);
 
+	expires = min_t(u64, tevt.local, tevt.global);
+
 	return cmp_next_hrtimer_event(basem, expires);
 }
 
-- 
2.39.2

[PATCH v10 12/20] timers: Split out "get next timer interrupt" functionality

[Anna-Maria Behnsen]

 functionality

The functionality for getting the next timer interrupt in
get_next_timer_interrupt() is split into a separate function
fetch_next_timer_interrupt() to be usable by other call sites.

This is preparatory work for the conversion of the NOHZ timer
placement to a pull at expiry time model. No functional change.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
---
v10: Update was required (change of preceding patches)
v9: Update was required (change of preceding patches)
v6: s/splitted/split
v5: Update commit message
v4: Fix typo in comment
---
 kernel/time/timer.c | 64 +++++++++++++++++++++++++++------------------
 1 file changed, 38 insertions(+), 26 deletions(-)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index f119b44e44e0..9fa759dd80f5 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1988,30 +1988,13 @@ static unsigned long next_timer_interrupt(struct timer_base *base,
 	return base->next_expiry;
 }
 
-static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
-					     bool *idle)
+static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem,
+						struct timer_base *base_local,
+						struct timer_base *base_global,
+						struct timer_events *tevt)
 {
-	struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX };
 	unsigned long nextevt, nextevt_local, nextevt_global;
-	struct timer_base *base_local, *base_global;
 	bool local_first;
-	u64 expires;
-
-	/*
-	 * Pretend that there is no timer pending if the cpu is offline.
-	 * Possible pending timers will be migrated later to an active cpu.
-	 */
-	if (cpu_is_offline(smp_processor_id())) {
-		if (idle)
-			*idle = true;
-		return tevt.local;
-	}
-
-	base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]);
-	base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]);
-
-	raw_spin_lock(&base_local->lock);
-	raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING);
 
 	nextevt_local = next_timer_interrupt(base_local, basej);
 	nextevt_global = next_timer_interrupt(base_global, basej);
@@ -2029,8 +2012,8 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 		/* If we missed a tick already, force 0 delta */
 		if (time_before(nextevt, basej))
 			nextevt = basej;
-		tevt.local = basem + (u64)(nextevt - basej) * TICK_NSEC;
-		goto forward;
+		tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC;
+		return nextevt;
 	}
 
 	/*
@@ -2040,12 +2023,41 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 	 * ignored. If the global queue is empty, nothing to do either.
 	 */
 	if (!local_first && base_global->timers_pending)
-		tevt.global = basem + (u64)(nextevt_global - basej) * TICK_NSEC;
+		tevt->global = basem + (u64)(nextevt_global - basej) * TICK_NSEC;
 
 	if (base_local->timers_pending)
-		tevt.local = basem + (u64)(nextevt_local - basej) * TICK_NSEC;
+		tevt->local = basem + (u64)(nextevt_local - basej) * TICK_NSEC;
+
+	return nextevt;
+}
+
+static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
+					     bool *idle)
+{
+	struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX };
+	struct timer_base *base_local, *base_global;
+	unsigned long nextevt;
+	u64 expires;
+
+	/*
+	 * Pretend that there is no timer pending if the cpu is offline.
+	 * Possible pending timers will be migrated later to an active cpu.
+	 */
+	if (cpu_is_offline(smp_processor_id())) {
+		if (idle)
+			*idle = true;
+		return tevt.local;
+	}
+
+	base_local = this_cpu_ptr(&timer_bases[BASE_LOCAL]);
+	base_global = this_cpu_ptr(&timer_bases[BASE_GLOBAL]);
+
+	raw_spin_lock(&base_local->lock);
+	raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING);
+
+	nextevt = fetch_next_timer_interrupt(basej, basem, base_local,
+					     base_global, &tevt);
 
-forward:
 	/*
 	 * We have a fresh next event. Check whether we can forward the
 	 * base.
-- 
2.39.2

[PATCH v10 13/20] timers: Add get next timer interrupt functionality for remote CPUs

[Anna-Maria Behnsen]

 remote CPUs

To prepare for the conversion of the NOHZ timer placement to a pull at
expiry time model it's required to have functionality available getting the
next timer interrupt on a remote CPU.

Locking of the timer bases and getting the information for the next timer
interrupt functionality is split into separate functions. This is required
to be compliant with lock ordering when the new model is in place.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
---
v10:
 - sparse annotations for locks

v8:
 - Update comment

v7:
 - Move functions into CONFIG_SMP && CONFIG_NO_HZ_COMMON section
 - change lock, fetch functions to be unconditional
 - split out unlock function into a separate function

v6:
 - introduce timer_lock_remote_bases() to fix race
---
 kernel/time/tick-internal.h | 10 +++++
 kernel/time/timer.c         | 80 ++++++++++++++++++++++++++++++++++---
 2 files changed, 85 insertions(+), 5 deletions(-)

diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index 47df30b871e4..8b0c28edbd09 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -8,6 +8,11 @@
 #include "timekeeping.h"
 #include "tick-sched.h"
 
+struct timer_events {
+	u64	local;
+	u64	global;
+};
+
 #ifdef CONFIG_GENERIC_CLOCKEVENTS
 
 # define TICK_DO_TIMER_NONE	-1
@@ -154,6 +159,11 @@ extern unsigned long tick_nohz_active;
 extern void timers_update_nohz(void);
 # ifdef CONFIG_SMP
 extern struct static_key_false timers_migration_enabled;
+extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem,
+					      struct timer_events *tevt,
+					      unsigned int cpu);
+extern void timer_lock_remote_bases(unsigned int cpu);
+extern void timer_unlock_remote_bases(unsigned int cpu);
 # endif
 #else /* CONFIG_NO_HZ_COMMON */
 static inline void timers_update_nohz(void) { }
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 9fa759dd80f5..3e2adfc15f3a 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -221,11 +221,6 @@ struct timer_base {
 
 static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]);
 
-struct timer_events {
-	u64	local;
-	u64	global;
-};
-
 #ifdef CONFIG_NO_HZ_COMMON
 
 static DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
@@ -2031,6 +2026,81 @@ static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem,
 	return nextevt;
 }
 
+# ifdef CONFIG_SMP
+/**
+ * fetch_next_timer_interrupt_remote() - Store next timers into @tevt
+ * @basej:	base time jiffies
+ * @basem:	base time clock monotonic
+ * @tevt:	Pointer to the storage for the expiry values
+ * @cpu:	Remote CPU
+ *
+ * Stores the next pending local and global timer expiry values in the
+ * struct pointed to by @tevt. If a queue is empty the corresponding
+ * field is set to KTIME_MAX. If local event expires before global
+ * event, global event is set to KTIME_MAX as well.
+ *
+ * Caller needs to make sure timer base locks are held (use
+ * timer_lock_remote_bases() for this purpose).
+ */
+void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem,
+				       struct timer_events *tevt,
+				       unsigned int cpu)
+{
+	struct timer_base *base_local, *base_global;
+
+	/* Preset local / global events */
+	tevt->local = tevt->global = KTIME_MAX;
+
+	base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu);
+	base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu);
+
+	lockdep_assert_held(&base_local->lock);
+	lockdep_assert_held(&base_global->lock);
+
+	fetch_next_timer_interrupt(basej, basem, base_local, base_global, tevt);
+}
+
+/**
+ * timer_unlock_remote_bases - unlock timer bases of cpu
+ * @cpu:	Remote CPU
+ *
+ * Unlocks the remote timer bases.
+ */
+void timer_unlock_remote_bases(unsigned int cpu)
+	__releases(timer_bases[BASE_LOCAL]->lock)
+	__releases(timer_bases[BASE_GLOBAL]->lock)
+{
+	struct timer_base *base_local, *base_global;
+
+	base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu);
+	base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu);
+
+	raw_spin_unlock(&base_global->lock);
+	raw_spin_unlock(&base_local->lock);
+}
+
+/**
+ * timer_lock_remote_bases - lock timer bases of cpu
+ * @cpu:	Remote CPU
+ *
+ * Locks the remote timer bases.
+ */
+void timer_lock_remote_bases(unsigned int cpu)
+	__acquires(timer_bases[BASE_LOCAL]->lock)
+	__acquires(timer_bases[BASE_GLOBAL]->lock)
+{
+	struct timer_base *base_local, *base_global;
+
+	base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu);
+	base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu);
+
+	lockdep_assert_irqs_disabled();
+
+	raw_spin_lock(&base_local->lock);
+	raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING);
+}
+# endif /* CONFIG_SMP */
+
 static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 					     bool *idle)
 {
-- 
2.39.2

[PATCH v10 14/20] timers: Restructure internal locking

[Anna-Maria Behnsen]

From: "Richard Cochran (linutronix GmbH)" <richardcochran@gmail.com>

Move the locking out from __run_timers() to the call sites, so the
protected section can be extended at the call site. Preparatory patch for
changing the NOHZ timer placement to a pull at expiry time model.

No functional change.

Signed-off-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
 kernel/time/timer.c | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 3e2adfc15f3a..ccb2a5c03b87 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -2223,11 +2223,7 @@ static inline void __run_timers(struct timer_base *base)
 	struct hlist_head heads[LVL_DEPTH];
 	int levels;
 
-	if (time_before(jiffies, base->next_expiry))
-		return;
-
-	timer_base_lock_expiry(base);
-	raw_spin_lock_irq(&base->lock);
+	lockdep_assert_held(&base->lock);
 
 	while (time_after_eq(jiffies, base->clk) &&
 	       time_after_eq(jiffies, base->next_expiry)) {
@@ -2251,21 +2247,36 @@ static inline void __run_timers(struct timer_base *base)
 		while (levels--)
 			expire_timers(base, heads + levels);
 	}
+}
+
+static void __run_timer_base(struct timer_base *base)
+{
+	if (time_before(jiffies, base->next_expiry))
+		return;
+
+	timer_base_lock_expiry(base);
+	raw_spin_lock_irq(&base->lock);
+	__run_timers(base);
 	raw_spin_unlock_irq(&base->lock);
 	timer_base_unlock_expiry(base);
 }
 
+static void run_timer_base(int index)
+{
+	struct timer_base *base = this_cpu_ptr(&timer_bases[index]);
+
+	__run_timer_base(base);
+}
+
 /*
  * This function runs timers and the timer-tq in bottom half context.
  */
 static __latent_entropy void run_timer_softirq(struct softirq_action *h)
 {
-	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_LOCAL]);
-
-	__run_timers(base);
+	run_timer_base(BASE_LOCAL);
 	if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) {
-		__run_timers(this_cpu_ptr(&timer_bases[BASE_GLOBAL]));
-		__run_timers(this_cpu_ptr(&timer_bases[BASE_DEF]));
+		run_timer_base(BASE_GLOBAL);
+		run_timer_base(BASE_DEF);
 	}
 }
 
-- 
2.39.2

[PATCH v10 15/20] timers: Check if timers base is handled already

[Anna-Maria Behnsen]

Due to the conversion of the NOHZ timer placement to a pull at expiry
time model, the per CPU timer bases with non pinned timers are no
longer handled only by the local CPU. In case a remote CPU already
expires the non pinned timers base of the local CPU, nothing more
needs to be done by the local CPU. A check at the begin of the expire
timers routine is required, because timer base lock is dropped before
executing the timer callback function.

This is a preparatory work, but has no functional impact right now.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
v10: s/cpu/CPU/ in commit message

v6: Drop double negation
---
 kernel/time/timer.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index ccb2a5c03b87..eb99297a96fe 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -2225,6 +2225,9 @@ static inline void __run_timers(struct timer_base *base)
 
 	lockdep_assert_held(&base->lock);
 
+	if (base->running_timer)
+		return;
+
 	while (time_after_eq(jiffies, base->clk) &&
 	       time_after_eq(jiffies, base->next_expiry)) {
 		levels = collect_expired_timers(base, heads);
-- 
2.39.2

[PATCH v10 16/20] tick/sched: Split out jiffies update helper function

[Anna-Maria Behnsen]

From: "Richard Cochran (linutronix GmbH)" <richardcochran@gmail.com>

The logic to get the time of the last jiffies update will be needed by
the timer pull model as well.

Move the code into a global function in anticipation of the new caller.

No functional change.

Signed-off-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
 kernel/time/tick-internal.h |  1 +
 kernel/time/tick-sched.c    | 18 +++++++++++++++---
 2 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index 8b0c28edbd09..ccf39befde85 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -157,6 +157,7 @@ static inline void tick_nohz_init(void) { }
 #ifdef CONFIG_NO_HZ_COMMON
 extern unsigned long tick_nohz_active;
 extern void timers_update_nohz(void);
+extern u64 get_jiffies_update(unsigned long *basej);
 # ifdef CONFIG_SMP
 extern struct static_key_false timers_migration_enabled;
 extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem,
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 27f1a2ae7f39..a3eb5165d439 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -799,18 +799,30 @@ static inline bool local_timer_softirq_pending(void)
 	return local_softirq_pending() & BIT(TIMER_SOFTIRQ);
 }
 
-static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu)
+/*
+ * Read jiffies and the time when jiffies were updated last
+ */
+u64 get_jiffies_update(unsigned long *basej)
 {
-	u64 basemono, next_tick, delta, expires;
 	unsigned long basejiff;
 	unsigned int seq;
+	u64 basemono;
 
-	/* Read jiffies and the time when jiffies were updated last */
 	do {
 		seq = read_seqcount_begin(&jiffies_seq);
 		basemono = last_jiffies_update;
 		basejiff = jiffies;
 	} while (read_seqcount_retry(&jiffies_seq, seq));
+	*basej = basejiff;
+	return basemono;
+}
+
+static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu)
+{
+	u64 basemono, next_tick, delta, expires;
+	unsigned long basejiff;
+
+	basemono = get_jiffies_update(&basejiff);
 	ts->last_jiffies = basejiff;
 	ts->timer_expires_base = basemono;
 
-- 
2.39.2

[PATCH v10 17/20] timers: Introduce function to check timer base is_idle flag

[Anna-Maria Behnsen]

 flag

To prepare for the conversion of the NOHZ timer placement to a pull at
expiry time model it's required to have a function that returns the value
of the is_idle flag of the timer base to keep the hierarchy states during
online in sync with timer base state.

No functional change.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
v10: Fix fallout of 0day: Move function definition of timer_base_is_idle()
     into SMP && NO_HZ_COMMON ifdef section

v9: new in v9
---
 kernel/time/tick-internal.h |  1 +
 kernel/time/timer.c         | 10 ++++++++++
 2 files changed, 11 insertions(+)

diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index ccf39befde85..7e3090109e33 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -165,6 +165,7 @@ extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem,
 					      unsigned int cpu);
 extern void timer_lock_remote_bases(unsigned int cpu);
 extern void timer_unlock_remote_bases(unsigned int cpu);
+extern bool timer_base_is_idle(void);
 # endif
 #else /* CONFIG_NO_HZ_COMMON */
 static inline void timers_update_nohz(void) { }
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index eb99297a96fe..3c49d8fdfd53 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -2099,6 +2099,16 @@ void timer_lock_remote_bases(unsigned int cpu)
 	raw_spin_lock(&base_local->lock);
 	raw_spin_lock_nested(&base_global->lock, SINGLE_DEPTH_NESTING);
 }
+
+/**
+ * timer_base_is_idle() - Return whether timer base is set idle
+ *
+ * Returns value of local timer base is_idle value.
+ */
+bool timer_base_is_idle(void)
+{
+	return __this_cpu_read(timer_bases[BASE_LOCAL].is_idle);
+}
 # endif /* CONFIG_SMP */
 
 static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
-- 
2.39.2

[PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Placing timers at enqueue time on a target CPU based on dubious heuristics
does not make any sense:

 1) Most timer wheel timers are canceled or rearmed before they expire.

 2) The heuristics to predict which CPU will be busy when the timer expires
    are wrong by definition.

So placing the timers at enqueue wastes precious cycles.

The proper solution to this problem is to always queue the timers on the
local CPU and allow the non pinned timers to be pulled onto a busy CPU at
expiry time.

Therefore split the timer storage into local pinned and global timers:
Local pinned timers are always expired on the CPU on which they have been
queued. Global timers can be expired on any CPU.

As long as a CPU is busy it expires both local and global timers. When a
CPU goes idle it arms for the first expiring local timer. If the first
expiring pinned (local) timer is before the first expiring movable timer,
then no action is required because the CPU will wake up before the first
movable timer expires. If the first expiring movable timer is before the
first expiring pinned (local) timer, then this timer is queued into an idle
timerqueue and eventually expired by another active CPU.

To avoid global locking the timerqueues are implemented as a hierarchy. The
lowest level of the hierarchy holds the CPUs. The CPUs are associated to
groups of 8, which are separated per node. If more than one CPU group
exist, then a second level in the hierarchy collects the groups. Depending
on the size of the system more than 2 levels are required. Each group has a
"migrator" which checks the timerqueue during the tick for remote expirable
timers.

If the last CPU in a group goes idle it reports the first expiring event in
the group up to the next group(s) in the hierarchy. If the last CPU goes
idle it arms its timer for the first system wide expiring timer to ensure
that no timer event is missed.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
v10:
  - Fix fallout of 0day (s/unsigned int/int)
  - Adress review remarks of bigeasy

v9:
  - Adapt to the changes of the preceding patches
  - Fix state inconsitency (when timer base is idle, cpu must also be
    marked as idle in hierarchy)
  - Make sure new timers are considered, when timer base is idle and a
    timer is enqueued into global queue (e.g. during interrupt) ->
    timer_use_tmigr()
  - Changes which are required due to the timer code change of marking the
    timer base idle in tick_nohz_stop_tick()

v8:
  - Review of Frederic:
    - Fix hotplug race (introduction of wakeup_recalc)
    - Make wakeup and wakeup_recalc logic consistent all over the place
    - Fix child/group state race and read it with locks held
  - Add more clarifying comments
  - Fix grammar all over the place
  - change integers which act as boolean value into bool
  - rewrite condition in tmigr_check_migrator() without negation
  - Improve update events logic with a check of the first event
  - Implement a quick forecast which is called when
    get_next_timer_interrupt() is executed.

v7:
  - Review remarks of Frederic and bigeasy:
    - change logic in tmigr_handle_remote_cpu()
    - s/kzalloc/kcalloc
    - move timer_expire_remote() into NO_HZ_COMMON && SMP config section
    - drop DBG_BUG_ON() makro and use only WARN_ON_ONCE()
    - remove leftovers from sibling logic during setup
  - Move timer_expire_remote() into tick-internal.h
  - Add documentation section about "Required event and timerqueue update
    after remote expiry"
  - Fix fallout of kernel test robot

v6:
  - Fix typos
  - Review remarks of Peter Zijlstra (locking, struct member cleanup, use
    atomic_try_cmpxchg(), update struct member descriptions)
  - Fix race in tmigr_handle_remote_cpu() (Frederic Weisbecker)

v5:
  - Review remarks of Frederic
  - Return nextevt when CPU is marked offline in timer migration hierarchy
    instead of KTIME_MAX
  - Fix update of group events issue, after remote expiring

v4:
  - Fold typo fix in comment into proper patch "timer: Split out "get next
    timer interrupt" functionality"
  - Update wrong comment for tmigr_state union definition
  - Fix fallout of kernel test robot
---
 create mode 100644 kernel/time/timer_migration.c
---
 include/linux/cpuhotplug.h    |    1 +
 kernel/time/Makefile          |    3 +
 kernel/time/tick-internal.h   |    1 +
 kernel/time/timer.c           |  113 ++-
 kernel/time/timer_migration.c | 1667 +++++++++++++++++++++++++++++++++
 kernel/time/timer_migration.h |  147 +++
 6 files changed, 1924 insertions(+), 8 deletions(-)
 create mode 100644 kernel/time/timer_migration.c
 create mode 100644 kernel/time/timer_migration.h

diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
index efc0c0b07efb..85a78c5a7c01 100644
--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h
@@ -245,6 +245,7 @@ enum cpuhp_state {
 	CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE,
 	CPUHP_AP_PERF_POWERPC_HV_GPCI_ONLINE,
 	CPUHP_AP_PERF_CSKY_ONLINE,
+	CPUHP_AP_TMIGR_ONLINE,
 	CPUHP_AP_WATCHDOG_ONLINE,
 	CPUHP_AP_WORKQUEUE_ONLINE,
 	CPUHP_AP_RANDOM_ONLINE,
diff --git a/kernel/time/Makefile b/kernel/time/Makefile
index 7e875e63ff3b..4af2a264a160 100644
--- a/kernel/time/Makefile
+++ b/kernel/time/Makefile
@@ -17,6 +17,9 @@ endif
 obj-$(CONFIG_GENERIC_SCHED_CLOCK)		+= sched_clock.o
 obj-$(CONFIG_TICK_ONESHOT)			+= tick-oneshot.o tick-sched.o
 obj-$(CONFIG_LEGACY_TIMER_TICK)			+= tick-legacy.o
+ifeq ($(CONFIG_SMP),y)
+ obj-$(CONFIG_NO_HZ_COMMON)			+= timer_migration.o
+endif
 obj-$(CONFIG_HAVE_GENERIC_VDSO)			+= vsyscall.o
 obj-$(CONFIG_DEBUG_FS)				+= timekeeping_debug.o
 obj-$(CONFIG_TEST_UDELAY)			+= test_udelay.o
diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index 7e3090109e33..a3243c4ac45f 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -166,6 +166,7 @@ extern void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem,
 extern void timer_lock_remote_bases(unsigned int cpu);
 extern void timer_unlock_remote_bases(unsigned int cpu);
 extern bool timer_base_is_idle(void);
+extern void timer_expire_remote(unsigned int cpu);
 # endif
 #else /* CONFIG_NO_HZ_COMMON */
 static inline void timers_update_nohz(void) { }
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 3c49d8fdfd53..a52f68be8dfd 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -53,6 +53,7 @@
 #include <asm/io.h>
 
 #include "tick-internal.h"
+#include "timer_migration.h"
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/timer.h>
@@ -2109,6 +2110,64 @@ bool timer_base_is_idle(void)
 {
 	return __this_cpu_read(timer_bases[BASE_LOCAL].is_idle);
 }
+
+static void __run_timer_base(struct timer_base *base);
+
+/**
+ * timer_expire_remote() - expire global timers of cpu
+ * @cpu:	Remote CPU
+ *
+ * Expire timers of global base of remote CPU.
+ */
+void timer_expire_remote(unsigned int cpu)
+{
+	struct timer_base *base = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu);
+
+	__run_timer_base(base);
+}
+
+static void timer_use_tmigr(unsigned long basej, u64 basem,
+			    unsigned long *nextevt, bool *tick_stop_path,
+			    bool timer_base_idle, struct timer_events *tevt)
+{
+	u64 next_tmigr;
+
+	if (timer_base_idle)
+		next_tmigr = tmigr_cpu_new_timer(tevt->global);
+	else if (tick_stop_path)
+		next_tmigr = tmigr_cpu_deactivate(tevt->global);
+	else
+		next_tmigr = tmigr_quick_check();
+
+	/*
+	 * If the CPU is the last going idle in timer migration hierarchy, make
+	 * sure the CPU will wake up in time to handle remote timers.
+	 * next_tmigr == KTIME_MAX if other CPUs are still active.
+	 */
+	if (next_tmigr < tevt->local) {
+		u64 tmp;
+
+		/* If we missed a tick already, force 0 delta */
+		if (next_tmigr < basem)
+			next_tmigr = basem;
+
+		tmp = div_u64(next_tmigr - basem, TICK_NSEC);
+
+		*nextevt = basej + (unsigned long)tmp;
+		tevt->local = next_tmigr;
+	}
+}
+# else
+static void timer_use_tmigr(unsigned long basej, u64 basem,
+			    unsigned long *nextevt, bool *tick_stop_path,
+			    bool timer_base_idle, struct timer_events *tevt)
+{
+	/*
+	 * Make sure first event is written into tevt->local to not miss a
+	 * timer on !SMP systems.
+	 */
+	tevt->local = min_t(u64, tevt->local, tevt->global);
+}
 # endif /* CONFIG_SMP */
 
 static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
@@ -2117,7 +2176,7 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 	struct timer_events tevt = { .local = KTIME_MAX, .global = KTIME_MAX };
 	struct timer_base *base_local, *base_global;
 	unsigned long nextevt;
-	u64 expires;
+	bool idle_is_possible;
 
 	/*
 	 * Pretend that there is no timer pending if the cpu is offline.
@@ -2138,6 +2197,22 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 	nextevt = fetch_next_timer_interrupt(basej, basem, base_local,
 					     base_global, &tevt);
 
+	/*
+	 * If the next event is only one jiffie ahead there is no need to call
+	 * timer migration hierarchy related functions. The value for the next
+	 * global timer in @tevt struct equals then KTIME_MAX. This is also
+	 * true, when the timer base is idle.
+	 *
+	 * The proper timer migration hierarchy function depends on the callsite
+	 * and whether timer base is idle or not. @nextevt will be updated when
+	 * this CPU needs to handle the first timer migration hierarchy
+	 * event. See timer_use_tmigr() for detailed information.
+	 */
+	idle_is_possible = time_after(nextevt, basej + 1);
+	if (idle_is_possible)
+		timer_use_tmigr(basej, basem, &nextevt, idle,
+				base_local->is_idle, &tevt);
+
 	/*
 	 * We have a fresh next event. Check whether we can forward the
 	 * base.
@@ -2150,7 +2225,10 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 	 */
 	if (idle) {
 		/*
-		 * Bases are idle if the next event is more than a tick away.
+		 * Bases are idle if the next event is more than a tick
+		 * away. Caution: @nextevt could have changed by enqueueing a
+		 * global timer into timer migration hierarchy. Therefore a new
+		 * check is required here.
 		 *
 		 * If the base is marked idle then any timer add operation must
 		 * forward the base clk itself to keep granularity small. This
@@ -2163,14 +2241,23 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 			trace_timer_base_idle(true, base_local->cpu);
 		}
 		*idle = base_local->is_idle;
+
+		/*
+		 * When timer base is not set idle, undo the effect of
+		 * tmigr_cpu_deactivate() to prevent inconsitent states - active
+		 * timer base but inactive timer migration hierarchy.
+		 *
+		 * When timer base was already marked idle, nothing will be
+		 * changed here.
+		 */
+		if (!base_local->is_idle && idle_is_possible)
+			tmigr_cpu_activate();
 	}
 
 	raw_spin_unlock(&base_global->lock);
 	raw_spin_unlock(&base_local->lock);
 
-	expires = min_t(u64, tevt.local, tevt.global);
-
-	return cmp_next_hrtimer_event(basem, expires);
+	return cmp_next_hrtimer_event(basem, tevt.local);
 }
 
 /**
@@ -2178,8 +2265,11 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
  * @basej:	base time jiffies
  * @basem:	base time clock monotonic
  *
- * Returns the tick aligned clock monotonic time of the next pending
- * timer or KTIME_MAX if no timer is pending.
+ * Returns the tick aligned clock monotonic time of the next pending timer or
+ * KTIME_MAX if no timer is pending. If timer of global base was queued into
+ * timer migration hierarchy, first global timer is not taken into account. If
+ * it was the last CPU of timer migration hierarchy going idle, first global
+ * event is taken into account.
  */
 u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
 {
@@ -2221,6 +2311,9 @@ void timer_clear_idle(void)
 	__this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false);
 	__this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false);
 	trace_timer_base_idle(false, smp_processor_id());
+
+	/* Activate without holding the timer_base->lock */
+	tmigr_cpu_activate();
 }
 #endif
 
@@ -2290,6 +2383,9 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h)
 	if (IS_ENABLED(CONFIG_NO_HZ_COMMON)) {
 		run_timer_base(BASE_GLOBAL);
 		run_timer_base(BASE_DEF);
+
+		if (is_timers_nohz_active())
+			tmigr_handle_remote();
 	}
 }
 
@@ -2304,7 +2400,8 @@ static void run_local_timers(void)
 
 	for (int i = 0; i < NR_BASES; i++, base++) {
 		/* Raise the softirq only if required. */
-		if (time_after_eq(jiffies, base->next_expiry)) {
+		if (time_after_eq(jiffies, base->next_expiry) ||
+		    (i == BASE_DEF && tmigr_requires_handle_remote())) {
 			raise_softirq(TIMER_SOFTIRQ);
 			return;
 		}
diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c
new file mode 100644
index 000000000000..de1905b0bae7
--- /dev/null
+++ b/kernel/time/timer_migration.c
@@ -0,0 +1,1667 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Infrastructure for migratable timers
+ *
+ * Copyright(C) 2022 linutronix GmbH
+ */
+#include <linux/cpuhotplug.h>
+#include <linux/slab.h>
+#include <linux/smp.h>
+#include <linux/spinlock.h>
+#include <linux/timerqueue.h>
+#include <trace/events/ipi.h>
+
+#include "timer_migration.h"
+#include "tick-internal.h"
+
+/*
+ * The timer migration mechanism is built on a hierarchy of groups. The
+ * lowest level group contains CPUs, the next level groups of CPU groups
+ * and so forth. The CPU groups are kept per node so for the normal case
+ * lock contention won't happen across nodes. Depending on the number of
+ * CPUs per node even the next level might be kept as groups of CPU groups
+ * per node and only the levels above cross the node topology.
+ *
+ * Example topology for a two node system with 24 CPUs each.
+ *
+ * LVL 2                           [GRP2:0]
+ *                              GRP1:0 = GRP1:M
+ *
+ * LVL 1            [GRP1:0]                      [GRP1:1]
+ *               GRP0:0 - GRP0:2               GRP0:3 - GRP0:5
+ *
+ * LVL 0  [GRP0:0]  [GRP0:1]  [GRP0:2]  [GRP0:3]  [GRP0:4]  [GRP0:5]
+ * CPUS     0-7       8-15      16-23     24-31     32-39     40-47
+ *
+ * The groups hold a timer queue of events sorted by expiry time. These
+ * queues are updated when CPUs go in idle. When they come out of idle
+ * ignore flag of events is set.
+ *
+ * Each group has a designated migrator CPU/group as long as a CPU/group is
+ * active in the group. This designated role is necessary to avoid that all
+ * active CPUs in a group try to migrate expired timers from other CPUs,
+ * which would result in massive lock bouncing.
+ *
+ * When a CPU is awake, it checks in it's own timer tick the group
+ * hierarchy up to the point where it is assigned the migrator role or if
+ * no CPU is active, it also checks the groups where no migrator is set
+ * (TMIGR_NONE).
+ *
+ * If it finds expired timers in one of the group queues it pulls them over
+ * from the idle CPU and runs the timer function. After that it updates the
+ * group and the parent groups if required.
+ *
+ * CPUs which go idle arm their CPU local timer hardware for the next local
+ * (pinned) timer event. If the next migratable timer expires after the
+ * next local timer or the CPU has no migratable timer pending then the
+ * CPU does not queue an event in the LVL0 group. If the next migratable
+ * timer expires before the next local timer then the CPU queues that timer
+ * in the LVL0 group. In both cases the CPU marks itself idle in the LVL0
+ * group.
+ *
+ * When CPU comes out of idle and when a group has at least a single active
+ * child, the ignore flag of the tmigr_event is set. This indicates, that
+ * the event is ignored even if it is still enqueued in the parent groups
+ * timer queue. It will be removed when touching the timer queue the next
+ * time. This spares locking in active path as the lock protects (after
+ * setup) only event information. For more information about locking,
+ * please read the section "Locking rules".
+ *
+ * If the CPU is the migrator of the group then it delegates that role to
+ * the next active CPU in the group or sets migrator to TMIGR_NONE when
+ * there is no active CPU in the group. This delegation needs to be
+ * propagated up the hierarchy so hand over from other leaves can happen at
+ * all hierarchy levels w/o doing a search.
+ *
+ * When the last CPU in the system goes idle, then it drops all migrator
+ * duties up to the top level of the hierarchy (LVL2 in the example). It
+ * then has to make sure, that it arms it's own local hardware timer for
+ * the earliest event in the system.
+ *
+ *
+ * Lifetime rules:
+ * ---------------
+ *
+ * The groups are built up at init time or when CPUs come online. They are
+ * not destroyed when a group becomes empty due to offlining. The group
+ * just won't participate in the hierarchy management anymore. Destroying
+ * groups would result in interesting race conditions which would just make
+ * the whole mechanism slow and complex.
+ *
+ *
+ * Locking rules:
+ * --------------
+ *
+ * For setting up new groups and handling events it's required to lock both
+ * child and parent group. The lock ordering is always bottom up. This also
+ * includes the per CPU locks in struct tmigr_cpu. For updating the migrator and
+ * active CPU/group information atomic_try_cmpxchg() is used instead and only
+ * the per CPU tmigr_cpu->lock is held.
+ *
+ * During the setup of groups tmigr_level_list is required. It is protected by
+ * @tmigr_mutex.
+ *
+ * When @timer_base->lock as well as tmigr related locks are required, the lock
+ * ordering is: first @timer_base->lock, afterwards tmigr related locks.
+ *
+ *
+ * Protection of the tmigr group state information:
+ * ------------------------------------------------
+ *
+ * The state information with the list of active children and migrator needs to
+ * be protected by a sequence counter. It prevents a race when updates in child
+ * groups are propagated in changed order. The state update is performed
+ * lockless and group wise. The following scenario describes what happens
+ * without updating the sequence counter:
+ *
+ * Therefore, let's take three groups and four CPUs (CPU2 and CPU3 as well
+ * as GRP0:1 will not change during the scenario):
+ *
+ *    LVL 1            [GRP1:0]
+ *                     migrator = GRP0:1
+ *                     active   = GRP0:0, GRP0:1
+ *                   /                \
+ *    LVL 0  [GRP0:0]                  [GRP0:1]
+ *           migrator = CPU0           migrator = CPU2
+ *           active   = CPU0           active   = CPU2
+ *              /         \                /         \
+ *    CPUs     0           1              2           3
+ *             active      idle           active      idle
+ *
+ *
+ * 1. CPU0 goes idle. As the update is performed group wise, in the first step
+ *    only GRP0:0 is updated. The update of GRP1:0 is pending as CPU0 has to
+ *    walk the hierarchy.
+ *
+ *    LVL 1            [GRP1:0]
+ *                     migrator = GRP0:1
+ *                     active   = GRP0:0, GRP0:1
+ *                   /                \
+ *    LVL 0  [GRP0:0]                  [GRP0:1]
+ *       --> migrator = TMIGR_NONE     migrator = CPU2
+ *       --> active   =                active   = CPU2
+ *              /         \                /         \
+ *    CPUs     0           1              2           3
+ *         --> idle        idle           active      idle
+ *
+ * 2. While CPU0 goes idle and continues to update the state, CPU1 comes out of
+ *    idle. CPU1 updates GRP0:0. The update for GRP1:0 is pending as CPU1 also
+ *    has to the hierarchy. Both CPUs (CPU0 and CPU1) now walk the hierarchy to
+ *    perform the needed update from their point of view. The currently visible
+ *    state looks the following:
+ *
+ *    LVL 1            [GRP1:0]
+ *                     migrator = GRP0:1
+ *                     active   = GRP0:0, GRP0:1
+ *                   /                \
+ *    LVL 0  [GRP0:0]                  [GRP0:1]
+ *       --> migrator = CPU1           migrator = CPU2
+ *       --> active   = CPU1           active   = CPU2
+ *              /         \                /         \
+ *    CPUs     0           1              2           3
+ *             idle    --> active         active      idle
+ *
+ * 3. Here is the race condition: CPU1 managed to propagate its changes (from
+ *    step 2) through the hierarchy to GRP1:0 before CPU0 (step 1) did. The
+ *    active members of GRP1:0 remain unchanged after the update since it is
+ *    still valid from CPU1 current point of view:
+ *
+ *    LVL 1            [GRP1:0]
+ *                 --> migrator = GRP0:1
+ *                 --> active   = GRP0:0, GRP0:1
+ *                   /                \
+ *    LVL 0  [GRP0:0]                  [GRP0:1]
+ *           migrator = CPU1           migrator = CPU2
+ *           active   = CPU1           active   = CPU2
+ *              /         \                /         \
+ *    CPUs     0           1              2           3
+ *             idle        active         active      idle
+ *
+ * 4. Now CPU0 finally propagates its changes (from step 1) to GRP1:0.
+ *
+ *    LVL 1            [GRP1:0]
+ *                 --> migrator = GRP0:1
+ *                 --> active   = GRP0:1
+ *                   /                \
+ *    LVL 0  [GRP0:0]                  [GRP0:1]
+ *           migrator = CPU1           migrator = CPU2
+ *           active   = CPU1           active   = CPU2
+ *              /         \                /         \
+ *    CPUs     0           1              2           3
+ *             idle        active         active      idle
+ *
+ *
+ * The race of CPU0 vs. CPU1 led to an inconsistent state in GRP1:0. CPU1 is
+ * active and is correctly listed as active in GRP0:0. However GRP1:0 does not
+ * have GRP0:0 listed as active, which is wrong. The sequence counter has been
+ * added to avoid inconsistent states during updates. The state is updated
+ * atomically only if all members, including the sequence counter, match the
+ * expected value (compare-and-exchange).
+ *
+ * Looking back at the previous example with the addition of the sequence
+ * counter: The update as performed by CPU0 in step 4 will fail. CPU1 changed
+ * the sequence number during the update in step 3 so the expected old value (as
+ * seen by CPU0 before starting the walk) does not match.
+ *
+ * Required event and timerqueue update after a remote expiry:
+ * -----------------------------------------------------------
+ *
+ * After expiring timers of a remote CPU, a walk through the hierarchy and
+ * update of events and timerqueues is required. It is obviously needed if there
+ * is a 'new' global timer but also if there is no new global timer but the
+ * remote CPU is still idle.
+ *
+ * 1. CPU0 and CPU1 are idle and have both a global timer expiring at the same
+ *    time. So both have an event enqueued in the timerqueue of GRP0:0. CPU3 is
+ *    also idle and has no global timer pending. CPU2 is the only active CPU and
+ *    thus also the migrator:
+ *
+ *    LVL 1            [GRP1:0]
+ *                     migrator = GRP0:1
+ *                     active   = GRP0:1
+ *                 --> timerqueue = evt-GRP0:0
+ *                   /                \
+ *    LVL 0  [GRP0:0]                  [GRP0:1]
+ *           migrator = TMIGR_NONE     migrator = CPU2
+ *           active   =                active   = CPU2
+ *           groupevt.ignore = false   groupevt.ignore = true
+ *           groupevt.cpu = CPU0       groupevt.cpu =
+ *           timerqueue = evt-CPU0,    timerqueue =
+ *                        evt-CPU1
+ *              /         \                /         \
+ *    CPUs     0           1              2           3
+ *             idle        idle           active      idle
+ *
+ * 2. CPU2 starts to expire remote timers. It starts with LVL0 group
+ *    GRP0:1. There is no event queued in the timerqueue, so CPU2 continues with
+ *    the parent of GRP0:1: GRP1:0. In GRP1:0 it dequeues the first event. It
+ *    looks at tmigr_event::cpu struct member and expires the pending timer(s)
+ *    of CPU0.
+ *
+ *    LVL 1            [GRP1:0]
+ *                     migrator = GRP0:1
+ *                     active   = GRP0:1
+ *                 --> timerqueue =
+ *                   /                \
+ *    LVL 0  [GRP0:0]                  [GRP0:1]
+ *           migrator = TMIGR_NONE     migrator = CPU2
+ *           active   =                active   = CPU2
+ *           groupevt.ignore = false   groupevt.ignore = true
+ *       --> groupevt.cpu = CPU0       groupevt.cpu =
+ *           timerqueue = evt-CPU0,    timerqueue =
+ *                        evt-CPU1
+ *              /         \                /         \
+ *    CPUs     0           1              2           3
+ *             idle        idle           active      idle
+ *
+ * 3. Some work has to be done after expiring the timers of CPU0. If we stop
+ *    here, then CPU1's pending global timer(s) will not expire in time and the
+ *    timerqueue of GRP0:0 has still an event for CPU0 enqueued which has just
+ *    been processed. So it is required to walk the hierarchy from CPU0's point
+ *    of view and update it accordingly. CPU0's event will be removed from the
+ *    timerqueue because it has no pending timer. If CPU0 would have a timer
+ *    pending then it has to expire after CPU1's first timer because all timers
+ *    from this period were just expired. Either way CPU1's event will be first
+ *    in GRP0:0's timerqueue and therefore set in the CPU field of the group
+ *    event which is then enqueued in GRP1:0's timerqueue as GRP0:0 is still not
+ *    active:
+ *
+ *    LVL 1            [GRP1:0]
+ *                     migrator = GRP0:1
+ *                     active   = GRP0:1
+ *                 --> timerqueue = evt-GRP0:0
+ *                   /                \
+ *    LVL 0  [GRP0:0]                  [GRP0:1]
+ *           migrator = TMIGR_NONE     migrator = CPU2
+ *           active   =                active   = CPU2
+ *           groupevt.ignore = false   groupevt.ignore = true
+ *       --> groupevt.cpu = CPU1       groupevt.cpu =
+ *       --> timerqueue = evt-CPU1     timerqueue =
+ *              /         \                /         \
+ *    CPUs     0           1              2           3
+ *             idle        idle           active      idle
+ *
+ * Now CPU2 (migrator) will continue step 2 at GRP1:0 and will expire the
+ * timer(s) of CPU1.
+ *
+ * The hierarchy walk in step 3 can be skipped if the migrator notices that a
+ * CPU of GRP0:0 is active again. The CPU will mark GRP0:0 active and take care
+ * of the group as migrator and any needed updates within the hierarchy.
+ */
+
+static DEFINE_MUTEX(tmigr_mutex);
+static struct list_head *tmigr_level_list __read_mostly;
+
+static unsigned int tmigr_hierarchy_levels __read_mostly;
+static unsigned int tmigr_crossnode_level __read_mostly;
+
+static DEFINE_PER_CPU(struct tmigr_cpu, tmigr_cpu);
+
+#define TMIGR_NONE	0xFF
+#define BIT_CNT		8
+
+static inline bool tmigr_is_not_available(struct tmigr_cpu *tmc)
+{
+	return !(tmc->tmgroup && tmc->online);
+}
+
+/*
+ * Returns true, when @childmask corresponds to the group migrator or when the
+ * group is not active - so no migrator is set.
+ */
+static bool tmigr_check_migrator(struct tmigr_group *group, u8 childmask)
+{
+	union tmigr_state s;
+
+	s.state = atomic_read(&group->migr_state);
+
+	if ((s.migrator == childmask) || (s.migrator == TMIGR_NONE))
+		return true;
+
+	return false;
+}
+
+static bool tmigr_check_migrator_and_lonely(struct tmigr_group *group, u8 childmask)
+{
+	bool lonely, migrator = false;
+	unsigned long active;
+	union tmigr_state s;
+
+	s.state = atomic_read(&group->migr_state);
+
+	if ((s.migrator == childmask) || (s.migrator == TMIGR_NONE))
+		migrator = true;
+
+	active = s.active;
+	lonely = bitmap_weight(&active, BIT_CNT) <= 1;
+
+	return (migrator && lonely);
+}
+
+static bool tmigr_check_lonely(struct tmigr_group *group)
+{
+	unsigned long active;
+	union tmigr_state s;
+
+	s.state = atomic_read(&group->migr_state);
+
+	active = s.active;
+
+	return bitmap_weight(&active, BIT_CNT) <= 1;
+}
+
+typedef bool (*up_f)(struct tmigr_group *, struct tmigr_group *, void *);
+
+static void __walk_groups(up_f up, void *data,
+			  struct tmigr_cpu *tmc)
+{
+	struct tmigr_group *child = NULL, *group = tmc->tmgroup;
+
+	do {
+		WARN_ON_ONCE(group->level >= tmigr_hierarchy_levels);
+
+		if (up(group, child, data))
+			break;
+
+		child = group;
+		group = group->parent;
+	} while (group);
+}
+
+static void walk_groups(up_f up, void *data, struct tmigr_cpu *tmc)
+{
+	lockdep_assert_held(&tmc->lock);
+
+	__walk_groups(up, data, tmc);
+}
+
+/**
+ * struct tmigr_walk - data required for walking the hierarchy
+ * @nextexp:		Next CPU event expiry information which is handed into
+ *			the timer migration code by the timer code
+ *			(get_next_timer_interrupt())
+ * @firstexp:		Contains the first event expiry information when last
+ *			active CPU of hierarchy is on the way to idle to make
+ *			sure CPU will be back in time.
+ * @evt:		Pointer to tmigr_event which needs to be queued (of idle
+ *			child group)
+ * @childmask:		childmask of child group
+ * @remote:		Is set, when the new timer path is executed in
+ *			tmigr_handle_remote_cpu()
+ */
+struct tmigr_walk {
+	u64			nextexp;
+	u64			firstexp;
+	struct tmigr_event	*evt;
+	u8			childmask;
+	bool			remote;
+};
+
+/**
+ * struct tmigr_remote_data - data required for remote expiry hierarchy walk
+ * @basej:		timer base in jiffies
+ * @now:		timer base monotonic
+ * @firstexp:		returns expiry of the first timer in the idle timer
+ *			migration hierarchy to make sure the timer is handled in
+ *			time; it is stored in the per CPU tmigr_cpu struct of
+ *			CPU which expires remote timers
+ * @childmask:		childmask of child group
+ * @check:		is set if there is the need to handle remote timers;
+ *			required in tmigr_check_handle_remote() only
+ * @tmc_active:		this flag indicates, whether the CPU which triggers
+ *			the hierarchy walk is !idle in the timer migration
+ *			hierarchy. When the CPU is idle and the whole hierarchy is
+ *			idle, only the first event of the top level has to be
+ *			considered.
+ */
+struct tmigr_remote_data {
+	unsigned long	basej;
+	u64		now;
+	u64		firstexp;
+	u8		childmask;
+	bool		check;
+	bool		tmc_active;
+};
+
+/*
+ * Returns the next event of the timerqueue @group->events
+ *
+ * Removes timers with ignore flag and update next_expiry of the group. Values
+ * of the group event are updated in tmigr_update_events() only.
+ */
+static struct tmigr_event *tmigr_next_groupevt(struct tmigr_group *group)
+{
+	struct timerqueue_node *node = NULL;
+	struct tmigr_event *evt = NULL;
+
+	lockdep_assert_held(&group->lock);
+
+	WRITE_ONCE(group->next_expiry, KTIME_MAX);
+
+	while ((node = timerqueue_getnext(&group->events))) {
+		evt = container_of(node, struct tmigr_event, nextevt);
+
+		if (!evt->ignore) {
+			WRITE_ONCE(group->next_expiry, evt->nextevt.expires);
+			return evt;
+		}
+
+		/*
+		 * Remove next timers with ignore flag, because the group lock
+		 * is held anyway
+		 */
+		if (!timerqueue_del(&group->events, node))
+			break;
+	}
+
+	return NULL;
+}
+
+/*
+ * Return the next event (with the expiry equal or before @now)
+ *
+ * Event, which is returned, is also removed from the queue.
+ */
+static struct tmigr_event *tmigr_next_expired_groupevt(struct tmigr_group *group,
+						     u64 now)
+{
+	struct tmigr_event *evt = tmigr_next_groupevt(group);
+
+	if (!evt || now < evt->nextevt.expires)
+		return NULL;
+
+	/*
+	 * The event is ready to expire. Remove it and update next group event.
+	 */
+	timerqueue_del(&group->events, &evt->nextevt);
+	tmigr_next_groupevt(group);
+
+	return evt;
+}
+
+static u64 tmigr_next_groupevt_expires(struct tmigr_group *group)
+{
+	struct tmigr_event *evt;
+
+	evt = tmigr_next_groupevt(group);
+
+	if (!evt)
+		return KTIME_MAX;
+	else
+		return evt->nextevt.expires;
+}
+
+static bool tmigr_active_up(struct tmigr_group *group,
+			    struct tmigr_group *child,
+			    void *ptr)
+{
+	union tmigr_state curstate, newstate;
+	struct tmigr_walk *data = ptr;
+	bool walk_done;
+	u8 childmask;
+
+	childmask = data->childmask;
+	curstate.state = atomic_read(&group->migr_state);
+
+	do {
+		newstate = curstate;
+		walk_done = true;
+
+		if (newstate.migrator == TMIGR_NONE) {
+			newstate.migrator = childmask;
+
+			/* Changes need to be propagated */
+			walk_done = false;
+		}
+
+		newstate.active |= childmask;
+		newstate.seq++;
+
+	} while (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state));
+
+
+	if ((walk_done == false) && group->parent)
+		data->childmask = group->childmask;
+
+	/*
+	 * The group is active (again). The group event might be still queued
+	 * into the parent group's timerqueue but can now be handled by the
+	 * migrator of this group. Therefore the ignore flag for the group event
+	 * is updated to reflect this.
+	 *
+	 * The update of the ignore flag in the active path is done lockless. In
+	 * worst case the migrator of the parent group observes the change too
+	 * late and expires remotely all events belonging to this group. The
+	 * lock is held while updating the ignore flag in idle path. So this
+	 * state change will not be lost.
+	 */
+	group->groupevt.ignore = true;
+
+	return walk_done;
+}
+
+static void __tmigr_cpu_activate(struct tmigr_cpu *tmc)
+{
+	struct tmigr_walk data;
+
+	data.childmask = tmc->childmask;
+
+	tmc->cpuevt.ignore = true;
+	WRITE_ONCE(tmc->wakeup, KTIME_MAX);
+	tmc->wakeup_recalc = false;
+
+	walk_groups(&tmigr_active_up, &data, tmc);
+}
+
+/**
+ * tmigr_cpu_activate() - set this CPU active in timer migration hierarchy
+ *
+ * Call site timer_clear_idle() is called with interrupts disabled.
+ */
+void tmigr_cpu_activate(void)
+{
+	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+
+	if (tmigr_is_not_available(tmc))
+		return;
+
+	if (WARN_ON_ONCE(!tmc->idle))
+		return;
+
+	raw_spin_lock(&tmc->lock);
+	tmc->idle = false;
+	__tmigr_cpu_activate(tmc);
+	raw_spin_unlock(&tmc->lock);
+}
+
+/*
+ * Returns true, if there is nothing to be propagated to the next level
+ *
+ * @data->firstexp is set to expiry of first gobal event of the (top level of
+ * the) hierarchy, but only when hierarchy is completely idle.
+ *
+ * This is the only place where the group event expiry value is set.
+ */
+static
+bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
+			 struct tmigr_walk *data, union tmigr_state childstate,
+			 union tmigr_state groupstate)
+{
+	struct tmigr_event *evt, *first_childevt;
+	bool walk_done, remote = data->remote;
+	bool leftmost_change = false;
+	u64 nextexp;
+
+	if (child) {
+		raw_spin_lock(&child->lock);
+		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
+
+		if (childstate.active) {
+			walk_done = true;
+			goto unlock;
+		}
+
+		first_childevt = tmigr_next_groupevt(child);
+		nextexp = child->next_expiry;
+		evt = &child->groupevt;
+	} else {
+		nextexp = data->nextexp;
+
+		first_childevt = evt = data->evt;
+
+		/*
+		 * Walking the hierarchy is required in any case when a
+		 * remote expiry was done before. This ensures to not lose
+		 * already queued events in non active groups (see section
+		 * "Required event and timerqueue update after a remote
+		 * expiry" in the documentation at the top).
+		 *
+		 * The two call sites which are executed without a remote expiry
+		 * before, are not prevented from propagating changes through
+		 * the hierarchy by the return:
+		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
+		 *    is never set.
+		 *  - tmigr_inactive_up() takes care of the propagation by
+		 *    itself and ignores the return value. But an immediate
+		 *    return is required because nothing has to be done in this
+		 *    level as the event could be ignored.
+		 */
+		if (evt->ignore && !remote)
+			return true;
+
+		raw_spin_lock(&group->lock);
+	}
+
+	if (nextexp == KTIME_MAX) {
+		evt->ignore = true;
+
+		/*
+		 * When the next child event could be ignored (nextexp is
+		 * KTIME_MAX) and there was no remote timer handling before or
+		 * the group is already active, there is no need to walk the
+		 * hierarchy even if there is a parent group.
+		 *
+		 * The other way round: even if the event could be ignored, but
+		 * if a remote timer handling was executed before and the group
+		 * is not active, walking the hierarchy is required to not miss
+		 * an enqueued timer in the non active group. The enqueued timer
+		 * of the group needs to be propagated to a higher level to
+		 * ensure it is handled.
+		 */
+		if (!remote || groupstate.active) {
+			walk_done = true;
+			goto unlock;
+		}
+	} else {
+		/*
+		 * An update of @evt->cpu and @evt->ignore flag is required only
+		 * when @child is set (the child is equal or higher than lvl0),
+		 * but it doesn't matter if it is written once more to the per
+		 * CPU event; make the update unconditional.
+		 */
+		evt->cpu = first_childevt->cpu;
+		evt->ignore = false;
+	}
+
+	walk_done = !group->parent;
+
+	/*
+	 * If the child event is already queued in the group, remove it from the
+	 * queue when the expiry time changed only.
+	 */
+	if (timerqueue_node_queued(&evt->nextevt)) {
+		if (evt->nextevt.expires == nextexp)
+			goto check_toplvl;
+
+		leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt;
+		if (!timerqueue_del(&group->events, &evt->nextevt))
+			WRITE_ONCE(group->next_expiry, KTIME_MAX);
+	}
+
+	evt->nextevt.expires = nextexp;
+
+	if (timerqueue_add(&group->events, &evt->nextevt)) {
+		leftmost_change = true;
+		WRITE_ONCE(group->next_expiry, nextexp);
+	}
+
+check_toplvl:
+	if (walk_done && (groupstate.migrator == TMIGR_NONE)) {
+		/*
+		 * Nothing to do when first event didn't changed and update was
+		 * done during remote timer handling.
+		 */
+		if (remote && !leftmost_change)
+			goto unlock;
+		/*
+		 * The top level group is idle and it has to be ensured the
+		 * global timers are handled in time. (This could be optimized
+		 * by keeping track of the last global scheduled event and only
+		 * arming it on the CPU if the new event is earlier. Not sure if
+		 * its worth the complexity.)
+		 */
+		data->firstexp = tmigr_next_groupevt_expires(group);
+	}
+
+unlock:
+	raw_spin_unlock(&group->lock);
+
+	if (child)
+		raw_spin_unlock(&child->lock);
+
+	return walk_done;
+}
+
+static bool tmigr_new_timer_up(struct tmigr_group *group,
+			       struct tmigr_group *child,
+			       void *ptr)
+{
+	union tmigr_state childstate, groupstate;
+	struct tmigr_walk *data = ptr;
+
+	if (child)
+		childstate.state = atomic_read(&child->migr_state);
+	else
+		childstate.state = 0;
+
+	groupstate.state = atomic_read(&group->migr_state);
+
+	return tmigr_update_events(group, child, data, childstate, groupstate);
+}
+
+/*
+ * Returns the expiry of the next timer that needs to be handled. KTIME_MAX is
+ * returned, if an active CPU will handle all the timer migration hierarchy
+ * timers.
+ */
+static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp)
+{
+	struct tmigr_walk data = { .nextexp = nextexp,
+				   .firstexp = KTIME_MAX,
+				   .evt = &tmc->cpuevt };
+
+	lockdep_assert_held(&tmc->lock);
+
+	if (tmc->remote)
+		return KTIME_MAX;
+
+	tmc->cpuevt.ignore = false;
+	data.remote = false;
+
+	walk_groups(&tmigr_new_timer_up, &data, tmc);
+
+	/* If there is a new first global event, make sure it is handled */
+	return data.firstexp;
+}
+
+static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now,
+				   unsigned long jif)
+{
+	struct timer_events tevt;
+	struct tmigr_walk data;
+	struct tmigr_cpu *tmc;
+	u64 next = KTIME_MAX;
+
+	tmc = per_cpu_ptr(&tmigr_cpu, cpu);
+
+	raw_spin_lock_irq(&tmc->lock);
+
+	/*
+	 * If the remote CPU is offline then the timers have been migrated to
+	 * another CPU.
+	 *
+	 * If tmigr_cpu::remote is set, at the moment another CPU already
+	 * expires the timers of the remote CPU.
+	 *
+	 * If tmigr_event::ignore is set, then the CPU returns from idle and
+	 * takes care of its timers.
+	 *
+	 * If the next event expires in the future, then the event has been
+	 * updated and there are no timers to expire right now. The CPU which
+	 * updated the event takes care when hierarchy is completely
+	 * idle. Otherwise the migrator does it as the event is enqueued.
+	 */
+	if (!tmc->online || tmc->remote || tmc->cpuevt.ignore ||
+	    now < tmc->cpuevt.nextevt.expires) {
+		raw_spin_unlock_irq(&tmc->lock);
+		return next;
+	}
+
+	tmc->remote = true;
+	WRITE_ONCE(tmc->wakeup, KTIME_MAX);
+
+	/* Drop the lock to allow the remote CPU to exit idle */
+	raw_spin_unlock_irq(&tmc->lock);
+
+	if (cpu != smp_processor_id())
+		timer_expire_remote(cpu);
+
+	/*
+	 * Lock ordering needs to be preserved - timer_base locks before tmigr
+	 * related locks (see section "Locking rules" in the documentation at
+	 * the top). During fetching the next timer interrupt, also tmc->lock
+	 * needs to be held. Otherwise there is a possible race window against
+	 * the CPU itself when it comes out of idle, updates the first timer in
+	 * the hierarchy and goes back to idle.
+	 *
+	 * timer base locks are dropped as fast as possible: After checking
+	 * whether the remote CPU went offline in the meantime and after
+	 * fetching the next remote timer interrupt. Dropping the locks as fast
+	 * as possible keeps the locking region small and prevents holding
+	 * several (unnecessary) locks during walking the hierarchy for updating
+	 * the timerqueue and group events.
+	 */
+	local_irq_disable();
+	timer_lock_remote_bases(cpu);
+	raw_spin_lock(&tmc->lock);
+
+	/*
+	 * When the CPU went offline in the meantime, no hierarchy walk has to
+	 * be done for updating the queued events, because the walk was
+	 * already done during marking the CPU offline in the hierarchy.
+	 *
+	 * When the CPU is no longer idle, the CPU takes care of the timers and
+	 * also of the timers in the hierarchy.
+	 *
+	 * (See also section "Required event and timerqueue update after a
+	 * remote expiry" in the documentation at the top)
+	 */
+	if (!tmc->online || !tmc->idle) {
+		timer_unlock_remote_bases(cpu);
+		goto unlock;
+	} else {
+		/* next	event of CPU */
+		fetch_next_timer_interrupt_remote(jif, now, &tevt, cpu);
+	}
+
+	timer_unlock_remote_bases(cpu);
+
+	data.nextexp = tevt.global;
+	data.firstexp = KTIME_MAX;
+	data.evt = &tmc->cpuevt;
+	data.remote = true;
+
+	/*
+	 * The update is done even when there is no 'new' global timer pending
+	 * on the remote CPU (see section "Required event and timerqueue update
+	 * after a remote expiry" in the documentation at the top)
+	 */
+	walk_groups(&tmigr_new_timer_up, &data, tmc);
+
+	next = data.firstexp;
+
+unlock:
+	tmc->remote = false;
+	raw_spin_unlock_irq(&tmc->lock);
+
+	return next;
+}
+
+static bool tmigr_handle_remote_up(struct tmigr_group *group,
+				   struct tmigr_group *child,
+				   void *ptr)
+{
+	struct tmigr_remote_data *data = ptr;
+	u64 now, next = KTIME_MAX;
+	struct tmigr_event *evt;
+	unsigned long jif;
+	u8 childmask;
+
+	jif = data->basej;
+	now = data->now;
+
+	childmask = data->childmask;
+
+again:
+	/*
+	 * Handle the group only if @childmask is the migrator or if the
+	 * group has no migrator. Otherwise the group is active and is
+	 * handled by its own migrator.
+	 */
+	if (!tmigr_check_migrator(group, childmask))
+		return true;
+
+	raw_spin_lock_irq(&group->lock);
+
+	evt = tmigr_next_expired_groupevt(group, now);
+
+	if (evt) {
+		unsigned int remote_cpu = evt->cpu;
+
+		raw_spin_unlock_irq(&group->lock);
+
+		next = tmigr_handle_remote_cpu(remote_cpu, now, jif);
+
+		/* check if there is another event, that needs to be handled */
+		goto again;
+	} else {
+		raw_spin_unlock_irq(&group->lock);
+	}
+
+	/*
+	 * Update of childmask for the next level and keep track of the expiry
+	 * of the first event that needs to be handled
+	 */
+	data->childmask = group->childmask;
+	data->firstexp = next;
+
+	return false;
+}
+
+/**
+ * tmigr_handle_remote() - Handle global timers of remote idle CPUs
+ *
+ * Called from the timer soft interrupt with interrupts enabled.
+ */
+void tmigr_handle_remote(void)
+{
+	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+	struct tmigr_remote_data data;
+
+	if (tmigr_is_not_available(tmc))
+		return;
+
+	data.childmask = tmc->childmask;
+	data.firstexp = KTIME_MAX;
+
+	/*
+	 * NOTE: This is a doubled check because the migrator test will be done
+	 * in tmigr_handle_remote_up() anyway. Keep this check to speed up the
+	 * return when nothing has to be done.
+	 */
+	if (!tmigr_check_migrator(tmc->tmgroup, tmc->childmask))
+		return;
+
+	data.now = get_jiffies_update(&data.basej);
+
+	/*
+	 * Update @tmc->wakeup only at the end and do not reset @tmc->wakeup to
+	 * KTIME_MAX. Even if tmc->lock is not held during the whole remote
+	 * handling, tmc->wakeup is fine to be stale as it is called in
+	 * interrupt context and tick_nohz_next_event() is executed in interrupt
+	 * exit path only after processing the last pending interrupt.
+	 */
+
+	__walk_groups(&tmigr_handle_remote_up, &data, tmc);
+
+	raw_spin_lock_irq(&tmc->lock);
+	WRITE_ONCE(tmc->wakeup, data.firstexp);
+	raw_spin_unlock_irq(&tmc->lock);
+}
+
+static bool tmigr_requires_handle_remote_up(struct tmigr_group *group,
+					    struct tmigr_group *child,
+					    void *ptr)
+{
+	struct tmigr_remote_data *data = ptr;
+	u8 childmask;
+
+	childmask = data->childmask;
+
+	/*
+	 * Handle the group only if the child is the migrator or if the group
+	 * has no migrator. Otherwise the group is active and is handled by its
+	 * own migrator.
+	 */
+	if (!tmigr_check_migrator(group, childmask))
+		return true;
+
+	/*
+	 * When there is a parent group and the CPU which triggered the
+	 * hierarchy walk is not active, proceed the walk to reach the top level
+	 * group before reading the next_expiry value.
+	 */
+	if (group->parent && !data->tmc_active)
+		goto out;
+
+	/*
+	 * The lock is required on 32bit architectures to read the variable
+	 * consistently with a concurrent writer. On 64bit the lock is not
+	 * required because the read operation is not split and so it is always
+	 * consistent.
+	 */
+	if (IS_ENABLED(CONFIG_64BIT)) {
+		data->firstexp = READ_ONCE(group->next_expiry);
+		if (data->now >= data->firstexp) {
+			data->check = true;
+			return true;
+		}
+	} else {
+		raw_spin_lock(&group->lock);
+		data->firstexp = group->next_expiry;
+		if (data->now >= group->next_expiry) {
+			data->check = true;
+			raw_spin_unlock(&group->lock);
+			return true;
+		}
+		raw_spin_unlock(&group->lock);
+	}
+
+out:
+	/* Update of childmask for the next level */
+	data->childmask = group->childmask;
+	return false;
+}
+
+/**
+ * tmigr_requires_handle_remote() - Check the need of remote timer handling
+ *
+ * Must be called with interrupts disabled.
+ */
+int tmigr_requires_handle_remote(void)
+{
+	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+	struct tmigr_remote_data data;
+	unsigned int ret = 0;
+	unsigned long jif;
+
+	if (tmigr_is_not_available(tmc))
+		return ret;
+
+	data.now = get_jiffies_update(&jif);
+	data.childmask = tmc->childmask;
+	data.firstexp = KTIME_MAX;
+	data.tmc_active = !tmc->idle;
+	data.check = false;
+
+	/*
+	 * If the CPU is active, walk the hierarchy to check whether a remote
+	 * expiry is required.
+	 *
+	 * Check is done lockless as interrupts are disabled and @tmc->idle is
+	 * set only by the local CPU.
+	 */
+	if (!tmc->idle) {
+		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
+
+		if (data.firstexp != KTIME_MAX)
+			ret = 1;
+
+		return ret;
+	}
+
+	/*
+	 * If the CPU is idle, check whether the recalculation of @tmc->wakeup
+	 * is required. @tmc->wakeup_recalc is set, when the last active CPU
+	 * went offline. The last active CPU delegated the handling of the timer
+	 * migration hierarchy to another (this) CPU by updating this flag and
+	 * sending a reschedule.
+	 *
+	 * Racy lockless check is valid:
+	 * - @tmc->wakeup_recalc is set by the remote CPU before it issues
+	 *   reschedule IPI.
+	 * - As interrupts are disabled here this CPU will either observe
+	 *   @tmc->wakeup_recalc set before the reschedule IPI can be handled or
+	 *   it will observe it when this function is called again on return
+	 *   from handling the reschedule IPI.
+	 */
+	if (tmc->wakeup_recalc) {
+		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
+
+		if (data.firstexp != KTIME_MAX)
+			ret = 1;
+
+		raw_spin_lock(&tmc->lock);
+		WRITE_ONCE(tmc->wakeup, data.firstexp);
+		tmc->wakeup_recalc = false;
+		raw_spin_unlock(&tmc->lock);
+
+		return ret;
+	}
+
+	/*
+	 * When the CPU is idle and @tmc->wakeup is reliable as
+	 * @tmc->wakeup_recalc is not set, compare it with @data.now. The lock
+	 * is required on 32bit architectures to read the variable consistently
+	 * with a concurrent writer. On 64bit the lock is not required because
+	 * the read operation is not split and so it is always consistent.
+
+	 */
+	if (IS_ENABLED(CONFIG_64BIT)) {
+		if (data.now >= READ_ONCE(tmc->wakeup))
+			ret = 1;
+	} else {
+		raw_spin_lock(&tmc->lock);
+		if (data.now >= tmc->wakeup)
+			ret = 1;
+		raw_spin_unlock(&tmc->lock);
+	}
+
+	return ret;
+}
+
+/**
+ * tmigr_cpu_new_timer() - enqueue next global timer into hierarchy (idle tmc)
+ * @nextexp:	Next expiry of global timer (or KTIME_MAX if not)
+ *
+ * The CPU is already deactivated in the timer migration
+ * hierarchy. tick_nohz_get_sleep_length() calls tick_nohz_next_event()
+ * and thereby the timer idle path is executed once more. @tmc->wakeup
+ * holds the first timer, when the timer migration hierarchy is
+ * completely idle.
+ *
+ * Returns the first timer that needs to be handled by this CPU or KTIME_MAX if
+ * nothing needs to be done.
+ */
+u64 tmigr_cpu_new_timer(u64 nextexp)
+{
+	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+	u64 ret;
+
+	if (tmigr_is_not_available(tmc))
+		return nextexp;
+
+	raw_spin_lock(&tmc->lock);
+
+	ret = READ_ONCE(tmc->wakeup);
+	if (nextexp != KTIME_MAX) {
+		if (nextexp != tmc->cpuevt.nextevt.expires ||
+		    tmc->cpuevt.ignore) {
+			ret = tmigr_new_timer(tmc, nextexp);
+		}
+	} else if (tmc->wakeup_recalc) {
+		struct tmigr_remote_data data;
+
+		data.now = KTIME_MAX;
+		data.childmask = tmc->childmask;
+		data.firstexp = KTIME_MAX;
+		data.tmc_active = false;
+		data.check = false;
+
+		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
+
+		ret = data.firstexp;
+	}
+	tmc->wakeup_recalc = false;
+
+	/*
+	 * Make sure the reevaluation of timers in idle path will not miss an
+	 * event.
+	 */
+	WRITE_ONCE(tmc->wakeup, ret);
+
+	raw_spin_unlock(&tmc->lock);
+	return ret;
+}
+
+static bool tmigr_inactive_up(struct tmigr_group *group,
+			      struct tmigr_group *child,
+			      void *ptr)
+{
+	union tmigr_state curstate, newstate, childstate;
+	struct tmigr_walk *data = ptr;
+	bool walk_done;
+	u8 childmask;
+
+	childmask = data->childmask;
+	curstate.state = atomic_read(&group->migr_state);
+	childstate.state = 0;
+
+	do {
+		if (child)
+			childstate.state = atomic_read(&child->migr_state);
+
+		newstate = curstate;
+		walk_done = true;
+
+		/* Reset active bit when the child is no longer active */
+		if (!childstate.active)
+			newstate.active &= ~childmask;
+
+		if (newstate.migrator == childmask) {
+			/*
+			 * Find a new migrator for the group, because the child
+			 * group is idle!
+			 */
+			if (!childstate.active) {
+				unsigned long new_migr_bit, active = newstate.active;
+
+				new_migr_bit = find_first_bit(&active, BIT_CNT);
+
+				if (new_migr_bit != BIT_CNT) {
+					newstate.migrator = BIT(new_migr_bit);
+				} else {
+					newstate.migrator = TMIGR_NONE;
+
+					/* Changes need to be propagated */
+					walk_done = false;
+				}
+			}
+		}
+
+		newstate.seq++;
+
+		WARN_ON_ONCE((newstate.migrator != TMIGR_NONE) && !(newstate.active));
+
+	} while (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state));
+
+	data->remote = false;
+
+	/* Event Handling */
+	tmigr_update_events(group, child, data, childstate, newstate);
+
+	if (group->parent && (walk_done == false))
+		data->childmask = group->childmask;
+
+	/*
+	 * data->firstexp was set by tmigr_update_events() and contains the
+	 * expiry of the first global event which needs to be handled. It
+	 * differs from KTIME_MAX if:
+	 * - group is the top level group and
+	 * - group is idle (which means CPU was the last active CPU in the
+	 *   hierarchy) and
+	 * - there is a pending event in the hierarchy
+	 */
+	if (data->firstexp != KTIME_MAX) {
+		WARN_ON_ONCE(group->parent);
+		/*
+		 * Top level path: If this CPU is about going offline and was
+		 * the last active CPU, wake up some random other CPU so it will
+		 * take over the migrator duty and program its timer
+		 * properly. Ideally wake the CPU with the closest expiry time,
+		 * but that's overkill to figure out.
+		 *
+		 * Set wakeup_recalc of remote CPU, to make sure the complete
+		 * idle hierarchy with enqueued timers is reevaluated.
+		 */
+		if (!(this_cpu_ptr(&tmigr_cpu)->online)) {
+			struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+			unsigned int cpu = smp_processor_id();
+			struct tmigr_cpu *tmc_resched;
+
+			cpu = cpumask_any_but(cpu_online_mask, cpu);
+			tmc_resched = per_cpu_ptr(&tmigr_cpu, cpu);
+
+			raw_spin_unlock(&tmc->lock);
+
+			raw_spin_lock(&tmc_resched->lock);
+			tmc_resched->wakeup_recalc = true;
+			raw_spin_unlock(&tmc_resched->lock);
+
+			raw_spin_lock(&tmc->lock);
+			smp_send_reschedule(cpu);
+		}
+	}
+
+	return walk_done;
+}
+
+static u64 __tmigr_cpu_deactivate(struct tmigr_cpu *tmc, u64 nextexp)
+{
+	struct tmigr_walk data = { .nextexp = nextexp,
+				   .firstexp = KTIME_MAX,
+				   .evt = &tmc->cpuevt,
+				   .childmask = tmc->childmask };
+
+	/*
+	 * If nextexp is KTIME_MAX, the CPU event will be ignored because the
+	 * local timer expires before the global timer, no global timer is set
+	 * or CPU goes offline.
+	 */
+	if (nextexp != KTIME_MAX)
+		tmc->cpuevt.ignore = false;
+
+	walk_groups(&tmigr_inactive_up, &data, tmc);
+	return data.firstexp;
+}
+
+/**
+ * tmigr_cpu_deactivate() - Put current CPU into inactive state
+ * @nextexp:	The next timer event expiry set in the current CPU
+ *
+ * Must be called with interrupts disabled.
+ *
+ * Return: the next event expiry of the current CPU or the next event expiry
+ * from the hierarchy if this CPU is the top level migrator or the hierarchy is
+ * completely idle.
+ */
+u64 tmigr_cpu_deactivate(u64 nextexp)
+{
+	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+	u64 ret;
+
+	if (tmigr_is_not_available(tmc))
+		return nextexp;
+
+	raw_spin_lock(&tmc->lock);
+
+	ret = __tmigr_cpu_deactivate(tmc, nextexp);
+
+	tmc->idle = true;
+
+	/*
+	 * Make sure the reevaluation of timers in idle path will not miss an
+	 * event.
+	 */
+	WRITE_ONCE(tmc->wakeup, ret);
+
+	raw_spin_unlock(&tmc->lock);
+	return ret;
+}
+
+/**
+ * tmigr_quick_check() - Quick forecast of next tmigr event when CPU wants to
+ *			 go idle
+ *
+ * Returns KTIME_MAX, when it is probable that nothing has to be done (not the
+ * only one in the level 0 group; and if it is the only one in level 0 group,
+ * but there are more than a single group active in top level)
+ *
+ * Returns first expiry of the top level group, when it is the only one in level
+ * 0 and top level also only has a single active child.
+ */
+u64 tmigr_quick_check(void)
+{
+	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+	struct tmigr_group *topgroup;
+	struct list_head lvllist;
+
+	if (tmigr_is_not_available(tmc))
+		return KTIME_MAX;
+
+	if (WARN_ON_ONCE(tmc->idle))
+		return KTIME_MAX;
+
+	if (!tmigr_check_migrator_and_lonely(tmc->tmgroup, tmc->childmask))
+		return KTIME_MAX;
+
+	for (int i = tmigr_hierarchy_levels; i > 0 ; i--) {
+		lvllist = tmigr_level_list[i - 1];
+		if (list_is_singular(&lvllist)) {
+			topgroup = list_first_entry(&lvllist, struct tmigr_group, list);
+
+			if (tmigr_check_lonely(topgroup))
+				return READ_ONCE(topgroup->next_expiry);
+		} else {
+			continue;
+		}
+	}
+
+	return KTIME_MAX;
+}
+
+static void tmigr_init_group(struct tmigr_group *group, unsigned int lvl,
+			     int node)
+{
+	union tmigr_state s;
+
+	raw_spin_lock_init(&group->lock);
+
+	group->level = lvl;
+	group->numa_node = lvl < tmigr_crossnode_level ? node : NUMA_NO_NODE;
+
+	group->num_children = 0;
+
+	s.migrator = TMIGR_NONE;
+	s.active = 0;
+	s.seq = 0;
+	atomic_set(&group->migr_state, s.state);
+
+	timerqueue_init_head(&group->events);
+	timerqueue_init(&group->groupevt.nextevt);
+	group->groupevt.nextevt.expires = KTIME_MAX;
+	WRITE_ONCE(group->next_expiry, KTIME_MAX);
+	group->groupevt.ignore = true;
+}
+
+static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node,
+					   unsigned int lvl)
+{
+	struct tmigr_group *tmp, *group = NULL;
+
+	lockdep_assert_held(&tmigr_mutex);
+
+	/* Try to attach to an existing group first */
+	list_for_each_entry(tmp, &tmigr_level_list[lvl], list) {
+		/*
+		 * If @lvl is below the cross NUMA node level, check whether
+		 * this group belongs to the same NUMA node.
+		 */
+		if (lvl < tmigr_crossnode_level && tmp->numa_node != node)
+			continue;
+
+		/* Capacity left? */
+		if (tmp->num_children >= TMIGR_CHILDREN_PER_GROUP)
+			continue;
+
+		/*
+		 * TODO: A possible further improvement: Make sure that all CPU
+		 * siblings end up in the same group of the lowest level of the
+		 * hierarchy. Rely on the topology sibling mask would be a
+		 * reasonable solution.
+		 */
+
+		group = tmp;
+		break;
+	}
+
+	if (group)
+		return group;
+
+	/* Allocate and	set up a new group */
+	group = kzalloc_node(sizeof(*group), GFP_KERNEL, node);
+	if (!group)
+		return ERR_PTR(-ENOMEM);
+
+	tmigr_init_group(group, lvl, node);
+
+	/* Setup successful. Add it to the hierarchy */
+	list_add(&group->list, &tmigr_level_list[lvl]);
+	return group;
+}
+
+static void tmigr_connect_child_parent(struct tmigr_group *child,
+				       struct tmigr_group *parent)
+{
+	union tmigr_state childstate;
+
+	raw_spin_lock_irq(&child->lock);
+	raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING);
+
+	child->parent = parent;
+	child->childmask = BIT(parent->num_children++);
+
+	raw_spin_unlock(&parent->lock);
+	raw_spin_unlock_irq(&child->lock);
+
+	/*
+	 * To prevent inconsistent states, active children need to be active in
+	 * the new parent as well. Inactive children are already marked inactive
+	 * in the parent group.
+	 */
+	childstate.state = atomic_read(&child->migr_state);
+	if (childstate.migrator != TMIGR_NONE) {
+		struct tmigr_walk data;
+
+		data.childmask = child->childmask;
+
+		/*
+		 * There is only one new level per time. When connecting the
+		 * child and the parent and set the child active when the parent
+		 * is inactive, the parent needs to be the uppermost
+		 * level. Otherwise there went something wrong!
+		 */
+		WARN_ON(!tmigr_active_up(parent, child, &data) && parent->parent);
+	}
+}
+
+static int tmigr_setup_groups(unsigned int cpu, unsigned int node)
+{
+	struct tmigr_group *group, *child, **stack;
+	int top = 0, err = 0, i = 0;
+	struct list_head *lvllist;
+
+	stack = kcalloc(tmigr_hierarchy_levels, sizeof(*stack), GFP_KERNEL);
+	if (!stack)
+		return -ENOMEM;
+
+	do {
+		group = tmigr_get_group(cpu, node, i);
+		if (IS_ERR(group)) {
+			err = PTR_ERR(group);
+			break;
+		}
+
+		top = i;
+		stack[i++] = group;
+
+		/*
+		 * When booting only less CPUs of a system than CPUs are
+		 * available, not all calculated hierarchy levels are required.
+		 *
+		 * The loop is aborted as soon as the highest level, which might
+		 * be different from tmigr_hierarchy_levels, contains only a
+		 * single group.
+		 */
+		if (group->parent || i == tmigr_hierarchy_levels ||
+		    (list_empty(&tmigr_level_list[i]) &&
+		     list_is_singular(&tmigr_level_list[i - 1])))
+			break;
+
+	} while (i < tmigr_hierarchy_levels);
+
+	do {
+		group = stack[--i];
+
+		if (err < 0) {
+			list_del(&group->list);
+			kfree(group);
+			continue;
+		}
+
+		WARN_ON_ONCE(i != group->level);
+
+		/*
+		 * Update tmc -> group / child -> group connection
+		 */
+		if (i == 0) {
+			struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+
+			raw_spin_lock_irq(&group->lock);
+
+			tmc->tmgroup = group;
+			tmc->childmask = BIT(group->num_children++);
+
+			raw_spin_unlock_irq(&group->lock);
+
+			/* There are no children that need to be connected */
+			continue;
+		} else {
+			child = stack[i - 1];
+			tmigr_connect_child_parent(child, group);
+		}
+
+		/* check if uppermost level was newly created */
+		if (top != i)
+			continue;
+
+		WARN_ON_ONCE(top == 0);
+
+		lvllist = &tmigr_level_list[top];
+		if (group->num_children == 1 && list_is_singular(lvllist)) {
+			lvllist = &tmigr_level_list[top - 1];
+			list_for_each_entry(child, lvllist, list) {
+				if (child->parent)
+					continue;
+
+				tmigr_connect_child_parent(child, group);
+			}
+		}
+	} while (i > 0);
+
+	kfree(stack);
+
+	return err;
+}
+
+static int tmigr_add_cpu(unsigned int cpu)
+{
+	int node = cpu_to_node(cpu);
+	int ret;
+
+	mutex_lock(&tmigr_mutex);
+	ret = tmigr_setup_groups(cpu, node);
+	mutex_unlock(&tmigr_mutex);
+
+	return ret;
+}
+
+static int tmigr_cpu_online(unsigned int cpu)
+{
+	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+	int ret;
+
+	/* First online attempt? Initialize CPU data */
+	if (!tmc->tmgroup) {
+		raw_spin_lock_init(&tmc->lock);
+
+		ret = tmigr_add_cpu(cpu);
+		if (ret < 0)
+			return ret;
+
+		if (tmc->childmask == 0)
+			return -EINVAL;
+
+		timerqueue_init(&tmc->cpuevt.nextevt);
+		tmc->cpuevt.nextevt.expires = KTIME_MAX;
+		tmc->cpuevt.ignore = true;
+		tmc->cpuevt.cpu = cpu;
+
+		tmc->remote = false;
+		WRITE_ONCE(tmc->wakeup, KTIME_MAX);
+	}
+	raw_spin_lock_irq(&tmc->lock);
+	tmc->idle = timer_base_is_idle();
+	if (!tmc->idle)
+		__tmigr_cpu_activate(tmc);
+	tmc->online = true;
+	raw_spin_unlock_irq(&tmc->lock);
+	return 0;
+}
+
+static int tmigr_cpu_offline(unsigned int cpu)
+{
+	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+
+	raw_spin_lock_irq(&tmc->lock);
+	tmc->online = false;
+	WRITE_ONCE(tmc->wakeup, KTIME_MAX);
+
+	/*
+	 * CPU has to handle the local events on his own, when on the way to
+	 * offline; Therefore nextevt value is set to KTIME_MAX
+	 */
+	__tmigr_cpu_deactivate(tmc, KTIME_MAX);
+	raw_spin_unlock_irq(&tmc->lock);
+
+	return 0;
+}
+
+static int __init tmigr_init(void)
+{
+	unsigned int cpulvl, nodelvl, cpus_per_node, i;
+	unsigned int nnodes = num_possible_nodes();
+	unsigned int ncpus = num_possible_cpus();
+	int ret = -ENOMEM;
+
+	BUILD_BUG_ON_NOT_POWER_OF_2(TMIGR_CHILDREN_PER_GROUP);
+
+	/* Nothing to do if running on UP */
+	if (ncpus == 1)
+		return 0;
+
+	/*
+	 * Calculate the required hierarchy levels. Unfortunately there is no
+	 * reliable information available, unless all possible CPUs have been
+	 * brought up and all NUMA nodes are populated.
+	 *
+	 * Estimate the number of levels with the number of possible nodes and
+	 * the number of possible CPUs. Assume CPUs are spread evenly across
+	 * nodes. We cannot rely on cpumask_of_node() because it only works for
+	 * online CPUs.
+	 */
+	cpus_per_node = DIV_ROUND_UP(ncpus, nnodes);
+
+	/* Calc the hierarchy levels required to hold the CPUs of a node */
+	cpulvl = DIV_ROUND_UP(order_base_2(cpus_per_node),
+			      ilog2(TMIGR_CHILDREN_PER_GROUP));
+
+	/* Calculate the extra levels to connect all nodes */
+	nodelvl = DIV_ROUND_UP(order_base_2(nnodes),
+			       ilog2(TMIGR_CHILDREN_PER_GROUP));
+
+	tmigr_hierarchy_levels = cpulvl + nodelvl;
+
+	/*
+	 * If a NUMA node spawns more than one CPU level group then the next
+	 * level(s) of the hierarchy contains groups which handle all CPU groups
+	 * of the same NUMA node. The level above goes across NUMA nodes. Store
+	 * this information for the setup code to decide in which level node
+	 * matching is no longer required.
+	 */
+	tmigr_crossnode_level = cpulvl;
+
+	tmigr_level_list = kcalloc(tmigr_hierarchy_levels, sizeof(struct list_head), GFP_KERNEL);
+	if (!tmigr_level_list)
+		goto err;
+
+	for (i = 0; i < tmigr_hierarchy_levels; i++)
+		INIT_LIST_HEAD(&tmigr_level_list[i]);
+
+	pr_info("Timer migration: %d hierarchy levels; %d children per group;"
+		" %d crossnode level\n",
+		tmigr_hierarchy_levels, TMIGR_CHILDREN_PER_GROUP,
+		tmigr_crossnode_level);
+
+	ret = cpuhp_setup_state(CPUHP_AP_TMIGR_ONLINE, "tmigr:online",
+				tmigr_cpu_online, tmigr_cpu_offline);
+	if (ret)
+		goto err;
+
+	return 0;
+
+err:
+	pr_err("Timer migration setup failed\n");
+	return ret;
+}
+late_initcall(tmigr_init);
diff --git a/kernel/time/timer_migration.h b/kernel/time/timer_migration.h
new file mode 100644
index 000000000000..c32947cf429b
--- /dev/null
+++ b/kernel/time/timer_migration.h
@@ -0,0 +1,147 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _KERNEL_TIME_MIGRATION_H
+#define _KERNEL_TIME_MIGRATION_H
+
+/* Per group capacity. Must be a power of 2! */
+#define TMIGR_CHILDREN_PER_GROUP 8
+
+/**
+ * struct tmigr_event - a timer event associated to a CPU
+ * @nextevt:	The node to enqueue an event in the parent group queue
+ * @cpu:	The CPU to which this event belongs
+ * @ignore:	Hint whether the event could be ignored; it is set when
+ *		CPU or group is active;
+ */
+struct tmigr_event {
+	struct timerqueue_node	nextevt;
+	unsigned int		cpu;
+	bool			ignore;
+};
+
+/**
+ * struct tmigr_group - timer migration hierarchy group
+ * @lock:		Lock protecting the event information and group hierarchy
+ *			information during setup
+ * @parent:		Pointer to the parent group
+ * @groupevt:		Next event of the group which is only used when the
+ *			group is !active. The group event is then queued into
+ *			the parent timer queue.
+ *			Ignore bit of @groupevt is set when the group is active.
+ * @next_expiry:	Base monotonic expiry time of the next event of the
+ *			group; It is used for the racy lockless check whether a
+ *			remote expiry is required; it is always reliable
+ * @events:		Timer queue for child events queued in the group
+ * @migr_state:		State of the group (see union tmigr_state)
+ * @level:		Hierarchy level of the group; Required during setup
+ * @numa_node:		Required for setup only to make sure CPU and low level
+ *			group information is NUMA local. It is set to NUMA node
+ *			as long as the group level is per NUMA node (level <
+ *			tmigr_crossnode_level); otherwise it is set to
+ *			NUMA_NO_NODE
+ * @num_children:	Counter of group children to make sure the group is only
+ *			filled with TMIGR_CHILDREN_PER_GROUP; Required for setup
+ *			only
+ * @childmask:		childmask of the group in the parent group; is set
+ *			during setup and will never change; can be read
+ *			lockless
+ * @list:		List head that is added to the per level
+ *			tmigr_level_list; is required during setup when a
+ *			new group needs to be connected to the existing
+ *			hierarchy groups
+ */
+struct tmigr_group {
+	raw_spinlock_t		lock;
+	struct tmigr_group	*parent;
+	struct tmigr_event	groupevt;
+	u64			next_expiry;
+	struct timerqueue_head	events;
+	atomic_t		migr_state;
+	unsigned int		level;
+	int			numa_node;
+	unsigned int		num_children;
+	u8			childmask;
+	struct list_head	list;
+};
+
+/**
+ * struct tmigr_cpu - timer migration per CPU group
+ * @lock:		Lock protecting the tmigr_cpu group information
+ * @online:		Indicates whether the CPU is online; In deactivate path
+ *			it is required to know whether the migrator in the top
+ *			level group is to be set offline, while a timer is
+ *			pending. Then another online CPU needs to be notified to
+ *			take over the migrator role. Furthermore the information
+ *			is required in CPU hotplug path as the CPU is able to go
+ *			idle before the timer migration hierarchy hotplug AP is
+ *			reached. During this phase, the CPU has to handle the
+ *			global timers on its own and must not act as a migrator.
+ * @idle:		Indicates whether the CPU is idle in the timer migration
+ *			hierarchy
+ * @remote:		Is set when timers of the CPU are expired remotely
+ * @wakeup_recalc:	Indicates, whether a recalculation of the @wakeup value
+ *			is required. @wakeup_recalc is only used by this CPU
+ *			when it is marked idle in the timer migration
+ *			hierarchy. It is set by a remote CPU which was the last
+ *			active CPU and is on the way to idle.
+ * @tmgroup:		Pointer to the parent group
+ * @childmask:		childmask of tmigr_cpu in the parent group
+ * @wakeup:		Stores the first timer when the timer migration
+ *			hierarchy is completely idle and remote expiry was done;
+ *			is returned to timer code in the idle path and is only
+ *			used in idle path; it is only valid, when @wakeup_recalc
+ *			is not set.
+ * @cpuevt:		CPU event which could be enqueued into the parent group
+ */
+struct tmigr_cpu {
+	raw_spinlock_t		lock;
+	bool			online;
+	bool			idle;
+	bool			remote;
+	bool			wakeup_recalc;
+	struct tmigr_group	*tmgroup;
+	u8			childmask;
+	u64			wakeup;
+	struct tmigr_event	cpuevt;
+};
+
+/**
+ * union tmigr_state - state of tmigr_group
+ * @state:	Combined version of the state - only used for atomic
+ *		read/cmpxchg function
+ * @struct:	Split version of the state - only use the struct members to
+ *		update information to stay independent of endianness
+ */
+union tmigr_state {
+	u32 state;
+	/**
+	 * struct - split state of tmigr_group
+	 * @active:	Contains each childmask bit of the active children
+	 * @migrator:	Contains childmask of the child which is migrator
+	 * @seq:	Sequence counter needs to be increased when an update
+	 *		to the tmigr_state is done. It prevents a race when
+	 *		updates in the child groups are propagated in changed
+	 *		order. Detailed information about the scenario is
+	 *		given in the documentation at the begin of
+	 *		timer_migration.c.
+	 */
+	struct {
+		u8	active;
+		u8	migrator;
+		u16	seq;
+	} __packed;
+};
+
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+extern void tmigr_handle_remote(void);
+extern int tmigr_requires_handle_remote(void);
+extern void tmigr_cpu_activate(void);
+extern u64 tmigr_cpu_deactivate(u64 nextevt);
+extern u64 tmigr_cpu_new_timer(u64 nextevt);
+extern u64 tmigr_quick_check(void);
+#else
+static inline void tmigr_handle_remote(void) { }
+static inline int tmigr_requires_handle_remote(void) { return 0; }
+static inline void tmigr_cpu_activate(void) { }
+#endif
+
+#endif
-- 
2.39.2

[PATCH v10 19/20] timer_migration: Add tracepoints

[Anna-Maria Behnsen]

The timer pull logic needs proper debugging aids. Add tracepoints so the
hierarchical idle machinery can be diagnosed.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
v10: Make an entry in MAINTAINERS file

v9: Add tmigr_cpu_new_timer_idle tracepoint

v8: Add wakeup value to tracepoints
---
---
 MAINTAINERS                            |   1 +
 include/trace/events/timer_migration.h | 297 +++++++++++++++++++++++++
 kernel/time/timer_migration.c          |  26 +++
 3 files changed, 324 insertions(+)
 create mode 100644 include/trace/events/timer_migration.h

diff --git a/MAINTAINERS b/MAINTAINERS
index a7c4cf8201e0..9f9f2a695082 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17244,6 +17244,7 @@ T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/core
 F:	fs/timerfd.c
 F:	include/linux/time_namespace.h
 F:	include/linux/timer*
+F:	include/trace/events/timer*
 F:	kernel/time/*timer*
 F:	kernel/time/namespace.c
 
diff --git a/include/trace/events/timer_migration.h b/include/trace/events/timer_migration.h
new file mode 100644
index 000000000000..a2e7e32058f8
--- /dev/null
+++ b/include/trace/events/timer_migration.h
@@ -0,0 +1,297 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM timer_migration
+
+#if !defined(_TRACE_TIMER_MIGRATION_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_TIMER_MIGRATION_H
+
+#include <linux/tracepoint.h>
+
+/* Group events */
+TRACE_EVENT(tmigr_group_set,
+
+	TP_PROTO(struct tmigr_group *group),
+
+	TP_ARGS(group),
+
+	TP_STRUCT__entry(
+		__field( void *,	group		)
+		__field( unsigned int,	lvl		)
+		__field( unsigned int,	numa_node	)
+	),
+
+	TP_fast_assign(
+		__entry->group		= group;
+		__entry->lvl		= group->level;
+		__entry->numa_node	= group->numa_node;
+	),
+
+	TP_printk("group=%p lvl=%d numa=%d",
+		  __entry->group, __entry->lvl, __entry->numa_node)
+);
+
+TRACE_EVENT(tmigr_connect_child_parent,
+
+	TP_PROTO(struct tmigr_group *child),
+
+	TP_ARGS(child),
+
+	TP_STRUCT__entry(
+		__field( void *,	child		)
+		__field( void *,	parent		)
+		__field( unsigned int,	lvl		)
+		__field( unsigned int,	numa_node	)
+		__field( unsigned int,	num_children	)
+		__field( u32,		childmask	)
+	),
+
+	TP_fast_assign(
+		__entry->child		= child;
+		__entry->parent		= child->parent;
+		__entry->lvl		= child->parent->level;
+		__entry->numa_node	= child->parent->numa_node;
+		__entry->numa_node	= child->parent->num_children;
+		__entry->childmask	= child->childmask;
+	),
+
+	TP_printk("group=%p childmask=%0x parent=%p lvl=%d numa=%d num_children=%d",
+		  __entry->child,  __entry->childmask, __entry->parent,
+		  __entry->lvl, __entry->numa_node, __entry->num_children)
+);
+
+TRACE_EVENT(tmigr_connect_cpu_parent,
+
+	TP_PROTO(struct tmigr_cpu *tmc),
+
+	TP_ARGS(tmc),
+
+	TP_STRUCT__entry(
+		__field( void *,	parent		)
+		__field( unsigned int,	cpu		)
+		__field( unsigned int,	lvl		)
+		__field( unsigned int,	numa_node	)
+		__field( unsigned int,	num_children	)
+		__field( u32,		childmask	)
+	),
+
+	TP_fast_assign(
+		__entry->parent		= tmc->tmgroup;
+		__entry->cpu		= tmc->cpuevt.cpu;
+		__entry->lvl		= tmc->tmgroup->level;
+		__entry->numa_node	= tmc->tmgroup->numa_node;
+		__entry->numa_node	= tmc->tmgroup->num_children;
+		__entry->childmask	= tmc->childmask;
+	),
+
+	TP_printk("cpu=%d childmask=%0x parent=%p lvl=%d numa=%d num_children=%d",
+		  __entry->cpu,	 __entry->childmask, __entry->parent,
+		  __entry->lvl, __entry->numa_node, __entry->num_children)
+);
+
+DECLARE_EVENT_CLASS(tmigr_group_and_cpu,
+
+	TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask),
+
+	TP_ARGS(group, state, childmask),
+
+	TP_STRUCT__entry(
+		__field( void *,	group		)
+		__field( void *,	parent		)
+		__field( unsigned int,	lvl		)
+		__field( unsigned int,	numa_node	)
+		__field( u8,		active		)
+		__field( u8,		migrator	)
+		__field( u32,		childmask	)
+	),
+
+	TP_fast_assign(
+		__entry->group		= group;
+		__entry->parent		= group->parent;
+		__entry->lvl		= group->level;
+		__entry->numa_node	= group->numa_node;
+		__entry->active		= state.active;
+		__entry->migrator	= state.migrator;
+		__entry->childmask	= childmask;
+	),
+
+	TP_printk("group=%p lvl=%d numa=%d active=%0x migrator=%0x "
+		  "parent=%p childmask=%0x",
+		  __entry->group, __entry->lvl, __entry->numa_node,
+		  __entry->active, __entry->migrator,
+		  __entry->parent, __entry->childmask)
+);
+
+DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_inactive,
+
+	TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask),
+
+	TP_ARGS(group, state, childmask)
+);
+
+DEFINE_EVENT(tmigr_group_and_cpu, tmigr_group_set_cpu_active,
+
+	TP_PROTO(struct tmigr_group *group, union tmigr_state state, u32 childmask),
+
+	TP_ARGS(group, state, childmask)
+);
+
+/* CPU events*/
+DECLARE_EVENT_CLASS(tmigr_cpugroup,
+
+	TP_PROTO(struct tmigr_cpu *tmc),
+
+	TP_ARGS(tmc),
+
+	TP_STRUCT__entry(
+		__field( void *,	parent)
+		__field( unsigned int,	cpu)
+		__field( u64,		wakeup)
+	),
+
+	TP_fast_assign(
+		__entry->cpu		= tmc->cpuevt.cpu;
+		__entry->parent		= tmc->tmgroup;
+		__entry->wakeup		= tmc->wakeup;
+	),
+
+	TP_printk("cpu=%d parent=%p wakeup=%llu", __entry->cpu, __entry->parent, __entry->wakeup)
+);
+
+DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_new_timer,
+
+	TP_PROTO(struct tmigr_cpu *tmc),
+
+	TP_ARGS(tmc)
+);
+
+DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_active,
+
+	TP_PROTO(struct tmigr_cpu *tmc),
+
+	TP_ARGS(tmc)
+);
+
+DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_online,
+
+	TP_PROTO(struct tmigr_cpu *tmc),
+
+	TP_ARGS(tmc)
+);
+
+DEFINE_EVENT(tmigr_cpugroup, tmigr_cpu_offline,
+
+	TP_PROTO(struct tmigr_cpu *tmc),
+
+	TP_ARGS(tmc)
+);
+
+DEFINE_EVENT(tmigr_cpugroup, tmigr_handle_remote_cpu,
+
+	TP_PROTO(struct tmigr_cpu *tmc),
+
+	TP_ARGS(tmc)
+);
+
+DECLARE_EVENT_CLASS(tmigr_idle,
+
+	TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt),
+
+	TP_ARGS(tmc, nextevt),
+
+	TP_STRUCT__entry(
+		__field( void *,	parent)
+		__field( unsigned int,	cpu)
+		__field( u64,		nextevt)
+		__field( u64,		wakeup)
+	),
+
+	TP_fast_assign(
+		__entry->cpu		= tmc->cpuevt.cpu;
+		__entry->parent		= tmc->tmgroup;
+		__entry->nextevt	= nextevt;
+		__entry->wakeup		= tmc->wakeup;
+	),
+
+	TP_printk("cpu=%d parent=%p nextevt=%llu wakeup=%llu",
+		  __entry->cpu, __entry->parent, __entry->nextevt, __entry->wakeup)
+);
+
+DEFINE_EVENT(tmigr_idle, tmigr_cpu_idle,
+
+	TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt),
+
+	TP_ARGS(tmc, nextevt)
+);
+
+DEFINE_EVENT(tmigr_idle, tmigr_cpu_new_timer_idle,
+
+	TP_PROTO(struct tmigr_cpu *tmc, u64 nextevt),
+
+	TP_ARGS(tmc, nextevt)
+);
+
+TRACE_EVENT(tmigr_update_events,
+
+	TP_PROTO(struct tmigr_group *child, struct tmigr_group *group,
+		 union tmigr_state childstate,	union tmigr_state groupstate,
+		 u64 nextevt),
+
+	TP_ARGS(child, group, childstate, groupstate, nextevt),
+
+	TP_STRUCT__entry(
+		__field( void *,	child			)
+		__field( void *,	group			)
+		__field( u64,		nextevt			)
+		__field( u64,		group_next_expiry	)
+		__field( unsigned int,	group_lvl		)
+		__field( u8,		child_active		)
+		__field( u8,		group_active		)
+		__field( unsigned int,	child_evtcpu		)
+		__field( u64,		child_evt_expiry	)
+	),
+
+	TP_fast_assign(
+		__entry->child			= child;
+		__entry->group			= group;
+		__entry->nextevt		= nextevt;
+		__entry->group_next_expiry	= group->next_expiry;
+		__entry->group_lvl		= group->level;
+		__entry->child_active		= childstate.active;
+		__entry->group_active		= groupstate.active;
+		__entry->child_evtcpu		= child ? child->groupevt.cpu : 0;
+		__entry->child_evt_expiry	= child ? child->groupevt.nextevt.expires : 0;
+	),
+
+	TP_printk("child=%p group=%p group_lvl=%d child_active=%0x group_active=%0x "
+		  "nextevt=%llu next_expiry=%llu child_evt_expiry=%llu child_evtcpu=%d",
+		  __entry->child, __entry->group, __entry->group_lvl, __entry->child_active,
+		  __entry->group_active,
+		  __entry->nextevt, __entry->group_next_expiry, __entry->child_evt_expiry,
+		  __entry->child_evtcpu)
+);
+
+TRACE_EVENT(tmigr_handle_remote,
+
+	TP_PROTO(struct tmigr_group *group),
+
+	TP_ARGS(group),
+
+	TP_STRUCT__entry(
+		__field( void * ,	group	)
+		__field( unsigned int ,	lvl	)
+	),
+
+	TP_fast_assign(
+		__entry->group		= group;
+		__entry->lvl		= group->level;
+	),
+
+	TP_printk("group=%p lvl=%d",
+		   __entry->group, __entry->lvl)
+);
+
+#endif /*  _TRACE_TIMER_MIGRATION_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c
index de1905b0bae7..d3b7200a9f4a 100644
--- a/kernel/time/timer_migration.c
+++ b/kernel/time/timer_migration.c
@@ -14,6 +14,9 @@
 #include "timer_migration.h"
 #include "tick-internal.h"
 
+#define CREATE_TRACE_POINTS
+#include <trace/events/timer_migration.h>
+
 /*
  * The timer migration mechanism is built on a hierarchy of groups. The
  * lowest level group contains CPUs, the next level groups of CPU groups
@@ -537,6 +540,8 @@ static bool tmigr_active_up(struct tmigr_group *group,
 	 */
 	group->groupevt.ignore = true;
 
+	trace_tmigr_group_set_cpu_active(group, newstate, childmask);
+
 	return walk_done;
 }
 
@@ -546,6 +551,8 @@ static void __tmigr_cpu_activate(struct tmigr_cpu *tmc)
 
 	data.childmask = tmc->childmask;
 
+	trace_tmigr_cpu_active(tmc);
+
 	tmc->cpuevt.ignore = true;
 	WRITE_ONCE(tmc->wakeup, KTIME_MAX);
 	tmc->wakeup_recalc = false;
@@ -703,6 +710,9 @@ bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
 		data->firstexp = tmigr_next_groupevt_expires(group);
 	}
 
+	trace_tmigr_update_events(child, group, childstate, groupstate,
+				  nextexp);
+
 unlock:
 	raw_spin_unlock(&group->lock);
 
@@ -745,6 +755,8 @@ static u64 tmigr_new_timer(struct tmigr_cpu *tmc, u64 nextexp)
 	if (tmc->remote)
 		return KTIME_MAX;
 
+	trace_tmigr_cpu_new_timer(tmc);
+
 	tmc->cpuevt.ignore = false;
 	data.remote = false;
 
@@ -787,6 +799,8 @@ static u64 tmigr_handle_remote_cpu(unsigned int cpu, u64 now,
 		return next;
 	}
 
+	trace_tmigr_handle_remote_cpu(tmc);
+
 	tmc->remote = true;
 	WRITE_ONCE(tmc->wakeup, KTIME_MAX);
 
@@ -872,6 +886,7 @@ static bool tmigr_handle_remote_up(struct tmigr_group *group,
 
 	childmask = data->childmask;
 
+	trace_tmigr_handle_remote(group);
 again:
 	/*
 	 * Handle the group only if @childmask is the migrator or if the
@@ -1140,6 +1155,7 @@ u64 tmigr_cpu_new_timer(u64 nextexp)
 	 */
 	WRITE_ONCE(tmc->wakeup, ret);
 
+	trace_tmigr_cpu_new_timer_idle(tmc, nextexp);
 	raw_spin_unlock(&tmc->lock);
 	return ret;
 }
@@ -1243,6 +1259,8 @@ static bool tmigr_inactive_up(struct tmigr_group *group,
 		}
 	}
 
+	trace_tmigr_group_set_cpu_inactive(group, newstate, childmask);
+
 	return walk_done;
 }
 
@@ -1295,6 +1313,7 @@ u64 tmigr_cpu_deactivate(u64 nextexp)
 	 */
 	WRITE_ONCE(tmc->wakeup, ret);
 
+	trace_tmigr_cpu_idle(tmc, nextexp);
 	raw_spin_unlock(&tmc->lock);
 	return ret;
 }
@@ -1407,6 +1426,7 @@ static struct tmigr_group *tmigr_get_group(unsigned int cpu, int node,
 
 	/* Setup successful. Add it to the hierarchy */
 	list_add(&group->list, &tmigr_level_list[lvl]);
+	trace_tmigr_group_set(group);
 	return group;
 }
 
@@ -1424,6 +1444,8 @@ static void tmigr_connect_child_parent(struct tmigr_group *child,
 	raw_spin_unlock(&parent->lock);
 	raw_spin_unlock_irq(&child->lock);
 
+	trace_tmigr_connect_child_parent(child);
+
 	/*
 	 * To prevent inconsistent states, active children need to be active in
 	 * the new parent as well. Inactive children are already marked inactive
@@ -1504,6 +1526,8 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node)
 
 			raw_spin_unlock_irq(&group->lock);
 
+			trace_tmigr_connect_cpu_parent(tmc);
+
 			/* There are no children that need to be connected */
 			continue;
 		} else {
@@ -1571,6 +1595,7 @@ static int tmigr_cpu_online(unsigned int cpu)
 		WRITE_ONCE(tmc->wakeup, KTIME_MAX);
 	}
 	raw_spin_lock_irq(&tmc->lock);
+	trace_tmigr_cpu_online(tmc);
 	tmc->idle = timer_base_is_idle();
 	if (!tmc->idle)
 		__tmigr_cpu_activate(tmc);
@@ -1592,6 +1617,7 @@ static int tmigr_cpu_offline(unsigned int cpu)
 	 * offline; Therefore nextevt value is set to KTIME_MAX
 	 */
 	__tmigr_cpu_deactivate(tmc, KTIME_MAX);
+	trace_tmigr_cpu_offline(tmc);
 	raw_spin_unlock_irq(&tmc->lock);
 
 	return 0;
-- 
2.39.2

[PATCH v10 20/20] timers: Always queue timers on the local CPU

[Anna-Maria Behnsen]

The timer pull model is in place so we can remove the heuristics which try
to guess the best target CPU at enqueue/modification time.

All non pinned timers are queued on the local CPU in the separate storage
and eventually pulled at expiry time to a remote CPU.

Originally-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
v9:
 - Update to the changes of the preceding patches

v6:
 - Update TIMER_PINNED flag description.

v5:
 - Move WARN_ONCE() in add_timer_on() into a previous patch
 - Fold crystallball magic related hunks into this patch

v4: Update comment about TIMER_PINNED flag (heristic is removed)
---
 include/linux/timer.h | 14 ++++----------
 kernel/time/timer.c   | 34 +++++++++++++---------------------
 2 files changed, 17 insertions(+), 31 deletions(-)

diff --git a/include/linux/timer.h b/include/linux/timer.h
index 404bb31a95c7..4dd59e4e5681 100644
--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -50,16 +50,10 @@ struct timer_list {
  * workqueue locking issues. It's not meant for executing random crap
  * with interrupts disabled. Abuse is monitored!
  *
- * @TIMER_PINNED: A pinned timer will not be affected by any timer
- * placement heuristics (like, NOHZ) and will always expire on the CPU
- * on which the timer was enqueued.
- *
- * Note: Because enqueuing of timers can migrate the timer from one
- * CPU to another, pinned timers are not guaranteed to stay on the
- * initialy selected CPU.  They move to the CPU on which the enqueue
- * function is invoked via mod_timer() or add_timer().  If the timer
- * should be placed on a particular CPU, then add_timer_on() has to be
- * used.
+ * @TIMER_PINNED: A pinned timer will always expire on the CPU on which the
+ * timer was enqueued. When a particular CPU is required, add_timer_on()
+ * has to be used. Enqueue via mod_timer() and add_timer() is always done
+ * on the local CPU.
  */
 #define TIMER_CPUMASK		0x0003FFFF
 #define TIMER_MIGRATING		0x00040000
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index a52f68be8dfd..8d53718af21d 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -590,10 +590,13 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer)
 
 	/*
 	 * We might have to IPI the remote CPU if the base is idle and the
-	 * timer is not deferrable. If the other CPU is on the way to idle
-	 * then it can't set base->is_idle as we hold the base lock:
+	 * timer is pinned. If it is a non pinned timer, it is only queued
+	 * on the remote CPU, when timer was running during queueing. Then
+	 * everything is handled by remote CPU anyway. If the other CPU is
+	 * on the way to idle then it can't set base->is_idle as we hold
+	 * the base lock:
 	 */
-	if (base->is_idle)
+	if (base->is_idle && timer->flags & TIMER_PINNED)
 		wake_up_nohz_cpu(base->cpu);
 }
 
@@ -941,17 +944,6 @@ static inline struct timer_base *get_timer_base(u32 tflags)
 	return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK);
 }
 
-static inline struct timer_base *
-get_target_base(struct timer_base *base, unsigned tflags)
-{
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
-	if (static_branch_likely(&timers_migration_enabled) &&
-	    !(tflags & TIMER_PINNED))
-		return get_timer_cpu_base(tflags, get_nohz_timer_target());
-#endif
-	return get_timer_this_cpu_base(tflags);
-}
-
 static inline void __forward_timer_base(struct timer_base *base,
 					unsigned long basej)
 {
@@ -1106,7 +1098,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires, unsigned int option
 	if (!ret && (options & MOD_TIMER_PENDING_ONLY))
 		goto out_unlock;
 
-	new_base = get_target_base(base, timer->flags);
+	new_base = get_timer_this_cpu_base(timer->flags);
 
 	if (base != new_base) {
 		/*
@@ -2237,7 +2229,7 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
 		 * granularity skew (by design).
 		 */
 		if (!base_local->is_idle && time_after(nextevt, basej + 1)) {
-			base_local->is_idle = base_global->is_idle = true;
+			base_local->is_idle = true;
 			trace_timer_base_idle(true, base_local->cpu);
 		}
 		*idle = base_local->is_idle;
@@ -2303,13 +2295,13 @@ u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle)
 void timer_clear_idle(void)
 {
 	/*
-	 * We do this unlocked. The worst outcome is a remote enqueue sending
-	 * a pointless IPI, but taking the lock would just make the window for
-	 * sending the IPI a few instructions smaller for the cost of taking
-	 * the lock in the exit from idle path.
+	 * We do this unlocked. The worst outcome is a remote pinned timer
+	 * enqueue sending a pointless IPI, but taking the lock would just
+	 * make the window for sending the IPI a few instructions smaller
+	 * for the cost of taking the lock in the exit from idle
+	 * path. Required for BASE_LOCAL only.
 	 */
 	__this_cpu_write(timer_bases[BASE_LOCAL].is_idle, false);
-	__this_cpu_write(timer_bases[BASE_GLOBAL].is_idle, false);
 	trace_timer_base_idle(false, smp_processor_id());
 
 	/* Activate without holding the timer_base->lock */
-- 
2.39.2

Re: [PATCH v10 01/20] timers: Restructure get_next_timer_interrupt()

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:24PM +0100, Anna-Maria Behnsen a écrit :
> get_next_timer_interrupt() contains two parts for the next timer interrupt
> calculation. Those two parts are separated by forwarding the base
> clock. But the second part does not depend on the forwarded base
> clock.
> 
> Therefore restructure get_next_timer_interrupt() to keep things together
> which belong together.
> 
> No functional change.
> 
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Re: [PATCH v10 02/20] timers: Split out get next timer interrupt

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:25PM +0100, Anna-Maria Behnsen a écrit :
> Split out get_next_timer_interrupt() to be able to extend it and make it
> reusable for other call sites.
> 
> No functional change.
> 
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Re: [PATCH v10 03/20] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:26PM +0100, Anna-Maria Behnsen a écrit :
> @@ -889,12 +884,41 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu)
>  static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>  {
>  	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
> +	unsigned long basejiff = ts->last_jiffies;
>  	u64 basemono = ts->timer_expires_base;
> -	u64 expires = ts->timer_expires;
> +	bool timer_idle;
> +	u64 expires;
>  
>  	/* Make sure we won't be trying to stop it twice in a row. */
>  	ts->timer_expires_base = 0;
>  
> +	/*
> +	 * Now the tick should be stopped definitely - so the timer base needs
> +	 * to be marked idle as well to not miss a newly queued timer.
> +	 */
> +	expires = timer_base_try_to_set_idle(basejiff, basemono, &timer_idle);
> +	if (!timer_idle) {
> +		/*
> +		 * Do not clear tick_stopped here when it was already set - it

Can that really happen? Looking at __get_next_timer_interrupt(), you're making a
behavioural change: if base->is_idle was previously set and the next timer is
now below/equal a jiffy, base->is_idle is not going to be cleared by
__get_next_timer_interrupt().

Therefore you shouldn't observe ts->tick_stopped && !timer_idle

But I'm assuming that behavioural change wasn't intended?

> +		 * will be retained on the next idle iteration when the tick
> +		 * expired earlier than expected.

I'm a bit confused by this sentence.

> +		 */
> +		expires = basemono + TICK_NSEC;

Do you need this line?

> @@ -1147,11 +1175,6 @@ void tick_nohz_idle_stop_tick(void)
>  void tick_nohz_idle_retain_tick(void)
>  {
>  	tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched));

Looks like the content of tick_nohz_retain_tick() can move here now.

> -	/*
> -	 * Undo the effect of get_next_timer_interrupt() called from
> -	 * tick_nohz_next_event().
> -	 */
> -	timer_clear_idle();
>  }

Thanks.

Re: [PATCH v10 04/20] timers: Optimization for timer_base_try_to_set_idle()

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:27PM +0100, Anna-Maria Behnsen a écrit :
> When tick is stopped also the timer base is_idle flag is set. When
> reentering the timer_base_try_to_set_idle() with the tick stopped, there is
> no need to check whether the timer base needs to be set idle again. When a
> timer was enqueued in the meantime, this is already handled by the
> tick_nohz_next_event() call which was executed before
> tick_nohz_stop_tick().
> 
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
> ---
>  kernel/time/tick-sched.c |  2 +-
>  kernel/time/timer.c      | 11 ++++++++---
>  2 files changed, 9 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> index c6223afc801f..27f1a2ae7f39 100644
> --- a/kernel/time/tick-sched.c
> +++ b/kernel/time/tick-sched.c
> @@ -886,7 +886,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>  	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
>  	unsigned long basejiff = ts->last_jiffies;
>  	u64 basemono = ts->timer_expires_base;
> -	bool timer_idle;
> +	bool timer_idle = ts->tick_stopped;
>  	u64 expires;
>  
>  	/* Make sure we won't be trying to stop it twice in a row. */
> diff --git a/kernel/time/timer.c b/kernel/time/timer.c
> index 3a668060692e..2f69a485a070 100644
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -1999,13 +1999,18 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
>   * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases
>   * @basej:	base time jiffies
>   * @basem:	base time clock monotonic
> - * @idle:	pointer to store the value of timer_base->is_idle
> + * @idle:	pointer to store the value of timer_base->is_idle on return;
> + *		*idle contains the information whether tick was already stopped
>   *
> - * Returns the tick aligned clock monotonic time of the next pending
> - * timer or KTIME_MAX if no timer is pending.
> + * Returns the tick aligned clock monotonic time of the next pending timer or
> + * KTIME_MAX if no timer is pending. When tick was already stopped KTIME_MAX is
> + * returned as well.
>   */
>  u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle)
>  {
> +	if (*idle)
> +		return KTIME_MAX;

Ok now I see the reason behind the behavioural change.

So either:

* We remove the old behaviour consisting in clearing base->is_idle if the new
  next timer is within a jiffy while the tick is stopped. But then the changelog
  from the previous patch should state that and comments must be clarified.

or:

* We restore the old behaviour, making things a bit more complicated I guess.

Thanks.

Re: [PATCH v10 05/20] timers: Introduce add_timer() variants which modify timer flags

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:28PM +0100, Anna-Maria Behnsen a écrit :
>  timer flags

Yes ;-)

> 
> A timer might be used as a pinned timer (using add_timer_on()) and later on
> as non-pinned timer using add_timer(). When the "NOHZ timer pull at expiry
> model" is in place, the TIMER_PINNED flag is required to be used whenever a
> timer needs to expire on a dedicated CPU. Otherwise the flag must not be
> set if expiration on a dedicated CPU is not required.
> 
> add_timer_on()'s behavior will be changed during the preparation patches
> for the "NOHZ timer pull at expiry model" to unconditionally set
> TIMER_PINNED flag. To be able to clear/ set the flag when queueing a
> timer, two variants of add_timer() are introduced.
> 
> This is a preparatory patch and has no functional change.
> 
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Re: [PATCH v10 03/20] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:26PM +0100, Anna-Maria Behnsen a écrit :
>> @@ -889,12 +884,41 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu)
>>  static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>>  {
>>  	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
>> +	unsigned long basejiff = ts->last_jiffies;
>>  	u64 basemono = ts->timer_expires_base;
>> -	u64 expires = ts->timer_expires;
>> +	bool timer_idle;
>> +	u64 expires;
>>  
>>  	/* Make sure we won't be trying to stop it twice in a row. */
>>  	ts->timer_expires_base = 0;
>>  
>> +	/*
>> +	 * Now the tick should be stopped definitely - so the timer base needs
>> +	 * to be marked idle as well to not miss a newly queued timer.
>> +	 */
>> +	expires = timer_base_try_to_set_idle(basejiff, basemono, &timer_idle);
>> +	if (!timer_idle) {
>> +		/*
>> +		 * Do not clear tick_stopped here when it was already set - it
>
> Can that really happen? Looking at __get_next_timer_interrupt(), you're making a
> behavioural change: if base->is_idle was previously set and the next timer is
> now below/equal a jiffy, base->is_idle is not going to be cleared by
> __get_next_timer_interrupt().
>
> Therefore you shouldn't observe ts->tick_stopped && !timer_idle
>
> But I'm assuming that behavioural change wasn't intended?

It was intended to keep tick_stopped and base->is_idle in sync. So when
tick_stopped is set also base->is_idle needs to be set and dropping it
before tick_stopped is dropped will break the plan to keep it in sync.

>> +		 * will be retained on the next idle iteration when the tick
>> +		 * expired earlier than expected.
>
> I'm a bit confused by this sentence.

Me too :) It is there because of a previous version and I didn't cleaned
it up properly.

>> +		 */
>> +		expires = basemono + TICK_NSEC;
>
> Do you need this line?

No. After revisiting it once more, it is not required, as it should be
set properly by the return value of timer_base_try_to_set_idle(). So I
should be able to completely drop this first part of the if statement.

>
>> @@ -1147,11 +1175,6 @@ void tick_nohz_idle_stop_tick(void)
>>  void tick_nohz_idle_retain_tick(void)
>>  {
>>  	tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched));
>
> Looks like the content of tick_nohz_retain_tick() can move here now.

I can do this.

>> -	/*
>> -	 * Undo the effect of get_next_timer_interrupt() called from
>> -	 * tick_nohz_next_event().
>> -	 */
>> -	timer_clear_idle();
>>  }
>
> Thanks.

Thanks,

	Anna-Maria

Re: [PATCH v10 04/20] timers: Optimization for timer_base_try_to_set_idle()

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:27PM +0100, Anna-Maria Behnsen a écrit :
>> When tick is stopped also the timer base is_idle flag is set. When
>> reentering the timer_base_try_to_set_idle() with the tick stopped, there is
>> no need to check whether the timer base needs to be set idle again. When a
>> timer was enqueued in the meantime, this is already handled by the
>> tick_nohz_next_event() call which was executed before
>> tick_nohz_stop_tick().
>> 
>> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
>> ---
>>  kernel/time/tick-sched.c |  2 +-
>>  kernel/time/timer.c      | 11 ++++++++---
>>  2 files changed, 9 insertions(+), 4 deletions(-)
>> 
>> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
>> index c6223afc801f..27f1a2ae7f39 100644
>> --- a/kernel/time/tick-sched.c
>> +++ b/kernel/time/tick-sched.c
>> @@ -886,7 +886,7 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>>  	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
>>  	unsigned long basejiff = ts->last_jiffies;
>>  	u64 basemono = ts->timer_expires_base;
>> -	bool timer_idle;
>> +	bool timer_idle = ts->tick_stopped;
>>  	u64 expires;
>>  
>>  	/* Make sure we won't be trying to stop it twice in a row. */
>> diff --git a/kernel/time/timer.c b/kernel/time/timer.c
>> index 3a668060692e..2f69a485a070 100644
>> --- a/kernel/time/timer.c
>> +++ b/kernel/time/timer.c
>> @@ -1999,13 +1999,18 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
>>   * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases
>>   * @basej:	base time jiffies
>>   * @basem:	base time clock monotonic
>> - * @idle:	pointer to store the value of timer_base->is_idle
>> + * @idle:	pointer to store the value of timer_base->is_idle on return;
>> + *		*idle contains the information whether tick was already stopped
>>   *
>> - * Returns the tick aligned clock monotonic time of the next pending
>> - * timer or KTIME_MAX if no timer is pending.
>> + * Returns the tick aligned clock monotonic time of the next pending timer or
>> + * KTIME_MAX if no timer is pending. When tick was already stopped KTIME_MAX is
>> + * returned as well.
>>   */
>>  u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle)
>>  {
>> +	if (*idle)
>> +		return KTIME_MAX;
>
> Ok now I see the reason behind the behavioural change.
>
> So either:
>
> * We remove the old behaviour consisting in clearing base->is_idle if the new
>   next timer is within a jiffy while the tick is stopped. But then the changelog
>   from the previous patch should state that and comments must be clarified.
>

I would like to take 'either'. I thought the changelog already mentioned
it. But maybe I have to make it more explicit. I'll go and rework the
comments once more.

Thanks,

	Anna-Maria

Re: [PATCH v10 05/20] timers: Introduce add_timer() variants which modify timer flags

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:28PM +0100, Anna-Maria Behnsen a écrit :
>>  timer flags
>
> Yes ;-)

Something got broken when importing the quilt series into git... This
also happend to some other patches/commit messages. I'll have a look at
it and will fix it.

Thanks,

	Anna-Maria

Re: [PATCH v10 03/20] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Frederic Weisbecker]

Le Mon, Jan 22, 2024 at 12:45:03PM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Mon, Jan 15, 2024 at 03:37:26PM +0100, Anna-Maria Behnsen a écrit :
> >> @@ -889,12 +884,41 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu)
> >>  static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
> >>  {
> >>  	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
> >> +	unsigned long basejiff = ts->last_jiffies;
> >>  	u64 basemono = ts->timer_expires_base;
> >> -	u64 expires = ts->timer_expires;
> >> +	bool timer_idle;
> >> +	u64 expires;
> >>  
> >>  	/* Make sure we won't be trying to stop it twice in a row. */
> >>  	ts->timer_expires_base = 0;
> >>  
> >> +	/*
> >> +	 * Now the tick should be stopped definitely - so the timer base needs
> >> +	 * to be marked idle as well to not miss a newly queued timer.
> >> +	 */
> >> +	expires = timer_base_try_to_set_idle(basejiff, basemono, &timer_idle);
> >> +	if (!timer_idle) {
> >> +		/*
> >> +		 * Do not clear tick_stopped here when it was already set - it
> >
> > Can that really happen? Looking at __get_next_timer_interrupt(), you're making a
> > behavioural change: if base->is_idle was previously set and the next timer is
> > now below/equal a jiffy, base->is_idle is not going to be cleared by
> > __get_next_timer_interrupt().
> >
> > Therefore you shouldn't observe ts->tick_stopped && !timer_idle
> >
> > But I'm assuming that behavioural change wasn't intended?
> 
> It was intended to keep tick_stopped and base->is_idle in sync. So when
> tick_stopped is set also base->is_idle needs to be set and dropping it
> before tick_stopped is dropped will break the plan to keep it in sync.

Ok that sounds good.

Thanks!

Re: [PATCH v10 04/20] timers: Optimization for timer_base_try_to_set_idle()

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:27PM +0100, Anna-Maria Behnsen a écrit :
> When tick is stopped also the timer base is_idle flag is set. When
> reentering the timer_base_try_to_set_idle() with the tick stopped, there is
> no need to check whether the timer base needs to be set idle again. When a
> timer was enqueued in the meantime, this is already handled by the
> tick_nohz_next_event() call which was executed before
> tick_nohz_stop_tick().
> 
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Re: [PATCH v10 09/20] timers: Split next timer interrupt logic

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:32PM +0100, Anna-Maria Behnsen a écrit :
> Split the logic for getting next timer interrupt (no matter of recalculated
> or already stored in base->next_expiry) into a separate function named
> next_timer_interrupt(). Make it available to local call sites only.
> 
> No functional change.
> 
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Re: [PATCH v10 14/20] timers: Restructure internal locking

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:37PM +0100, Anna-Maria Behnsen a écrit :
> From: "Richard Cochran (linutronix GmbH)" <richardcochran@gmail.com>
> 
> Move the locking out from __run_timers() to the call sites, so the
> protected section can be extended at the call site. Preparatory patch for
> changing the NOHZ timer placement to a pull at expiry time model.
> 
> No functional change.
> 
> Signed-off-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Re: [PATCH v10 15/20] timers: Check if timers base is handled already

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:38PM +0100, Anna-Maria Behnsen a écrit :
> Due to the conversion of the NOHZ timer placement to a pull at expiry
> time model, the per CPU timer bases with non pinned timers are no
> longer handled only by the local CPU. In case a remote CPU already
> expires the non pinned timers base of the local CPU, nothing more
> needs to be done by the local CPU. A check at the begin of the expire
> timers routine is required, because timer base lock is dropped before
> executing the timer callback function.
> 
> This is a preparatory work, but has no functional impact right now.
> 
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Re: [PATCH v10 16/20] tick/sched: Split out jiffies update helper function

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:39PM +0100, Anna-Maria Behnsen a écrit :
> From: "Richard Cochran (linutronix GmbH)" <richardcochran@gmail.com>
> 
> The logic to get the time of the last jiffies update will be needed by
> the timer pull model as well.
> 
> Move the code into a global function in anticipation of the new caller.
> 
> No functional change.
> 
> Signed-off-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Re: [PATCH v10 17/20] timers: Introduce function to check timer base is_idle flag

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:40PM +0100, Anna-Maria Behnsen a écrit :
>  flag
> 
> To prepare for the conversion of the NOHZ timer placement to a pull at
> expiry time model it's required to have a function that returns the value
> of the is_idle flag of the timer base to keep the hierarchy states during
> online in sync with timer base state.
> 
> No functional change.
> 
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

On Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen wrote:
> +/**
> + * tmigr_quick_check() - Quick forecast of next tmigr event when CPU wants to
> + *			 go idle
> + *
> + * Returns KTIME_MAX, when it is probable that nothing has to be done (not the
> + * only one in the level 0 group; and if it is the only one in level 0 group,
> + * but there are more than a single group active in top level)
> + *
> + * Returns first expiry of the top level group, when it is the only one in level
> + * 0 and top level also only has a single active child.
> + */
> +u64 tmigr_quick_check(void)
> +{
> +	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
> +	struct tmigr_group *topgroup;
> +	struct list_head lvllist;
> +
> +	if (tmigr_is_not_available(tmc))
> +		return KTIME_MAX;

Offline CPUs are supposed to handle their own global timers.

So instead of returning KTIME_MAX here, shouldn't we pass instead
tevt->global as a parameter and return that value?

Otherwise the quick check will simply ignore the next global event of this CPU
if it's before the next local event.

> +
> +	if (WARN_ON_ONCE(tmc->idle))
> +		return KTIME_MAX;

Same here I guess...

> +
> +	if (!tmigr_check_migrator_and_lonely(tmc->tmgroup, tmc->childmask))
> +		return KTIME_MAX;

This one makes sense.

> +
> +	for (int i = tmigr_hierarchy_levels; i > 0 ; i--) {
> +		lvllist = tmigr_level_list[i - 1];
> +		if (list_is_singular(&lvllist)) {
> +			topgroup = list_first_entry(&lvllist, struct
> tmigr_group, list);

Is it safe against concurrent allocation failure in hotplug?

If the list is seen singular, then concurrently a CPU comes up and creates/add
a new group. The current CPU actually fetches it instead of the current group
because it's not singular anymore. But then some higher level group
allocation fails and the newly added first entry is removed.

list_is_singular() looks safe. But list_first_entry isn't. You can create
list_first_entry_rcu:

#define list_first_entry_rcu(ptr, type, member) \
	list_entry_rcu((ptr)->next, type, member)

Protected inside rcu_read_lock() until the below READ_ONCE().

And then use list_del_rcu/list_add_rcu/kfree_rcu on the update side.

Isn't it possible to walk through group->parent instead?

> +
> +			if (tmigr_check_lonely(topgroup))
> +				return READ_ONCE(topgroup->next_expiry);
> +		} else {
> +			continue;
> +		}
> +	}
> +
> +	return KTIME_MAX;

I'm less sure about that return value.

Thanks.

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

On Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen wrote:
> + * Protection of the tmigr group state information:
> + * ------------------------------------------------
> + *
> + * The state information with the list of active children and migrator needs to
> + * be protected by a sequence counter. It prevents a race when updates in child
> + * groups are propagated in changed order. The state update is performed
> + * lockless and group wise. The following scenario describes what happens
> + * without updating the sequence counter:
> + *
> + * Therefore, let's take three groups and four CPUs (CPU2 and CPU3 as well
> + * as GRP0:1 will not change during the scenario):
> + *
> + *    LVL 1            [GRP1:0]
> + *                     migrator = GRP0:1
> + *                     active   = GRP0:0, GRP0:1
> + *                   /                \
> + *    LVL 0  [GRP0:0]                  [GRP0:1]
> + *           migrator = CPU0           migrator = CPU2
> + *           active   = CPU0           active   = CPU2
> + *              /         \                /         \
> + *    CPUs     0           1              2           3
> + *             active      idle           active      idle
> + *
> + *
> + * 1. CPU0 goes idle. As the update is performed group wise, in the first step
> + *    only GRP0:0 is updated. The update of GRP1:0 is pending as CPU0 has to
> + *    walk the hierarchy.
> + *
> + *    LVL 1            [GRP1:0]
> + *                     migrator = GRP0:1
> + *                     active   = GRP0:0, GRP0:1
> + *                   /                \
> + *    LVL 0  [GRP0:0]                  [GRP0:1]
> + *       --> migrator = TMIGR_NONE     migrator = CPU2
> + *       --> active   =                active   = CPU2
> + *              /         \                /         \
> + *    CPUs     0           1              2           3
> + *         --> idle        idle           active      idle
> + *
> + * 2. While CPU0 goes idle and continues to update the state, CPU1 comes out of
> + *    idle. CPU1 updates GRP0:0. The update for GRP1:0 is pending as CPU1 also
> + *    has to the hierarchy. Both CPUs (CPU0 and CPU1) now walk the hierarchy to
> + *    perform the needed update from their point of view. The currently visible
> + *    state looks the following:
> + *
> + *    LVL 1            [GRP1:0]
> + *                     migrator = GRP0:1
> + *                     active   = GRP0:0, GRP0:1
> + *                   /                \
> + *    LVL 0  [GRP0:0]                  [GRP0:1]
> + *       --> migrator = CPU1           migrator = CPU2
> + *       --> active   = CPU1           active   = CPU2
> + *              /         \                /         \
> + *    CPUs     0           1              2           3
> + *             idle    --> active         active      idle
> + *
> + * 3. Here is the race condition: CPU1 managed to propagate its changes (from
> + *    step 2) through the hierarchy to GRP1:0 before CPU0 (step 1) did. The
> + *    active members of GRP1:0 remain unchanged after the update since it is
> + *    still valid from CPU1 current point of view:
> + *
> + *    LVL 1            [GRP1:0]
> + *                 --> migrator = GRP0:1
> + *                 --> active   = GRP0:0, GRP0:1
> + *                   /                \
> + *    LVL 0  [GRP0:0]                  [GRP0:1]
> + *           migrator = CPU1           migrator = CPU2
> + *           active   = CPU1           active   = CPU2
> + *              /         \                /         \
> + *    CPUs     0           1              2           3
> + *             idle        active         active      idle

So let's take this scenario and suppose we are at this stage. CPU 1
has propagated the state up to [GRP1:0] and CPU 0 is going to do it
but hasn't yet.

> +static bool tmigr_inactive_up(struct tmigr_group *group,
> +			      struct tmigr_group *child,
> +			      void *ptr)
> +{
> +	union tmigr_state curstate, newstate, childstate;
> +	struct tmigr_walk *data = ptr;
> +	bool walk_done;
> +	u8 childmask;
> +
> +	childmask = data->childmask;
> +	curstate.state = atomic_read(&group->migr_state);

And now suppose CPU 0 arrives here and sees the group->migr_state change
performed by CPU 1. So it's all good, right? The below atomic_cmpxchg()
will success on the first take.

> +	childstate.state = 0;
> +
> +	do {
> +		if (child)
> +			childstate.state = atomic_read(&child->migr_state);

But then how do you guarantee that CPU 0 will load here the version of
child->migr_state modified by CPU 1? What prevents from loading the stale value?
The one that was modified by CPU 0 instead? Nothing because the two above reads
are unordered. As a result, CPU 0 may ignore the fact that CPU 1 is up and
wrongly report GRP0:0 as active up to GRP1:0.

One way to solve this is to change the above atomic_read(&group->migr_state)
into atomic_read_acquire(&group->migr_state). It's cheap and pairs with the
order enforced by the upwards successful cmpxchg calls.

> +
> +		newstate = curstate;
> +		walk_done = true;
> +
> +		/* Reset active bit when the child is no longer active */
> +		if (!childstate.active)
> +			newstate.active &= ~childmask;
> +
> +		if (newstate.migrator == childmask) {
> +			/*
> +			 * Find a new migrator for the group, because the child
> +			 * group is idle!
> +			 */
> +			if (!childstate.active) {
> +				unsigned long new_migr_bit, active = newstate.active;
> +
> +				new_migr_bit = find_first_bit(&active, BIT_CNT);
> +
> +				if (new_migr_bit != BIT_CNT) {
> +					newstate.migrator = BIT(new_migr_bit);
> +				} else {
> +					newstate.migrator = TMIGR_NONE;
> +
> +					/* Changes need to be propagated */
> +					walk_done = false;
> +				}
> +			}
> +		}
> +
> +		newstate.seq++;
> +
> +		WARN_ON_ONCE((newstate.migrator != TMIGR_NONE) && !(newstate.active));
> +
> +	} while (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state));

Similarly, I seem to remember that a failing cmpxchg() doesn't imply a full
memory barrier. If it's the case, you may need to reload &group->migr_state
using an acquire barrier. But lemme check that...

Thanks.

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> +static bool tmigr_inactive_up(struct tmigr_group *group,
> +			      struct tmigr_group *child,
> +			      void *ptr)
> +{
> +	union tmigr_state curstate, newstate, childstate;
> +	struct tmigr_walk *data = ptr;
> +	bool walk_done;
> +	u8 childmask;
> +
> +	childmask = data->childmask;
> +	curstate.state = atomic_read(&group->migr_state);
> +	childstate.state = 0;
> +
> +	do {

So I got the confirmation from Boqun (+Cc) and Paul that a failing cmpxchg
may not order the load of the old value against subsequent loads. And
that may apply to atomic_try_cmpxchg() as well.

Therefore you not only need to turn group->migr_state read into
an atomic_read_acquire() but you also need to do this on each iteration
of this loop. For example you can move the read_acquire right here.

Thanks.

> +		if (child)
> +			childstate.state = atomic_read(&child->migr_state);
> +
> +		newstate = curstate;
> +		walk_done = true;
> +
> +		/* Reset active bit when the child is no longer active */
> +		if (!childstate.active)
> +			newstate.active &= ~childmask;
> +
> +		if (newstate.migrator == childmask) {
> +			/*
> +			 * Find a new migrator for the group, because the child
> +			 * group is idle!
> +			 */
> +			if (!childstate.active) {
> +				unsigned long new_migr_bit, active = newstate.active;
> +
> +				new_migr_bit = find_first_bit(&active, BIT_CNT);
> +
> +				if (new_migr_bit != BIT_CNT) {
> +					newstate.migrator = BIT(new_migr_bit);
> +				} else {
> +					newstate.migrator = TMIGR_NONE;
> +
> +					/* Changes need to be propagated */
> +					walk_done = false;
> +				}
> +			}
> +		}
> +
> +		newstate.seq++;
> +
> +		WARN_ON_ONCE((newstate.migrator != TMIGR_NONE) && !(newstate.active));
> +
> +	} while (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state));

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> On Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen wrote:
>> +/**
>> + * tmigr_quick_check() - Quick forecast of next tmigr event when CPU wants to
>> + *			 go idle
>> + *
>> + * Returns KTIME_MAX, when it is probable that nothing has to be done (not the
>> + * only one in the level 0 group; and if it is the only one in level 0 group,
>> + * but there are more than a single group active in top level)
>> + *
>> + * Returns first expiry of the top level group, when it is the only one in level
>> + * 0 and top level also only has a single active child.
>> + */
>> +u64 tmigr_quick_check(void)
>> +{
>> +	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
>> +	struct tmigr_group *topgroup;
>> +	struct list_head lvllist;
>> +
>> +	if (tmigr_is_not_available(tmc))
>> +		return KTIME_MAX;
>
> Offline CPUs are supposed to handle their own global timers.
>
> So instead of returning KTIME_MAX here, shouldn't we pass instead
> tevt->global as a parameter and return that value?
>
> Otherwise the quick check will simply ignore the next global event of this CPU
> if it's before the next local event.

I thought about this as well. I skipped it to keep it as simple as
possible - as this is only a forecast and might change (CPUs will come
online/go offline/deletion of timers...). But I can integrate it to keep
it more precise.

>> +
>> +	if (WARN_ON_ONCE(tmc->idle))
>> +		return KTIME_MAX;
>
> Same here I guess...

Yes.

>> +
>> +	if (!tmigr_check_migrator_and_lonely(tmc->tmgroup, tmc->childmask))
>> +		return KTIME_MAX;
>
> This one makes sense.
>
>> +
>> +	for (int i = tmigr_hierarchy_levels; i > 0 ; i--) {
>> +		lvllist = tmigr_level_list[i - 1];
>> +		if (list_is_singular(&lvllist)) {
>> +			topgroup = list_first_entry(&lvllist, struct
>> tmigr_group, list);
>
> Is it safe against concurrent allocation failure in hotplug?

As you pointed out, it isn't!

> If the list is seen singular, then concurrently a CPU comes up and creates/add
> a new group. The current CPU actually fetches it instead of the current group
> because it's not singular anymore. But then some higher level group
> allocation fails and the newly added first entry is removed.
>
> list_is_singular() looks safe. But list_first_entry isn't. You can create
> list_first_entry_rcu:
>
> #define list_first_entry_rcu(ptr, type, member) \
> 	list_entry_rcu((ptr)->next, type, member)
>
> Protected inside rcu_read_lock() until the below READ_ONCE().
>
> And then use list_del_rcu/list_add_rcu/kfree_rcu on the update side.
>
> Isn't it possible to walk through group->parent instead?

Yes. Sure! This is possible and maybe also easier. I cannot remember why
I implemented it this way...

>> +
>> +			if (tmigr_check_lonely(topgroup))
>> +				return READ_ONCE(topgroup->next_expiry);

When I hand in tevt->global as a parameter, I'll need to compare the
first expiry of the toplevel group and the tevt->global value and return
the earlier expiry. Only a single child is active in top level, so it
might be that this CPU is the last active CPU in hierarchy.

I didn't check all the way to the top whether all groups are
'lonely'. So when the top level group has only a single active child, it
is also possible that the child of the top level group has two active
children... Then a return of KTIME_MAX would be also a more precise
forecast.

This quick check is there to keep the overhead minimal when checking
whether it might be possible to go idle. So I don't know, if we should
add this additional check per level (which is pretty simple when using
group->parent for walking the hierarchy). What do you think?

>> +		} else {
>> +			continue;
>> +		}
>> +	}
>> +
>> +	return KTIME_MAX;
>
> I'm less sure about that return value.


This is ok, because there is not only a single child active in top
level. This CPU is definitely not the last active CPU in hierarchy. So
it is likely that some other CPU will handle tevt->global of this CPU.

I'll add a comment :)

Thanks,

	Anna-Maria

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> +int tmigr_requires_handle_remote(void)
> +{
> +	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
> +	struct tmigr_remote_data data;
> +	unsigned int ret = 0;
> +	unsigned long jif;
> +
> +	if (tmigr_is_not_available(tmc))
> +		return ret;
> +
> +	data.now = get_jiffies_update(&jif);
> +	data.childmask = tmc->childmask;
> +	data.firstexp = KTIME_MAX;
> +	data.tmc_active = !tmc->idle;
> +	data.check = false;
> +
> +	/*
> +	 * If the CPU is active, walk the hierarchy to check whether a remote
> +	 * expiry is required.
> +	 *
> +	 * Check is done lockless as interrupts are disabled and @tmc->idle is
> +	 * set only by the local CPU.
> +	 */
> +	if (!tmc->idle) {
> +		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
> +
> +		if (data.firstexp != KTIME_MAX)
> +			ret = 1;
> +
> +		return ret;
> +	}
> +
> +	/*
> +	 * If the CPU is idle, check whether the recalculation of @tmc->wakeup
> +	 * is required. @tmc->wakeup_recalc is set, when the last active CPU
> +	 * went offline. The last active CPU delegated the handling of the timer
> +	 * migration hierarchy to another (this) CPU by updating this flag and
> +	 * sending a reschedule.
> +	 *
> +	 * Racy lockless check is valid:
> +	 * - @tmc->wakeup_recalc is set by the remote CPU before it issues
> +	 *   reschedule IPI.
> +	 * - As interrupts are disabled here this CPU will either observe
> +	 *   @tmc->wakeup_recalc set before the reschedule IPI can be handled or
> +	 *   it will observe it when this function is called again on return
> +	 *   from handling the reschedule IPI.
> +	 */
> +	if (tmc->wakeup_recalc) {
> +		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
> +
> +		if (data.firstexp != KTIME_MAX)
> +			ret = 1;
> +
> +		raw_spin_lock(&tmc->lock);
> +		WRITE_ONCE(tmc->wakeup, data.firstexp);
> +		tmc->wakeup_recalc = false;
> +		raw_spin_unlock(&tmc->lock);

Suppose we have:

            [GRP1:0]
            migrator = GRP0:1
            active   = GRP0:0, GRP0:1
              /                \
     [GRP0:0]                  [GRP0:1]
     migrator = CPU 1         migrator = CPU 3
     active   = CPU 1         active   = CPU 3
       /         \               /         \
CPUs  0           1             2           3
     idle        active        idle        active

CPU 0 and CPU 2 have no timer.
CPU 1 has a timer in a few millisecs.

            [GRP1:0]
            migrator = GRP0:1
            active   = GRP0:1
              /                \
     [GRP0:0]                  [GRP0:1]
     migrator = NONE           migrator = CPU 3
     active   = NONE           active   = CPU 3
       /         \                /         \
CPUs  0           1              2           3
     idle        idle         idle        active


CPU 1 went idle, CPU 3 will take care of CPU 1's timer. Then come two
things happening at the same time: CPU 0 has a timer interrupt, due to
RCU callbacks handling for example, and CPU 3 goes offline:

CPU 0                                   CPU 3
-----                                   -----
                                        // On top level [GRP1:0], just set migrator = TMIGR_NONE
                                        tmigr_inactive_up() {
                                            cpu = cpumask_any_but(cpu_online_mask, cpu);
                                            //cpu == 0
                                            tmc_resched = per_cpu_ptr(&tmigr_cpu, CPU 0);
                                            raw_spin_lock(&tmc_resched->lock);
                                            tmc_resched->wakeup_recalc = true;
                                            raw_spin_unlock(&tmc_resched->lock);
// timer interrupt
run_local_timers() {
    tmigr_requires_handle_remote() {
        data.firstexp = KTIME_MAX;
        // CPU 0 sees the tmc_resched->wakeup_recalc
        // latest update
        if (tmc->wakeup_recalc) {
            tmigr_requires_handle_remote_up() {
                // CPU 0 doesn't see GRP0:0 
                // latest update from CPU 1,
                // because it has no locking
                // and does a racy check.
        	    if (!tmigr_check_migrator(group, childmask))
                    return true;
            }
            raw_spin_lock(&tmc->lock);
            WRITE_ONCE(tmc->wakeup, data.firstexp);
            tmc->wakeup_recalc = false;
            raw_spin_unlock(&tmc->lock)
            return 0;
        }
                                            // IPI is sent only now
		                                    smp_send_reschedule(cpu);
                                            }


There is nothing that prevents CPU 0 from not seeing the hierarchy updates from
other CPUs since it checks the migrators in a racy way. As a result the timer of
CPU 1 may be ignored by CPU 0.

You'd need to lock the tmc while calling tmigr_requires_handle_remote_up(), so
that CPU 0 "inherits" everything that CPU 3 has seen, and that includes changes
from CPU 1.


But I see that tmigr_cpu_new_timer() does it right. Wouldn't it be possible to
exlusively let tmigr_cpu_new_timer() handle the wakeup_recalc thing? This is
going to be called after the end of the IRQ (whether timer interrupt or sched
IPI) in any case.

Thanks.

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> +static bool tmigr_inactive_up(struct tmigr_group *group,
>> +			      struct tmigr_group *child,
>> +			      void *ptr)
>> +{
>> +	union tmigr_state curstate, newstate, childstate;
>> +	struct tmigr_walk *data = ptr;
>> +	bool walk_done;
>> +	u8 childmask;
>> +
>> +	childmask = data->childmask;
>> +	curstate.state = atomic_read(&group->migr_state);
>> +	childstate.state = 0;
>> +
>> +	do {
>
> So I got the confirmation from Boqun (+Cc) and Paul that a failing cmpxchg
> may not order the load of the old value against subsequent loads. And
> that may apply to atomic_try_cmpxchg() as well.
>
> Therefore you not only need to turn group->migr_state read into
> an atomic_read_acquire() but you also need to do this on each iteration
> of this loop. For example you can move the read_acquire right here.

I tried to read and understand more about the memory barriers especially
the acquire/release stuff. So please correct me whenever I'm wrong.

We have to make sure that the child/group state values contain the last
updates and prevent reordering to be able to rely on those values.

So I understand, that we need the atomic_read_acquire() here for the
child state, because we change the group state accordingly and need to
make sure, that it contains the last update of it. The cmpxchg which
writes the child state is (on success) a full memory barrier. And the
atomic_read_acquire() makes sure all preceding "critical sections"
(which ends with the full memory barrier) are visible. Is this right?

To make sure the proper states are used, atomic_read_acquire() is then
also required in:
  - tmigr_check_migrator()
  - tmigr_check_migrator_and_lonely()
  - tmigr_check_lonely()
  - tmigr_new_timer_up() (for childstate and groupstate)
  - tmigr_connect_child_parent()
Right?

Regarding the pairing of acquire: What happens when two
atomic_read_acquire() are executed afterwards without pairing 1:1 with a
release or stronger memory barrier?

Now I want to understand the case for the group state here and also in
active_up path. When reading it without acquire, it is possible, that
not all changes are visible due to reordering,... . But then the worst
outcome would be that the cmpxchg fails and the loop has to be done once
more? Is this right?

I know that memory barriers are not for free and redo the loop is also
not for free. But I don't know which of both is worse. At least in
inactive_up() path, we are not in the critical path. In active_up() it
would be good to take the less expensive option.

I want to understand the atomic_try_cmpxchg_acquire() variant: The Read
is an acquire, so even if the compare/write fails, the value which is
handed back is the one which was update last with a succesful cmpxchg
and then we can rely on this value?

Thanks a lot in advance for the help to understand this topic a little
better!

	Anna-Maria

>
> Thanks.
>
>> +		if (child)
>> +			childstate.state = atomic_read(&child->migr_state);
>> +
>> +		newstate = curstate;
>> +		walk_done = true;
>> +
>> +		/* Reset active bit when the child is no longer active */
>> +		if (!childstate.active)
>> +			newstate.active &= ~childmask;
>> +
>> +		if (newstate.migrator == childmask) {
>> +			/*
>> +			 * Find a new migrator for the group, because the child
>> +			 * group is idle!
>> +			 */
>> +			if (!childstate.active) {
>> +				unsigned long new_migr_bit, active = newstate.active;
>> +
>> +				new_migr_bit = find_first_bit(&active, BIT_CNT);
>> +
>> +				if (new_migr_bit != BIT_CNT) {
>> +					newstate.migrator = BIT(new_migr_bit);
>> +				} else {
>> +					newstate.migrator = TMIGR_NONE;
>> +
>> +					/* Changes need to be propagated */
>> +					walk_done = false;
>> +				}
>> +			}
>> +		}
>> +
>> +		newstate.seq++;
>> +
>> +		WARN_ON_ONCE((newstate.migrator != TMIGR_NONE) && !(newstate.active));
>> +
>> +	} while (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state));

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Paul E. McKenney]

On Sat, Jan 27, 2024 at 11:54:46PM +0100, Frederic Weisbecker wrote:
> Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> > +static bool tmigr_inactive_up(struct tmigr_group *group,
> > +			      struct tmigr_group *child,
> > +			      void *ptr)
> > +{
> > +	union tmigr_state curstate, newstate, childstate;
> > +	struct tmigr_walk *data = ptr;
> > +	bool walk_done;
> > +	u8 childmask;
> > +
> > +	childmask = data->childmask;
> > +	curstate.state = atomic_read(&group->migr_state);
> > +	childstate.state = 0;
> > +
> > +	do {
> 
> So I got the confirmation from Boqun (+Cc) and Paul that a failing cmpxchg
> may not order the load of the old value against subsequent loads. And
> that may apply to atomic_try_cmpxchg() as well.

Plus we checked with Mark Rutland, who agreed and who went further kindly
submitted a patch to clarify the documentation:

https://lore.kernel.org/lkml/20240129122250.1086874-1-mark.rutland@arm.com/

Here is an LKMM litmus test demonstrating that failing cmpxchg() does not
provide ordering:

------------------------------------------------------------------------

C cmpxchg-fail-order

{}

P0(int *x, int *y, int *z)
{
      int r0;
      int r1;

      WRITE_ONCE(*x, 1);
      r1 = cmpxchg(z, 1, 0);
      r0 = READ_ONCE(*y);
}

P1(int *x, int *y, int *z)
{
      int r0;
      int r1;
 
      WRITE_ONCE(*y, 1);
      r1 = cmpxchg(z, 1, 0);
      r0 = READ_ONCE(*x);
}

locations[0:r1;1:r1]
exists (0:r0=0 /\ 1:r0=0)

------------------------------------------------------------------------

Yes, this is cmpxchg() rather than atomic_cmpxchg(), but both have the
same ordering properties.

Here P0() is one thread and P1() the other.  Their parameters are the
global shared variables.  The "{}" preceding P0() is where initialization
could be placed, but this test is fine with the default initialization
of zero, for example, given that both instances of cmpxchg() specify
the value 1 as the old value (and thus both instances will fail).

The "locations" clause says to print the final values of P0()'s and P1()'s
local variables r1, where the leading "0:" selects P0() and the leading
"1:" selects P1().  And yes, the first process must be named "P0" and
subsequent processes' names must consist of "P" followed by a consecutive
number, so P0(), P1(), P2(), P3(), ...

The "exists" clause gives an expression to be tested "at the end of time".
Again, the "0:" and "1:" select P0() and P1(), respectively.  The "="
tests for equality.  The "/\" is boolean AND.  Boolean OR would be
"\/" and boolean NOT "~".  Parentheses may be used.  A variable name on
the left-hand side of a relational operator denotes the value of that
variable, but on the right-hand side its address.

The value of any variable mentioned in the "exists" clause is printed as
part of the "States" list shown below.

Running this with the herd7 tool does the moral equivalent of a full
state-space search, and gets the following:

------------------------------------------------------------------------

$ herd7 -conf linux-kernel.cfg /tmp/cmpxchg.litmus
Test cmpxchg-fail-order Allowed
States 4
0:r0=0; 0:r1=0; 1:r0=0; 1:r1=0;
0:r0=0; 0:r1=0; 1:r0=1; 1:r1=0;
0:r0=1; 0:r1=0; 1:r0=0; 1:r1=0;
0:r0=1; 0:r1=0; 1:r0=1; 1:r1=0;
Ok
Witnesses
Positive: 1 Negative: 3
Condition exists (0:r0=0 /\ 1:r0=0)
Observation cmpxchg-fail-order Sometimes 1 3
Time cmpxchg-fail-order 0.00
Hash=564afea251867c6127350213c7eb388d

------------------------------------------------------------------------

Here, there are four possible combinations of final values for the two
r0 local variables, and all of them can happen, including the combination
where both are zero, which is the combination called out by the "exists"
clause.  The "Sometimes" says that the expression in the "exists" clause
is sometimes true (other options being "Always" and "Never").

The fact that both instances of r0 can be zero shows that failing
cmpxchg() really does not provide ordering.  And again, ordering for
atomic_try_cmpxchg() is the same as that for cmpxchg().

						Thanx, Paul

> Therefore you not only need to turn group->migr_state read into
> an atomic_read_acquire() but you also need to do this on each iteration
> of this loop. For example you can move the read_acquire right here.
> 
> Thanks.
> 
> > +		if (child)
> > +			childstate.state = atomic_read(&child->migr_state);
> > +
> > +		newstate = curstate;
> > +		walk_done = true;
> > +
> > +		/* Reset active bit when the child is no longer active */
> > +		if (!childstate.active)
> > +			newstate.active &= ~childmask;
> > +
> > +		if (newstate.migrator == childmask) {
> > +			/*
> > +			 * Find a new migrator for the group, because the child
> > +			 * group is idle!
> > +			 */
> > +			if (!childstate.active) {
> > +				unsigned long new_migr_bit, active = newstate.active;
> > +
> > +				new_migr_bit = find_first_bit(&active, BIT_CNT);
> > +
> > +				if (new_migr_bit != BIT_CNT) {
> > +					newstate.migrator = BIT(new_migr_bit);
> > +				} else {
> > +					newstate.migrator = TMIGR_NONE;
> > +
> > +					/* Changes need to be propagated */
> > +					walk_done = false;
> > +				}
> > +			}
> > +		}
> > +
> > +		newstate.seq++;
> > +
> > +		WARN_ON_ONCE((newstate.migrator != TMIGR_NONE) && !(newstate.active));
> > +
> > +	} while (!atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state));

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Jan 29, 2024 at 11:50:39AM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> >> +static bool tmigr_inactive_up(struct tmigr_group *group,
> >> +			      struct tmigr_group *child,
> >> +			      void *ptr)
> >> +{
> >> +	union tmigr_state curstate, newstate, childstate;
> >> +	struct tmigr_walk *data = ptr;
> >> +	bool walk_done;
> >> +	u8 childmask;
> >> +
> >> +	childmask = data->childmask;
> >> +	curstate.state = atomic_read(&group->migr_state);
> >> +	childstate.state = 0;
> >> +
> >> +	do {
> >
> > So I got the confirmation from Boqun (+Cc) and Paul that a failing cmpxchg
> > may not order the load of the old value against subsequent loads. And
> > that may apply to atomic_try_cmpxchg() as well.
> >
> > Therefore you not only need to turn group->migr_state read into
> > an atomic_read_acquire() but you also need to do this on each iteration
> > of this loop. For example you can move the read_acquire right here.
> 
> I tried to read and understand more about the memory barriers especially
> the acquire/release stuff. So please correct me whenever I'm wrong.
> 
> We have to make sure that the child/group state values contain the last
> updates and prevent reordering to be able to rely on those values.
> 
> So I understand, that we need the atomic_read_acquire() here for the
> child state, because we change the group state accordingly and need to
> make sure, that it contains the last update of it. The cmpxchg which
> writes the child state is (on success) a full memory barrier. And the
> atomic_read_acquire() makes sure all preceding "critical sections"
> (which ends with the full memory barrier) are visible. Is this right?

Right. And BTW I'm being suggested by Paul to actually avoid
atomic_read_acquire() after cmpxchg() failure because that implies an
error prone re-read. So pick up your favourite between smp_rmb() or
smp_mb__after_atomic().

With the latter this could look like:

curstate.state = atomic_read_acquire(&group->migr_state);
for (;;) {
    childstate.state = atomic_read(&child->migr_state);
    ...
    if (atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state))
       break;
    smp_mb__after_atomic();
}

> 
> To make sure the proper states are used, atomic_read_acquire() is then
> also required in:
>   - tmigr_check_migrator()
>   - tmigr_check_migrator_and_lonely()
>   - tmigr_check_lonely()

Not sure about those. I'll check them.

>   - tmigr_new_timer_up() (for childstate and groupstate)

Actually you need to fix some ordering there that I suggested a while ago :)
See https://lore.kernel.org/all/ZIhKT3h7Dc0G3xoU@lothringen/

>   - tmigr_connect_child_parent()
> Right?
> 
> Regarding the pairing of acquire: What happens when two
> atomic_read_acquire() are executed afterwards without pairing 1:1 with a
> release or stronger memory barrier?

I think I'll need an example.

> 
> Now I want to understand the case for the group state here and also in
> active_up path. When reading it without acquire, it is possible, that
> not all changes are visible due to reordering,... . But then the worst
> outcome would be that the cmpxchg fails and the loop has to be done once
> more? Is this right?

Right. This one looks good as it doesn't depend on the child's value.

> 
> I know that memory barriers are not for free and redo the loop is also
> not for free. But I don't know which of both is worse. At least in
> inactive_up() path, we are not in the critical path. In active_up() it
> would be good to take the less expensive option.

I don't think you need to change the active_up(), from a quick glance.

> 
> I want to understand the atomic_try_cmpxchg_acquire() variant: The Read
> is an acquire, so even if the compare/write fails, the value which is
> handed back is the one which was update last with a succesful cmpxchg
> and then we can rely on this value?

So cmpxchg_acquire() provides a weaker ordering than cmpxchg(). Instead
of issuing a full memory barrier, it issues an acquire barrier, which is
really not what you want since you actually want to order what precedes
the cmpxchg() with the write that it performs. At the very least you would
actually need cmpxchg_release().

And most importantly, neither cmpxchg(), cmpxchg_release() nor cmpxchg_acquire()
guarantee any ordering on failure.

Thanks.

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 29, 2024 at 11:50:39AM +0100, Anna-Maria Behnsen a écrit :
>> Frederic Weisbecker <frederic@kernel.org> writes:
>> 
>> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> >> +static bool tmigr_inactive_up(struct tmigr_group *group,
>> >> +			      struct tmigr_group *child,
>> >> +			      void *ptr)
>> >> +{
>> >> +	union tmigr_state curstate, newstate, childstate;
>> >> +	struct tmigr_walk *data = ptr;
>> >> +	bool walk_done;
>> >> +	u8 childmask;
>> >> +
>> >> +	childmask = data->childmask;
>> >> +	curstate.state = atomic_read(&group->migr_state);
>> >> +	childstate.state = 0;
>> >> +
>> >> +	do {
>> >
>> > So I got the confirmation from Boqun (+Cc) and Paul that a failing cmpxchg
>> > may not order the load of the old value against subsequent loads. And
>> > that may apply to atomic_try_cmpxchg() as well.
>> >
>> > Therefore you not only need to turn group->migr_state read into
>> > an atomic_read_acquire() but you also need to do this on each iteration
>> > of this loop. For example you can move the read_acquire right here.
>> 
>> I tried to read and understand more about the memory barriers especially
>> the acquire/release stuff. So please correct me whenever I'm wrong.
>> 
>> We have to make sure that the child/group state values contain the last
>> updates and prevent reordering to be able to rely on those values.
>> 
>> So I understand, that we need the atomic_read_acquire() here for the
>> child state, because we change the group state accordingly and need to
>> make sure, that it contains the last update of it. The cmpxchg which
>> writes the child state is (on success) a full memory barrier. And the
>> atomic_read_acquire() makes sure all preceding "critical sections"
>> (which ends with the full memory barrier) are visible. Is this right?
>
> Right. And BTW I'm being suggested by Paul to actually avoid
> atomic_read_acquire() after cmpxchg() failure because that implies an
> error prone re-read. So pick up your favourite between smp_rmb() or
> smp_mb__after_atomic().
>
> With the latter this could look like:
>
> curstate.state = atomic_read_acquire(&group->migr_state);
> for (;;) {
>     childstate.state = atomic_read(&child->migr_state);
>     ...
>     if (atomic_try_cmpxchg(&group->migr_state, &curstate.state, newstate.state))
>        break;
>     smp_mb__after_atomic();
> }

I'll take this.

>> 
>> To make sure the proper states are used, atomic_read_acquire() is then
>> also required in:
>>   - tmigr_check_migrator()
>>   - tmigr_check_migrator_and_lonely()
>>   - tmigr_check_lonely()
>
> Not sure about those. I'll check them.

Please ignore - I was on the wrong track and John helped me out (at
least another step) of my "memory barrier understanding mess". So I have
no more questions regarding the memory barriers here - at least at the
moment.

>>   - tmigr_new_timer_up() (for childstate and groupstate)
>
> Actually you need to fix some ordering there that I suggested a while ago :)
> See https://lore.kernel.org/all/ZIhKT3h7Dc0G3xoU@lothringen/

I quickly opened it and it seems that I completely missed it
somehow. I'm sorry!

Thanks for your patience!

	Anna-Maria

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Sun, Jan 28, 2024 at 04:58:55PM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> >> +
> >> +			if (tmigr_check_lonely(topgroup))
> >> +				return READ_ONCE(topgroup->next_expiry);
> 
> When I hand in tevt->global as a parameter, I'll need to compare the
> first expiry of the toplevel group and the tevt->global value and return
> the earlier expiry. Only a single child is active in top level, so it
> might be that this CPU is the last active CPU in hierarchy.
> 
> I didn't check all the way to the top whether all groups are
> 'lonely'. So when the top level group has only a single active child, it
> is also possible that the child of the top level group has two active
> children... Then a return of KTIME_MAX would be also a more precise
> forecast.
> 
> This quick check is there to keep the overhead minimal when checking
> whether it might be possible to go idle. So I don't know, if we should
> add this additional check per level (which is pretty simple when using
> group->parent for walking the hierarchy). What do you think?

Not sure. Maybe if the tree never exceeds 3 levels (does it?) it's ok to
do the walk?

Thanks.

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> +/*
> + * Returns true, if there is nothing to be propagated to the next level
> + *
> + * @data->firstexp is set to expiry of first gobal event of the (top level of
> + * the) hierarchy, but only when hierarchy is completely idle.
> + *
> + * This is the only place where the group event expiry value is set.
> + */
> +static
> +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
> +			 struct tmigr_walk *data, union tmigr_state childstate,
> +			 union tmigr_state groupstate)
> +{
> +	struct tmigr_event *evt, *first_childevt;
> +	bool walk_done, remote = data->remote;
> +	bool leftmost_change = false;
> +	u64 nextexp;
> +
> +	if (child) {
> +		raw_spin_lock(&child->lock);
> +		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
> +
> +		if (childstate.active) {

Since you're going to do the atomic_read(&group->migr_state)
within the group->lock, you may as well do the atomic_read(&child->migr_state)
within the child->lock. It won't hurt and simplifies the picture
in the mind. Then you can add the following comment to outline the ordering
expectations:

/*
 * Observing child->migr_state.active means that:
 *
 * 1) Either the child is effectively active, then it's fine to stop here
 *
 * 2) Or we are racing with a CPU going inactive and this childstate is actually
 *    not active anymore but tmigr_inactive_up() hasn't yet called tmigr_update_event()
 *    on it. It's fine to stop here because that pending call will take care
 *    of the rest of the propagation.
 *
 * 3) In any case it's impossible to observe childstate.active when a racing
 *    CPU made it inactive and also called tmigr_update_event() on it. The
 *    group->lock enforces ordering such that ->migr_state changes
 *    in tmigr_inactive_up() are released by group->lock UNLOCK on the
 *    subsequent call to tmigr_update_event() and then acquired by
 *    child->lock LOCK in tmigr_new_timer() -> tmigr_update_event().
 */
 
> +			walk_done = true;
> +			goto unlock;
> +		}
> +
> +		first_childevt = tmigr_next_groupevt(child);
> +		nextexp = child->next_expiry;
> +		evt = &child->groupevt;
> +	} else {
> +		nextexp = data->nextexp;
> +
> +		first_childevt = evt = data->evt;
> +
> +		/*
> +		 * Walking the hierarchy is required in any case when a
> +		 * remote expiry was done before. This ensures to not lose
> +		 * already queued events in non active groups (see section
> +		 * "Required event and timerqueue update after a remote
> +		 * expiry" in the documentation at the top).
> +		 *
> +		 * The two call sites which are executed without a remote expiry
> +		 * before, are not prevented from propagating changes through
> +		 * the hierarchy by the return:
> +		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
> +		 *    is never set.
> +		 *  - tmigr_inactive_up() takes care of the propagation by
> +		 *    itself and ignores the return value. But an immediate
> +		 *    return is required because nothing has to be done in this
> +		 *    level as the event could be ignored.
> +		 */
> +		if (evt->ignore && !remote)
> +			return true;
> +
> +		raw_spin_lock(&group->lock);
> +	}
> +
> +	if (nextexp == KTIME_MAX) {
> +		evt->ignore = true;
> +
> +		/*
> +		 * When the next child event could be ignored (nextexp is
> +		 * KTIME_MAX) and there was no remote timer handling before or
> +		 * the group is already active, there is no need to walk the
> +		 * hierarchy even if there is a parent group.
> +		 *
> +		 * The other way round: even if the event could be ignored, but
> +		 * if a remote timer handling was executed before and the group
> +		 * is not active, walking the hierarchy is required to not miss
> +		 * an enqueued timer in the non active group. The enqueued timer
> +		 * of the group needs to be propagated to a higher level to
> +		 * ensure it is handled.
> +		 */
> +		if (!remote || groupstate.active) {

Same here, fetching group->migr_state.active from within the lock simplifies
the mind mapping.

Thanks.

> +			walk_done = true;
> +			goto unlock;
> +		}
> +	} else {
> +		/*
> +		 * An update of @evt->cpu and @evt->ignore flag is required only
> +		 * when @child is set (the child is equal or higher than lvl0),
> +		 * but it doesn't matter if it is written once more to the per
> +		 * CPU event; make the update unconditional.
> +		 */
> +		evt->cpu = first_childevt->cpu;
> +		evt->ignore = false;
> +	}
> +
> +	walk_done = !group->parent;
> +
> +	/*
> +	 * If the child event is already queued in the group, remove it from the
> +	 * queue when the expiry time changed only.
> +	 */
> +	if (timerqueue_node_queued(&evt->nextevt)) {
> +		if (evt->nextevt.expires == nextexp)
> +			goto check_toplvl;
> +
> +		leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt;
> +		if (!timerqueue_del(&group->events, &evt->nextevt))
> +			WRITE_ONCE(group->next_expiry, KTIME_MAX);
> +	}
> +
> +	evt->nextevt.expires = nextexp;
> +
> +	if (timerqueue_add(&group->events, &evt->nextevt)) {
> +		leftmost_change = true;
> +		WRITE_ONCE(group->next_expiry, nextexp);
> +	}
> +
> +check_toplvl:
> +	if (walk_done && (groupstate.migrator == TMIGR_NONE)) {
> +		/*
> +		 * Nothing to do when first event didn't changed and update was
> +		 * done during remote timer handling.
> +		 */
> +		if (remote && !leftmost_change)
> +			goto unlock;
> +		/*
> +		 * The top level group is idle and it has to be ensured the
> +		 * global timers are handled in time. (This could be optimized
> +		 * by keeping track of the last global scheduled event and only
> +		 * arming it on the CPU if the new event is earlier. Not sure if
> +		 * its worth the complexity.)
> +		 */
> +		data->firstexp = tmigr_next_groupevt_expires(group);
> +	}
> +
> +unlock:
> +	raw_spin_unlock(&group->lock);
> +
> +	if (child)
> +		raw_spin_unlock(&child->lock);
> +
> +	return walk_done;
> +}

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Sun, Jan 28, 2024 at 04:58:55PM +0100, Anna-Maria Behnsen a écrit :
>> Frederic Weisbecker <frederic@kernel.org> writes:
>> >> +
>> >> +			if (tmigr_check_lonely(topgroup))
>> >> +				return READ_ONCE(topgroup->next_expiry);
>> 
>> When I hand in tevt->global as a parameter, I'll need to compare the
>> first expiry of the toplevel group and the tevt->global value and return
>> the earlier expiry. Only a single child is active in top level, so it
>> might be that this CPU is the last active CPU in hierarchy.
>> 
>> I didn't check all the way to the top whether all groups are
>> 'lonely'. So when the top level group has only a single active child, it
>> is also possible that the child of the top level group has two active
>> children... Then a return of KTIME_MAX would be also a more precise
>> forecast.
>> 
>> This quick check is there to keep the overhead minimal when checking
>> whether it might be possible to go idle. So I don't know, if we should
>> add this additional check per level (which is pretty simple when using
>> group->parent for walking the hierarchy). What do you think?
>
> Not sure. Maybe if the tree never exceeds 3 levels (does it?) it's ok to
> do the walk?

Levels depend on the number of CPUs and NUMA nodes

Take 4096 CPUs, all evenly distributed across 4 NUMA nodes, then you
will end up with 6 levels.

Thanks,

	Anna-Maria

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> +int tmigr_requires_handle_remote(void)
>> +{
>> +	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
>> +	struct tmigr_remote_data data;
>> +	unsigned int ret = 0;
>> +	unsigned long jif;
>> +
>> +	if (tmigr_is_not_available(tmc))
>> +		return ret;
>> +
>> +	data.now = get_jiffies_update(&jif);
>> +	data.childmask = tmc->childmask;
>> +	data.firstexp = KTIME_MAX;
>> +	data.tmc_active = !tmc->idle;
>> +	data.check = false;
>> +
>> +	/*
>> +	 * If the CPU is active, walk the hierarchy to check whether a remote
>> +	 * expiry is required.
>> +	 *
>> +	 * Check is done lockless as interrupts are disabled and @tmc->idle is
>> +	 * set only by the local CPU.
>> +	 */
>> +	if (!tmc->idle) {
>> +		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
>> +
>> +		if (data.firstexp != KTIME_MAX)
>> +			ret = 1;
>> +
>> +		return ret;
>> +	}
>> +
>> +	/*
>> +	 * If the CPU is idle, check whether the recalculation of @tmc->wakeup
>> +	 * is required. @tmc->wakeup_recalc is set, when the last active CPU
>> +	 * went offline. The last active CPU delegated the handling of the timer
>> +	 * migration hierarchy to another (this) CPU by updating this flag and
>> +	 * sending a reschedule.
>> +	 *
>> +	 * Racy lockless check is valid:
>> +	 * - @tmc->wakeup_recalc is set by the remote CPU before it issues
>> +	 *   reschedule IPI.
>> +	 * - As interrupts are disabled here this CPU will either observe
>> +	 *   @tmc->wakeup_recalc set before the reschedule IPI can be handled or
>> +	 *   it will observe it when this function is called again on return
>> +	 *   from handling the reschedule IPI.
>> +	 */
>> +	if (tmc->wakeup_recalc) {
>> +		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
>> +
>> +		if (data.firstexp != KTIME_MAX)
>> +			ret = 1;
>> +
>> +		raw_spin_lock(&tmc->lock);
>> +		WRITE_ONCE(tmc->wakeup, data.firstexp);
>> +		tmc->wakeup_recalc = false;
>> +		raw_spin_unlock(&tmc->lock);
>
> Suppose we have:
>
>             [GRP1:0]
>             migrator = GRP0:1
>             active   = GRP0:0, GRP0:1
>               /                \
>      [GRP0:0]                  [GRP0:1]
>      migrator = CPU 1         migrator = CPU 3
>      active   = CPU 1         active   = CPU 3
>        /         \               /         \
> CPUs  0           1             2           3
>      idle        active        idle        active
>
> CPU 0 and CPU 2 have no timer.
> CPU 1 has a timer in a few millisecs.
>
>             [GRP1:0]
>             migrator = GRP0:1
>             active   = GRP0:1
>               /                \
>      [GRP0:0]                  [GRP0:1]
>      migrator = NONE           migrator = CPU 3
>      active   = NONE           active   = CPU 3
>        /         \                /         \
> CPUs  0           1              2           3
>      idle        idle         idle        active
>
>
> CPU 1 went idle, CPU 3 will take care of CPU 1's timer. Then come two
> things happening at the same time: CPU 0 has a timer interrupt, due to
> RCU callbacks handling for example, and CPU 3 goes offline:
>
> CPU 0                                   CPU 3
> -----                                   -----
>                                         // On top level [GRP1:0], just set migrator = TMIGR_NONE
>                                         tmigr_inactive_up() {
>                                             cpu = cpumask_any_but(cpu_online_mask, cpu);
>                                             //cpu == 0
>                                             tmc_resched = per_cpu_ptr(&tmigr_cpu, CPU 0);
>                                             raw_spin_lock(&tmc_resched->lock);
>                                             tmc_resched->wakeup_recalc = true;
>                                             raw_spin_unlock(&tmc_resched->lock);
> // timer interrupt
> run_local_timers() {
>     tmigr_requires_handle_remote() {
>         data.firstexp = KTIME_MAX;
>         // CPU 0 sees the tmc_resched->wakeup_recalc
>         // latest update
>         if (tmc->wakeup_recalc) {
>             tmigr_requires_handle_remote_up() {
>                 // CPU 0 doesn't see GRP0:0 
>                 // latest update from CPU 1,
>                 // because it has no locking
>                 // and does a racy check.
>         	    if (!tmigr_check_migrator(group, childmask))
>                     return true;
>             }
>             raw_spin_lock(&tmc->lock);
>             WRITE_ONCE(tmc->wakeup, data.firstexp);
>             tmc->wakeup_recalc = false;
>             raw_spin_unlock(&tmc->lock)
>             return 0;
>         }
>                                             // IPI is sent only now
> 		                                    smp_send_reschedule(cpu);
>                                             }
>
>
> There is nothing that prevents CPU 0 from not seeing the hierarchy updates from
> other CPUs since it checks the migrators in a racy way. As a result the timer of
> CPU 1 may be ignored by CPU 0.
>
> You'd need to lock the tmc while calling tmigr_requires_handle_remote_up(), so
> that CPU 0 "inherits" everything that CPU 3 has seen, and that includes changes
> from CPU 1.
>

puhh. ok. But for the !idle case the lockless walk of
tmigr_requires_handle_remote_up() is ok? It's also possible, that the
CPU misses an update of the state - another CPU goes idle and selects
this CPU as the new migrator. And this CPU reads a stale value where the
other CPU is migrator. But this will be revisited on the next
tick. hmm...

>
> But I see that tmigr_cpu_new_timer() does it right. Wouldn't it be possible to
> exlusively let tmigr_cpu_new_timer() handle the wakeup_recalc thing? This is
> going to be called after the end of the IRQ (whether timer interrupt or sched
> IPI) in any case.

Should work, yes. But when a timer has to be handled right away and it
is checked after the end of the IRQ, then the tick might be reprogrammed
so that CPU comes out of idle, or am I wrong?

But, while I'm having a deeper look at the code - I completely destroyed
the logic to use the 'check' value of the tmigr_remote_date struct for
making a decision whether to raise a timer softirq or not... Whenever
the expiry in the top level is !KTIME_MAX, then
tmigr_require_handle_remote returns 1. Oh no. Also the struct member
description is not up to date.

Thanks,

	Anna-Maria

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Tue, Jan 30, 2024 at 06:56:32PM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> > CPU 1 went idle, CPU 3 will take care of CPU 1's timer. Then come two
> > things happening at the same time: CPU 0 has a timer interrupt, due to
> > RCU callbacks handling for example, and CPU 3 goes offline:
> >
> > CPU 0                                   CPU 3
> > -----                                   -----
> >                                         // On top level [GRP1:0], just set migrator = TMIGR_NONE
> >                                         tmigr_inactive_up() {
> >                                             cpu = cpumask_any_but(cpu_online_mask, cpu);
> >                                             //cpu == 0
> >                                             tmc_resched = per_cpu_ptr(&tmigr_cpu, CPU 0);
> >                                             raw_spin_lock(&tmc_resched->lock);
> >                                             tmc_resched->wakeup_recalc = true;
> >                                             raw_spin_unlock(&tmc_resched->lock);
> > // timer interrupt
> > run_local_timers() {
> >     tmigr_requires_handle_remote() {
> >         data.firstexp = KTIME_MAX;
> >         // CPU 0 sees the tmc_resched->wakeup_recalc
> >         // latest update
> >         if (tmc->wakeup_recalc) {
> >             tmigr_requires_handle_remote_up() {
> >                 // CPU 0 doesn't see GRP0:0 
> >                 // latest update from CPU 1,
> >                 // because it has no locking
> >                 // and does a racy check.
> >         	    if (!tmigr_check_migrator(group, childmask))
> >                     return true;
> >             }
> >             raw_spin_lock(&tmc->lock);
> >             WRITE_ONCE(tmc->wakeup, data.firstexp);
> >             tmc->wakeup_recalc = false;
> >             raw_spin_unlock(&tmc->lock)
> >             return 0;
> >         }
> >                                             // IPI is sent only now
> > 		                                    smp_send_reschedule(cpu);
> >                                             }
> >
> >
> > There is nothing that prevents CPU 0 from not seeing the hierarchy updates from
> > other CPUs since it checks the migrators in a racy way. As a result the timer of
> > CPU 1 may be ignored by CPU 0.
> >
> > You'd need to lock the tmc while calling tmigr_requires_handle_remote_up(), so
> > that CPU 0 "inherits" everything that CPU 3 has seen, and that includes changes
> > from CPU 1.
> >
> 
> puhh. ok. But for the !idle case the lockless walk of
> tmigr_requires_handle_remote_up() is ok?

Looks ok to me. It's racy but if the !idle migrator doesn't notice in the
current tick, it will in the next one.

> It's also possible, that the
> CPU misses an update of the state - another CPU goes idle and selects
> this CPU as the new migrator. And this CPU reads a stale value where the
> other CPU is migrator. But this will be revisited on the next
> tick. hmm...

Exactly, and I'm not worried. There has to be strong ordering with atomics
or locking in the idle case because the CPU goes to sleep and it must make
sure not to miss a timer. But in the !idle case the check is periodic, so you
don't need any of that. We can live with an unnoticed timer for a tick or two.

> 
> >
> > But I see that tmigr_cpu_new_timer() does it right. Wouldn't it be possible to
> > exlusively let tmigr_cpu_new_timer() handle the wakeup_recalc thing? This is
> > going to be called after the end of the IRQ (whether timer interrupt or sched
> > IPI) in any case.
> 
> Should work, yes. But when a timer has to be handled right away and it
> is checked after the end of the IRQ, then the tick might be reprogrammed
> so that CPU comes out of idle, or am I wrong?

If there is a pending timer, it can wait a tick. That's what happens if
we wait for tmigr_cpu_new_timer() to handle it.

But you know what, let's make it more simple. CPU down hotplug is not a
fast path and it doesn't deserve so many optimizations. Just remove ->wakeup_recalc
entirely and if the offlining CPU detects it's the last active CPU in the
hierarchy, just queue an empty work to the first online CPU. It will briefly
force that CPU out of idle and trigger an activate. Then either the CPU
periodically checks remote timers or it will go back idle and notice.

Something like this (untested):

diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c
index de1905b0bae7..0f15215ef257 100644
--- a/kernel/time/timer_migration.c
+++ b/kernel/time/timer_migration.c
@@ -548,7 +548,6 @@ static void __tmigr_cpu_activate(struct tmigr_cpu *tmc)
 
 	tmc->cpuevt.ignore = true;
 	WRITE_ONCE(tmc->wakeup, KTIME_MAX);
-	tmc->wakeup_recalc = false;
 
 	walk_groups(&tmigr_active_up, &data, tmc);
 }
@@ -1041,41 +1040,11 @@ int tmigr_requires_handle_remote(void)
 	}
 
 	/*
-	 * If the CPU is idle, check whether the recalculation of @tmc->wakeup
-	 * is required. @tmc->wakeup_recalc is set, when the last active CPU
-	 * went offline. The last active CPU delegated the handling of the timer
-	 * migration hierarchy to another (this) CPU by updating this flag and
-	 * sending a reschedule.
-	 *
-	 * Racy lockless check is valid:
-	 * - @tmc->wakeup_recalc is set by the remote CPU before it issues
-	 *   reschedule IPI.
-	 * - As interrupts are disabled here this CPU will either observe
-	 *   @tmc->wakeup_recalc set before the reschedule IPI can be handled or
-	 *   it will observe it when this function is called again on return
-	 *   from handling the reschedule IPI.
-	 */
-	if (tmc->wakeup_recalc) {
-		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
-
-		if (data.firstexp != KTIME_MAX)
-			ret = 1;
-
-		raw_spin_lock(&tmc->lock);
-		WRITE_ONCE(tmc->wakeup, data.firstexp);
-		tmc->wakeup_recalc = false;
-		raw_spin_unlock(&tmc->lock);
-
-		return ret;
-	}
-
-	/*
-	 * When the CPU is idle and @tmc->wakeup is reliable as
-	 * @tmc->wakeup_recalc is not set, compare it with @data.now. The lock
-	 * is required on 32bit architectures to read the variable consistently
-	 * with a concurrent writer. On 64bit the lock is not required because
-	 * the read operation is not split and so it is always consistent.
-
+	 * When the CPU is idle and @tmc->wakeup is reliable, compare it with
+	 * @data.now. The lock is required on 32bit architectures to read the
+	 * variable consistently with a concurrent writer. On 64bit the lock
+	 * is not required because the read operation is not split and so it is
+	 * always consistent.
 	 */
 	if (IS_ENABLED(CONFIG_64BIT)) {
 		if (data.now >= READ_ONCE(tmc->wakeup))
@@ -1119,21 +1088,7 @@ u64 tmigr_cpu_new_timer(u64 nextexp)
 		    tmc->cpuevt.ignore) {
 			ret = tmigr_new_timer(tmc, nextexp);
 		}
-	} else if (tmc->wakeup_recalc) {
-		struct tmigr_remote_data data;
-
-		data.now = KTIME_MAX;
-		data.childmask = tmc->childmask;
-		data.firstexp = KTIME_MAX;
-		data.tmc_active = false;
-		data.check = false;
-
-		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
-
-		ret = data.firstexp;
 	}
-	tmc->wakeup_recalc = false;
-
 	/*
 	 * Make sure the reevaluation of timers in idle path will not miss an
 	 * event.
@@ -1212,36 +1167,7 @@ static bool tmigr_inactive_up(struct tmigr_group *group,
 	 *   hierarchy) and
 	 * - there is a pending event in the hierarchy
 	 */
-	if (data->firstexp != KTIME_MAX) {
-		WARN_ON_ONCE(group->parent);
-		/*
-		 * Top level path: If this CPU is about going offline and was
-		 * the last active CPU, wake up some random other CPU so it will
-		 * take over the migrator duty and program its timer
-		 * properly. Ideally wake the CPU with the closest expiry time,
-		 * but that's overkill to figure out.
-		 *
-		 * Set wakeup_recalc of remote CPU, to make sure the complete
-		 * idle hierarchy with enqueued timers is reevaluated.
-		 */
-		if (!(this_cpu_ptr(&tmigr_cpu)->online)) {
-			struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
-			unsigned int cpu = smp_processor_id();
-			struct tmigr_cpu *tmc_resched;
-
-			cpu = cpumask_any_but(cpu_online_mask, cpu);
-			tmc_resched = per_cpu_ptr(&tmigr_cpu, cpu);
-
-			raw_spin_unlock(&tmc->lock);
-
-			raw_spin_lock(&tmc_resched->lock);
-			tmc_resched->wakeup_recalc = true;
-			raw_spin_unlock(&tmc_resched->lock);
-
-			raw_spin_lock(&tmc->lock);
-			smp_send_reschedule(cpu);
-		}
-	}
+	WARN_ON_ONCE(data->firstexp != KTIME_MAX && group->parent);
 
 	return walk_done;
 }
@@ -1579,9 +1505,20 @@ static int tmigr_cpu_online(unsigned int cpu)
 	return 0;
 }
 
+long tmigr_trigger_active(void *unused)
+{
+	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+
+	WARN_ON_ONCE(!tmc->online || tmc->idle);
+
+	return 0;
+}
+
 static int tmigr_cpu_offline(unsigned int cpu)
 {
 	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
+	int migrator;
+	u64 firstexp;
 
 	raw_spin_lock_irq(&tmc->lock);
 	tmc->online = false;
@@ -1591,9 +1528,14 @@ static int tmigr_cpu_offline(unsigned int cpu)
 	 * CPU has to handle the local events on his own, when on the way to
 	 * offline; Therefore nextevt value is set to KTIME_MAX
 	 */
-	__tmigr_cpu_deactivate(tmc, KTIME_MAX);
+	firstexp = __tmigr_cpu_deactivate(tmc, KTIME_MAX);
 	raw_spin_unlock_irq(&tmc->lock);
 
+	if (firstexp != KTIME_MAX) {
+		migrator = cpumask_any_but(cpu_online_mask, cpu);
+		work_on_cpu(migrator, tmigr_trigger_active, NULL);
+	}
+
 	return 0;
 }
 
diff --git a/kernel/time/timer_migration.h b/kernel/time/timer_migration.h
index c32947cf429b..c556d5824792 100644
--- a/kernel/time/timer_migration.h
+++ b/kernel/time/timer_migration.h
@@ -78,18 +78,12 @@ struct tmigr_group {
  * @idle:		Indicates whether the CPU is idle in the timer migration
  *			hierarchy
  * @remote:		Is set when timers of the CPU are expired remotely
- * @wakeup_recalc:	Indicates, whether a recalculation of the @wakeup value
- *			is required. @wakeup_recalc is only used by this CPU
- *			when it is marked idle in the timer migration
- *			hierarchy. It is set by a remote CPU which was the last
- *			active CPU and is on the way to idle.
  * @tmgroup:		Pointer to the parent group
  * @childmask:		childmask of tmigr_cpu in the parent group
  * @wakeup:		Stores the first timer when the timer migration
  *			hierarchy is completely idle and remote expiry was done;
  *			is returned to timer code in the idle path and is only
- *			used in idle path; it is only valid, when @wakeup_recalc
- *			is not set.
+ *			used in idle path.
  * @cpuevt:		CPU event which could be enqueued into the parent group
  */
 struct tmigr_cpu {
@@ -97,7 +91,6 @@ struct tmigr_cpu {
 	bool			online;
 	bool			idle;
 	bool			remote;
-	bool			wakeup_recalc;
 	struct tmigr_group	*tmgroup;
 	u8			childmask;
 	u64			wakeup;

Re: [PATCH v10 00/20] timers: Move from a push remote at enqueue to a pull at expiry model

[Christian Loehle]

On 15/01/2024 14:37, Anna-Maria Behnsen wrote:
> Hi,
> 
> the cleanup patches are already applied and so the contains only two parts:
> 
> - Patches 1 - 4: timer base idle marking rework with two preparatory
>   changes. See the section below for more details.
> 
> - Patches 5 - 20: Updated timer pull model on top of timer idle rework
> 
> 
> The queue is available here:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel timers/pushpull
> 
> 
> Move marking timer bases as idle into tick_nohz_stop_tick()
> -----------------------------------------------------------
> 
> The idle marking of timer bases is done in get_next_timer_interrupt()
> whenever possible. The timer bases are idle, even if the tick will not be
> stopped. This lead to an IPI when a new first timer is enqueued remote. To
> prevent this, setting timer_base->in_idle flag is postponed to
> tick_nohz_stop_tick().
> 
> Furthermore this synchronizes the states of timer base is_idle and
> tick_stopped. With the timer pull model in place, also the idle state in
> the hierarchy of a CPU is synchronized with the other idle related states.
> 
> 
> Timer pull model
> ----------------
> 
> Placing timers at enqueue time on a target CPU based on dubious heuristics
> does not make any sense:
> 
>  1) Most timer wheel timers are canceled or rearmed before they expire.
> 
>  2) The heuristics to predict which CPU will be busy when the timer expires
>     are wrong by definition.
> 
> So placing the timers at enqueue wastes precious cycles.
> 
> The proper solution to this problem is to always queue the timers on the
> local CPU and allow the non pinned timers to be pulled onto a busy CPU at
> expiry time.
> 
> Therefore split the timer storage into local pinned and global timers:
> Local pinned timers are always expired on the CPU on which they have been
> queued. Global timers can be expired on any CPU.
> 
> As long as a CPU is busy it expires both local and global timers. When a
> CPU goes idle it arms for the first expiring local timer. If the first
> expiring pinned (local) timer is before the first expiring movable timer,
> then no action is required because the CPU will wake up before the first
> movable timer expires. If the first expiring movable timer is before the
> first expiring pinned (local) timer, then this timer is queued into a idle
> timerqueue and eventually expired by some other active CPU.
> 
> To avoid global locking the timerqueues are implemented as a hierarchy. The
> lowest level of the hierarchy holds the CPUs. The CPUs are associated to
> groups of 8, which are separated per node. If more than one CPU group
> exist, then a second level in the hierarchy collects the groups. Depending
> on the size of the system more than 2 levels are required. Each group has a
> "migrator" which checks the timerqueue during the tick for remote timers to
> be expired.
> 
> If the last CPU in a group goes idle it reports the first expiring event in
> the group up to the next group(s) in the hierarchy. If the last CPU goes
> idle it arms its timer for the first system wide expiring timer to ensure
> that no timer event is missed.
> 
> 
> Testing
> ~~~~~~~
> 
> Enqueue
> ^^^^^^^
> 
> The impact of wasting cycles during enqueue by using the heuristic in
> contrast to always queuing the timer on the local CPU was measured with a
> micro benchmark. Therefore a timer is enqueued and dequeued in a loop with
> 1000 repetitions on a isolated CPU. The time the loop takes is measured. A
> quarter of the remaining CPUs was kept busy. This measurement was repeated
> several times. With the patch queue the average duration was reduced by
> approximately 25%.
> 
> 	145ns	plain v6
> 	109ns	v6 with patch queue
> 
> 
> Furthermore the impact of residence in deep idle states of an idle system
> was investigated. The patch queue doesn't downgrade this behavior.
> 
> dbench test
> ^^^^^^^^^^^
> 
> A dbench test starting X pairs of client servers are used to create load on
> the system. The measurable value is the throughput. The tests were executed
> on a zen3 machine. The base is the tip tree branch timers/core which is
> based on a v6.6-rc1.
> 
> governor menu
> 
> NR	timers/core	pull-model	impact
> ----------------------------------------------
> 1	353.19 (0.19)	353.45 (0.30)	0.07%
> 2	700.10 (0.96)	687.00 (0.20)	-1.87%
> 4	1329.37 (0.63)	1282.91 (0.64)	-3.49%
> 8	2561.16 (1.28)	2493.56	(1.76)	-2.64%
> 16	4959.96 (0.80)	4914.59 (0.64)	-0.91%
> 32	9741.92 (3.44)	8979.83 (1.13)	-7.82%
> 64	16535.40 (2.84)	16388.47 (4.02)	-0.89%
> 128	22136.83 (2.42)	23174.50 (1.43)	4.69%
> 256	39256.77 (4.48)	38994.00 (0.39)	-0.67%
> 512	36799.03 (1.83)	38091.10 (0.63)	3.51%
> 1024	32903.03 (0.86)	35370.70 (0.89)	7.50%
> 
> 
> governor teo
> 
> NR	timers/core	pull-model	impact
> ----------------------------------------------
> 1	350.83 (1.27)	352.45 (0.96)	0.46%
> 2	699.52 (0.85)	690.10 (0.54)	-1.35%
> 4	1339.53 (1.99)	1294.71 (2.71)	-3.35%
> 8	2574.10 (0.76)	2495.46 (1.97)	-3.06%
> 16	4898.50 (1.74)	4783.06 (1.64)	-2.36%
> 32	9115.50 (4.63)	9037.83 (1.58)	-0.85%
> 64	16663.90 (3.80)	16042.00 (1.72)	-3.73%
> 128	25044.93 (1.11)	23250.03 (1.08)	-7.17%
> 256	38059.53 (1.70)	39658.57 (2.98)	4.20%
> 512	36369.30 (0.39)	38890.13 (0.36)	6.93%
> 1024	33956.83 (1.14)	35514.83 (0.29)	4.59%
> 
> 
> 
> Ping Pong Oberservation
> ^^^^^^^^^^^^^^^^^^^^^^^
> 
> During testing on a mostly idle machine a ping pong game could be observed:
> a process_timeout timer is expired remotely on a non idle CPU. Then the CPU
> where the schedule_timeout() was executed to enqueue the timer comes out of
> idle and restarts the timer using schedule_timeout() and goes back to idle
> again. This is due to the fair scheduler which tries to keep the task on
> the CPU which it previously executed on.
> 
> 
> 
> 
> Possible Next Steps
> ~~~~~~~~~~~~~~~~~~~
> 
> Simple deferrable timers are no longer required as they can be converted to
> global timers. If a CPU goes idle, a formerly deferrable timer will not
> prevent the CPU to sleep as long as possible. Only the last migrator CPU
> has to take care of them. Deferrable timers with timer pinned flags needs
> to be expired on the specified CPU but must not prevent CPU from going
> idle. They require their own timer base which is never taken into account
> when calculating the next expiry time. This conversation and required
> cleanup will be done in a follow up series.
> 
> 
> v9..v10: https://lore.kernel.org/r/20231201092654.34614-1-anna-maria@linutronix.de/
>   - Address review Feedback of Bigeasy
> 
> 
> v8..v9: https://lore.kernel.org/r/20231004123454.15691-1-anna-maria@linutronix.de
>   - Address review feedback
>   - Add more minor cleanup fixes
>   - fixes inconsistent idle related states
> 
> 
> v7..v8: https://lore.kernel.org/r/20230524070629.6377-1-anna-maria@linutronix.de
>   - Address review feedback
>   - Move marking timer base idle into tick_nohz_stop_tick()
>   - Look ahead function to determine possible sleep lenght
> 
> 
> v6..v7:
>   - Address review feedback of Frederic and bigeasy
>   - Change lock, unlock fetch next timer interrupt logic after remote expiry
>   - Move timer_expire_remote() into tick-internal.h
>   - Add documentation section about "Required event and timerqueue update
>     after remote expiry"
>   - Fix fallout of kernel test robot
> 
> 
> v5..v6:
> 
>   - Address review of Frederic Weisbecker and Peter Zijlstra (spelling,
>     locking, race in tmigr_handle_remote_cpu())
> 
>   - unconditionally set TIMER_PINNED flag in add_timer_on(); introduce
>     add_timer() variants which set/unset TIMER_PINNED flag; drop fixing
>     add_timer_on() call sites, as TIMER_PINNED flag is set implicitly;
>     Fixing workqueue to use add_timer_global() instead of simply
>     add_timer() for unbound work.
> 
>   - Drop support for siblings to end up in the same level 0 group (could be
>     added again in a better way as an improvement later on)
> 
>   - Do not send IPI for new first deferrable timers
> 
> v4..v5:
>   - address review feedback of Frederic Weisbecker
>   - fix issue with group timer update after remote expiry
> 
> v3..v4:
>   - address review feedback of Frederic Weisbecker
>   - address kernel test robot fallout
>   - Move patch 16 "add_timer_on(): Make sure callers have TIMER_PINNED
>     flag" at the begin of the queue to prevent timers to end up in global
>     timer base when they were queued using add_timer_on()
>   - Fix some comments and typos
> 
> v2..v3: https://lore.kernel.org/r/20170418111102.490432548@linutronix.de/
>   - Minimize usage of locks by storing data using atomic_cmpxchg() for
>     migrator information and information about active cpus.
> 
> 
> Thanks,
> 
> 	Anna-Maria
> 
> 
> 
> 
> Anna-Maria Behnsen (18):
>   timers: Restructure get_next_timer_interrupt()
>   timers: Split out get next timer interrupt
>   timers: Move marking timer bases idle into tick_nohz_stop_tick()
>   timers: Optimization for timer_base_try_to_set_idle()
>   timers: Introduce add_timer() variants which modify timer flags
>   workqueue: Use global variant for add_timer()
>   timers: add_timer_on(): Make sure TIMER_PINNED flag is set
>   timers: Ease code in run_local_timers()
>   timers: Split next timer interrupt logic
>   timers: Keep the pinned timers separate from the others
>   timers: Retrieve next expiry of pinned/non-pinned timers separately
>   timers: Split out "get next timer interrupt" functionality
>   timers: Add get next timer interrupt functionality for remote CPUs
>   timers: Check if timers base is handled already
>   timers: Introduce function to check timer base is_idle flag
>   timers: Implement the hierarchical pull model
>   timer_migration: Add tracepoints
>   timers: Always queue timers on the local CPU
> 
> Richard Cochran (linutronix GmbH) (2):
>   timers: Restructure internal locking
>   tick/sched: Split out jiffies update helper function
> 
>  MAINTAINERS                            |    1 +
>  include/linux/cpuhotplug.h             |    1 +
>  include/linux/timer.h                  |   16 +-
>  include/trace/events/timer_migration.h |  297 +++++
>  kernel/time/Makefile                   |    3 +
>  kernel/time/tick-internal.h            |   14 +
>  kernel/time/tick-sched.c               |   65 +-
>  kernel/time/timer.c                    |  505 +++++--
>  kernel/time/timer_migration.c          | 1693 ++++++++++++++++++++++++
>  kernel/time/timer_migration.h          |  147 ++
>  kernel/workqueue.c                     |    2 +-
>  11 files changed, 2629 insertions(+), 115 deletions(-)
>  create mode 100644 include/trace/events/timer_migration.h
>  create mode 100644 kernel/time/timer_migration.c
>  create mode 100644 kernel/time/timer_migration.h

Hi Anna-Maria,
I did some quick measurements on a pixel6 Android 14 with 6.6 kernel baseline.
The workload is 5 iterations of uibenchjanktests (~70 Min runtime total).
Backport of timers/pushpull up to:
6b7e23d1f495 ("timers: Always queue timers on the local CPU").

Power:
+------------+--------+------------+-------+-----------+
|  channel   | metric |    tag     | value | perc_diff |
+------------+--------+------------+-------+-----------+
|    CPU     | gmean  | mainline_5 | 196.6 |   0.0%    |
|  CPU-Big   | gmean  | mainline_5 | 65.3  |   0.0%    |
| CPU-Little | gmean  | mainline_5 | 99.6  |   0.0%    |
|  CPU-Mid   | gmean  | mainline_5 | 31.6  |   0.0%    |
|    GPU     | gmean  | mainline_5 | 36.7  |   0.0%    |
|   Total    | gmean  | mainline_5 | 233.3 |   0.0%    |
|    CPU     | gmean  | pushpull_5 | 195.9 |  -0.35%   |
|  CPU-Big   | gmean  | pushpull_5 | 64.8  |  -0.85%   |
| CPU-Little | gmean  | pushpull_5 | 98.5  |  -1.12%   |
|  CPU-Mid   | gmean  | pushpull_5 | 32.6  |   3.13%   |
|    GPU     | gmean  | pushpull_5 | 36.8  |   0.19%   |
|   Total    | gmean  | pushpull_5 | 232.6 |  -0.26%   |
+------------+--------+------------+-------+-----------+
(Slightly skewed in favor of mainline because of starting
temperature.)

Idle residency:
+------------+---------+------------+--------+
|    tag     | cluster | idle_state |  time  |
+------------+---------+------------+--------+
| mainline_5 | little  |    -1.0    | 518.42 |
| mainline_5 | little  |    0.0     | 238.28 |
| mainline_5 | little  |    1.0     |  19.7  |
| mainline_5 |   mid   |    -1.0    | 201.0  |
| mainline_5 |   mid   |    0.0     | 335.26 |
| mainline_5 |   mid   |    1.0     | 240.15 |
| mainline_5 |   big   |    -1.0    | 173.86 |
| mainline_5 |   big   |    0.0     | 330.93 |
| mainline_5 |   big   |    1.0     | 271.61 |
| pushpull_5 | little  |    -1.0    | 526.45 |
| pushpull_5 | little  |    0.0     | 257.77 |
| pushpull_5 | little  |    1.0     |  5.18  |
| pushpull_5 |   mid   |    -1.0    | 220.98 |
| pushpull_5 |   mid   |    0.0     | 347.43 |
| pushpull_5 |   mid   |    1.0     | 220.98 |
| pushpull_5 |   big   |    -1.0    | 177.36 |
| pushpull_5 |   big   |    0.0     | 331.61 |
| pushpull_5 |   big   |    1.0     | 280.42 |
+------------+---------+------------+--------+

We can see the improvement we were hoping for:
Longer idle times on the big cores.

For completeness here are the idle misses:
+------------+-------+--------------------+
|    tag     | type  |     count_perc     |
+------------+-------+--------------------+
| mainline_5 | False | 3.4829999999999997 |
| mainline_5 | True  | 15.639000000000001 |
| pushpull_5 | False |       3.487        |
| pushpull_5 | True  |       15.881       |
+------------+-------+--------------------+

If there is anything else you would like to see some data on, please
let me know.

Kind Regards,
Christian

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

[...]

>
> But you know what, let's make it more simple. CPU down hotplug is not a
> fast path and it doesn't deserve so many optimizations. Just remove ->wakeup_recalc
> entirely and if the offlining CPU detects it's the last active CPU in the
> hierarchy, just queue an empty work to the first online CPU. It will briefly
> force that CPU out of idle and trigger an activate. Then either the CPU
> periodically checks remote timers or it will go back idle and notice.
>

I'll have a look at it and give it a try! Thanks!

> Something like this (untested):
>
> diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c
> index de1905b0bae7..0f15215ef257 100644
> --- a/kernel/time/timer_migration.c
> +++ b/kernel/time/timer_migration.c
> @@ -548,7 +548,6 @@ static void __tmigr_cpu_activate(struct tmigr_cpu *tmc)
>  
>  	tmc->cpuevt.ignore = true;
>  	WRITE_ONCE(tmc->wakeup, KTIME_MAX);
> -	tmc->wakeup_recalc = false;
>  
>  	walk_groups(&tmigr_active_up, &data, tmc);
>  }
> @@ -1041,41 +1040,11 @@ int tmigr_requires_handle_remote(void)
>  	}
>  
>  	/*
> -	 * If the CPU is idle, check whether the recalculation of @tmc->wakeup
> -	 * is required. @tmc->wakeup_recalc is set, when the last active CPU
> -	 * went offline. The last active CPU delegated the handling of the timer
> -	 * migration hierarchy to another (this) CPU by updating this flag and
> -	 * sending a reschedule.
> -	 *
> -	 * Racy lockless check is valid:
> -	 * - @tmc->wakeup_recalc is set by the remote CPU before it issues
> -	 *   reschedule IPI.
> -	 * - As interrupts are disabled here this CPU will either observe
> -	 *   @tmc->wakeup_recalc set before the reschedule IPI can be handled or
> -	 *   it will observe it when this function is called again on return
> -	 *   from handling the reschedule IPI.
> -	 */
> -	if (tmc->wakeup_recalc) {
> -		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
> -
> -		if (data.firstexp != KTIME_MAX)
> -			ret = 1;
> -
> -		raw_spin_lock(&tmc->lock);
> -		WRITE_ONCE(tmc->wakeup, data.firstexp);
> -		tmc->wakeup_recalc = false;
> -		raw_spin_unlock(&tmc->lock);
> -
> -		return ret;
> -	}
> -
> -	/*
> -	 * When the CPU is idle and @tmc->wakeup is reliable as
> -	 * @tmc->wakeup_recalc is not set, compare it with @data.now. The lock
> -	 * is required on 32bit architectures to read the variable consistently
> -	 * with a concurrent writer. On 64bit the lock is not required because
> -	 * the read operation is not split and so it is always consistent.
> -
> +	 * When the CPU is idle and @tmc->wakeup is reliable, compare it with
> +	 * @data.now. The lock is required on 32bit architectures to read the
> +	 * variable consistently with a concurrent writer. On 64bit the lock
> +	 * is not required because the read operation is not split and so it is
> +	 * always consistent.
>  	 */
>  	if (IS_ENABLED(CONFIG_64BIT)) {
>  		if (data.now >= READ_ONCE(tmc->wakeup))
> @@ -1119,21 +1088,7 @@ u64 tmigr_cpu_new_timer(u64 nextexp)
>  		    tmc->cpuevt.ignore) {
>  			ret = tmigr_new_timer(tmc, nextexp);
>  		}
> -	} else if (tmc->wakeup_recalc) {
> -		struct tmigr_remote_data data;
> -
> -		data.now = KTIME_MAX;
> -		data.childmask = tmc->childmask;
> -		data.firstexp = KTIME_MAX;
> -		data.tmc_active = false;
> -		data.check = false;
> -
> -		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
> -
> -		ret = data.firstexp;
>  	}
> -	tmc->wakeup_recalc = false;
> -
>  	/*
>  	 * Make sure the reevaluation of timers in idle path will not miss an
>  	 * event.
> @@ -1212,36 +1167,7 @@ static bool tmigr_inactive_up(struct tmigr_group *group,
>  	 *   hierarchy) and
>  	 * - there is a pending event in the hierarchy
>  	 */
> -	if (data->firstexp != KTIME_MAX) {
> -		WARN_ON_ONCE(group->parent);
> -		/*
> -		 * Top level path: If this CPU is about going offline and was
> -		 * the last active CPU, wake up some random other CPU so it will
> -		 * take over the migrator duty and program its timer
> -		 * properly. Ideally wake the CPU with the closest expiry time,
> -		 * but that's overkill to figure out.
> -		 *
> -		 * Set wakeup_recalc of remote CPU, to make sure the complete
> -		 * idle hierarchy with enqueued timers is reevaluated.
> -		 */
> -		if (!(this_cpu_ptr(&tmigr_cpu)->online)) {
> -			struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
> -			unsigned int cpu = smp_processor_id();
> -			struct tmigr_cpu *tmc_resched;
> -
> -			cpu = cpumask_any_but(cpu_online_mask, cpu);
> -			tmc_resched = per_cpu_ptr(&tmigr_cpu, cpu);
> -
> -			raw_spin_unlock(&tmc->lock);
> -
> -			raw_spin_lock(&tmc_resched->lock);
> -			tmc_resched->wakeup_recalc = true;
> -			raw_spin_unlock(&tmc_resched->lock);
> -
> -			raw_spin_lock(&tmc->lock);
> -			smp_send_reschedule(cpu);
> -		}
> -	}
> +	WARN_ON_ONCE(data->firstexp != KTIME_MAX && group->parent);
>  
>  	return walk_done;
>  }
> @@ -1579,9 +1505,20 @@ static int tmigr_cpu_online(unsigned int cpu)
>  	return 0;
>  }
>  
> +long tmigr_trigger_active(void *unused)
> +{
> +	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
> +
> +	WARN_ON_ONCE(!tmc->online || tmc->idle);
> +
> +	return 0;
> +}
> +
>  static int tmigr_cpu_offline(unsigned int cpu)
>  {
>  	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
> +	int migrator;
> +	u64 firstexp;
>  
>  	raw_spin_lock_irq(&tmc->lock);
>  	tmc->online = false;
> @@ -1591,9 +1528,14 @@ static int tmigr_cpu_offline(unsigned int cpu)
>  	 * CPU has to handle the local events on his own, when on the way to
>  	 * offline; Therefore nextevt value is set to KTIME_MAX
>  	 */
> -	__tmigr_cpu_deactivate(tmc, KTIME_MAX);
> +	firstexp = __tmigr_cpu_deactivate(tmc, KTIME_MAX);
>  	raw_spin_unlock_irq(&tmc->lock);
>  
> +	if (firstexp != KTIME_MAX) {
> +		migrator = cpumask_any_but(cpu_online_mask, cpu);
> +		work_on_cpu(migrator, tmigr_trigger_active, NULL);
> +	}
> +
>  	return 0;
>  }
>  
> diff --git a/kernel/time/timer_migration.h b/kernel/time/timer_migration.h
> index c32947cf429b..c556d5824792 100644
> --- a/kernel/time/timer_migration.h
> +++ b/kernel/time/timer_migration.h
> @@ -78,18 +78,12 @@ struct tmigr_group {
>   * @idle:		Indicates whether the CPU is idle in the timer migration
>   *			hierarchy
>   * @remote:		Is set when timers of the CPU are expired remotely
> - * @wakeup_recalc:	Indicates, whether a recalculation of the @wakeup value
> - *			is required. @wakeup_recalc is only used by this CPU
> - *			when it is marked idle in the timer migration
> - *			hierarchy. It is set by a remote CPU which was the last
> - *			active CPU and is on the way to idle.
>   * @tmgroup:		Pointer to the parent group
>   * @childmask:		childmask of tmigr_cpu in the parent group
>   * @wakeup:		Stores the first timer when the timer migration
>   *			hierarchy is completely idle and remote expiry was done;
>   *			is returned to timer code in the idle path and is only
> - *			used in idle path; it is only valid, when @wakeup_recalc
> - *			is not set.
> + *			used in idle path.
>   * @cpuevt:		CPU event which could be enqueued into the parent group
>   */
>  struct tmigr_cpu {
> @@ -97,7 +91,6 @@ struct tmigr_cpu {
>  	bool			online;
>  	bool			idle;
>  	bool			remote;
> -	bool			wakeup_recalc;
>  	struct tmigr_group	*tmgroup;
>  	u8			childmask;
>  	u64			wakeup;

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> +/*
>> + * Returns true, if there is nothing to be propagated to the next level
>> + *
>> + * @data->firstexp is set to expiry of first gobal event of the (top level of
>> + * the) hierarchy, but only when hierarchy is completely idle.
>> + *
>> + * This is the only place where the group event expiry value is set.
>> + */
>> +static
>> +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
>> +			 struct tmigr_walk *data, union tmigr_state childstate,
>> +			 union tmigr_state groupstate)
>> +{
>> +	struct tmigr_event *evt, *first_childevt;
>> +	bool walk_done, remote = data->remote;
>> +	bool leftmost_change = false;
>> +	u64 nextexp;
>> +
>> +	if (child) {
>> +		raw_spin_lock(&child->lock);
>> +		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
>> +
>> +		if (childstate.active) {
>
> Since you're going to do the atomic_read(&group->migr_state)
> within the group->lock, you may as well do the atomic_read(&child->migr_state)
> within the child->lock. It won't hurt and simplifies the picture
> in the mind.

Already changed it this way.

> Then you can add the following comment to outline the ordering
> expectations:
>
> /*
>  * Observing child->migr_state.active means that:
>  *
>  * 1) Either the child is effectively active, then it's fine to stop here
>  *
>  * 2) Or we are racing with a CPU going inactive and this childstate is actually
>  *    not active anymore but tmigr_inactive_up() hasn't yet called tmigr_update_event()
>  *    on it. It's fine to stop here because that pending call will take care
>  *    of the rest of the propagation.
>  *
>  * 3) In any case it's impossible to observe childstate.active when a racing
>  *    CPU made it inactive and also called tmigr_update_event() on it. The
>  *    group->lock enforces ordering such that ->migr_state changes
>  *    in tmigr_inactive_up() are released by group->lock UNLOCK on the
>  *    subsequent call to tmigr_update_event() and then acquired by
>  *    child->lock LOCK in tmigr_new_timer() -> tmigr_update_event().
>  */

I'll add the comment! Thanks

>> +			walk_done = true;
>> +			goto unlock;
>> +		}
>> +
>> +		first_childevt = tmigr_next_groupevt(child);
>> +		nextexp = child->next_expiry;
>> +		evt = &child->groupevt;
>> +	} else {
>> +		nextexp = data->nextexp;
>> +
>> +		first_childevt = evt = data->evt;
>> +
>> +		/*
>> +		 * Walking the hierarchy is required in any case when a
>> +		 * remote expiry was done before. This ensures to not lose
>> +		 * already queued events in non active groups (see section
>> +		 * "Required event and timerqueue update after a remote
>> +		 * expiry" in the documentation at the top).
>> +		 *
>> +		 * The two call sites which are executed without a remote expiry
>> +		 * before, are not prevented from propagating changes through
>> +		 * the hierarchy by the return:
>> +		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
>> +		 *    is never set.
>> +		 *  - tmigr_inactive_up() takes care of the propagation by
>> +		 *    itself and ignores the return value. But an immediate
>> +		 *    return is required because nothing has to be done in this
>> +		 *    level as the event could be ignored.
>> +		 */
>> +		if (evt->ignore && !remote)
>> +			return true;
>> +
>> +		raw_spin_lock(&group->lock);
>> +	}
>> +
>> +	if (nextexp == KTIME_MAX) {
>> +		evt->ignore = true;
>> +
>> +		/*
>> +		 * When the next child event could be ignored (nextexp is
>> +		 * KTIME_MAX) and there was no remote timer handling before or
>> +		 * the group is already active, there is no need to walk the
>> +		 * hierarchy even if there is a parent group.
>> +		 *
>> +		 * The other way round: even if the event could be ignored, but
>> +		 * if a remote timer handling was executed before and the group
>> +		 * is not active, walking the hierarchy is required to not miss
>> +		 * an enqueued timer in the non active group. The enqueued timer
>> +		 * of the group needs to be propagated to a higher level to
>> +		 * ensure it is handled.
>> +		 */
>> +		if (!remote || groupstate.active) {
>
> Same here, fetching group->migr_state.active from within the lock simplifies
> the mind mapping.

Sure. Already changed it.

Thanks,
	Anna-Maria

Re: [PATCH v10 00/20] timers: Move from a push remote at enqueue to a pull at expiry model

[Anna-Maria Behnsen]

Hi,

Christian Loehle <christian.loehle@arm.com> writes:

> On 15/01/2024 14:37, Anna-Maria Behnsen wrote:
>> Hi,
>> 
>> the cleanup patches are already applied and so the contains only two parts:
>> 
>> - Patches 1 - 4: timer base idle marking rework with two preparatory
>>   changes. See the section below for more details.
>> 
>> - Patches 5 - 20: Updated timer pull model on top of timer idle rework
>> 
>> 
>> The queue is available here:
>> 
>>   git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel timers/pushpull
>> 
>> 
>> Move marking timer bases as idle into tick_nohz_stop_tick()
>> -----------------------------------------------------------
>> 
>> The idle marking of timer bases is done in get_next_timer_interrupt()
>> whenever possible. The timer bases are idle, even if the tick will not be
>> stopped. This lead to an IPI when a new first timer is enqueued remote. To
>> prevent this, setting timer_base->in_idle flag is postponed to
>> tick_nohz_stop_tick().
>> 
>> Furthermore this synchronizes the states of timer base is_idle and
>> tick_stopped. With the timer pull model in place, also the idle state in
>> the hierarchy of a CPU is synchronized with the other idle related states.
>> 
>> 
>> Timer pull model
>> ----------------
>> 
>> Placing timers at enqueue time on a target CPU based on dubious heuristics
>> does not make any sense:
>> 
>>  1) Most timer wheel timers are canceled or rearmed before they expire.
>> 
>>  2) The heuristics to predict which CPU will be busy when the timer expires
>>     are wrong by definition.
>> 
>> So placing the timers at enqueue wastes precious cycles.
>> 
>> The proper solution to this problem is to always queue the timers on the
>> local CPU and allow the non pinned timers to be pulled onto a busy CPU at
>> expiry time.
>> 
>> Therefore split the timer storage into local pinned and global timers:
>> Local pinned timers are always expired on the CPU on which they have been
>> queued. Global timers can be expired on any CPU.
>> 
>> As long as a CPU is busy it expires both local and global timers. When a
>> CPU goes idle it arms for the first expiring local timer. If the first
>> expiring pinned (local) timer is before the first expiring movable timer,
>> then no action is required because the CPU will wake up before the first
>> movable timer expires. If the first expiring movable timer is before the
>> first expiring pinned (local) timer, then this timer is queued into a idle
>> timerqueue and eventually expired by some other active CPU.
>> 
>> To avoid global locking the timerqueues are implemented as a hierarchy. The
>> lowest level of the hierarchy holds the CPUs. The CPUs are associated to
>> groups of 8, which are separated per node. If more than one CPU group
>> exist, then a second level in the hierarchy collects the groups. Depending
>> on the size of the system more than 2 levels are required. Each group has a
>> "migrator" which checks the timerqueue during the tick for remote timers to
>> be expired.
>> 
>> If the last CPU in a group goes idle it reports the first expiring event in
>> the group up to the next group(s) in the hierarchy. If the last CPU goes
>> idle it arms its timer for the first system wide expiring timer to ensure
>> that no timer event is missed.
>> 
>> 
>> Testing
>> ~~~~~~~
>> 
>> Enqueue
>> ^^^^^^^
>> 
>> The impact of wasting cycles during enqueue by using the heuristic in
>> contrast to always queuing the timer on the local CPU was measured with a
>> micro benchmark. Therefore a timer is enqueued and dequeued in a loop with
>> 1000 repetitions on a isolated CPU. The time the loop takes is measured. A
>> quarter of the remaining CPUs was kept busy. This measurement was repeated
>> several times. With the patch queue the average duration was reduced by
>> approximately 25%.
>> 
>> 	145ns	plain v6
>> 	109ns	v6 with patch queue
>> 
>> 
>> Furthermore the impact of residence in deep idle states of an idle system
>> was investigated. The patch queue doesn't downgrade this behavior.
>> 
>> dbench test
>> ^^^^^^^^^^^
>> 
>> A dbench test starting X pairs of client servers are used to create load on
>> the system. The measurable value is the throughput. The tests were executed
>> on a zen3 machine. The base is the tip tree branch timers/core which is
>> based on a v6.6-rc1.
>> 
>> governor menu
>> 
>> NR	timers/core	pull-model	impact
>> ----------------------------------------------
>> 1	353.19 (0.19)	353.45 (0.30)	0.07%
>> 2	700.10 (0.96)	687.00 (0.20)	-1.87%
>> 4	1329.37 (0.63)	1282.91 (0.64)	-3.49%
>> 8	2561.16 (1.28)	2493.56	(1.76)	-2.64%
>> 16	4959.96 (0.80)	4914.59 (0.64)	-0.91%
>> 32	9741.92 (3.44)	8979.83 (1.13)	-7.82%
>> 64	16535.40 (2.84)	16388.47 (4.02)	-0.89%
>> 128	22136.83 (2.42)	23174.50 (1.43)	4.69%
>> 256	39256.77 (4.48)	38994.00 (0.39)	-0.67%
>> 512	36799.03 (1.83)	38091.10 (0.63)	3.51%
>> 1024	32903.03 (0.86)	35370.70 (0.89)	7.50%
>> 
>> 
>> governor teo
>> 
>> NR	timers/core	pull-model	impact
>> ----------------------------------------------
>> 1	350.83 (1.27)	352.45 (0.96)	0.46%
>> 2	699.52 (0.85)	690.10 (0.54)	-1.35%
>> 4	1339.53 (1.99)	1294.71 (2.71)	-3.35%
>> 8	2574.10 (0.76)	2495.46 (1.97)	-3.06%
>> 16	4898.50 (1.74)	4783.06 (1.64)	-2.36%
>> 32	9115.50 (4.63)	9037.83 (1.58)	-0.85%
>> 64	16663.90 (3.80)	16042.00 (1.72)	-3.73%
>> 128	25044.93 (1.11)	23250.03 (1.08)	-7.17%
>> 256	38059.53 (1.70)	39658.57 (2.98)	4.20%
>> 512	36369.30 (0.39)	38890.13 (0.36)	6.93%
>> 1024	33956.83 (1.14)	35514.83 (0.29)	4.59%
>> 
>> 
>> 
>> Ping Pong Oberservation
>> ^^^^^^^^^^^^^^^^^^^^^^^
>> 
>> During testing on a mostly idle machine a ping pong game could be observed:
>> a process_timeout timer is expired remotely on a non idle CPU. Then the CPU
>> where the schedule_timeout() was executed to enqueue the timer comes out of
>> idle and restarts the timer using schedule_timeout() and goes back to idle
>> again. This is due to the fair scheduler which tries to keep the task on
>> the CPU which it previously executed on.
>> 
>> 
>> 
>> 
>> Possible Next Steps
>> ~~~~~~~~~~~~~~~~~~~
>> 
>> Simple deferrable timers are no longer required as they can be converted to
>> global timers. If a CPU goes idle, a formerly deferrable timer will not
>> prevent the CPU to sleep as long as possible. Only the last migrator CPU
>> has to take care of them. Deferrable timers with timer pinned flags needs
>> to be expired on the specified CPU but must not prevent CPU from going
>> idle. They require their own timer base which is never taken into account
>> when calculating the next expiry time. This conversation and required
>> cleanup will be done in a follow up series.
>> 
>> 
>> v9..v10: https://lore.kernel.org/r/20231201092654.34614-1-anna-maria@linutronix.de/
>>   - Address review Feedback of Bigeasy
>> 
>> 
>> v8..v9: https://lore.kernel.org/r/20231004123454.15691-1-anna-maria@linutronix.de
>>   - Address review feedback
>>   - Add more minor cleanup fixes
>>   - fixes inconsistent idle related states
>> 
>> 
>> v7..v8: https://lore.kernel.org/r/20230524070629.6377-1-anna-maria@linutronix.de
>>   - Address review feedback
>>   - Move marking timer base idle into tick_nohz_stop_tick()
>>   - Look ahead function to determine possible sleep lenght
>> 
>> 
>> v6..v7:
>>   - Address review feedback of Frederic and bigeasy
>>   - Change lock, unlock fetch next timer interrupt logic after remote expiry
>>   - Move timer_expire_remote() into tick-internal.h
>>   - Add documentation section about "Required event and timerqueue update
>>     after remote expiry"
>>   - Fix fallout of kernel test robot
>> 
>> 
>> v5..v6:
>> 
>>   - Address review of Frederic Weisbecker and Peter Zijlstra (spelling,
>>     locking, race in tmigr_handle_remote_cpu())
>> 
>>   - unconditionally set TIMER_PINNED flag in add_timer_on(); introduce
>>     add_timer() variants which set/unset TIMER_PINNED flag; drop fixing
>>     add_timer_on() call sites, as TIMER_PINNED flag is set implicitly;
>>     Fixing workqueue to use add_timer_global() instead of simply
>>     add_timer() for unbound work.
>> 
>>   - Drop support for siblings to end up in the same level 0 group (could be
>>     added again in a better way as an improvement later on)
>> 
>>   - Do not send IPI for new first deferrable timers
>> 
>> v4..v5:
>>   - address review feedback of Frederic Weisbecker
>>   - fix issue with group timer update after remote expiry
>> 
>> v3..v4:
>>   - address review feedback of Frederic Weisbecker
>>   - address kernel test robot fallout
>>   - Move patch 16 "add_timer_on(): Make sure callers have TIMER_PINNED
>>     flag" at the begin of the queue to prevent timers to end up in global
>>     timer base when they were queued using add_timer_on()
>>   - Fix some comments and typos
>> 
>> v2..v3: https://lore.kernel.org/r/20170418111102.490432548@linutronix.de/
>>   - Minimize usage of locks by storing data using atomic_cmpxchg() for
>>     migrator information and information about active cpus.
>> 
>> 
>> Thanks,
>> 
>> 	Anna-Maria
>> 
>> 
>> 
>> 
>> Anna-Maria Behnsen (18):
>>   timers: Restructure get_next_timer_interrupt()
>>   timers: Split out get next timer interrupt
>>   timers: Move marking timer bases idle into tick_nohz_stop_tick()
>>   timers: Optimization for timer_base_try_to_set_idle()
>>   timers: Introduce add_timer() variants which modify timer flags
>>   workqueue: Use global variant for add_timer()
>>   timers: add_timer_on(): Make sure TIMER_PINNED flag is set
>>   timers: Ease code in run_local_timers()
>>   timers: Split next timer interrupt logic
>>   timers: Keep the pinned timers separate from the others
>>   timers: Retrieve next expiry of pinned/non-pinned timers separately
>>   timers: Split out "get next timer interrupt" functionality
>>   timers: Add get next timer interrupt functionality for remote CPUs
>>   timers: Check if timers base is handled already
>>   timers: Introduce function to check timer base is_idle flag
>>   timers: Implement the hierarchical pull model
>>   timer_migration: Add tracepoints
>>   timers: Always queue timers on the local CPU
>> 
>> Richard Cochran (linutronix GmbH) (2):
>>   timers: Restructure internal locking
>>   tick/sched: Split out jiffies update helper function
>> 
>>  MAINTAINERS                            |    1 +
>>  include/linux/cpuhotplug.h             |    1 +
>>  include/linux/timer.h                  |   16 +-
>>  include/trace/events/timer_migration.h |  297 +++++
>>  kernel/time/Makefile                   |    3 +
>>  kernel/time/tick-internal.h            |   14 +
>>  kernel/time/tick-sched.c               |   65 +-
>>  kernel/time/timer.c                    |  505 +++++--
>>  kernel/time/timer_migration.c          | 1693 ++++++++++++++++++++++++
>>  kernel/time/timer_migration.h          |  147 ++
>>  kernel/workqueue.c                     |    2 +-
>>  11 files changed, 2629 insertions(+), 115 deletions(-)
>>  create mode 100644 include/trace/events/timer_migration.h
>>  create mode 100644 kernel/time/timer_migration.c
>>  create mode 100644 kernel/time/timer_migration.h
>
> Hi Anna-Maria,
> I did some quick measurements on a pixel6 Android 14 with 6.6 kernel baseline.
> The workload is 5 iterations of uibenchjanktests (~70 Min runtime total).
> Backport of timers/pushpull up to:
> 6b7e23d1f495 ("timers: Always queue timers on the local CPU").
>
> Power:
> +------------+--------+------------+-------+-----------+
> |  channel   | metric |    tag     | value | perc_diff |
> +------------+--------+------------+-------+-----------+
> |    CPU     | gmean  | mainline_5 | 196.6 |   0.0%    |
> |  CPU-Big   | gmean  | mainline_5 | 65.3  |   0.0%    |
> | CPU-Little | gmean  | mainline_5 | 99.6  |   0.0%    |
> |  CPU-Mid   | gmean  | mainline_5 | 31.6  |   0.0%    |
> |    GPU     | gmean  | mainline_5 | 36.7  |   0.0%    |
> |   Total    | gmean  | mainline_5 | 233.3 |   0.0%    |
> |    CPU     | gmean  | pushpull_5 | 195.9 |  -0.35%   |
> |  CPU-Big   | gmean  | pushpull_5 | 64.8  |  -0.85%   |
> | CPU-Little | gmean  | pushpull_5 | 98.5  |  -1.12%   |
> |  CPU-Mid   | gmean  | pushpull_5 | 32.6  |   3.13%   |
> |    GPU     | gmean  | pushpull_5 | 36.8  |   0.19%   |
> |   Total    | gmean  | pushpull_5 | 232.6 |  -0.26%   |
> +------------+--------+------------+-------+-----------+
> (Slightly skewed in favor of mainline because of starting
> temperature.)
>
> Idle residency:
> +------------+---------+------------+--------+
> |    tag     | cluster | idle_state |  time  |
> +------------+---------+------------+--------+
> | mainline_5 | little  |    -1.0    | 518.42 |
> | mainline_5 | little  |    0.0     | 238.28 |
> | mainline_5 | little  |    1.0     |  19.7  |
> | mainline_5 |   mid   |    -1.0    | 201.0  |
> | mainline_5 |   mid   |    0.0     | 335.26 |
> | mainline_5 |   mid   |    1.0     | 240.15 |
> | mainline_5 |   big   |    -1.0    | 173.86 |
> | mainline_5 |   big   |    0.0     | 330.93 |
> | mainline_5 |   big   |    1.0     | 271.61 |
> | pushpull_5 | little  |    -1.0    | 526.45 |
> | pushpull_5 | little  |    0.0     | 257.77 |
> | pushpull_5 | little  |    1.0     |  5.18  |
> | pushpull_5 |   mid   |    -1.0    | 220.98 |
> | pushpull_5 |   mid   |    0.0     | 347.43 |
> | pushpull_5 |   mid   |    1.0     | 220.98 |
> | pushpull_5 |   big   |    -1.0    | 177.36 |
> | pushpull_5 |   big   |    0.0     | 331.61 |
> | pushpull_5 |   big   |    1.0     | 280.42 |
> +------------+---------+------------+--------+
>
> We can see the improvement we were hoping for:
> Longer idle times on the big cores.

This is good to know, that it works and I didn't break this in the
meantime :)

Thanks a lot for testing!

	Anna-Maria

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> +static void tmigr_connect_child_parent(struct tmigr_group *child,
> +				       struct tmigr_group *parent)
> +{
> +	union tmigr_state childstate;
> +
> +	raw_spin_lock_irq(&child->lock);
> +	raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING);
> +
> +	child->parent = parent;
> +	child->childmask = BIT(parent->num_children++);
> +
> +	raw_spin_unlock(&parent->lock);
> +	raw_spin_unlock_irq(&child->lock);
> +
> +	/*
> +	 * To prevent inconsistent states, active children need to be active in
> +	 * the new parent as well. Inactive children are already marked inactive
> +	 * in the parent group.
> +	 */
> +	childstate.state = atomic_read(&child->migr_state);
> +	if (childstate.migrator != TMIGR_NONE) {

Is it possible here to connect a running online child (not one that we just
created) to a new parent? If not, is it possible that a newly created child is
not TMIGR_NONE?


> +		struct tmigr_walk data;
> +
> +		data.childmask = child->childmask;
> +
> +		/*
> +		 * There is only one new level per time. When connecting the
> +		 * child and the parent and set the child active when the parent
> +		 * is inactive, the parent needs to be the uppermost
> +		 * level. Otherwise there went something wrong!
> +		 */
> +		WARN_ON(!tmigr_active_up(parent, child, &data) && parent->parent);
> +	}
> +}
[...]
> +static int tmigr_cpu_online(unsigned int cpu)
> +{
> +	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
> +	int ret;
> +
> +	/* First online attempt? Initialize CPU data */
> +	if (!tmc->tmgroup) {
> +		raw_spin_lock_init(&tmc->lock);
> +
> +		ret = tmigr_add_cpu(cpu);
> +		if (ret < 0)
> +			return ret;
> +
> +		if (tmc->childmask == 0)
> +			return -EINVAL;
> +
> +		timerqueue_init(&tmc->cpuevt.nextevt);
> +		tmc->cpuevt.nextevt.expires = KTIME_MAX;
> +		tmc->cpuevt.ignore = true;
> +		tmc->cpuevt.cpu = cpu;
> +
> +		tmc->remote = false;
> +		WRITE_ONCE(tmc->wakeup, KTIME_MAX);
> +	}
> +	raw_spin_lock_irq(&tmc->lock);
> +	tmc->idle = timer_base_is_idle();
> +	if (!tmc->idle)
> +		__tmigr_cpu_activate(tmc);

Heh, I was about to say that it's impossible that timer_base_is_idle()
at this stage but actually if we run in nohz_full...

It happens so that nohz_full is deactivated until rcutree_online_cpu()
which calls tick_dep_clear() but it's a pure coincidence that might
disappear one day. So yes, let's keep it that way.

Thanks.

> +	tmc->online = true;
> +	raw_spin_unlock_irq(&tmc->lock);
> +	return 0;
> +}

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> +static void tmigr_connect_child_parent(struct tmigr_group *child,
>> +				       struct tmigr_group *parent)
>> +{
>> +	union tmigr_state childstate;
>> +
>> +	raw_spin_lock_irq(&child->lock);
>> +	raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING);
>> +
>> +	child->parent = parent;
>> +	child->childmask = BIT(parent->num_children++);
>> +
>> +	raw_spin_unlock(&parent->lock);
>> +	raw_spin_unlock_irq(&child->lock);
>> +
>> +	/*
>> +	 * To prevent inconsistent states, active children need to be active in
>> +	 * the new parent as well. Inactive children are already marked inactive
>> +	 * in the parent group.
>> +	 */
>> +	childstate.state = atomic_read(&child->migr_state);
>> +	if (childstate.migrator != TMIGR_NONE) {
>
> Is it possible here to connect a running online child (not one that we just
> created) to a new parent?

connect_child_parent() is only executed for the just created ones. So,
yes in theory this would be possible, but it doesn't happen as
tmigr_setup_groups() takes care to make it right (hopefully :)). When a
LVL0 group has some space left, only the connection between tmc and the
LVL0 group is done in tmigr_setup_groups(). If there is no space left in
LVL0 group, then a new group is created and depending on the levels
which has to be created only executed for the new ones.

> If not, is it possible that a newly created child is
> not TMIGR_NONE?

Yes. See tmigr_cpu_online(). When new groups have to be created starting
from LVL0, then they are not active - so TMIGR_NONE is set. Activating
the new online CPU is done afterwards.

But if it is required to add also a new level at the top, then it is
mandatory to propagate the active state of the already existing child to
the new parent. The connect_child_parent() is then also executed for the
formerly top level group (child) to the newly created group (parent).

>
>> +		struct tmigr_walk data;
>> +
>> +		data.childmask = child->childmask;
>> +
>> +		/*
>> +		 * There is only one new level per time. When connecting the
>> +		 * child and the parent and set the child active when the parent
>> +		 * is inactive, the parent needs to be the uppermost
>> +		 * level. Otherwise there went something wrong!
>> +		 */
>> +		WARN_ON(!tmigr_active_up(parent, child, &data) && parent->parent);
>> +	}
>> +}
> [...]
>> +static int tmigr_cpu_online(unsigned int cpu)
>> +{
>> +	struct tmigr_cpu *tmc = this_cpu_ptr(&tmigr_cpu);
>> +	int ret;
>> +
>> +	/* First online attempt? Initialize CPU data */
>> +	if (!tmc->tmgroup) {
>> +		raw_spin_lock_init(&tmc->lock);
>> +
>> +		ret = tmigr_add_cpu(cpu);
>> +		if (ret < 0)
>> +			return ret;
>> +
>> +		if (tmc->childmask == 0)
>> +			return -EINVAL;
>> +
>> +		timerqueue_init(&tmc->cpuevt.nextevt);
>> +		tmc->cpuevt.nextevt.expires = KTIME_MAX;
>> +		tmc->cpuevt.ignore = true;
>> +		tmc->cpuevt.cpu = cpu;
>> +
>> +		tmc->remote = false;
>> +		WRITE_ONCE(tmc->wakeup, KTIME_MAX);
>> +	}
>> +	raw_spin_lock_irq(&tmc->lock);
>> +	tmc->idle = timer_base_is_idle();
>> +	if (!tmc->idle)
>> +		__tmigr_cpu_activate(tmc);
>
> Heh, I was about to say that it's impossible that timer_base_is_idle()
> at this stage but actually if we run in nohz_full...
>
> It happens so that nohz_full is deactivated until rcutree_online_cpu()
> which calls tick_dep_clear() but it's a pure coincidence that might
> disappear one day. So yes, let's keep it that way.

I instrumented the code (with NOHZ FULL and NOHZ_IDLE) to make sure the
timer migration hierarchy state 'idle' is in sync with the timer base
'idle'. And this was one part where it was possible that it runs out of
sync as I remember correctly. But if I understood you correctly, this
shouldn't happen at the moment?

Thanks,

	Anna-Maria

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> +static bool tmigr_handle_remote_up(struct tmigr_group *group,
> +				   struct tmigr_group *child,
> +				   void *ptr)
> +{
> +	struct tmigr_remote_data *data = ptr;
> +	u64 now, next = KTIME_MAX;
> +	struct tmigr_event *evt;
> +	unsigned long jif;
> +	u8 childmask;
> +
> +	jif = data->basej;
> +	now = data->now;
> +
> +	childmask = data->childmask;
> +
> +again:
> +	/*
> +	 * Handle the group only if @childmask is the migrator or if the
> +	 * group has no migrator. Otherwise the group is active and is
> +	 * handled by its own migrator.
> +	 */
> +	if (!tmigr_check_migrator(group, childmask))
> +		return true;
> +
> +	raw_spin_lock_irq(&group->lock);
> +
> +	evt = tmigr_next_expired_groupevt(group, now);
> +
> +	if (evt) {
> +		unsigned int remote_cpu = evt->cpu;
> +
> +		raw_spin_unlock_irq(&group->lock);
> +
> +		next = tmigr_handle_remote_cpu(remote_cpu, now, jif);
> +
> +		/* check if there is another event, that needs to be handled */
> +		goto again;
> +	} else {
> +		raw_spin_unlock_irq(&group->lock);
> +	}
> +
> +	/*
> +	 * Update of childmask for the next level and keep track of the expiry
> +	 * of the first event that needs to be handled
> +	 */
> +	data->childmask = group->childmask;
> +	data->firstexp = next;

So assume we have:

            [GRP1:0]
        migrator = [GRP0:0]
        active   = [GRP0:0]
        nextevt  = TIMER3
        /                    \
    [GRP0:0]                  [GRP0:1]
 migrator = CPU0           migrator = NONE
 active   = CPU0           active   = NONE
 nextevt  = KTIME_MAX      nextevt  = TIMER3
    /         \                /         \
   0           1              2           3
  idle       idle           idle         idle (TIMER3)

Then CPU 0 goes idle:

            [GRP1:0]
        migrator = NONE
        active   = NONE
        nextevt  = TIMER3
        /                    \
    [GRP0:0]                  [GRP0:1]
 migrator = NONE           migrator = NONE
 active   = NONE           active   = NONE
 nextevt  = KTIME_MAX      nextevt  = TIMER3
    /         \                /         \
   0           1              2           3
  idle       idle           idle         idle (TIMER3)

CPU 0 is the idle migrator and its tmc->wakeup is TIMER3.
But CPU 0 has a local timer that expires before TIMER3.

When that timer interrupt fires, it raises the softirq, which
executes on IRQ tail. So CPU0 eventually calls tmigr_handle_remote()
before TIMER3 has expired.

This leads to tmigr_next_expired_groupevt() to return NULL and then
data->firstexp = KTIME_MAX and then tmc->wakeup = KTIME_MAX.

Later on, tmigr_new_timer() is called with a KTIME_MAX global
event and so tmc->wakeup stays with KTIME_MAX, ignoring TIMER3.

It looks like you need to handle the tmigr_next_expired_groupevt()
case returning NULL.

Thanks.

Re: [PATCH v10 19/20] timer_migration: Add tracepoints

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:42PM +0100, Anna-Maria Behnsen a écrit :
> +TRACE_EVENT(tmigr_connect_child_parent,
> +
> +	TP_PROTO(struct tmigr_group *child),
> +
> +	TP_ARGS(child),
> +
> +	TP_STRUCT__entry(
> +		__field( void *,	child		)
> +		__field( void *,	parent		)
> +		__field( unsigned int,	lvl		)
> +		__field( unsigned int,	numa_node	)
> +		__field( unsigned int,	num_children	)
> +		__field( u32,		childmask	)
> +	),
> +
> +	TP_fast_assign(
> +		__entry->child		= child;
> +		__entry->parent		= child->parent;
> +		__entry->lvl		= child->parent->level;
> +		__entry->numa_node	= child->parent->numa_node;
> +		__entry->numa_node	= child->parent->num_children;

__entry->num_children ?

> +		__entry->childmask	= child->childmask;
> +	),
> +
> +	TP_printk("group=%p childmask=%0x parent=%p lvl=%d numa=%d num_children=%d",
> +		  __entry->child,  __entry->childmask, __entry->parent,
> +		  __entry->lvl, __entry->numa_node, __entry->num_children)
> +);
> +
> +TRACE_EVENT(tmigr_connect_cpu_parent,
> +
> +	TP_PROTO(struct tmigr_cpu *tmc),
> +
> +	TP_ARGS(tmc),
> +
> +	TP_STRUCT__entry(
> +		__field( void *,	parent		)
> +		__field( unsigned int,	cpu		)
> +		__field( unsigned int,	lvl		)
> +		__field( unsigned int,	numa_node	)
> +		__field( unsigned int,	num_children	)
> +		__field( u32,		childmask	)
> +	),
> +
> +	TP_fast_assign(
> +		__entry->parent		= tmc->tmgroup;
> +		__entry->cpu		= tmc->cpuevt.cpu;
> +		__entry->lvl		= tmc->tmgroup->level;
> +		__entry->numa_node	= tmc->tmgroup->numa_node;
> +		__entry->numa_node	= tmc->tmgroup->num_children;

Ditto.

Thanks.

Re: [PATCH v10 20/20] timers: Always queue timers on the local CPU

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:43PM +0100, Anna-Maria Behnsen a écrit :
> The timer pull model is in place so we can remove the heuristics which try
> to guess the best target CPU at enqueue/modification time.
> 
> All non pinned timers are queued on the local CPU in the separate storage
> and eventually pulled at expiry time to a remote CPU.
> 
> Originally-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Just one detail below:

> @@ -590,10 +590,13 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer)
>  
>  	/*
>  	 * We might have to IPI the remote CPU if the base is idle and the
> -	 * timer is not deferrable. If the other CPU is on the way to idle
> -	 * then it can't set base->is_idle as we hold the base lock:
> +	 * timer is pinned. If it is a non pinned timer, it is only queued
> +	 * on the remote CPU, when timer was running during queueing. Then
> +	 * everything is handled by remote CPU anyway. If the other CPU is
> +	 * on the way to idle then it can't set base->is_idle as we hold
> +	 * the base lock:
>  	 */
> -	if (base->is_idle)
> +	if (base->is_idle && timer->flags & TIMER_PINNED)

Is the TIMER_PINNED test necessary? If base->is_idle, then the timer
is now guaranteed to be TIMER_PINNED, right?

Thanks.

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Thu, Feb 01, 2024 at 05:15:37PM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> >> +static void tmigr_connect_child_parent(struct tmigr_group *child,
> >> +				       struct tmigr_group *parent)
> >> +{
> >> +	union tmigr_state childstate;
> >> +
> >> +	raw_spin_lock_irq(&child->lock);
> >> +	raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING);
> >> +
> >> +	child->parent = parent;
> >> +	child->childmask = BIT(parent->num_children++);
> >> +
> >> +	raw_spin_unlock(&parent->lock);
> >> +	raw_spin_unlock_irq(&child->lock);
> >> +
> >> +	/*
> >> +	 * To prevent inconsistent states, active children need to be active in
> >> +	 * the new parent as well. Inactive children are already marked inactive
> >> +	 * in the parent group.
> >> +	 */
> >> +	childstate.state = atomic_read(&child->migr_state);
> >> +	if (childstate.migrator != TMIGR_NONE) {
> >
> > Is it possible here to connect a running online child (not one that we just
> > created) to a new parent?
> 
> connect_child_parent() is only executed for the just created ones. So,
> yes in theory this would be possible, but it doesn't happen as
> tmigr_setup_groups() takes care to make it right (hopefully :)). When a
> LVL0 group has some space left, only the connection between tmc and the
> LVL0 group is done in tmigr_setup_groups(). If there is no space left in
> LVL0 group, then a new group is created and depending on the levels
> which has to be created only executed for the new ones.
> 
> > If not, is it possible that a newly created child is
> > not TMIGR_NONE?
> 
> Yes. See tmigr_cpu_online(). When new groups have to be created starting
> from LVL0, then they are not active - so TMIGR_NONE is set. Activating
> the new online CPU is done afterwards.
> 
> But if it is required to add also a new level at the top, then it is
> mandatory to propagate the active state of the already existing child to
> the new parent. The connect_child_parent() is then also executed for the
> formerly top level group (child) to the newly created group (parent).

Ah and this is why we have the "if (childstate.migrator != TMIGR_NONE)"
branch, right?

> > Heh, I was about to say that it's impossible that timer_base_is_idle()
> > at this stage but actually if we run in nohz_full...
> >
> > It happens so that nohz_full is deactivated until rcutree_online_cpu()
> > which calls tick_dep_clear() but it's a pure coincidence that might
> > disappear one day. So yes, let's keep it that way.
> 
> I instrumented the code (with NOHZ FULL and NOHZ_IDLE) to make sure the
> timer migration hierarchy state 'idle' is in sync with the timer base
> 'idle'. And this was one part where it was possible that it runs out of
> sync as I remember correctly. But if I understood you correctly, this
> shouldn't happen at the moment?

Well, it's not supposed to :-)

Thanks.

> 
> Thanks,
> 
> 	Anna-Maria
> 

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Thu, Feb 01, 2024 at 05:15:37PM +0100, Anna-Maria Behnsen a écrit :
>> Frederic Weisbecker <frederic@kernel.org> writes:
>> 
>> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> >> +static void tmigr_connect_child_parent(struct tmigr_group *child,
>> >> +				       struct tmigr_group *parent)
>> >> +{
>> >> +	union tmigr_state childstate;
>> >> +
>> >> +	raw_spin_lock_irq(&child->lock);
>> >> +	raw_spin_lock_nested(&parent->lock, SINGLE_DEPTH_NESTING);
>> >> +
>> >> +	child->parent = parent;
>> >> +	child->childmask = BIT(parent->num_children++);
>> >> +
>> >> +	raw_spin_unlock(&parent->lock);
>> >> +	raw_spin_unlock_irq(&child->lock);
>> >> +
>> >> +	/*
>> >> +	 * To prevent inconsistent states, active children need to be active in
>> >> +	 * the new parent as well. Inactive children are already marked inactive
>> >> +	 * in the parent group.
>> >> +	 */
>> >> +	childstate.state = atomic_read(&child->migr_state);
>> >> +	if (childstate.migrator != TMIGR_NONE) {
>> >
>> > Is it possible here to connect a running online child (not one that we just
>> > created) to a new parent?
>> 
>> connect_child_parent() is only executed for the just created ones. So,
>> yes in theory this would be possible, but it doesn't happen as
>> tmigr_setup_groups() takes care to make it right (hopefully :)). When a
>> LVL0 group has some space left, only the connection between tmc and the
>> LVL0 group is done in tmigr_setup_groups(). If there is no space left in
>> LVL0 group, then a new group is created and depending on the levels
>> which has to be created only executed for the new ones.
>> 
>> > If not, is it possible that a newly created child is
>> > not TMIGR_NONE?
>> 
>> Yes. See tmigr_cpu_online(). When new groups have to be created starting
>> from LVL0, then they are not active - so TMIGR_NONE is set. Activating
>> the new online CPU is done afterwards.
>> 
>> But if it is required to add also a new level at the top, then it is
>> mandatory to propagate the active state of the already existing child to
>> the new parent. The connect_child_parent() is then also executed for the
>> formerly top level group (child) to the newly created group (parent).
>
> Ah and this is why we have the "if (childstate.migrator != TMIGR_NONE)"
> branch, right?

yes - I see, comments would be helpful here :)

>> > Heh, I was about to say that it's impossible that timer_base_is_idle()
>> > at this stage but actually if we run in nohz_full...
>> >
>> > It happens so that nohz_full is deactivated until rcutree_online_cpu()
>> > which calls tick_dep_clear() but it's a pure coincidence that might
>> > disappear one day. So yes, let's keep it that way.
>> 
>> I instrumented the code (with NOHZ FULL and NOHZ_IDLE) to make sure the
>> timer migration hierarchy state 'idle' is in sync with the timer base
>> 'idle'. And this was one part where it was possible that it runs out of
>> sync as I remember correctly. But if I understood you correctly, this
>> shouldn't happen at the moment?
>
> Well, it's not supposed to :-)

Hmm, let me double check this and run the tests on the instrumented
version...

Thanks,

	Anna-Maria

Re: [PATCH v10 20/20] timers: Always queue timers on the local CPU

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:43PM +0100, Anna-Maria Behnsen a écrit :
>> The timer pull model is in place so we can remove the heuristics which try
>> to guess the best target CPU at enqueue/modification time.
>> 
>> All non pinned timers are queued on the local CPU in the separate storage
>> and eventually pulled at expiry time to a remote CPU.
>> 
>> Originally-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
>> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
>
> Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
>
> Just one detail below:
>
>> @@ -590,10 +590,13 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer)
>>  
>>  	/*
>>  	 * We might have to IPI the remote CPU if the base is idle and the
>> -	 * timer is not deferrable. If the other CPU is on the way to idle
>> -	 * then it can't set base->is_idle as we hold the base lock:
>> +	 * timer is pinned. If it is a non pinned timer, it is only queued
>> +	 * on the remote CPU, when timer was running during queueing. Then
>> +	 * everything is handled by remote CPU anyway. If the other CPU is
>> +	 * on the way to idle then it can't set base->is_idle as we hold
>> +	 * the base lock:
>>  	 */
>> -	if (base->is_idle)
>> +	if (base->is_idle && timer->flags & TIMER_PINNED)
>
> Is the TIMER_PINNED test necessary? If base->is_idle, then the timer
> is now guaranteed to be TIMER_PINNED, right?
>

Yes, you are right. Should I drop it? To clarify it, I could add a

  WARN_ON_ONCE(!timer->flags & TIMER_PINNED)

instead.

Thanks,

	Anna-Maria

Re: [PATCH v10 20/20] timers: Always queue timers on the local CPU

[Frederic Weisbecker]

On Thu, Feb 01, 2024 at 09:58:38PM +0100, Anna-Maria Behnsen wrote:
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Mon, Jan 15, 2024 at 03:37:43PM +0100, Anna-Maria Behnsen a écrit :
> >> The timer pull model is in place so we can remove the heuristics which try
> >> to guess the best target CPU at enqueue/modification time.
> >> 
> >> All non pinned timers are queued on the local CPU in the separate storage
> >> and eventually pulled at expiry time to a remote CPU.
> >> 
> >> Originally-by: Richard Cochran (linutronix GmbH) <richardcochran@gmail.com>
> >> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
> >
> > Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
> >
> > Just one detail below:
> >
> >> @@ -590,10 +590,13 @@ trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer)
> >>  
> >>  	/*
> >>  	 * We might have to IPI the remote CPU if the base is idle and the
> >> -	 * timer is not deferrable. If the other CPU is on the way to idle
> >> -	 * then it can't set base->is_idle as we hold the base lock:
> >> +	 * timer is pinned. If it is a non pinned timer, it is only queued
> >> +	 * on the remote CPU, when timer was running during queueing. Then
> >> +	 * everything is handled by remote CPU anyway. If the other CPU is
> >> +	 * on the way to idle then it can't set base->is_idle as we hold
> >> +	 * the base lock:
> >>  	 */
> >> -	if (base->is_idle)
> >> +	if (base->is_idle && timer->flags & TIMER_PINNED)
> >
> > Is the TIMER_PINNED test necessary? If base->is_idle, then the timer
> > is now guaranteed to be TIMER_PINNED, right?
> >
> 
> Yes, you are right. Should I drop it? To clarify it, I could add a
> 
>   WARN_ON_ONCE(!timer->flags & TIMER_PINNED)

Yep, that looks good!

Thanks.

> 
> instead.
> 
> Thanks,
> 
> 	Anna-Maria
> 

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> +/*
> + * Returns true, if there is nothing to be propagated to the next level
> + *
> + * @data->firstexp is set to expiry of first gobal event of the (top level of
> + * the) hierarchy, but only when hierarchy is completely idle.
> + *
> + * This is the only place where the group event expiry value is set.
> + */
> +static
> +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
> +			 struct tmigr_walk *data, union tmigr_state childstate,
> +			 union tmigr_state groupstate)
> +{
> +	struct tmigr_event *evt, *first_childevt;
> +	bool walk_done, remote = data->remote;
> +	bool leftmost_change = false;
> +	u64 nextexp;
> +
> +	if (child) {
> +		raw_spin_lock(&child->lock);
> +		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
> +
> +		if (childstate.active) {
> +			walk_done = true;
> +			goto unlock;
> +		}
> +
> +		first_childevt = tmigr_next_groupevt(child);
> +		nextexp = child->next_expiry;
> +		evt = &child->groupevt;
> +	} else {
> +		nextexp = data->nextexp;
> +
> +		first_childevt = evt = data->evt;
> +
> +		/*
> +		 * Walking the hierarchy is required in any case when a
> +		 * remote expiry was done before. This ensures to not lose
> +		 * already queued events in non active groups (see section
> +		 * "Required event and timerqueue update after a remote
> +		 * expiry" in the documentation at the top).
> +		 *
> +		 * The two call sites which are executed without a remote expiry
> +		 * before, are not prevented from propagating changes through
> +		 * the hierarchy by the return:
> +		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
> +		 *    is never set.
> +		 *  - tmigr_inactive_up() takes care of the propagation by
> +		 *    itself and ignores the return value. But an immediate
> +		 *    return is required because nothing has to be done in this
> +		 *    level as the event could be ignored.
> +		 */
> +		if (evt->ignore && !remote)
> +			return true;
> +
> +		raw_spin_lock(&group->lock);
> +	}
> +
> +	if (nextexp == KTIME_MAX) {
> +		evt->ignore = true;
> +
> +		/*
> +		 * When the next child event could be ignored (nextexp is
> +		 * KTIME_MAX) and there was no remote timer handling before or
> +		 * the group is already active, there is no need to walk the
> +		 * hierarchy even if there is a parent group.
> +		 *
> +		 * The other way round: even if the event could be ignored, but
> +		 * if a remote timer handling was executed before and the group
> +		 * is not active, walking the hierarchy is required to not miss
> +		 * an enqueued timer in the non active group. The enqueued timer
> +		 * of the group needs to be propagated to a higher level to
> +		 * ensure it is handled.
> +		 */
> +		if (!remote || groupstate.active) {
> +			walk_done = true;
> +			goto unlock;
> +		}
> +	} else {
> +		/*
> +		 * An update of @evt->cpu and @evt->ignore flag is required only
> +		 * when @child is set (the child is equal or higher than lvl0),
> +		 * but it doesn't matter if it is written once more to the per
> +		 * CPU event; make the update unconditional.
> +		 */
> +		evt->cpu = first_childevt->cpu;
> +		evt->ignore = false;
> +	}
> +
> +	walk_done = !group->parent;
> +
> +	/*
> +	 * If the child event is already queued in the group, remove it from the
> +	 * queue when the expiry time changed only.
> +	 */
> +	if (timerqueue_node_queued(&evt->nextevt)) {
> +		if (evt->nextevt.expires == nextexp)
> +			goto check_toplvl;
> +
> +		leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt;
> +		if (!timerqueue_del(&group->events, &evt->nextevt))
> +			WRITE_ONCE(group->next_expiry, KTIME_MAX);
> +	}
> +
> +	evt->nextevt.expires = nextexp;
> +
> +	if (timerqueue_add(&group->events, &evt->nextevt)) {
> +		leftmost_change = true;
> +		WRITE_ONCE(group->next_expiry, nextexp);
> +	}
> +
> +check_toplvl:
> +	if (walk_done && (groupstate.migrator == TMIGR_NONE)) {
> +		/*
> +		 * Nothing to do when first event didn't changed and update was
> +		 * done during remote timer handling.
> +		 */
> +		if (remote && !leftmost_change)

So if the first timer in the list hasn't changed, and that first timer belongs
to another CPU (and another group) than the tmc for which we are remotely
handling timers and re-propagating timers up, then data->firstexp will be
after the leftmost timer expiration (data->firstexp could even be KTIME_MAX
in the worst case), and so will be tmc->wakeup for the caller of
tmigr_handle_remote()?

Thanks.


> +			goto unlock;
> +		/*
> +		 * The top level group is idle and it has to be ensured the
> +		 * global timers are handled in time. (This could be optimized
> +		 * by keeping track of the last global scheduled event and only
> +		 * arming it on the CPU if the new event is earlier. Not sure if
> +		 * its worth the complexity.)
> +		 */
> +		data->firstexp = tmigr_next_groupevt_expires(group);
> +	}
> +
> +unlock:
> +	raw_spin_unlock(&group->lock);
> +
> +	if (child)
> +		raw_spin_unlock(&child->lock);
> +
> +	return walk_done;
> +}

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> +/*
> + * Returns true, if there is nothing to be propagated to the next level
> + *
> + * @data->firstexp is set to expiry of first gobal event of the (top level of
> + * the) hierarchy, but only when hierarchy is completely idle.
> + *
> + * This is the only place where the group event expiry value is set.
> + */
> +static
> +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
> +			 struct tmigr_walk *data, union tmigr_state childstate,
> +			 union tmigr_state groupstate)
> +{
> +	struct tmigr_event *evt, *first_childevt;
> +	bool walk_done, remote = data->remote;
> +	bool leftmost_change = false;
> +	u64 nextexp;
> +
> +	if (child) {
> +		raw_spin_lock(&child->lock);
> +		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
> +
> +		if (childstate.active) {
> +			walk_done = true;
> +			goto unlock;
> +		}
> +
> +		first_childevt = tmigr_next_groupevt(child);
> +		nextexp = child->next_expiry;
> +		evt = &child->groupevt;
> +	} else {
> +		nextexp = data->nextexp;
> +
> +		first_childevt = evt = data->evt;
> +
> +		/*
> +		 * Walking the hierarchy is required in any case when a
> +		 * remote expiry was done before. This ensures to not lose
> +		 * already queued events in non active groups (see section
> +		 * "Required event and timerqueue update after a remote
> +		 * expiry" in the documentation at the top).
> +		 *
> +		 * The two call sites which are executed without a remote expiry
> +		 * before, are not prevented from propagating changes through
> +		 * the hierarchy by the return:
> +		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
> +		 *    is never set.
> +		 *  - tmigr_inactive_up() takes care of the propagation by
> +		 *    itself and ignores the return value. But an immediate
> +		 *    return is required because nothing has to be done in this
> +		 *    level as the event could be ignored.
> +		 */
> +		if (evt->ignore && !remote)
> +			return true;
> +
> +		raw_spin_lock(&group->lock);
> +	}
> +
> +	if (nextexp == KTIME_MAX) {
> +		evt->ignore = true;
> +
> +		/*
> +		 * When the next child event could be ignored (nextexp is
> +		 * KTIME_MAX) and there was no remote timer handling before or
> +		 * the group is already active, there is no need to walk the
> +		 * hierarchy even if there is a parent group.
> +		 *
> +		 * The other way round: even if the event could be ignored, but
> +		 * if a remote timer handling was executed before and the group
> +		 * is not active, walking the hierarchy is required to not miss
> +		 * an enqueued timer in the non active group. The enqueued timer
> +		 * of the group needs to be propagated to a higher level to
> +		 * ensure it is handled.
> +		 */
> +		if (!remote || groupstate.active) {
> +			walk_done = true;
> +			goto unlock;

So if the current tmc going inactive was the migrator for the whole hierarchy
and it is reaching here the top-level, this assumes that if none of this tmc's
groups have a timer, then it can just return. But what if the top level has
timers from other children? Who is going to handle them then?

Should this be "goto check_toplvl" instead?

Thanks.

> +		}
> +	} else {
> +		/*
> +		 * An update of @evt->cpu and @evt->ignore flag is required only
> +		 * when @child is set (the child is equal or higher than lvl0),
> +		 * but it doesn't matter if it is written once more to the per
> +		 * CPU event; make the update unconditional.
> +		 */
> +		evt->cpu = first_childevt->cpu;
> +		evt->ignore = false;
> +	}
> +
> +	walk_done = !group->parent;
> +
> +	/*
> +	 * If the child event is already queued in the group, remove it from the
> +	 * queue when the expiry time changed only.
> +	 */
> +	if (timerqueue_node_queued(&evt->nextevt)) {
> +		if (evt->nextevt.expires == nextexp)
> +			goto check_toplvl;
> +
> +		leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt;
> +		if (!timerqueue_del(&group->events, &evt->nextevt))
> +			WRITE_ONCE(group->next_expiry, KTIME_MAX);
> +	}
> +
> +	evt->nextevt.expires = nextexp;
> +
> +	if (timerqueue_add(&group->events, &evt->nextevt)) {
> +		leftmost_change = true;
> +		WRITE_ONCE(group->next_expiry, nextexp);
> +	}
> +
> +check_toplvl:
> +	if (walk_done && (groupstate.migrator == TMIGR_NONE)) {
> +		/*
> +		 * Nothing to do when first event didn't changed and update was
> +		 * done during remote timer handling.
> +		 */
> +		if (remote && !leftmost_change)
> +			goto unlock;
> +		/*
> +		 * The top level group is idle and it has to be ensured the
> +		 * global timers are handled in time. (This could be optimized
> +		 * by keeping track of the last global scheduled event and only
> +		 * arming it on the CPU if the new event is earlier. Not sure if
> +		 * its worth the complexity.)
> +		 */
> +		data->firstexp = tmigr_next_groupevt_expires(group);
> +	}
> +
> +unlock:
> +	raw_spin_unlock(&group->lock);
> +
> +	if (child)
> +		raw_spin_unlock(&child->lock);
> +
> +	return walk_done;
> +}

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Anna-Maria Behnsen <anna-maria@linutronix.de> writes:

> Frederic Weisbecker <frederic@kernel.org> writes:
>
>> Le Thu, Feb 01, 2024 at 05:15:37PM +0100, Anna-Maria Behnsen a écrit :
>>> Frederic Weisbecker <frederic@kernel.org> writes:
>>> 
>>> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>>> > Heh, I was about to say that it's impossible that timer_base_is_idle()
>>> > at this stage but actually if we run in nohz_full...
>>> >
>>> > It happens so that nohz_full is deactivated until rcutree_online_cpu()
>>> > which calls tick_dep_clear() but it's a pure coincidence that might
>>> > disappear one day. So yes, let's keep it that way.
>>> 
>>> I instrumented the code (with NOHZ FULL and NOHZ_IDLE) to make sure the
>>> timer migration hierarchy state 'idle' is in sync with the timer base
>>> 'idle'. And this was one part where it was possible that it runs out of
>>> sync as I remember correctly. But if I understood you correctly, this
>>> shouldn't happen at the moment?
>>
>> Well, it's not supposed to :-)
>
> Hmm, let me double check this and run the tests on the instrumented
> version...

I added a prinkt() to verify what I think I remember. I was able to see
the prints. So it seems, that the coincidence that nohz_full is
deactivated until rcutree_online_cpu() already disappeared.

--- a/kernel/time/timer_migration.c
+++ b/kernel/time/timer_migration.c
@@ -1672,6 +1672,8 @@ static int tmigr_cpu_online(unsigned int
 	tmc->idle = timer_base_is_idle();
 	if (!tmc->idle)
 		__tmigr_cpu_activate(tmc);
+	else
+		printk("TIMER BASE IS IDLE\n");
 	tmc->online = true;
 	raw_spin_unlock_irq(&tmc->lock);
 	return 0;

Thanks,

        Anna-Maria

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> +static bool tmigr_handle_remote_up(struct tmigr_group *group,
>> +				   struct tmigr_group *child,
>> +				   void *ptr)
>> +{
>> +	struct tmigr_remote_data *data = ptr;
>> +	u64 now, next = KTIME_MAX;
>> +	struct tmigr_event *evt;
>> +	unsigned long jif;
>> +	u8 childmask;
>> +
>> +	jif = data->basej;
>> +	now = data->now;
>> +
>> +	childmask = data->childmask;
>> +
>> +again:
>> +	/*
>> +	 * Handle the group only if @childmask is the migrator or if the
>> +	 * group has no migrator. Otherwise the group is active and is
>> +	 * handled by its own migrator.
>> +	 */
>> +	if (!tmigr_check_migrator(group, childmask))
>> +		return true;
>> +
>> +	raw_spin_lock_irq(&group->lock);
>> +
>> +	evt = tmigr_next_expired_groupevt(group, now);
>> +
>> +	if (evt) {
>> +		unsigned int remote_cpu = evt->cpu;
>> +
>> +		raw_spin_unlock_irq(&group->lock);
>> +
>> +		next = tmigr_handle_remote_cpu(remote_cpu, now, jif);
>> +
>> +		/* check if there is another event, that needs to be handled */
>> +		goto again;
>> +	} else {
>> +		raw_spin_unlock_irq(&group->lock);
>> +	}
>> +
>> +	/*
>> +	 * Update of childmask for the next level and keep track of the expiry
>> +	 * of the first event that needs to be handled
>> +	 */
>> +	data->childmask = group->childmask;
>> +	data->firstexp = next;
>
> So assume we have:
>
>             [GRP1:0]
>         migrator = [GRP0:0]
>         active   = [GRP0:0]
>         nextevt  = TIMER3
>         /                    \
>     [GRP0:0]                  [GRP0:1]
>  migrator = CPU0           migrator = NONE
>  active   = CPU0           active   = NONE
>  nextevt  = KTIME_MAX      nextevt  = TIMER3
>     /         \                /         \
>    0           1              2           3
>   idle       idle           idle         idle (TIMER3)
>
> Then CPU 0 goes idle:
>
>             [GRP1:0]
>         migrator = NONE
>         active   = NONE
>         nextevt  = TIMER3
>         /                    \
>     [GRP0:0]                  [GRP0:1]
>  migrator = NONE           migrator = NONE
>  active   = NONE           active   = NONE
>  nextevt  = KTIME_MAX      nextevt  = TIMER3
>     /         \                /         \
>    0           1              2           3
>   idle       idle           idle         idle (TIMER3)
>
> CPU 0 is the idle migrator and its tmc->wakeup is TIMER3.
> But CPU 0 has a local timer that expires before TIMER3.
>
> When that timer interrupt fires, it raises the softirq, which
> executes on IRQ tail. So CPU0 eventually calls tmigr_handle_remote()
> before TIMER3 has expired.
>
> This leads to tmigr_next_expired_groupevt() to return NULL and then
> data->firstexp = KTIME_MAX and then tmc->wakeup = KTIME_MAX.
>
> Later on, tmigr_new_timer() is called with a KTIME_MAX global
> event and so tmc->wakeup stays with KTIME_MAX, ignoring TIMER3.
>
> It looks like you need to handle the tmigr_next_expired_groupevt()
> case returning NULL.

Yes. You are right. group->next_expiry is updated via
tmigr_next_expired_groupevt(). So I should take this value to rely on
for the firstevt.

But with this, we do not need the return value of
tmigr_handle_remote_cpu(). When the upperst level is inactive and there
is a timer (e.g. propagated there by tmigr_handle_remote_cpu()), the
return value of tmigr_handle_remote_cpu() is the next_expiry of the top
level group. But this value is also noticed by walking the hierarchy up
to the top level with tmigr_handle_remote_up().

Thanks,

	Anna-Maria

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Feb 05, 2024 at 04:59:45PM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> >> +static bool tmigr_handle_remote_up(struct tmigr_group *group,
> >> +				   struct tmigr_group *child,
> >> +				   void *ptr)
> >> +{
> >> +	struct tmigr_remote_data *data = ptr;
> >> +	u64 now, next = KTIME_MAX;
> >> +	struct tmigr_event *evt;
> >> +	unsigned long jif;
> >> +	u8 childmask;
> >> +
> >> +	jif = data->basej;
> >> +	now = data->now;
> >> +
> >> +	childmask = data->childmask;
> >> +
> >> +again:
> >> +	/*
> >> +	 * Handle the group only if @childmask is the migrator or if the
> >> +	 * group has no migrator. Otherwise the group is active and is
> >> +	 * handled by its own migrator.
> >> +	 */
> >> +	if (!tmigr_check_migrator(group, childmask))
> >> +		return true;
> >> +
> >> +	raw_spin_lock_irq(&group->lock);
> >> +
> >> +	evt = tmigr_next_expired_groupevt(group, now);
> >> +
> >> +	if (evt) {
> >> +		unsigned int remote_cpu = evt->cpu;
> >> +
> >> +		raw_spin_unlock_irq(&group->lock);
> >> +
> >> +		next = tmigr_handle_remote_cpu(remote_cpu, now, jif);
> >> +
> >> +		/* check if there is another event, that needs to be handled */
> >> +		goto again;
> >> +	} else {
> >> +		raw_spin_unlock_irq(&group->lock);
> >> +	}
> >> +
> >> +	/*
> >> +	 * Update of childmask for the next level and keep track of the expiry
> >> +	 * of the first event that needs to be handled
> >> +	 */
> >> +	data->childmask = group->childmask;
> >> +	data->firstexp = next;
> >
> > So assume we have:
> >
> >             [GRP1:0]
> >         migrator = [GRP0:0]
> >         active   = [GRP0:0]
> >         nextevt  = TIMER3
> >         /                    \
> >     [GRP0:0]                  [GRP0:1]
> >  migrator = CPU0           migrator = NONE
> >  active   = CPU0           active   = NONE
> >  nextevt  = KTIME_MAX      nextevt  = TIMER3
> >     /         \                /         \
> >    0           1              2           3
> >   idle       idle           idle         idle (TIMER3)
> >
> > Then CPU 0 goes idle:
> >
> >             [GRP1:0]
> >         migrator = NONE
> >         active   = NONE
> >         nextevt  = TIMER3
> >         /                    \
> >     [GRP0:0]                  [GRP0:1]
> >  migrator = NONE           migrator = NONE
> >  active   = NONE           active   = NONE
> >  nextevt  = KTIME_MAX      nextevt  = TIMER3
> >     /         \                /         \
> >    0           1              2           3
> >   idle       idle           idle         idle (TIMER3)
> >
> > CPU 0 is the idle migrator and its tmc->wakeup is TIMER3.
> > But CPU 0 has a local timer that expires before TIMER3.
> >
> > When that timer interrupt fires, it raises the softirq, which
> > executes on IRQ tail. So CPU0 eventually calls tmigr_handle_remote()
> > before TIMER3 has expired.
> >
> > This leads to tmigr_next_expired_groupevt() to return NULL and then
> > data->firstexp = KTIME_MAX and then tmc->wakeup = KTIME_MAX.
> >
> > Later on, tmigr_new_timer() is called with a KTIME_MAX global
> > event and so tmc->wakeup stays with KTIME_MAX, ignoring TIMER3.
> >
> > It looks like you need to handle the tmigr_next_expired_groupevt()
> > case returning NULL.
> 
> Yes. You are right. group->next_expiry is updated via
> tmigr_next_expired_groupevt(). So I should take this value to rely on
> for the firstevt.
> 
> But with this, we do not need the return value of
> tmigr_handle_remote_cpu(). When the upperst level is inactive and there
> is a timer (e.g. propagated there by tmigr_handle_remote_cpu()), the
> return value of tmigr_handle_remote_cpu() is the next_expiry of the top
> level group. But this value is also noticed by walking the hierarchy up
> to the top level with tmigr_handle_remote_up().

Indeed that's a nice simplification!

This also cure an issue with KTIME_MAX overwriting data->firstexp in
tmigr_handle_remote_up() while walking upper level after an expiration
on below level.

Thanks.

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Mon, Feb 05, 2024 at 02:29:34PM +0100, Anna-Maria Behnsen a écrit :
> Anna-Maria Behnsen <anna-maria@linutronix.de> writes:
> 
> > Frederic Weisbecker <frederic@kernel.org> writes:
> >
> >> Le Thu, Feb 01, 2024 at 05:15:37PM +0100, Anna-Maria Behnsen a écrit :
> >>> Frederic Weisbecker <frederic@kernel.org> writes:
> >>> 
> >>> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> >>> > Heh, I was about to say that it's impossible that timer_base_is_idle()
> >>> > at this stage but actually if we run in nohz_full...
> >>> >
> >>> > It happens so that nohz_full is deactivated until rcutree_online_cpu()
> >>> > which calls tick_dep_clear() but it's a pure coincidence that might
> >>> > disappear one day. So yes, let's keep it that way.
> >>> 
> >>> I instrumented the code (with NOHZ FULL and NOHZ_IDLE) to make sure the
> >>> timer migration hierarchy state 'idle' is in sync with the timer base
> >>> 'idle'. And this was one part where it was possible that it runs out of
> >>> sync as I remember correctly. But if I understood you correctly, this
> >>> shouldn't happen at the moment?
> >>
> >> Well, it's not supposed to :-)
> >
> > Hmm, let me double check this and run the tests on the instrumented
> > version...
> 
> I added a prinkt() to verify what I think I remember. I was able to see
> the prints. So it seems, that the coincidence that nohz_full is
> deactivated until rcutree_online_cpu() already disappeared.

Nice, then I guess it can become a WARN_ON.

Thanks.

> 
> --- a/kernel/time/timer_migration.c
> +++ b/kernel/time/timer_migration.c
> @@ -1672,6 +1672,8 @@ static int tmigr_cpu_online(unsigned int
>  	tmc->idle = timer_base_is_idle();
>  	if (!tmc->idle)
>  		__tmigr_cpu_activate(tmc);
> +	else
> +		printk("TIMER BASE IS IDLE\n");
>  	tmc->online = true;
>  	raw_spin_unlock_irq(&tmc->lock);
>  	return 0;
> 
> Thanks,
> 
>         Anna-Maria
> 

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Feb 05, 2024 at 02:29:34PM +0100, Anna-Maria Behnsen a écrit :
>> Anna-Maria Behnsen <anna-maria@linutronix.de> writes:
>> 
>> > Frederic Weisbecker <frederic@kernel.org> writes:
>> >
>> >> Le Thu, Feb 01, 2024 at 05:15:37PM +0100, Anna-Maria Behnsen a écrit :
>> >>> Frederic Weisbecker <frederic@kernel.org> writes:
>> >>> 
>> >>> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> >>> > Heh, I was about to say that it's impossible that timer_base_is_idle()
>> >>> > at this stage but actually if we run in nohz_full...
>> >>> >
>> >>> > It happens so that nohz_full is deactivated until rcutree_online_cpu()
>> >>> > which calls tick_dep_clear() but it's a pure coincidence that might
>> >>> > disappear one day. So yes, let's keep it that way.
>> >>> 
>> >>> I instrumented the code (with NOHZ FULL and NOHZ_IDLE) to make sure the
>> >>> timer migration hierarchy state 'idle' is in sync with the timer base
>> >>> 'idle'. And this was one part where it was possible that it runs out of
>> >>> sync as I remember correctly. But if I understood you correctly, this
>> >>> shouldn't happen at the moment?
>> >>
>> >> Well, it's not supposed to :-)
>> >
>> > Hmm, let me double check this and run the tests on the instrumented
>> > version...
>> 
>> I added a prinkt() to verify what I think I remember. I was able to see
>> the prints. So it seems, that the coincidence that nohz_full is
>> deactivated until rcutree_online_cpu() already disappeared.
>
> Nice, then I guess it can become a WARN_ON.

Either I misunderstood something, or wasn't able to explain what I
wanted to say.

I understood, that nohz full is disabled (by coincidence) until
rcutree_online_cpu() which comes after the timer migration CPU hotplug
AP. This means, that the check whether timer base is idle or not,
shouldn't be required in tmigr_cpu_online() to keep cpu idle or mark it
active in the hierarchy. But we could keep it in case coincidence
disappears. No?

So I added a printk() when timer base is idle in tmigr_cpu_online(). And
I was able to see the prints. This means, nohz full is _not_ disabled
when executing tmigr_cpu_online(), or am I wrong?

So when I replace the printk() with a WARN_ON() it will definitely
trigger. So I'm not sure if this is what you want to have :)

Thanks,

        Anna-Maria

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

On Tue, Feb 06, 2024 at 11:06:19AM +0100, Anna-Maria Behnsen wrote:
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Mon, Feb 05, 2024 at 02:29:34PM +0100, Anna-Maria Behnsen a écrit :
> >> Anna-Maria Behnsen <anna-maria@linutronix.de> writes:
> >> 
> >> > Frederic Weisbecker <frederic@kernel.org> writes:
> >> >
> >> >> Le Thu, Feb 01, 2024 at 05:15:37PM +0100, Anna-Maria Behnsen a écrit :
> >> >>> Frederic Weisbecker <frederic@kernel.org> writes:
> >> >>> 
> >> >>> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> >> >>> > Heh, I was about to say that it's impossible that timer_base_is_idle()
> >> >>> > at this stage but actually if we run in nohz_full...
> >> >>> >
> >> >>> > It happens so that nohz_full is deactivated until rcutree_online_cpu()
> >> >>> > which calls tick_dep_clear() but it's a pure coincidence that might
> >> >>> > disappear one day. So yes, let's keep it that way.
> >> >>> 
> >> >>> I instrumented the code (with NOHZ FULL and NOHZ_IDLE) to make sure the
> >> >>> timer migration hierarchy state 'idle' is in sync with the timer base
> >> >>> 'idle'. And this was one part where it was possible that it runs out of
> >> >>> sync as I remember correctly. But if I understood you correctly, this
> >> >>> shouldn't happen at the moment?
> >> >>
> >> >> Well, it's not supposed to :-)
> >> >
> >> > Hmm, let me double check this and run the tests on the instrumented
> >> > version...
> >> 
> >> I added a prinkt() to verify what I think I remember. I was able to see
> >> the prints. So it seems, that the coincidence that nohz_full is
> >> deactivated until rcutree_online_cpu() already disappeared.
> >
> > Nice, then I guess it can become a WARN_ON.
> 
> Either I misunderstood something, or wasn't able to explain what I
> wanted to say.
> 
> I understood, that nohz full is disabled (by coincidence) until
> rcutree_online_cpu() which comes after the timer migration CPU hotplug
> AP. This means, that the check whether timer base is idle or not,
> shouldn't be required in tmigr_cpu_online() to keep cpu idle or mark it
> active in the hierarchy. But we could keep it in case coincidence
> disappears. No?
> 
> So I added a printk() when timer base is idle in tmigr_cpu_online(). And
> I was able to see the prints. This means, nohz full is _not_ disabled
> when executing tmigr_cpu_online(), or am I wrong?
> 
> So when I replace the printk() with a WARN_ON() it will definitely
> trigger. So I'm not sure if this is what you want to have :)

Yes, silly me, I thought the tick dependency was set on all hotplug
operations but it's only cpu down. So this piece doesn't need to change
AFAICT.

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> +/*
>> + * Returns true, if there is nothing to be propagated to the next level
>> + *
>> + * @data->firstexp is set to expiry of first gobal event of the (top level of
>> + * the) hierarchy, but only when hierarchy is completely idle.
>> + *
>> + * This is the only place where the group event expiry value is set.
>> + */
>> +static
>> +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
>> +			 struct tmigr_walk *data, union tmigr_state childstate,
>> +			 union tmigr_state groupstate)
>> +{
>> +	struct tmigr_event *evt, *first_childevt;
>> +	bool walk_done, remote = data->remote;
>> +	bool leftmost_change = false;
>> +	u64 nextexp;
>> +
>> +	if (child) {
>> +		raw_spin_lock(&child->lock);
>> +		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
>> +
>> +		if (childstate.active) {
>> +			walk_done = true;
>> +			goto unlock;
>> +		}
>> +
>> +		first_childevt = tmigr_next_groupevt(child);
>> +		nextexp = child->next_expiry;
>> +		evt = &child->groupevt;
>> +	} else {
>> +		nextexp = data->nextexp;
>> +
>> +		first_childevt = evt = data->evt;
>> +
>> +		/*
>> +		 * Walking the hierarchy is required in any case when a
>> +		 * remote expiry was done before. This ensures to not lose
>> +		 * already queued events in non active groups (see section
>> +		 * "Required event and timerqueue update after a remote
>> +		 * expiry" in the documentation at the top).
>> +		 *
>> +		 * The two call sites which are executed without a remote expiry
>> +		 * before, are not prevented from propagating changes through
>> +		 * the hierarchy by the return:
>> +		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
>> +		 *    is never set.
>> +		 *  - tmigr_inactive_up() takes care of the propagation by
>> +		 *    itself and ignores the return value. But an immediate
>> +		 *    return is required because nothing has to be done in this
>> +		 *    level as the event could be ignored.
>> +		 */
>> +		if (evt->ignore && !remote)
>> +			return true;
>> +
>> +		raw_spin_lock(&group->lock);
>> +	}
>> +
>> +	if (nextexp == KTIME_MAX) {
>> +		evt->ignore = true;
>> +
>> +		/*
>> +		 * When the next child event could be ignored (nextexp is
>> +		 * KTIME_MAX) and there was no remote timer handling before or
>> +		 * the group is already active, there is no need to walk the
>> +		 * hierarchy even if there is a parent group.
>> +		 *
>> +		 * The other way round: even if the event could be ignored, but
>> +		 * if a remote timer handling was executed before and the group
>> +		 * is not active, walking the hierarchy is required to not miss
>> +		 * an enqueued timer in the non active group. The enqueued timer
>> +		 * of the group needs to be propagated to a higher level to
>> +		 * ensure it is handled.
>> +		 */
>> +		if (!remote || groupstate.active) {
>> +			walk_done = true;
>> +			goto unlock;
>> +		}
>> +	} else {
>> +		/*
>> +		 * An update of @evt->cpu and @evt->ignore flag is required only
>> +		 * when @child is set (the child is equal or higher than lvl0),
>> +		 * but it doesn't matter if it is written once more to the per
>> +		 * CPU event; make the update unconditional.
>> +		 */
>> +		evt->cpu = first_childevt->cpu;
>> +		evt->ignore = false;
>> +	}
>> +
>> +	walk_done = !group->parent;
>> +
>> +	/*
>> +	 * If the child event is already queued in the group, remove it from the
>> +	 * queue when the expiry time changed only.
>> +	 */
>> +	if (timerqueue_node_queued(&evt->nextevt)) {
>> +		if (evt->nextevt.expires == nextexp)
>> +			goto check_toplvl;
>> +
>> +		leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt;
>> +		if (!timerqueue_del(&group->events, &evt->nextevt))
>> +			WRITE_ONCE(group->next_expiry, KTIME_MAX);
>> +	}
>> +
>> +	evt->nextevt.expires = nextexp;
>> +
>> +	if (timerqueue_add(&group->events, &evt->nextevt)) {
>> +		leftmost_change = true;
>> +		WRITE_ONCE(group->next_expiry, nextexp);
>> +	}
>> +
>> +check_toplvl:
>> +	if (walk_done && (groupstate.migrator == TMIGR_NONE)) {
>> +		/*
>> +		 * Nothing to do when first event didn't changed and update was
>> +		 * done during remote timer handling.
>> +		 */
>> +		if (remote && !leftmost_change)
>
> So if the first timer in the list hasn't changed, and that first timer belongs
> to another CPU (and another group) than the tmc for which we are remotely
> handling timers and re-propagating timers up, then data->firstexp will be
> after the leftmost timer expiration (data->firstexp could even be KTIME_MAX
> in the worst case), and so will be tmc->wakeup for the caller of
> tmigr_handle_remote()?
>

This is related to the discussion regarding tmigr_handle_remote_up(). So
this should be also covered by the change I proposed there.

And then we definitely do not need the update of data->firstevt here, as
we are still on the way to top to make sure all events are handled. And
the first event which needs to be handled by the migrator CPU is set by
the call to tmigr_next_expired_groupevt().

>
>> +			goto unlock;
>> +		/*
>> +		 * The top level group is idle and it has to be ensured the
>> +		 * global timers are handled in time. (This could be optimized
>> +		 * by keeping track of the last global scheduled event and only
>> +		 * arming it on the CPU if the new event is earlier. Not sure if
>> +		 * its worth the complexity.)
>> +		 */
>> +		data->firstexp = tmigr_next_groupevt_expires(group);
>> +	}
>> +
>> +unlock:
>> +	raw_spin_unlock(&group->lock);
>> +
>> +	if (child)
>> +		raw_spin_unlock(&child->lock);
>> +
>> +	return walk_done;
>> +}

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Tue, Feb 06, 2024 at 12:03:32PM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> >> +/*
> >> + * Returns true, if there is nothing to be propagated to the next level
> >> + *
> >> + * @data->firstexp is set to expiry of first gobal event of the (top level of
> >> + * the) hierarchy, but only when hierarchy is completely idle.
> >> + *
> >> + * This is the only place where the group event expiry value is set.
> >> + */
> >> +static
> >> +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
> >> +			 struct tmigr_walk *data, union tmigr_state childstate,
> >> +			 union tmigr_state groupstate)
> >> +{
> >> +	struct tmigr_event *evt, *first_childevt;
> >> +	bool walk_done, remote = data->remote;
> >> +	bool leftmost_change = false;
> >> +	u64 nextexp;
> >> +
> >> +	if (child) {
> >> +		raw_spin_lock(&child->lock);
> >> +		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
> >> +
> >> +		if (childstate.active) {
> >> +			walk_done = true;
> >> +			goto unlock;
> >> +		}
> >> +
> >> +		first_childevt = tmigr_next_groupevt(child);
> >> +		nextexp = child->next_expiry;
> >> +		evt = &child->groupevt;
> >> +	} else {
> >> +		nextexp = data->nextexp;
> >> +
> >> +		first_childevt = evt = data->evt;
> >> +
> >> +		/*
> >> +		 * Walking the hierarchy is required in any case when a
> >> +		 * remote expiry was done before. This ensures to not lose
> >> +		 * already queued events in non active groups (see section
> >> +		 * "Required event and timerqueue update after a remote
> >> +		 * expiry" in the documentation at the top).
> >> +		 *
> >> +		 * The two call sites which are executed without a remote expiry
> >> +		 * before, are not prevented from propagating changes through
> >> +		 * the hierarchy by the return:
> >> +		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
> >> +		 *    is never set.
> >> +		 *  - tmigr_inactive_up() takes care of the propagation by
> >> +		 *    itself and ignores the return value. But an immediate
> >> +		 *    return is required because nothing has to be done in this
> >> +		 *    level as the event could be ignored.
> >> +		 */
> >> +		if (evt->ignore && !remote)
> >> +			return true;
> >> +
> >> +		raw_spin_lock(&group->lock);
> >> +	}
> >> +
> >> +	if (nextexp == KTIME_MAX) {
> >> +		evt->ignore = true;
> >> +
> >> +		/*
> >> +		 * When the next child event could be ignored (nextexp is
> >> +		 * KTIME_MAX) and there was no remote timer handling before or
> >> +		 * the group is already active, there is no need to walk the
> >> +		 * hierarchy even if there is a parent group.
> >> +		 *
> >> +		 * The other way round: even if the event could be ignored, but
> >> +		 * if a remote timer handling was executed before and the group
> >> +		 * is not active, walking the hierarchy is required to not miss
> >> +		 * an enqueued timer in the non active group. The enqueued timer
> >> +		 * of the group needs to be propagated to a higher level to
> >> +		 * ensure it is handled.
> >> +		 */
> >> +		if (!remote || groupstate.active) {
> >> +			walk_done = true;
> >> +			goto unlock;
> >> +		}
> >> +	} else {
> >> +		/*
> >> +		 * An update of @evt->cpu and @evt->ignore flag is required only
> >> +		 * when @child is set (the child is equal or higher than lvl0),
> >> +		 * but it doesn't matter if it is written once more to the per
> >> +		 * CPU event; make the update unconditional.
> >> +		 */
> >> +		evt->cpu = first_childevt->cpu;
> >> +		evt->ignore = false;
> >> +	}
> >> +
> >> +	walk_done = !group->parent;
> >> +
> >> +	/*
> >> +	 * If the child event is already queued in the group, remove it from the
> >> +	 * queue when the expiry time changed only.
> >> +	 */
> >> +	if (timerqueue_node_queued(&evt->nextevt)) {
> >> +		if (evt->nextevt.expires == nextexp)
> >> +			goto check_toplvl;
> >> +
> >> +		leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt;
> >> +		if (!timerqueue_del(&group->events, &evt->nextevt))
> >> +			WRITE_ONCE(group->next_expiry, KTIME_MAX);
> >> +	}
> >> +
> >> +	evt->nextevt.expires = nextexp;
> >> +
> >> +	if (timerqueue_add(&group->events, &evt->nextevt)) {
> >> +		leftmost_change = true;
> >> +		WRITE_ONCE(group->next_expiry, nextexp);
> >> +	}
> >> +
> >> +check_toplvl:
> >> +	if (walk_done && (groupstate.migrator == TMIGR_NONE)) {
> >> +		/*
> >> +		 * Nothing to do when first event didn't changed and update was
> >> +		 * done during remote timer handling.
> >> +		 */
> >> +		if (remote && !leftmost_change)
> >
> > So if the first timer in the list hasn't changed, and that first timer belongs
> > to another CPU (and another group) than the tmc for which we are remotely
> > handling timers and re-propagating timers up, then data->firstexp will be
> > after the leftmost timer expiration (data->firstexp could even be KTIME_MAX
> > in the worst case), and so will be tmc->wakeup for the caller of
> > tmigr_handle_remote()?
> >
> 
> This is related to the discussion regarding tmigr_handle_remote_up(). So
> this should be also covered by the change I proposed there.
> 
> And then we definitely do not need the update of data->firstevt here, as
> we are still on the way to top to make sure all events are handled. And
> the first event which needs to be handled by the migrator CPU is set by
> the call to tmigr_next_expired_groupevt().

Sounds good!

Thanks.

> 
> >
> >> +			goto unlock;
> >> +		/*
> >> +		 * The top level group is idle and it has to be ensured the
> >> +		 * global timers are handled in time. (This could be optimized
> >> +		 * by keeping track of the last global scheduled event and only
> >> +		 * arming it on the CPU if the new event is earlier. Not sure if
> >> +		 * its worth the complexity.)
> >> +		 */
> >> +		data->firstexp = tmigr_next_groupevt_expires(group);
> >> +	}
> >> +
> >> +unlock:
> >> +	raw_spin_unlock(&group->lock);
> >> +
> >> +	if (child)
> >> +		raw_spin_unlock(&child->lock);
> >> +
> >> +	return walk_done;
> >> +}

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> +/*
>> + * Returns true, if there is nothing to be propagated to the next level
>> + *
>> + * @data->firstexp is set to expiry of first gobal event of the (top level of
>> + * the) hierarchy, but only when hierarchy is completely idle.
>> + *
>> + * This is the only place where the group event expiry value is set.
>> + */
>> +static
>> +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
>> +			 struct tmigr_walk *data, union tmigr_state childstate,
>> +			 union tmigr_state groupstate)
>> +{
>> +	struct tmigr_event *evt, *first_childevt;
>> +	bool walk_done, remote = data->remote;
>> +	bool leftmost_change = false;
>> +	u64 nextexp;
>> +
>> +	if (child) {
>> +		raw_spin_lock(&child->lock);
>> +		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
>> +
>> +		if (childstate.active) {
>> +			walk_done = true;
>> +			goto unlock;
>> +		}
>> +
>> +		first_childevt = tmigr_next_groupevt(child);
>> +		nextexp = child->next_expiry;
>> +		evt = &child->groupevt;
>> +	} else {
>> +		nextexp = data->nextexp;
>> +
>> +		first_childevt = evt = data->evt;
>> +
>> +		/*
>> +		 * Walking the hierarchy is required in any case when a
>> +		 * remote expiry was done before. This ensures to not lose
>> +		 * already queued events in non active groups (see section
>> +		 * "Required event and timerqueue update after a remote
>> +		 * expiry" in the documentation at the top).
>> +		 *
>> +		 * The two call sites which are executed without a remote expiry
>> +		 * before, are not prevented from propagating changes through
>> +		 * the hierarchy by the return:
>> +		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
>> +		 *    is never set.
>> +		 *  - tmigr_inactive_up() takes care of the propagation by
>> +		 *    itself and ignores the return value. But an immediate
>> +		 *    return is required because nothing has to be done in this
>> +		 *    level as the event could be ignored.
>> +		 */
>> +		if (evt->ignore && !remote)
>> +			return true;
>> +
>> +		raw_spin_lock(&group->lock);
>> +	}
>> +
>> +	if (nextexp == KTIME_MAX) {
>> +		evt->ignore = true;
>> +
>> +		/*
>> +		 * When the next child event could be ignored (nextexp is
>> +		 * KTIME_MAX) and there was no remote timer handling before or
>> +		 * the group is already active, there is no need to walk the
>> +		 * hierarchy even if there is a parent group.
>> +		 *
>> +		 * The other way round: even if the event could be ignored, but
>> +		 * if a remote timer handling was executed before and the group
>> +		 * is not active, walking the hierarchy is required to not miss
>> +		 * an enqueued timer in the non active group. The enqueued timer
>> +		 * of the group needs to be propagated to a higher level to
>> +		 * ensure it is handled.
>> +		 */
>> +		if (!remote || groupstate.active) {
>> +			walk_done = true;
>> +			goto unlock;
>
> So if the current tmc going inactive was the migrator for the whole hierarchy
> and it is reaching here the top-level, this assumes that if none of this tmc's
> groups have a timer, then it can just return. But what if the top level has
> timers from other children? Who is going to handle them then?
>
> Should this be "goto check_toplvl" instead?
>

Simply replacing this goto will not work. Then we chould end up with a
'data->firstexp' set even if we do not want to have it (when remote is
not set).

There is another issue in here. When the event could be ignored and it
is propagated because of e.g. remote timer handling, then the timerqueue
dance is done nevertheless. It's not a big problem (as the ignore flag
is set and event is removed of queue when revisting the timer queue),
but its obviously more work than it is required to have.

Thanks

>> +		}
>> +	} else {
>> +		/*
>> +		 * An update of @evt->cpu and @evt->ignore flag is required only
>> +		 * when @child is set (the child is equal or higher than lvl0),
>> +		 * but it doesn't matter if it is written once more to the per
>> +		 * CPU event; make the update unconditional.
>> +		 */
>> +		evt->cpu = first_childevt->cpu;
>> +		evt->ignore = false;
>> +	}
>> +
>> +	walk_done = !group->parent;
>> +
>> +	/*
>> +	 * If the child event is already queued in the group, remove it from the
>> +	 * queue when the expiry time changed only.
>> +	 */
>> +	if (timerqueue_node_queued(&evt->nextevt)) {
>> +		if (evt->nextevt.expires == nextexp)
>> +			goto check_toplvl;
>> +
>> +		leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt;
>> +		if (!timerqueue_del(&group->events, &evt->nextevt))
>> +			WRITE_ONCE(group->next_expiry, KTIME_MAX);
>> +	}
>> +
>> +	evt->nextevt.expires = nextexp;
>> +
>> +	if (timerqueue_add(&group->events, &evt->nextevt)) {
>> +		leftmost_change = true;
>> +		WRITE_ONCE(group->next_expiry, nextexp);
>> +	}
>> +
>> +check_toplvl:
>> +	if (walk_done && (groupstate.migrator == TMIGR_NONE)) {
>> +		/*
>> +		 * Nothing to do when first event didn't changed and update was
>> +		 * done during remote timer handling.
>> +		 */
>> +		if (remote && !leftmost_change)
>> +			goto unlock;
>> +		/*
>> +		 * The top level group is idle and it has to be ensured the
>> +		 * global timers are handled in time. (This could be optimized
>> +		 * by keeping track of the last global scheduled event and only
>> +		 * arming it on the CPU if the new event is earlier. Not sure if
>> +		 * its worth the complexity.)
>> +		 */
>> +		data->firstexp = tmigr_next_groupevt_expires(group);
>> +	}
>> +
>> +unlock:
>> +	raw_spin_unlock(&group->lock);
>> +
>> +	if (child)
>> +		raw_spin_unlock(&child->lock);
>> +
>> +	return walk_done;
>> +}

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Tue, Feb 06, 2024 at 12:36:55PM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
> >> +/*
> >> + * Returns true, if there is nothing to be propagated to the next level
> >> + *
> >> + * @data->firstexp is set to expiry of first gobal event of the (top level of
> >> + * the) hierarchy, but only when hierarchy is completely idle.
> >> + *
> >> + * This is the only place where the group event expiry value is set.
> >> + */
> >> +static
> >> +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
> >> +			 struct tmigr_walk *data, union tmigr_state childstate,
> >> +			 union tmigr_state groupstate)
> >> +{
> >> +	struct tmigr_event *evt, *first_childevt;
> >> +	bool walk_done, remote = data->remote;
> >> +	bool leftmost_change = false;
> >> +	u64 nextexp;
> >> +
> >> +	if (child) {
> >> +		raw_spin_lock(&child->lock);
> >> +		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
> >> +
> >> +		if (childstate.active) {
> >> +			walk_done = true;
> >> +			goto unlock;
> >> +		}
> >> +
> >> +		first_childevt = tmigr_next_groupevt(child);
> >> +		nextexp = child->next_expiry;
> >> +		evt = &child->groupevt;
> >> +	} else {
> >> +		nextexp = data->nextexp;
> >> +
> >> +		first_childevt = evt = data->evt;
> >> +
> >> +		/*
> >> +		 * Walking the hierarchy is required in any case when a
> >> +		 * remote expiry was done before. This ensures to not lose
> >> +		 * already queued events in non active groups (see section
> >> +		 * "Required event and timerqueue update after a remote
> >> +		 * expiry" in the documentation at the top).
> >> +		 *
> >> +		 * The two call sites which are executed without a remote expiry
> >> +		 * before, are not prevented from propagating changes through
> >> +		 * the hierarchy by the return:
> >> +		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
> >> +		 *    is never set.
> >> +		 *  - tmigr_inactive_up() takes care of the propagation by
> >> +		 *    itself and ignores the return value. But an immediate
> >> +		 *    return is required because nothing has to be done in this
> >> +		 *    level as the event could be ignored.
> >> +		 */
> >> +		if (evt->ignore && !remote)
> >> +			return true;
> >> +
> >> +		raw_spin_lock(&group->lock);
> >> +	}
> >> +
> >> +	if (nextexp == KTIME_MAX) {
> >> +		evt->ignore = true;
> >> +
> >> +		/*
> >> +		 * When the next child event could be ignored (nextexp is
> >> +		 * KTIME_MAX) and there was no remote timer handling before or
> >> +		 * the group is already active, there is no need to walk the
> >> +		 * hierarchy even if there is a parent group.
> >> +		 *
> >> +		 * The other way round: even if the event could be ignored, but
> >> +		 * if a remote timer handling was executed before and the group
> >> +		 * is not active, walking the hierarchy is required to not miss
> >> +		 * an enqueued timer in the non active group. The enqueued timer
> >> +		 * of the group needs to be propagated to a higher level to
> >> +		 * ensure it is handled.
> >> +		 */
> >> +		if (!remote || groupstate.active) {
> >> +			walk_done = true;
> >> +			goto unlock;
> >
> > So if the current tmc going inactive was the migrator for the whole hierarchy
> > and it is reaching here the top-level, this assumes that if none of this tmc's
> > groups have a timer, then it can just return. But what if the top level has
> > timers from other children? Who is going to handle them then?
> >
> > Should this be "goto check_toplvl" instead?
> >
> 
> Simply replacing this goto will not work. Then we chould end up with a
> 'data->firstexp' set even if we do not want to have it (when remote is
> not set).

Hmm, but the (groupstate.migrator == TMIGR_NONE) condition should prevent that,
no?

> 
> There is another issue in here. When the event could be ignored and it
> is propagated because of e.g. remote timer handling, then the timerqueue
> dance is done nevertheless. It's not a big problem (as the ignore flag
> is set and event is removed of queue when revisting the timer queue),
> but its obviously more work than it is required to have.

Right. I guess it doesn't hurt to delete it from the timerqueue if present and
then update group->next_expiry accordingly. But it's certainly not useful
to requeue it :-)

Thanks.

> 
> Thanks
> 
> >> +		}
> >> +	} else {
> >> +		/*
> >> +		 * An update of @evt->cpu and @evt->ignore flag is required only
> >> +		 * when @child is set (the child is equal or higher than lvl0),
> >> +		 * but it doesn't matter if it is written once more to the per
> >> +		 * CPU event; make the update unconditional.
> >> +		 */
> >> +		evt->cpu = first_childevt->cpu;
> >> +		evt->ignore = false;
> >> +	}
> >> +
> >> +	walk_done = !group->parent;
> >> +
> >> +	/*
> >> +	 * If the child event is already queued in the group, remove it from the
> >> +	 * queue when the expiry time changed only.
> >> +	 */
> >> +	if (timerqueue_node_queued(&evt->nextevt)) {
> >> +		if (evt->nextevt.expires == nextexp)
> >> +			goto check_toplvl;
> >> +
> >> +		leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt;
> >> +		if (!timerqueue_del(&group->events, &evt->nextevt))
> >> +			WRITE_ONCE(group->next_expiry, KTIME_MAX);
> >> +	}
> >> +
> >> +	evt->nextevt.expires = nextexp;
> >> +
> >> +	if (timerqueue_add(&group->events, &evt->nextevt)) {
> >> +		leftmost_change = true;
> >> +		WRITE_ONCE(group->next_expiry, nextexp);
> >> +	}
> >> +
> >> +check_toplvl:
> >> +	if (walk_done && (groupstate.migrator == TMIGR_NONE)) {
> >> +		/*
> >> +		 * Nothing to do when first event didn't changed and update was
> >> +		 * done during remote timer handling.
> >> +		 */
> >> +		if (remote && !leftmost_change)
> >> +			goto unlock;
> >> +		/*
> >> +		 * The top level group is idle and it has to be ensured the
> >> +		 * global timers are handled in time. (This could be optimized
> >> +		 * by keeping track of the last global scheduled event and only
> >> +		 * arming it on the CPU if the new event is earlier. Not sure if
> >> +		 * its worth the complexity.)
> >> +		 */
> >> +		data->firstexp = tmigr_next_groupevt_expires(group);
> >> +	}
> >> +
> >> +unlock:
> >> +	raw_spin_unlock(&group->lock);
> >> +
> >> +	if (child)
> >> +		raw_spin_unlock(&child->lock);
> >> +
> >> +	return walk_done;
> >> +}

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Tue, Feb 06, 2024 at 12:36:55PM +0100, Anna-Maria Behnsen a écrit :
>> Frederic Weisbecker <frederic@kernel.org> writes:
>> 
>> > Le Mon, Jan 15, 2024 at 03:37:41PM +0100, Anna-Maria Behnsen a écrit :
>> >> +/*
>> >> + * Returns true, if there is nothing to be propagated to the next level
>> >> + *
>> >> + * @data->firstexp is set to expiry of first gobal event of the (top level of
>> >> + * the) hierarchy, but only when hierarchy is completely idle.
>> >> + *
>> >> + * This is the only place where the group event expiry value is set.
>> >> + */
>> >> +static
>> >> +bool tmigr_update_events(struct tmigr_group *group, struct tmigr_group *child,
>> >> +			 struct tmigr_walk *data, union tmigr_state childstate,
>> >> +			 union tmigr_state groupstate)
>> >> +{
>> >> +	struct tmigr_event *evt, *first_childevt;
>> >> +	bool walk_done, remote = data->remote;
>> >> +	bool leftmost_change = false;
>> >> +	u64 nextexp;
>> >> +
>> >> +	if (child) {
>> >> +		raw_spin_lock(&child->lock);
>> >> +		raw_spin_lock_nested(&group->lock, SINGLE_DEPTH_NESTING);
>> >> +
>> >> +		if (childstate.active) {
>> >> +			walk_done = true;
>> >> +			goto unlock;
>> >> +		}
>> >> +
>> >> +		first_childevt = tmigr_next_groupevt(child);
>> >> +		nextexp = child->next_expiry;
>> >> +		evt = &child->groupevt;
>> >> +	} else {
>> >> +		nextexp = data->nextexp;
>> >> +
>> >> +		first_childevt = evt = data->evt;
>> >> +
>> >> +		/*
>> >> +		 * Walking the hierarchy is required in any case when a
>> >> +		 * remote expiry was done before. This ensures to not lose
>> >> +		 * already queued events in non active groups (see section
>> >> +		 * "Required event and timerqueue update after a remote
>> >> +		 * expiry" in the documentation at the top).
>> >> +		 *
>> >> +		 * The two call sites which are executed without a remote expiry
>> >> +		 * before, are not prevented from propagating changes through
>> >> +		 * the hierarchy by the return:
>> >> +		 *  - When entering this path by tmigr_new_timer(), @evt->ignore
>> >> +		 *    is never set.
>> >> +		 *  - tmigr_inactive_up() takes care of the propagation by
>> >> +		 *    itself and ignores the return value. But an immediate
>> >> +		 *    return is required because nothing has to be done in this
>> >> +		 *    level as the event could be ignored.
>> >> +		 */
>> >> +		if (evt->ignore && !remote)
>> >> +			return true;
>> >> +
>> >> +		raw_spin_lock(&group->lock);
>> >> +	}
>> >> +
>> >> +	if (nextexp == KTIME_MAX) {
>> >> +		evt->ignore = true;
>> >> +
>> >> +		/*
>> >> +		 * When the next child event could be ignored (nextexp is
>> >> +		 * KTIME_MAX) and there was no remote timer handling before or
>> >> +		 * the group is already active, there is no need to walk the
>> >> +		 * hierarchy even if there is a parent group.
>> >> +		 *
>> >> +		 * The other way round: even if the event could be ignored, but
>> >> +		 * if a remote timer handling was executed before and the group
>> >> +		 * is not active, walking the hierarchy is required to not miss
>> >> +		 * an enqueued timer in the non active group. The enqueued timer
>> >> +		 * of the group needs to be propagated to a higher level to
>> >> +		 * ensure it is handled.
>> >> +		 */
>> >> +		if (!remote || groupstate.active) {
>> >> +			walk_done = true;
>> >> +			goto unlock;
>> >
>> > So if the current tmc going inactive was the migrator for the whole hierarchy
>> > and it is reaching here the top-level, this assumes that if none of this tmc's
>> > groups have a timer, then it can just return. But what if the top level has
>> > timers from other children? Who is going to handle them then?
>> >
>> > Should this be "goto check_toplvl" instead?
>> >
>> 
>> Simply replacing this goto will not work. Then we chould end up with a
>> 'data->firstexp' set even if we do not want to have it (when remote is
>> not set).
>
> Hmm, but the (groupstate.migrator == TMIGR_NONE) condition should prevent that,
> no?
>

When remote is not set, and the group is not active (but it's not top
level), walk_done will be set to true. Then goto check_toplevel. There
walk_done and the groupstate migrator check is both true. But it is not
top level... and the next goto unlock will also not happen and
data->firstexp will be set. No?

>> 
>> There is another issue in here. When the event could be ignored and it
>> is propagated because of e.g. remote timer handling, then the timerqueue
>> dance is done nevertheless. It's not a big problem (as the ignore flag
>> is set and event is removed of queue when revisting the timer queue),
>> but its obviously more work than it is required to have.
>
> Right. I guess it doesn't hurt to delete it from the timerqueue if present and
> then update group->next_expiry accordingly. But it's certainly not useful
> to requeue it :-)
>

This would be the plan to keep the delete but drop the requeue in this
case.

With those two things, the tmigr_update_events() wants to be changed a
little more.

Thanks

> Thanks.
>
>> 
>> Thanks
>> 
>> >> +		}
>> >> +	} else {
>> >> +		/*
>> >> +		 * An update of @evt->cpu and @evt->ignore flag is required only
>> >> +		 * when @child is set (the child is equal or higher than lvl0),
>> >> +		 * but it doesn't matter if it is written once more to the per
>> >> +		 * CPU event; make the update unconditional.
>> >> +		 */
>> >> +		evt->cpu = first_childevt->cpu;
>> >> +		evt->ignore = false;
>> >> +	}
>> >> +
>> >> +	walk_done = !group->parent;
>> >> +
>> >> +	/*
>> >> +	 * If the child event is already queued in the group, remove it from the
>> >> +	 * queue when the expiry time changed only.
>> >> +	 */
>> >> +	if (timerqueue_node_queued(&evt->nextevt)) {
>> >> +		if (evt->nextevt.expires == nextexp)
>> >> +			goto check_toplvl;
>> >> +
>> >> +		leftmost_change = timerqueue_getnext(&group->events) == &evt->nextevt;
>> >> +		if (!timerqueue_del(&group->events, &evt->nextevt))
>> >> +			WRITE_ONCE(group->next_expiry, KTIME_MAX);
>> >> +	}
>> >> +
>> >> +	evt->nextevt.expires = nextexp;
>> >> +
>> >> +	if (timerqueue_add(&group->events, &evt->nextevt)) {
>> >> +		leftmost_change = true;
>> >> +		WRITE_ONCE(group->next_expiry, nextexp);
>> >> +	}
>> >> +
>> >> +check_toplvl:
>> >> +	if (walk_done && (groupstate.migrator == TMIGR_NONE)) {
>> >> +		/*
>> >> +		 * Nothing to do when first event didn't changed and update was
>> >> +		 * done during remote timer handling.
>> >> +		 */
>> >> +		if (remote && !leftmost_change)
>> >> +			goto unlock;
>> >> +		/*
>> >> +		 * The top level group is idle and it has to be ensured the
>> >> +		 * global timers are handled in time. (This could be optimized
>> >> +		 * by keeping track of the last global scheduled event and only
>> >> +		 * arming it on the CPU if the new event is earlier. Not sure if
>> >> +		 * its worth the complexity.)
>> >> +		 */
>> >> +		data->firstexp = tmigr_next_groupevt_expires(group);
>> >> +	}
>> >> +
>> >> +unlock:
>> >> +	raw_spin_unlock(&group->lock);
>> >> +
>> >> +	if (child)
>> >> +		raw_spin_unlock(&child->lock);
>> >> +
>> >> +	return walk_done;
>> >> +}

Re: [PATCH v10 18/20] timers: Implement the hierarchical pull model

[Frederic Weisbecker]

Le Tue, Feb 06, 2024 at 03:13:23PM +0100, Anna-Maria Behnsen a écrit :
> When remote is not set, and the group is not active (but it's not top
> level), walk_done will be set to true. Then goto check_toplevel. There
> walk_done and the groupstate migrator check is both true. But it is not
> top level... and the next goto unlock will also not happen and
> data->firstexp will be set. No?

Bah yes of course, walk_done doesn't mean we are in the top then, you're right.

> 
> >> 
> >> There is another issue in here. When the event could be ignored and it
> >> is propagated because of e.g. remote timer handling, then the timerqueue
> >> dance is done nevertheless. It's not a big problem (as the ignore flag
> >> is set and event is removed of queue when revisting the timer queue),
> >> but its obviously more work than it is required to have.
> >
> > Right. I guess it doesn't hurt to delete it from the timerqueue if present and
> > then update group->next_expiry accordingly. But it's certainly not useful
> > to requeue it :-)
> >
> 
> This would be the plan to keep the delete but drop the requeue in this
> case.
> 
> With those two things, the tmigr_update_events() wants to be changed a
> little more.

For the best!

Thanks.

[PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Anna-Maria Behnsen]

The timer base is marked idle when get_next_timer_interrupt() is
executed. But the decision whether the tick will be stopped and whether the
system is able to go idle is done later. When the timer bases is marked
idle and a new first timer is enqueued remote an IPI is raised. Even if it
is not required because the tick is not stopped and the timer base is
evaluated again at the next tick.

To prevent this, the timer base is marked idle in tick_nohz_stop_tick() and
get_next_timer_interrupt() is streamlined by only looking for the next timer
interrupt. All other work is postponed to timer_base_try_to_set_idle() which is
called by tick_nohz_stop_tick(). timer_base_try_to_set_idle() never resets
timer_base::is_idle state. This is done when the tick is restarted via
tick_nohz_restart_sched_tick().

With this, tick_sched::tick_stopped and timer_base::is_idle are always in
sync. So there is no longer the need to execute timer_clear_idle() in
tick_nohz_idle_retain_tick(). This was required before, as
tick_nohz_next_event() set timer_base::is_idle even if the tick would not be
stopped. So timer_clear_idle() is only executed, when timer base is idle. So the
check whether timer base is idle, is now no longer required as well.

While at it fix some nearby whitespace damage as well.

Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
---
v10a:
 - Drop the unnecessary if branch which handles return value of
   timer_base_try_to_set_idle()
 - Do not open code 'tick_nohz_retain_tick()' and keep
   tick_nohz_idle_retain_tick() as is.
---
 kernel/time/tick-internal.h |  1 +
 kernel/time/tick-sched.c    | 40 +++++++++++++++++--------
 kernel/time/timer.c         | 60 ++++++++++++++++++++++++++-----------
 3 files changed, 71 insertions(+), 30 deletions(-)

diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index 481b7ab65e2c..47df30b871e4 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -163,6 +163,7 @@ static inline void timers_update_nohz(void) { }
 DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases);
 
 extern u64 get_next_timer_interrupt(unsigned long basej, u64 basem);
+u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle);
 void timer_clear_idle(void);
 
 #define CLOCK_SET_WALL							\
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 01fb50c1b17e..4c7ccb1c9307 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -849,11 +849,6 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu)
 	 */
 	delta = next_tick - basemono;
 	if (delta <= (u64)TICK_NSEC) {
-		/*
-		 * Tell the timer code that the base is not idle, i.e. undo
-		 * the effect of get_next_timer_interrupt():
-		 */
-		timer_clear_idle();
 		/*
 		 * We've not stopped the tick yet, and there's a timer in the
 		 * next period, so no point in stopping it either, bail.
@@ -889,12 +884,34 @@ static ktime_t tick_nohz_next_event(struct tick_sched *ts, int cpu)
 static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
 {
 	struct clock_event_device *dev = __this_cpu_read(tick_cpu_device.evtdev);
+	unsigned long basejiff = ts->last_jiffies;
 	u64 basemono = ts->timer_expires_base;
-	u64 expires = ts->timer_expires;
+	bool timer_idle;
+	u64 expires;
 
 	/* Make sure we won't be trying to stop it twice in a row. */
 	ts->timer_expires_base = 0;
 
+	/*
+	 * Now the tick should be stopped definitely - so the timer base needs
+	 * to be marked idle as well to not miss a newly queued timer.
+	 */
+	expires = timer_base_try_to_set_idle(basejiff, basemono, &timer_idle);
+	if (expires > ts->timer_expires) {
+		/*
+		 * This path could only happen when the first timer was removed
+		 * between calculating the possible sleep length and now (when
+		 * high resolution mode is not active, timer could also be a
+		 * hrtimer).
+		 *
+		 * We have to stick to the original calculated expiry value to
+		 * not stop the tick for too long with a shallow C-state (which
+		 * was programmed by cpuidle because of an early next expiration
+		 * value).
+		 */
+		expires = ts->timer_expires;
+	}
+
 	/*
 	 * If this CPU is the one which updates jiffies, then give up
 	 * the assignment and let it be taken by the CPU which runs
@@ -930,6 +947,10 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
 	 * scheduler tick in tick_nohz_restart_sched_tick().
 	 */
 	if (!ts->tick_stopped) {
+		/* If the timer base is not idle, retain the tick. */
+		if (!timer_idle)
+			return;
+
 		calc_load_nohz_start();
 		quiet_vmstat();
 
@@ -991,7 +1012,7 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now)
 	touch_softlockup_watchdog_sched();
 
 	/* Cancel the scheduled timer and restore the tick: */
-	ts->tick_stopped  = 0;
+	ts->tick_stopped = 0;
 	tick_nohz_restart(ts, now);
 }
 
@@ -1147,11 +1168,6 @@ void tick_nohz_idle_stop_tick(void)
 void tick_nohz_idle_retain_tick(void)
 {
 	tick_nohz_retain_tick(this_cpu_ptr(&tick_cpu_sched));
-	/*
-	 * Undo the effect of get_next_timer_interrupt() called from
-	 * tick_nohz_next_event().
-	 */
-	timer_clear_idle();
 }
 
 /**
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 2aea55d53416..3a668060692e 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1911,19 +1911,22 @@ static u64 cmp_next_hrtimer_event(u64 basem, u64 expires)
 	return DIV_ROUND_UP_ULL(nextevt, TICK_NSEC) * TICK_NSEC;
 }
 
-static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem)
+static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem,
+					     bool *idle)
 {
 	struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
 	unsigned long nextevt = basej + NEXT_TIMER_MAX_DELTA;
 	u64 expires = KTIME_MAX;
-	bool was_idle;
 
 	/*
 	 * Pretend that there is no timer pending if the cpu is offline.
 	 * Possible pending timers will be migrated later to an active cpu.
 	 */
-	if (cpu_is_offline(smp_processor_id()))
+	if (cpu_is_offline(smp_processor_id())) {
+		if (idle)
+			*idle = true;
 		return expires;
+	}
 
 	raw_spin_lock(&base->lock);
 	if (base->next_expiry_recalc)
@@ -1953,17 +1956,26 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem)
 	__forward_timer_base(base, basej);
 
 	/*
-	 * Base is idle if the next event is more than a tick away.
-	 *
-	 * If the base is marked idle then any timer add operation must forward
-	 * the base clk itself to keep granularity small. This idle logic is
-	 * only maintained for the BASE_STD base, deferrable timers may still
-	 * see large granularity skew (by design).
+	 * Set base->is_idle only when caller is timer_base_try_to_set_idle()
 	 */
-	was_idle = base->is_idle;
-	base->is_idle = time_after(nextevt, basej + 1);
-	if (was_idle != base->is_idle)
-		trace_timer_base_idle(base->is_idle, base->cpu);
+	if (idle) {
+		/*
+		 * Base is idle if the next event is more than a tick away.
+		 *
+		 * If the base is marked idle then any timer add operation must
+		 * forward the base clk itself to keep granularity small. This
+		 * idle logic is only maintained for the BASE_STD base,
+		 * deferrable timers may still see large granularity skew (by
+		 * design).
+		 */
+		if (!base->is_idle) {
+			if (time_after(nextevt, basej + 1)) {
+				base->is_idle = true;
+				trace_timer_base_idle(true, base->cpu);
+			}
+		}
+		*idle = base->is_idle;
+	}
 
 	raw_spin_unlock(&base->lock);
 
@@ -1980,7 +1992,21 @@ static inline u64 __get_next_timer_interrupt(unsigned long basej, u64 basem)
  */
 u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
 {
-	return __get_next_timer_interrupt(basej, basem);
+	return __get_next_timer_interrupt(basej, basem, NULL);
+}
+
+/**
+ * timer_base_try_to_set_idle() - Try to set the idle state of the timer bases
+ * @basej:	base time jiffies
+ * @basem:	base time clock monotonic
+ * @idle:	pointer to store the value of timer_base->is_idle
+ *
+ * Returns the tick aligned clock monotonic time of the next pending
+ * timer or KTIME_MAX if no timer is pending.
+ */
+u64 timer_base_try_to_set_idle(unsigned long basej, u64 basem, bool *idle)
+{
+	return __get_next_timer_interrupt(basej, basem, idle);
 }
 
 /**
@@ -1998,10 +2024,8 @@ void timer_clear_idle(void)
 	 * sending the IPI a few instructions smaller for the cost of taking
 	 * the lock in the exit from idle path.
 	 */
-	if (base->is_idle) {
-		base->is_idle = false;
-		trace_timer_base_idle(false, smp_processor_id());
-	}
+	base->is_idle = false;
+	trace_timer_base_idle(false, smp_processor_id());
 }
 #endif
 
-- 
2.39.2

Re: [PATCH v10 13/20] timers: Add get next timer interrupt functionality for remote CPUs

[Frederic Weisbecker]

Le Mon, Jan 15, 2024 at 03:37:36PM +0100, Anna-Maria Behnsen a écrit :
> +# ifdef CONFIG_SMP
> +/**
> + * fetch_next_timer_interrupt_remote() - Store next timers into @tevt
> + * @basej:	base time jiffies
> + * @basem:	base time clock monotonic
> + * @tevt:	Pointer to the storage for the expiry values
> + * @cpu:	Remote CPU
> + *
> + * Stores the next pending local and global timer expiry values in the
> + * struct pointed to by @tevt. If a queue is empty the corresponding
> + * field is set to KTIME_MAX. If local event expires before global
> + * event, global event is set to KTIME_MAX as well.
> + *
> + * Caller needs to make sure timer base locks are held (use
> + * timer_lock_remote_bases() for this purpose).
> + */
> +void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem,
> +				       struct timer_events *tevt,
> +				       unsigned int cpu)
> +{
> +	struct timer_base *base_local, *base_global;
> +
> +	/* Preset local / global events */
> +	tevt->local = tevt->global = KTIME_MAX;
> +
> +	base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu);
> +	base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu);
> +
> +	lockdep_assert_held(&base_local->lock);
> +	lockdep_assert_held(&base_global->lock);
> +
> +	fetch_next_timer_interrupt(basej, basem, base_local, base_global, tevt);

If the next timer is global and it is <= jiffies + 1, the result will be
returned in tevt.local only and not on tevt.global. So a remote fetch may miss it.

For this to work on both local and remote fetch, you may need:

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 320eb4ceafa2..64ce9a7760f5 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -2004,6 +2007,8 @@ static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem,
 		if (time_before(nextevt, basej))
 			nextevt = basej;
 		tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC;
+		if (!local_first)
+			tevt->global = tevt->local;
 		return nextevt;
 	}
 

Re: [PATCH v10 13/20] timers: Add get next timer interrupt functionality for remote CPUs

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Jan 15, 2024 at 03:37:36PM +0100, Anna-Maria Behnsen a écrit :
>> +# ifdef CONFIG_SMP
>> +/**
>> + * fetch_next_timer_interrupt_remote() - Store next timers into @tevt
>> + * @basej:	base time jiffies
>> + * @basem:	base time clock monotonic
>> + * @tevt:	Pointer to the storage for the expiry values
>> + * @cpu:	Remote CPU
>> + *
>> + * Stores the next pending local and global timer expiry values in the
>> + * struct pointed to by @tevt. If a queue is empty the corresponding
>> + * field is set to KTIME_MAX. If local event expires before global
>> + * event, global event is set to KTIME_MAX as well.
>> + *
>> + * Caller needs to make sure timer base locks are held (use
>> + * timer_lock_remote_bases() for this purpose).
>> + */
>> +void fetch_next_timer_interrupt_remote(unsigned long basej, u64 basem,
>> +				       struct timer_events *tevt,
>> +				       unsigned int cpu)
>> +{
>> +	struct timer_base *base_local, *base_global;
>> +
>> +	/* Preset local / global events */
>> +	tevt->local = tevt->global = KTIME_MAX;
>> +
>> +	base_local = per_cpu_ptr(&timer_bases[BASE_LOCAL], cpu);
>> +	base_global = per_cpu_ptr(&timer_bases[BASE_GLOBAL], cpu);
>> +
>> +	lockdep_assert_held(&base_local->lock);
>> +	lockdep_assert_held(&base_global->lock);
>> +
>> +	fetch_next_timer_interrupt(basej, basem, base_local, base_global, tevt);
>
> If the next timer is global and it is <= jiffies + 1, the result will be
> returned in tevt.local only and not on tevt.global. So a remote fetch may miss it.

Oh no. But yes, sounds reasonable.

> For this to work on both local and remote fetch, you may need:
>
> diff --git a/kernel/time/timer.c b/kernel/time/timer.c
> index 320eb4ceafa2..64ce9a7760f5 100644
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -2004,6 +2007,8 @@ static unsigned long fetch_next_timer_interrupt(unsigned long basej, u64 basem,
>  		if (time_before(nextevt, basej))
>  			nextevt = basej;
>  		tevt->local = basem + (u64)(nextevt - basej) * TICK_NSEC;
> +		if (!local_first)
> +			tevt->global = tevt->local;
>  		return nextevt;
>  	}
>  

Will fix it - with a big comment explaining why this is required for
remote call sites and will not hurt when executed on the local cpu.

Thanks a lot!

Re: [PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Frederic Weisbecker]

Le Mon, Feb 19, 2024 at 09:52:36AM +0100, Anna-Maria Behnsen a écrit :
> The timer base is marked idle when get_next_timer_interrupt() is
> executed. But the decision whether the tick will be stopped and whether the
> system is able to go idle is done later. When the timer bases is marked
> idle and a new first timer is enqueued remote an IPI is raised. Even if it
> is not required because the tick is not stopped and the timer base is
> evaluated again at the next tick.
> 
> To prevent this, the timer base is marked idle in tick_nohz_stop_tick() and
> get_next_timer_interrupt() is streamlined by only looking for the next timer
> interrupt. All other work is postponed to timer_base_try_to_set_idle() which is
> called by tick_nohz_stop_tick(). timer_base_try_to_set_idle() never resets
> timer_base::is_idle state. This is done when the tick is restarted via
> tick_nohz_restart_sched_tick().
> 
> With this, tick_sched::tick_stopped and timer_base::is_idle are always in
> sync. So there is no longer the need to execute timer_clear_idle() in
> tick_nohz_idle_retain_tick(). This was required before, as
> tick_nohz_next_event() set timer_base::is_idle even if the tick would not be
> stopped. So timer_clear_idle() is only executed, when timer base is idle. So the
> check whether timer base is idle, is now no longer required as well.
> 
> While at it fix some nearby whitespace damage as well.
> 
> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

Just a small detail below that can be fixed in a further patch:

> @@ -930,6 +947,10 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>  	 * scheduler tick in tick_nohz_restart_sched_tick().
>  	 */
>  	if (!ts->tick_stopped) {
> +		/* If the timer base is not idle, retain the tick. */
> +		if (!timer_idle)
> +			return;

This happens after tick_do_timer_cpu has been set to TICK_DO_TIMER_NONE. Ideally
it would be better to do it before. Not that it hurts in practice: another CPU
or this one will take the duty. But it looks weird to stop halfway.

Thanks!

Re: [PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Mon, Feb 19, 2024 at 09:52:36AM +0100, Anna-Maria Behnsen a écrit :
>> The timer base is marked idle when get_next_timer_interrupt() is
>> executed. But the decision whether the tick will be stopped and whether the
>> system is able to go idle is done later. When the timer bases is marked
>> idle and a new first timer is enqueued remote an IPI is raised. Even if it
>> is not required because the tick is not stopped and the timer base is
>> evaluated again at the next tick.
>> 
>> To prevent this, the timer base is marked idle in tick_nohz_stop_tick() and
>> get_next_timer_interrupt() is streamlined by only looking for the next timer
>> interrupt. All other work is postponed to timer_base_try_to_set_idle() which is
>> called by tick_nohz_stop_tick(). timer_base_try_to_set_idle() never resets
>> timer_base::is_idle state. This is done when the tick is restarted via
>> tick_nohz_restart_sched_tick().
>> 
>> With this, tick_sched::tick_stopped and timer_base::is_idle are always in
>> sync. So there is no longer the need to execute timer_clear_idle() in
>> tick_nohz_idle_retain_tick(). This was required before, as
>> tick_nohz_next_event() set timer_base::is_idle even if the tick would not be
>> stopped. So timer_clear_idle() is only executed, when timer base is idle. So the
>> check whether timer base is idle, is now no longer required as well.
>> 
>> While at it fix some nearby whitespace damage as well.
>> 
>> Signed-off-by: Anna-Maria Behnsen <anna-maria@linutronix.de>
>
> Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
>
> Just a small detail below that can be fixed in a further patch:
>
>> @@ -930,6 +947,10 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>>  	 * scheduler tick in tick_nohz_restart_sched_tick().
>>  	 */
>>  	if (!ts->tick_stopped) {
>> +		/* If the timer base is not idle, retain the tick. */
>> +		if (!timer_idle)
>> +			return;
>
> This happens after tick_do_timer_cpu has been set to TICK_DO_TIMER_NONE. Ideally
> it would be better to do it before. Not that it hurts in practice: another CPU
> or this one will take the duty. But it looks weird to stop halfway.
>

Yes, you are right. I would prefere, to clean it up directly and add
another patch before this patch which simply moves the
TICK_DO_TIMER_NONE related block after the !ts->tick_stopped
block. Because a changed order shouldn't be a problem at the moment as
well, or am I wrong?

Thanks,

	Anna-Maria

---8<----
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 01fb50c1b17e..b93f0e6f273f 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -895,21 +895,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
 	/* Make sure we won't be trying to stop it twice in a row. */
 	ts->timer_expires_base = 0;
 
-	/*
-	 * If this CPU is the one which updates jiffies, then give up
-	 * the assignment and let it be taken by the CPU which runs
-	 * the tick timer next, which might be this CPU as well. If we
-	 * don't drop this here, the jiffies might be stale and
-	 * do_timer() never gets invoked. Keep track of the fact that it
-	 * was the one which had the do_timer() duty last.
-	 */
-	if (cpu == tick_do_timer_cpu) {
-		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
-		ts->do_timer_last = 1;
-	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
-		ts->do_timer_last = 0;
-	}
-
 	/* Skip reprogram of event if it's not changed */
 	if (ts->tick_stopped && (expires == ts->next_tick)) {
 		/* Sanity check: make sure clockevent is actually programmed */
@@ -938,6 +923,21 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
 		trace_tick_stop(1, TICK_DEP_MASK_NONE);
 	}
 
+	/*
+	 * If this CPU is the one which updates jiffies, then give up
+	 * the assignment and let it be taken by the CPU which runs
+	 * the tick timer next, which might be this CPU as well. If we
+	 * don't drop this here, the jiffies might be stale and
+	 * do_timer() never gets invoked. Keep track of the fact that it
+	 * was the one which had the do_timer() duty last.
+	 */
+	if (cpu == tick_do_timer_cpu) {
+		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
+		ts->do_timer_last = 1;
+	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
+		ts->do_timer_last = 0;
+	}
+
 	ts->next_tick = expires;
 
 	/*

Re: [PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Frederic Weisbecker]

Le Tue, Feb 20, 2024 at 11:48:19AM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> index 01fb50c1b17e..b93f0e6f273f 100644
> --- a/kernel/time/tick-sched.c
> +++ b/kernel/time/tick-sched.c
> @@ -895,21 +895,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>  	/* Make sure we won't be trying to stop it twice in a row. */
>  	ts->timer_expires_base = 0;
>  
> -	/*
> -	 * If this CPU is the one which updates jiffies, then give up
> -	 * the assignment and let it be taken by the CPU which runs
> -	 * the tick timer next, which might be this CPU as well. If we
> -	 * don't drop this here, the jiffies might be stale and
> -	 * do_timer() never gets invoked. Keep track of the fact that it
> -	 * was the one which had the do_timer() duty last.
> -	 */
> -	if (cpu == tick_do_timer_cpu) {
> -		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
> -		ts->do_timer_last = 1;
> -	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
> -		ts->do_timer_last = 0;
> -	}
> -
>  	/* Skip reprogram of event if it's not changed */
>  	if (ts->tick_stopped && (expires == ts->next_tick)) {
>  		/* Sanity check: make sure clockevent is actually programmed */

That should work but then you lose the optimization that resets
ts->do_timer_last even if the next timer hasn't changed.

Thanks.



> @@ -938,6 +923,21 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>  		trace_tick_stop(1, TICK_DEP_MASK_NONE);
>  	}
>  
> +	/*
> +	 * If this CPU is the one which updates jiffies, then give up
> +	 * the assignment and let it be taken by the CPU which runs
> +	 * the tick timer next, which might be this CPU as well. If we
> +	 * don't drop this here, the jiffies might be stale and
> +	 * do_timer() never gets invoked. Keep track of the fact that it
> +	 * was the one which had the do_timer() duty last.
> +	 */
> +	if (cpu == tick_do_timer_cpu) {
> +		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
> +		ts->do_timer_last = 1;
> +	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
> +		ts->do_timer_last = 0;
> +	}
> +
>  	ts->next_tick = expires;
>  
>  	/*

Re: [PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Tue, Feb 20, 2024 at 11:48:19AM +0100, Anna-Maria Behnsen a écrit :
>> Frederic Weisbecker <frederic@kernel.org> writes:
>> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
>> index 01fb50c1b17e..b93f0e6f273f 100644
>> --- a/kernel/time/tick-sched.c
>> +++ b/kernel/time/tick-sched.c
>> @@ -895,21 +895,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>>  	/* Make sure we won't be trying to stop it twice in a row. */
>>  	ts->timer_expires_base = 0;
>>  
>> -	/*
>> -	 * If this CPU is the one which updates jiffies, then give up
>> -	 * the assignment and let it be taken by the CPU which runs
>> -	 * the tick timer next, which might be this CPU as well. If we
>> -	 * don't drop this here, the jiffies might be stale and
>> -	 * do_timer() never gets invoked. Keep track of the fact that it
>> -	 * was the one which had the do_timer() duty last.
>> -	 */
>> -	if (cpu == tick_do_timer_cpu) {
>> -		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
>> -		ts->do_timer_last = 1;
>> -	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
>> -		ts->do_timer_last = 0;
>> -	}
>> -
>>  	/* Skip reprogram of event if it's not changed */
>>  	if (ts->tick_stopped && (expires == ts->next_tick)) {
>>  		/* Sanity check: make sure clockevent is actually programmed */
>
> That should work but then you lose the optimization that resets
> ts->do_timer_last even if the next timer hasn't changed.
>

Beside of this optimization thing, I see onther problem. But I'm not
sure, if I understood it correctly: When the CPU drops the
tick_do_timer_cpu assignment and stops the tick, it is possible, that
this CPU nevertheless executes tick_sched_do_timer() and then reassigns
to tick_do_timer_cpu?

Then it is mandatory that we have this drop the assignment also in the
path when the tick is already stopped. Otherwise the problem described
in the comment could happen with stale jiffies, no?

Thanks

> Thanks.
>
>
>
>> @@ -938,6 +923,21 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>>  		trace_tick_stop(1, TICK_DEP_MASK_NONE);
>>  	}
>>  
>> +	/*
>> +	 * If this CPU is the one which updates jiffies, then give up
>> +	 * the assignment and let it be taken by the CPU which runs
>> +	 * the tick timer next, which might be this CPU as well. If we
>> +	 * don't drop this here, the jiffies might be stale and
>> +	 * do_timer() never gets invoked. Keep track of the fact that it
>> +	 * was the one which had the do_timer() duty last.
>> +	 */
>> +	if (cpu == tick_do_timer_cpu) {
>> +		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
>> +		ts->do_timer_last = 1;
>> +	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
>> +		ts->do_timer_last = 0;
>> +	}
>> +
>>  	ts->next_tick = expires;
>>  
>>  	/*

Re: [PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Frederic Weisbecker]

Le Tue, Feb 20, 2024 at 01:02:18PM +0100, Anna-Maria Behnsen a écrit :
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Tue, Feb 20, 2024 at 11:48:19AM +0100, Anna-Maria Behnsen a écrit :
> >> Frederic Weisbecker <frederic@kernel.org> writes:
> >> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> >> index 01fb50c1b17e..b93f0e6f273f 100644
> >> --- a/kernel/time/tick-sched.c
> >> +++ b/kernel/time/tick-sched.c
> >> @@ -895,21 +895,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
> >>  	/* Make sure we won't be trying to stop it twice in a row. */
> >>  	ts->timer_expires_base = 0;
> >>  
> >> -	/*
> >> -	 * If this CPU is the one which updates jiffies, then give up
> >> -	 * the assignment and let it be taken by the CPU which runs
> >> -	 * the tick timer next, which might be this CPU as well. If we
> >> -	 * don't drop this here, the jiffies might be stale and
> >> -	 * do_timer() never gets invoked. Keep track of the fact that it
> >> -	 * was the one which had the do_timer() duty last.
> >> -	 */
> >> -	if (cpu == tick_do_timer_cpu) {
> >> -		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
> >> -		ts->do_timer_last = 1;
> >> -	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
> >> -		ts->do_timer_last = 0;
> >> -	}
> >> -
> >>  	/* Skip reprogram of event if it's not changed */
> >>  	if (ts->tick_stopped && (expires == ts->next_tick)) {
> >>  		/* Sanity check: make sure clockevent is actually programmed */
> >
> > That should work but then you lose the optimization that resets
> > ts->do_timer_last even if the next timer hasn't changed.
> >
> 
> Beside of this optimization thing, I see onther problem. But I'm not
> sure, if I understood it correctly: When the CPU drops the
> tick_do_timer_cpu assignment and stops the tick, it is possible, that
> this CPU nevertheless executes tick_sched_do_timer() and then reassigns
> to tick_do_timer_cpu?

Yes but in this case a timer interrupt has executed and ts->next_tick
is cleared, so the above skip reprogramm branch is not taken.

Thanks.

> 
> Then it is mandatory that we have this drop the assignment also in the
> path when the tick is already stopped. Otherwise the problem described
> in the comment could happen with stale jiffies, no?
> 
> Thanks
> 
> > Thanks.
> >
> >
> >
> >> @@ -938,6 +923,21 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
> >>  		trace_tick_stop(1, TICK_DEP_MASK_NONE);
> >>  	}
> >>  
> >> +	/*
> >> +	 * If this CPU is the one which updates jiffies, then give up
> >> +	 * the assignment and let it be taken by the CPU which runs
> >> +	 * the tick timer next, which might be this CPU as well. If we
> >> +	 * don't drop this here, the jiffies might be stale and
> >> +	 * do_timer() never gets invoked. Keep track of the fact that it
> >> +	 * was the one which had the do_timer() duty last.
> >> +	 */
> >> +	if (cpu == tick_do_timer_cpu) {
> >> +		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
> >> +		ts->do_timer_last = 1;
> >> +	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
> >> +		ts->do_timer_last = 0;
> >> +	}
> >> +
> >>  	ts->next_tick = expires;
> >>  
> >>  	/*

Re: [PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> Le Tue, Feb 20, 2024 at 01:02:18PM +0100, Anna-Maria Behnsen a écrit :
>> Frederic Weisbecker <frederic@kernel.org> writes:
>> 
>> > Le Tue, Feb 20, 2024 at 11:48:19AM +0100, Anna-Maria Behnsen a écrit :
>> >> Frederic Weisbecker <frederic@kernel.org> writes:
>> >> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
>> >> index 01fb50c1b17e..b93f0e6f273f 100644
>> >> --- a/kernel/time/tick-sched.c
>> >> +++ b/kernel/time/tick-sched.c
>> >> @@ -895,21 +895,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>> >>  	/* Make sure we won't be trying to stop it twice in a row. */
>> >>  	ts->timer_expires_base = 0;
>> >>  
>> >> -	/*
>> >> -	 * If this CPU is the one which updates jiffies, then give up
>> >> -	 * the assignment and let it be taken by the CPU which runs
>> >> -	 * the tick timer next, which might be this CPU as well. If we
>> >> -	 * don't drop this here, the jiffies might be stale and
>> >> -	 * do_timer() never gets invoked. Keep track of the fact that it
>> >> -	 * was the one which had the do_timer() duty last.
>> >> -	 */
>> >> -	if (cpu == tick_do_timer_cpu) {
>> >> -		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
>> >> -		ts->do_timer_last = 1;
>> >> -	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
>> >> -		ts->do_timer_last = 0;
>> >> -	}
>> >> -
>> >>  	/* Skip reprogram of event if it's not changed */
>> >>  	if (ts->tick_stopped && (expires == ts->next_tick)) {
>> >>  		/* Sanity check: make sure clockevent is actually programmed */
>> >
>> > That should work but then you lose the optimization that resets
>> > ts->do_timer_last even if the next timer hasn't changed.
>> >
>> 
>> Beside of this optimization thing, I see onther problem. But I'm not
>> sure, if I understood it correctly: When the CPU drops the
>> tick_do_timer_cpu assignment and stops the tick, it is possible, that
>> this CPU nevertheless executes tick_sched_do_timer() and then reassigns
>> to tick_do_timer_cpu?
>
> Yes but in this case a timer interrupt has executed and ts->next_tick
> is cleared, so the above skip reprogramm branch is not taken.
>

Yes... So I need to change it without dropping the
optimization. Otherwise someone might complain about it.

Two possible solutions:

a) split out this if/else thing for dropping the tick_do_timer_cpu
   assignment into a separate function and call it:
   - before the return in the skip reprogramm branch
   - and after the if clause which contains stopping the tick (where it
     is executed in the current proposal)

b) Take my current proposal and add before the return in the skip
   reprogramm branch the following lines:

   if (tick_do_timer_cpu != TICK_DO_TIMER_NONE)
   	ts->do_timer_last = 0;

   as the first part of the tick_do_timer_cpu/last logic shouldn't be
   required (because then also ts->next_tick is already cleared).

What do you prefere? Or do you prefere something else?

Thanks

> Thanks.
>
>> 
>> Then it is mandatory that we have this drop the assignment also in the
>> path when the tick is already stopped. Otherwise the problem described
>> in the comment could happen with stale jiffies, no?
>> 
>> Thanks
>> 
>> > Thanks.
>> >
>> >
>> >
>> >> @@ -938,6 +923,21 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>> >>  		trace_tick_stop(1, TICK_DEP_MASK_NONE);
>> >>  	}
>> >>  
>> >> +	/*
>> >> +	 * If this CPU is the one which updates jiffies, then give up
>> >> +	 * the assignment and let it be taken by the CPU which runs
>> >> +	 * the tick timer next, which might be this CPU as well. If we
>> >> +	 * don't drop this here, the jiffies might be stale and
>> >> +	 * do_timer() never gets invoked. Keep track of the fact that it
>> >> +	 * was the one which had the do_timer() duty last.
>> >> +	 */
>> >> +	if (cpu == tick_do_timer_cpu) {
>> >> +		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
>> >> +		ts->do_timer_last = 1;
>> >> +	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
>> >> +		ts->do_timer_last = 0;
>> >> +	}
>> >> +
>> >>  	ts->next_tick = expires;
>> >>  
>> >>  	/*

Re: [PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Frederic Weisbecker]

On Tue, Feb 20, 2024 at 03:00:57PM +0100, Anna-Maria Behnsen wrote:
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > Le Tue, Feb 20, 2024 at 01:02:18PM +0100, Anna-Maria Behnsen a écrit :
> >> Frederic Weisbecker <frederic@kernel.org> writes:
> >> 
> >> > Le Tue, Feb 20, 2024 at 11:48:19AM +0100, Anna-Maria Behnsen a écrit :
> >> >> Frederic Weisbecker <frederic@kernel.org> writes:
> >> >> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> >> >> index 01fb50c1b17e..b93f0e6f273f 100644
> >> >> --- a/kernel/time/tick-sched.c
> >> >> +++ b/kernel/time/tick-sched.c
> >> >> @@ -895,21 +895,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
> >> >>  	/* Make sure we won't be trying to stop it twice in a row. */
> >> >>  	ts->timer_expires_base = 0;
> >> >>  
> >> >> -	/*
> >> >> -	 * If this CPU is the one which updates jiffies, then give up
> >> >> -	 * the assignment and let it be taken by the CPU which runs
> >> >> -	 * the tick timer next, which might be this CPU as well. If we
> >> >> -	 * don't drop this here, the jiffies might be stale and
> >> >> -	 * do_timer() never gets invoked. Keep track of the fact that it
> >> >> -	 * was the one which had the do_timer() duty last.
> >> >> -	 */
> >> >> -	if (cpu == tick_do_timer_cpu) {
> >> >> -		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
> >> >> -		ts->do_timer_last = 1;
> >> >> -	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
> >> >> -		ts->do_timer_last = 0;
> >> >> -	}
> >> >> -
> >> >>  	/* Skip reprogram of event if it's not changed */
> >> >>  	if (ts->tick_stopped && (expires == ts->next_tick)) {
> >> >>  		/* Sanity check: make sure clockevent is actually programmed */
> >> >
> >> > That should work but then you lose the optimization that resets
> >> > ts->do_timer_last even if the next timer hasn't changed.
> >> >
> >> 
> >> Beside of this optimization thing, I see onther problem. But I'm not
> >> sure, if I understood it correctly: When the CPU drops the
> >> tick_do_timer_cpu assignment and stops the tick, it is possible, that
> >> this CPU nevertheless executes tick_sched_do_timer() and then reassigns
> >> to tick_do_timer_cpu?
> >
> > Yes but in this case a timer interrupt has executed and ts->next_tick
> > is cleared, so the above skip reprogramm branch is not taken.
> >
> 
> Yes... So I need to change it without dropping the
> optimization. Otherwise someone might complain about it.
> 
> Two possible solutions:
> 
> a) split out this if/else thing for dropping the tick_do_timer_cpu
>    assignment into a separate function and call it:
>    - before the return in the skip reprogramm branch
>    - and after the if clause which contains stopping the tick (where it
>      is executed in the current proposal)
> 
> b) Take my current proposal and add before the return in the skip
>    reprogramm branch the following lines:
> 
>    if (tick_do_timer_cpu != TICK_DO_TIMER_NONE)
>    	ts->do_timer_last = 0;
> 
>    as the first part of the tick_do_timer_cpu/last logic shouldn't be
>    required (because then also ts->next_tick is already cleared).
> 
> What do you prefere? Or do you prefere something else?

Wouldn't the following work? If timer_idle is false, then the tick isn't
even stopped and there is nothing to do? So you can early return.

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index fdd57f1af1d7..1b2984acafbd 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -924,6 +924,9 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
 		expires = ts->timer_expires;
 	}
 
+	if (!timer_idle)
+		return;
+
 	/*
 	 * If this CPU is the one which updates jiffies, then give up
 	 * the assignment and let it be taken by the CPU which runs

Re: [PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Anna-Maria Behnsen]

Frederic Weisbecker <frederic@kernel.org> writes:

> On Tue, Feb 20, 2024 at 03:00:57PM +0100, Anna-Maria Behnsen wrote:
>> Frederic Weisbecker <frederic@kernel.org> writes:
>> 
>> > Le Tue, Feb 20, 2024 at 01:02:18PM +0100, Anna-Maria Behnsen a écrit :
>> >> Frederic Weisbecker <frederic@kernel.org> writes:
>> >> 
>> >> > Le Tue, Feb 20, 2024 at 11:48:19AM +0100, Anna-Maria Behnsen a écrit :
>> >> >> Frederic Weisbecker <frederic@kernel.org> writes:
>> >> >> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
>> >> >> index 01fb50c1b17e..b93f0e6f273f 100644
>> >> >> --- a/kernel/time/tick-sched.c
>> >> >> +++ b/kernel/time/tick-sched.c
>> >> >> @@ -895,21 +895,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>> >> >>  	/* Make sure we won't be trying to stop it twice in a row. */
>> >> >>  	ts->timer_expires_base = 0;
>> >> >>  
>> >> >> -	/*
>> >> >> -	 * If this CPU is the one which updates jiffies, then give up
>> >> >> -	 * the assignment and let it be taken by the CPU which runs
>> >> >> -	 * the tick timer next, which might be this CPU as well. If we
>> >> >> -	 * don't drop this here, the jiffies might be stale and
>> >> >> -	 * do_timer() never gets invoked. Keep track of the fact that it
>> >> >> -	 * was the one which had the do_timer() duty last.
>> >> >> -	 */
>> >> >> -	if (cpu == tick_do_timer_cpu) {
>> >> >> -		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
>> >> >> -		ts->do_timer_last = 1;
>> >> >> -	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
>> >> >> -		ts->do_timer_last = 0;
>> >> >> -	}
>> >> >> -
>> >> >>  	/* Skip reprogram of event if it's not changed */
>> >> >>  	if (ts->tick_stopped && (expires == ts->next_tick)) {
>> >> >>  		/* Sanity check: make sure clockevent is actually programmed */
>> >> >
>> >> > That should work but then you lose the optimization that resets
>> >> > ts->do_timer_last even if the next timer hasn't changed.
>> >> >
>> >> 
>> >> Beside of this optimization thing, I see onther problem. But I'm not
>> >> sure, if I understood it correctly: When the CPU drops the
>> >> tick_do_timer_cpu assignment and stops the tick, it is possible, that
>> >> this CPU nevertheless executes tick_sched_do_timer() and then reassigns
>> >> to tick_do_timer_cpu?
>> >
>> > Yes but in this case a timer interrupt has executed and ts->next_tick
>> > is cleared, so the above skip reprogramm branch is not taken.
>> >
>> 
>> Yes... So I need to change it without dropping the
>> optimization. Otherwise someone might complain about it.
>> 
>> Two possible solutions:
>> 
>> a) split out this if/else thing for dropping the tick_do_timer_cpu
>>    assignment into a separate function and call it:
>>    - before the return in the skip reprogramm branch
>>    - and after the if clause which contains stopping the tick (where it
>>      is executed in the current proposal)
>> 
>> b) Take my current proposal and add before the return in the skip
>>    reprogramm branch the following lines:
>> 
>>    if (tick_do_timer_cpu != TICK_DO_TIMER_NONE)
>>    	ts->do_timer_last = 0;
>> 
>>    as the first part of the tick_do_timer_cpu/last logic shouldn't be
>>    required (because then also ts->next_tick is already cleared).
>> 
>> What do you prefere? Or do you prefere something else?
>
> Wouldn't the following work? If timer_idle is false, then the tick isn't
> even stopped and there is nothing to do? So you can early return.
>
> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> index fdd57f1af1d7..1b2984acafbd 100644
> --- a/kernel/time/tick-sched.c
> +++ b/kernel/time/tick-sched.c
> @@ -924,6 +924,9 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
>  		expires = ts->timer_expires;
>  	}
>  
> +	if (!timer_idle)
> +		return;
> +
>  	/*
>  	 * If this CPU is the one which updates jiffies, then give up
>  	 * the assignment and let it be taken by the CPU which runs

Yes... And then I can drop the if (!timer_idle) thing inside
!ts->tick_stopped branch.

Re: [PATCH v10a] timers: Move marking timer bases idle into tick_nohz_stop_tick()

[Frederic Weisbecker]

On Tue, Feb 20, 2024 at 04:23:26PM +0100, Anna-Maria Behnsen wrote:
> Frederic Weisbecker <frederic@kernel.org> writes:
> 
> > On Tue, Feb 20, 2024 at 03:00:57PM +0100, Anna-Maria Behnsen wrote:
> >> Frederic Weisbecker <frederic@kernel.org> writes:
> >> 
> >> > Le Tue, Feb 20, 2024 at 01:02:18PM +0100, Anna-Maria Behnsen a écrit :
> >> >> Frederic Weisbecker <frederic@kernel.org> writes:
> >> >> 
> >> >> > Le Tue, Feb 20, 2024 at 11:48:19AM +0100, Anna-Maria Behnsen a écrit :
> >> >> >> Frederic Weisbecker <frederic@kernel.org> writes:
> >> >> >> diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> >> >> >> index 01fb50c1b17e..b93f0e6f273f 100644
> >> >> >> --- a/kernel/time/tick-sched.c
> >> >> >> +++ b/kernel/time/tick-sched.c
> >> >> >> @@ -895,21 +895,6 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
> >> >> >>  	/* Make sure we won't be trying to stop it twice in a row. */
> >> >> >>  	ts->timer_expires_base = 0;
> >> >> >>  
> >> >> >> -	/*
> >> >> >> -	 * If this CPU is the one which updates jiffies, then give up
> >> >> >> -	 * the assignment and let it be taken by the CPU which runs
> >> >> >> -	 * the tick timer next, which might be this CPU as well. If we
> >> >> >> -	 * don't drop this here, the jiffies might be stale and
> >> >> >> -	 * do_timer() never gets invoked. Keep track of the fact that it
> >> >> >> -	 * was the one which had the do_timer() duty last.
> >> >> >> -	 */
> >> >> >> -	if (cpu == tick_do_timer_cpu) {
> >> >> >> -		tick_do_timer_cpu = TICK_DO_TIMER_NONE;
> >> >> >> -		ts->do_timer_last = 1;
> >> >> >> -	} else if (tick_do_timer_cpu != TICK_DO_TIMER_NONE) {
> >> >> >> -		ts->do_timer_last = 0;
> >> >> >> -	}
> >> >> >> -
> >> >> >>  	/* Skip reprogram of event if it's not changed */
> >> >> >>  	if (ts->tick_stopped && (expires == ts->next_tick)) {
> >> >> >>  		/* Sanity check: make sure clockevent is actually programmed */
> >> >> >
> >> >> > That should work but then you lose the optimization that resets
> >> >> > ts->do_timer_last even if the next timer hasn't changed.
> >> >> >
> >> >> 
> >> >> Beside of this optimization thing, I see onther problem. But I'm not
> >> >> sure, if I understood it correctly: When the CPU drops the
> >> >> tick_do_timer_cpu assignment and stops the tick, it is possible, that
> >> >> this CPU nevertheless executes tick_sched_do_timer() and then reassigns
> >> >> to tick_do_timer_cpu?
> >> >
> >> > Yes but in this case a timer interrupt has executed and ts->next_tick
> >> > is cleared, so the above skip reprogramm branch is not taken.
> >> >
> >> 
> >> Yes... So I need to change it without dropping the
> >> optimization. Otherwise someone might complain about it.
> >> 
> >> Two possible solutions:
> >> 
> >> a) split out this if/else thing for dropping the tick_do_timer_cpu
> >>    assignment into a separate function and call it:
> >>    - before the return in the skip reprogramm branch
> >>    - and after the if clause which contains stopping the tick (where it
> >>      is executed in the current proposal)
> >> 
> >> b) Take my current proposal and add before the return in the skip
> >>    reprogramm branch the following lines:
> >> 
> >>    if (tick_do_timer_cpu != TICK_DO_TIMER_NONE)
> >>    	ts->do_timer_last = 0;
> >> 
> >>    as the first part of the tick_do_timer_cpu/last logic shouldn't be
> >>    required (because then also ts->next_tick is already cleared).
> >> 
> >> What do you prefere? Or do you prefere something else?
> >
> > Wouldn't the following work? If timer_idle is false, then the tick isn't
> > even stopped and there is nothing to do? So you can early return.
> >
> > diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
> > index fdd57f1af1d7..1b2984acafbd 100644
> > --- a/kernel/time/tick-sched.c
> > +++ b/kernel/time/tick-sched.c
> > @@ -924,6 +924,9 @@ static void tick_nohz_stop_tick(struct tick_sched *ts, int cpu)
> >  		expires = ts->timer_expires;
> >  	}
> >  
> > +	if (!timer_idle)
> > +		return;
> > +
> >  	/*
> >  	 * If this CPU is the one which updates jiffies, then give up
> >  	 * the assignment and let it be taken by the CPU which runs
> 
> Yes... And then I can drop the if (!timer_idle) thing inside
> !ts->tick_stopped branch.
> 

Right!