All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers
@ 2017-12-21 10:41 Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 01/36] timers: Use static keys for migrate_enable/nohz_active Anna-Maria Gleixner
                   ` (36 more replies)
  0 siblings, 37 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz

There are quite some places in the kernel which use a combination of
hrtimers and tasklets to make use of the precise expiry of hrtimers, which
schedule a tasklet to bring the actual function into softirq context.

This was introduced when the previous hrtimer softirq code was
removed. That code was implemented by expiring the timer in hard irq
context and then deferring the execution of the callback into softirq
context. That caused a lot of pointless shuffling between the rbtree and a
linked list.

In recent discussions it turned out that more potential users of hrtimers
in softirq context might come up. Aside of that the RT patches need this
functionality as well to defer hrtimers into softirq context if their
callbacks are not interrupt safe on RT.

This series implements a new approach by adding SOFT hrtimer mode and
instead of doing the list shuffle, timers started with this mode are put
into separate soft expiry hrtimer queues. These queues are evaluated only
when the hardirq context detects that the first expiring timer in the
softirq queues has expired. That makes the overhead in the hardirq context
minimal.

The series reworks the code to reuse as much as possible from the existing
facilities for the new softirq hrtimers and integrates them with all
flavours of hrtimers (HIGH_RES=y/n - NOHZ=y/n).

To achieve this quite some of the conditionals in the existing code are
removed for the price of adding some pointless data and state tracking to
the HIGH_RES=n case. That's minimal, but well worth it as it increases the
readability and maintainability of the code.

The first part of the series implements the new functionality and the
second part converts the hrtimer/tasklet users to make use of it and
removes struct hrtimer_tasklet and the surrounding helper functions.

This series is available from git as well:

	git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git WIP.timers

Thanks,

        Anna-Maria

---
v3..v4:

  - [PATCH 10/36] "hrtimer: Switch for loop to _ffs() evaluation": Add
    Joe Perches' annotation
    
  - [PATCH 28/36] "hrtimer: Implement support for softirq based
    hrtimers":
     - WARN_ON_ONCE() on non equality of HRTIMER_MODE_SOFT bit and
       hrtimer.is_soft at the begin of hrtimer_start_range_ns() and
       without holding hrtimer base lock to prevent a deadlock
     - Fold fix for remote soft hrtimer enqueue: Bug was reported-by
       Bert Schulze and fixed by Sebastian Siewior:
       https://lkml.kernel.org/r/20171214104755.7bnfwfv6mer2toe2@breakpoint.cc
     - Ensure to update softirq expires next after migrating hrtimer
       lists
     - Add bh_disable/enable() with a comment in hrtimers_dead_cpu():
       https://lkml.kernel.org/r/20171219085843.l55fasrfdqdyta5z@breakpoint.cc
     - Fix comment before __hrtimer_get_next_event(): use
       HRTIMER_ACTIVE_ALL instead of HRTIMER_ACTIVE

v2..v3:

  - Use static keys for migrate_enable and nohz_active
  - fix missing struct documentation
  - integrate Sebastian Siewiors suggestions and fix bug he mentioned:
    https://lkml.kernel.org/r/20171110124224.ykr6n4z7zgypojnb@breakpoint.cc


v1..v2:

  - integration of Peter Zijlstras patch:
    https://lkml.kernel.org/r/20170927164025.GI17526@worktop.programming.kicks-ass.net
  - using hrtimer_mode instead of additional hrtimer clock bases
  - folding the fix for updating the base offsets:
    https://lkml.kernel.org/r/20171006102820.ou4wpm56ed6m3ewr@linutronix.de
  - rework of 08/25 - 10/25 (all of those patches are facing reduction of
    conditional code) and make hrtimer_force_reprogram() unconditional as well
  - integration of new versions of "ALSA/dummy: Replace tasklet with
    softirq hrtimer" and "net/cdc_ncm: Replace tasklet with softirq
    hrtimer"
  - additional hrtimer/tasklet user conversion: "net/mvpp2: Replace tasklet
    with softirq hrtimer"
  - additional fixes of several wrong comments
  - update hrtimer tracing (mode and clock bases)


 drivers/net/ethernet/marvell/mvpp2.c  |   62 +--
 drivers/net/wireless/mac80211_hwsim.c |   44 +-
 drivers/usb/gadget/function/f_ncm.c   |   30 -
 include/linux/hrtimer.h               |  113 +++--
 include/linux/interrupt.h             |   25 -
 include/net/xfrm.h                    |    2 
 include/trace/events/timer.h          |   37 +
 kernel/softirq.c                      |   51 --
 kernel/time/hrtimer.c                 |  650 +++++++++++++++++++++-------------
 kernel/time/tick-internal.h           |   13 
 kernel/time/tick-sched.c              |    2 
 kernel/time/timer.c                   |   98 ++---
 net/can/bcm.c                         |  156 ++------
 net/xfrm/xfrm_state.c                 |   30 -
 sound/drivers/dummy.c                 |   27 -
 15 files changed, 709 insertions(+), 631 deletions(-)

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v4 01/36] timers: Use static keys for migrate_enable/nohz_active
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2017-12-22 15:45   ` [PATCH v5 " Sebastian Andrzej Siewior
                     ` (2 more replies)
  2017-12-21 10:41 ` [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment Anna-Maria Gleixner
                   ` (35 subsequent siblings)
  36 siblings, 3 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

From: Thomas Gleixner <tglx@linutronix.de>

The members migrate_enable and nohz_active in the timer/hrtimer per CPU
bases have been introduced to avoid accessing global variables for these
decisions.

Still that results in a (cache hot) load and conditional branch, which can
be avoided by using static keys.

Implement it with static keys and optimize for the most critical case of
high performance networking which tends to disable the timer migration
functionality.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h     |  4 --
 kernel/time/hrtimer.c       | 17 +++------
 kernel/time/tick-internal.h | 19 ++++++----
 kernel/time/tick-sched.c    |  2 +-
 kernel/time/timer.c         | 91 +++++++++++++++++++++++----------------------
 5 files changed, 65 insertions(+), 68 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 012c37fdb688..79b2a8d29d8c 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -153,8 +153,6 @@ enum  hrtimer_base_type {
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
- * @migration_enabled:	The migration of hrtimers to other cpus is enabled
- * @nohz_active:	The nohz functionality is enabled
  * @expires_next:	absolute time of the next event which was scheduled
  *			via clock_set_next_event()
  * @next_timer:		Pointer to the first expiring timer
@@ -178,8 +176,6 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
-	bool				migration_enabled;
-	bool				nohz_active;
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
 					hres_active	: 1,
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index d32520840fde..69d203d8b12d 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -178,23 +178,16 @@ hrtimer_check_target(struct hrtimer *timer, struct hrtimer_clock_base *new_base)
 #endif
 }
 
-#ifdef CONFIG_NO_HZ_COMMON
-static inline
-struct hrtimer_cpu_base *get_target_base(struct hrtimer_cpu_base *base,
-					 int pinned)
-{
-	if (pinned || !base->migration_enabled)
-		return base;
-	return &per_cpu(hrtimer_bases, get_nohz_timer_target());
-}
-#else
 static inline
 struct hrtimer_cpu_base *get_target_base(struct hrtimer_cpu_base *base,
 					 int pinned)
 {
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+	if (static_branch_unlikely(&timers_migration_enabled) && !pinned)
+		return &per_cpu(hrtimer_bases, get_nohz_timer_target());
+#endif
 	return base;
 }
-#endif
 
 /*
  * We switch the timer base to a power-optimized selected CPU target,
@@ -969,7 +962,7 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 		 * Kick to reschedule the next tick to handle the new timer
 		 * on dynticks target.
 		 */
-		if (new_base->cpu_base->nohz_active)
+		if (is_timers_nohz_active())
 			wake_up_nohz_cpu(new_base->cpu_base->cpu);
 	} else {
 		hrtimer_reprogram(timer, new_base);
diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index f8e1845aa464..4ac74dff59f0 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -150,14 +150,19 @@ static inline void tick_nohz_init(void) { }
 
 #ifdef CONFIG_NO_HZ_COMMON
 extern unsigned long tick_nohz_active;
-#else
+extern void timers_update_nohz(void);
+extern struct static_key_false timers_nohz_active;
+static inline bool is_timers_nohz_active(void)
+{
+	return static_branch_unlikely(&timers_nohz_active);
+}
+# ifdef CONFIG_SMP
+extern struct static_key_false timers_migration_enabled;
+# endif
+#else /* CONFIG_NO_HZ_COMMON */
+static inline void timers_update_nohz(void) { }
 #define tick_nohz_active (0)
-#endif
-
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
-extern void timers_update_migration(bool update_nohz);
-#else
-static inline void timers_update_migration(bool update_nohz) { }
+static inline bool is_timers_nohz_active(void) { return false; }
 #endif
 
 DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases);
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 99578f06c8d4..f371f25f3510 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1079,7 +1079,7 @@ static inline void tick_nohz_activate(struct tick_sched *ts, int mode)
 	ts->nohz_mode = mode;
 	/* One update is enough */
 	if (!test_and_set_bit(0, &tick_nohz_active))
-		timers_update_migration(true);
+		timers_update_nohz();
 }
 
 /**
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index ffebcf878fba..1e2140a23044 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -200,8 +200,6 @@ struct timer_base {
 	unsigned long		clk;
 	unsigned long		next_expiry;
 	unsigned int		cpu;
-	bool			migration_enabled;
-	bool			nohz_active;
 	bool			is_idle;
 	bool			must_forward_clk;
 	DECLARE_BITMAP(pending_map, WHEEL_SIZE);
@@ -210,45 +208,59 @@ struct timer_base {
 
 static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]);
 
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+#ifdef CONFIG_NO_HZ_COMMON
+
+DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
+static DEFINE_MUTEX(timer_keys_mutex);
+
+static void timer_update_keys(struct work_struct *work);
+static DECLARE_WORK(timer_update_work, timer_update_keys);
+
+#ifdef CONFIG_SMP
 unsigned int sysctl_timer_migration = 1;
 
-void timers_update_migration(bool update_nohz)
+DEFINE_STATIC_KEY_FALSE(timers_migration_enabled);
+
+static void timers_update_migration(void)
 {
 	bool on = sysctl_timer_migration && tick_nohz_active;
-	unsigned int cpu;
 
-	/* Avoid the loop, if nothing to update */
-	if (this_cpu_read(timer_bases[BASE_STD].migration_enabled) == on)
-		return;
+	if (on)
+		static_branch_enable(&timers_migration_enabled);
+	else
+		static_branch_disable(&timers_migration_enabled);
+}
+#else
+static inline void timers_update_migration(void) { }
+#endif /* !CONFIG_SMP */
 
-	for_each_possible_cpu(cpu) {
-		per_cpu(timer_bases[BASE_STD].migration_enabled, cpu) = on;
-		per_cpu(timer_bases[BASE_DEF].migration_enabled, cpu) = on;
-		per_cpu(hrtimer_bases.migration_enabled, cpu) = on;
-		if (!update_nohz)
-			continue;
-		per_cpu(timer_bases[BASE_STD].nohz_active, cpu) = true;
-		per_cpu(timer_bases[BASE_DEF].nohz_active, cpu) = true;
-		per_cpu(hrtimer_bases.nohz_active, cpu) = true;
-	}
+static void timer_update_keys(struct work_struct *work)
+{
+	mutex_lock(&timer_keys_mutex);
+	timers_update_migration();
+	static_branch_enable(&timers_nohz_active);
+	mutex_unlock(&timer_keys_mutex);
+}
+
+void timers_update_nohz(void)
+{
+	schedule_work(&timer_update_work);
 }
 
 int timer_migration_handler(struct ctl_table *table, int write,
 			    void __user *buffer, size_t *lenp,
 			    loff_t *ppos)
 {
-	static DEFINE_MUTEX(mutex);
 	int ret;
 
-	mutex_lock(&mutex);
+	mutex_lock(&timer_keys_mutex);
 	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
 	if (!ret && write)
-		timers_update_migration(false);
-	mutex_unlock(&mutex);
+		timers_update_migration();
+	mutex_unlock(&timer_keys_mutex);
 	return ret;
 }
-#endif
+#endif /* NO_HZ_COMMON */
 
 static unsigned long round_jiffies_common(unsigned long j, int cpu,
 		bool force_up)
@@ -534,7 +546,7 @@ __internal_add_timer(struct timer_base *base, struct timer_list *timer)
 static void
 trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer)
 {
-	if (!IS_ENABLED(CONFIG_NO_HZ_COMMON) || !base->nohz_active)
+	if (!is_timers_nohz_active())
 		return;
 
 	/*
@@ -826,7 +838,7 @@ static inline struct timer_base *get_timer_cpu_base(u32 tflags, u32 cpu)
 	 * If the timer is deferrable and nohz is active then we need to use
 	 * the deferrable base.
 	 */
-	if (IS_ENABLED(CONFIG_NO_HZ_COMMON) && base->nohz_active &&
+	if (is_timers_nohz_active() &&
 	    (tflags & TIMER_DEFERRABLE))
 		base = per_cpu_ptr(&timer_bases[BASE_DEF], cpu);
 	return base;
@@ -840,7 +852,7 @@ static inline struct timer_base *get_timer_this_cpu_base(u32 tflags)
 	 * If the timer is deferrable and nohz is active then we need to use
 	 * the deferrable base.
 	 */
-	if (IS_ENABLED(CONFIG_NO_HZ_COMMON) && base->nohz_active &&
+	if (is_timers_nohz_active() &&
 	    (tflags & TIMER_DEFERRABLE))
 		base = this_cpu_ptr(&timer_bases[BASE_DEF]);
 	return base;
@@ -851,21 +863,20 @@ static inline struct timer_base *get_timer_base(u32 tflags)
 	return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK);
 }
 
-#ifdef CONFIG_NO_HZ_COMMON
 static inline struct timer_base *
 get_target_base(struct timer_base *base, unsigned tflags)
 {
-#ifdef CONFIG_SMP
-	if ((tflags & TIMER_PINNED) || !base->migration_enabled)
-		return get_timer_this_cpu_base(tflags);
-	return get_timer_cpu_base(tflags, get_nohz_timer_target());
-#else
-	return get_timer_this_cpu_base(tflags);
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+	if (static_branch_unlikely(&timers_migration_enabled) &&
+	    !(tflags & TIMER_PINNED))
+		return get_timer_cpu_base(tflags, get_nohz_timer_target());
 #endif
+	return get_timer_this_cpu_base(tflags);
 }
 
 static inline void forward_timer_base(struct timer_base *base)
 {
+#ifdef CONFIG_NO_HZ_COMMON
 	unsigned long jnow;
 
 	/*
@@ -889,16 +900,8 @@ static inline void forward_timer_base(struct timer_base *base)
 		base->clk = jnow;
 	else
 		base->clk = base->next_expiry;
-}
-#else
-static inline struct timer_base *
-get_target_base(struct timer_base *base, unsigned tflags)
-{
-	return get_timer_this_cpu_base(tflags);
-}
-
-static inline void forward_timer_base(struct timer_base *base) { }
 #endif
+}
 
 
 /*
@@ -1684,7 +1687,7 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h)
 	base->must_forward_clk = false;
 
 	__run_timers(base);
-	if (IS_ENABLED(CONFIG_NO_HZ_COMMON) && base->nohz_active)
+	if (is_timers_nohz_active())
 		__run_timers(this_cpu_ptr(&timer_bases[BASE_DEF]));
 }
 
@@ -1698,7 +1701,7 @@ void run_local_timers(void)
 	hrtimer_run_queues();
 	/* Raise the softirq only if required. */
 	if (time_before(jiffies, base->clk)) {
-		if (!IS_ENABLED(CONFIG_NO_HZ_COMMON) || !base->nohz_active)
+		if (!is_timers_nohz_active())
 			return;
 		/* CPU is awake, so check the deferrable base. */
 		base++;
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 01/36] timers: Use static keys for migrate_enable/nohz_active Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-11 18:28   ` Frederic Weisbecker
                     ` (2 more replies)
  2017-12-21 10:41 ` [PATCH v4 03/36] hrtimer: Fix kerneldoc for struct hrtimer_cpu_base Anna-Maria Gleixner
                   ` (34 subsequent siblings)
  36 siblings, 3 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

From: Thomas Gleixner <tglx@linutronix.de>

The protection of a hrtimer which runs its callback against migration to a
different CPU has nothing to do with hard interrupt context.

The protection against migration of a hrtimer running the expiry callback
is the pointer in the cpu_base which holds a pointer to the currently
running timer. This pointer is evaluated in the code which potentially
switches the timer base and makes sure it's kept on the CPU on which the
callback is running.

Reported-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 69d203d8b12d..aee49c0c58b9 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1195,9 +1195,9 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 		timer->is_rel = false;
 
 	/*
-	 * Because we run timers from hardirq context, there is no chance
-	 * they get migrated to another cpu, therefore its safe to unlock
-	 * the timer base.
+	 * The timer is marked as running in the cpu base, so it is
+	 * protected against migration to a different CPU even if the lock
+	 * is dropped.
 	 */
 	raw_spin_unlock(&cpu_base->lock);
 	trace_hrtimer_expire_entry(timer, now);
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 03/36] hrtimer: Fix kerneldoc for struct hrtimer_cpu_base
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 01/36] timers: Use static keys for migrate_enable/nohz_active Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:41   ` [tip:timers/core] hrtimer: Fix kerneldoc syntax for 'struct hrtimer_cpu_base' tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 04/36] hrtimer: Cleanup clock argument in schedule_hrtimeout_range_clock() Anna-Maria Gleixner
                   ` (33 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

The sequence '/**' marks the start of a struct description. Add the
missing second asterisk. While at it adapt the ordering of the struct
members to the struct definition and document the purpose of
expires_next more precisely.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 79b2a8d29d8c..b3a382be8db0 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -144,7 +144,7 @@ enum  hrtimer_base_type {
 	HRTIMER_MAX_CLOCK_BASES,
 };
 
-/*
+/**
  * struct hrtimer_cpu_base - the per cpu clock bases
  * @lock:		lock protecting the base and associated clock bases
  *			and timers
@@ -153,12 +153,12 @@ enum  hrtimer_base_type {
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
- * @expires_next:	absolute time of the next event which was scheduled
- *			via clock_set_next_event()
- * @next_timer:		Pointer to the first expiring timer
  * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hres_active:	State of high resolution mode
  * @hang_detected:	The last hrtimer interrupt detected a hang
+ * @expires_next:	absolute time of the next event, is required for remote
+ *			hrtimer enqueue
+ * @next_timer:		Pointer to the first expiring timer
  * @nr_events:		Total number of hrtimer interrupt events
  * @nr_retries:		Total number of hrtimer interrupt retries
  * @nr_hangs:		Total number of hrtimer interrupt hangs
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 04/36] hrtimer: Cleanup clock argument in schedule_hrtimeout_range_clock()
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (2 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 03/36] hrtimer: Fix kerneldoc for struct hrtimer_cpu_base Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:41   ` [tip:timers/core] hrtimer: Clean up the 'int clock' parameter of schedule_hrtimeout_range_clock() tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 05/36] hrtimer: Fix hrtimer function description Anna-Maria Gleixner
                   ` (32 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

schedule_hrtimeout_range_clock() uses an integer for the clock id
instead of the predefined type "clockid_t". The ID of the clock is
indicated in hrtimer code as clock_id. Therefore change the name of
the variable as well to make it consistent.

While at it, clean up the description for the function parameters clock_id
and mode. The clock modes and the clock ids are not restricted as the
comment suggests. Fix the mode description as well for the callers of
schedule_hrtimeout_range_clock().

No functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h |  2 +-
 kernel/time/hrtimer.c   | 12 ++++++------
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index b3a382be8db0..931ce9c89c93 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -462,7 +462,7 @@ extern int schedule_hrtimeout_range(ktime_t *expires, u64 delta,
 extern int schedule_hrtimeout_range_clock(ktime_t *expires,
 					  u64 delta,
 					  const enum hrtimer_mode mode,
-					  int clock);
+					  clockid_t clock_id);
 extern int schedule_hrtimeout(ktime_t *expires, const enum hrtimer_mode mode);
 
 /* Soft interrupt function to run the hrtimer queues: */
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index aee49c0c58b9..74a64db375d1 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1662,12 +1662,12 @@ void __init hrtimers_init(void)
  * schedule_hrtimeout_range_clock - sleep until timeout
  * @expires:	timeout value (ktime_t)
  * @delta:	slack in expires timeout (ktime_t)
- * @mode:	timer mode, HRTIMER_MODE_ABS or HRTIMER_MODE_REL
- * @clock:	timer clock, CLOCK_MONOTONIC or CLOCK_REALTIME
+ * @mode:	timer mode
+ * @clock_id:	timer clock to be used
  */
 int __sched
 schedule_hrtimeout_range_clock(ktime_t *expires, u64 delta,
-			       const enum hrtimer_mode mode, int clock)
+			       const enum hrtimer_mode mode, clockid_t clock_id)
 {
 	struct hrtimer_sleeper t;
 
@@ -1688,7 +1688,7 @@ schedule_hrtimeout_range_clock(ktime_t *expires, u64 delta,
 		return -EINTR;
 	}
 
-	hrtimer_init_on_stack(&t.timer, clock, mode);
+	hrtimer_init_on_stack(&t.timer, clock_id, mode);
 	hrtimer_set_expires_range_ns(&t.timer, *expires, delta);
 
 	hrtimer_init_sleeper(&t, current);
@@ -1710,7 +1710,7 @@ schedule_hrtimeout_range_clock(ktime_t *expires, u64 delta,
  * schedule_hrtimeout_range - sleep until timeout
  * @expires:	timeout value (ktime_t)
  * @delta:	slack in expires timeout (ktime_t)
- * @mode:	timer mode, HRTIMER_MODE_ABS or HRTIMER_MODE_REL
+ * @mode:	timer mode
  *
  * Make the current task sleep until the given expiry time has
  * elapsed. The routine will return immediately unless
@@ -1749,7 +1749,7 @@ EXPORT_SYMBOL_GPL(schedule_hrtimeout_range);
 /**
  * schedule_hrtimeout - sleep until timeout
  * @expires:	timeout value (ktime_t)
- * @mode:	timer mode, HRTIMER_MODE_ABS or HRTIMER_MODE_REL
+ * @mode:	timer mode
  *
  * Make the current task sleep until the given expiry time has
  * elapsed. The routine will return immediately unless
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 05/36] hrtimer: Fix hrtimer function description
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (3 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 04/36] hrtimer: Cleanup clock argument in schedule_hrtimeout_range_clock() Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:42   ` [tip:timers/core] hrtimer: Fix hrtimer_start[_range_ns]() function descriptions tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 06/36] hrtimer: Ensure POSIX compliance (relative CLOCK_REALTIME hrtimers) Anna-Maria Gleixner
                   ` (31 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

The hrtimer_start[_range_ns]() starts a timer reliable on this CPU only
when HRTIMER_MODE_PINNED is set. Furthermore the HRTIMER_MODE_PINNED mode
is not considered, when a hrtimer is initialized.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h | 6 +++---
 kernel/time/hrtimer.c   | 9 +++++----
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 931ce9c89c93..4e6a8841dcbe 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -361,11 +361,11 @@ extern void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 				   u64 range_ns, const enum hrtimer_mode mode);
 
 /**
- * hrtimer_start - (re)start an hrtimer on the current CPU
+ * hrtimer_start - (re)start an hrtimer
  * @timer:	the timer to be added
  * @tim:	expiry time
- * @mode:	expiry mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL)
+ * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
  */
 static inline void hrtimer_start(struct hrtimer *timer, ktime_t tim,
 				 const enum hrtimer_mode mode)
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 74a64db375d1..ae7b29c4e03e 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -924,12 +924,12 @@ static inline ktime_t hrtimer_update_lowres(struct hrtimer *timer, ktime_t tim,
 }
 
 /**
- * hrtimer_start_range_ns - (re)start an hrtimer on the current CPU
+ * hrtimer_start_range_ns - (re)start an hrtimer
  * @timer:	the timer to be added
  * @tim:	expiry time
  * @delta_ns:	"slack" range for the timer
- * @mode:	expiry mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL)
+ * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
  */
 void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 			    u64 delta_ns, const enum hrtimer_mode mode)
@@ -1107,7 +1107,8 @@ static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
  * hrtimer_init - initialize a timer to the given clock
  * @timer:	the timer to be initialized
  * @clock_id:	the clock to be used
- * @mode:	timer mode abs/rel
+ * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
+ *		relative (HRTIMER_MODE_REL); pinned is not considered here!
  */
 void hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 		  enum hrtimer_mode mode)
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 06/36] hrtimer: Ensure POSIX compliance (relative CLOCK_REALTIME hrtimers)
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (4 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 05/36] hrtimer: Fix hrtimer function description Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:42   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 07/36] hrtimer: Cleanup hrtimer_mode enum Anna-Maria Gleixner
                   ` (30 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

POSIX specification defines, that relative CLOCK_REALTIME timers are not
affected by clock modifications. Those timers have to use CLOCK_MONOTONIC
to ensure POSIX compliance.

The introduction of the additional mode HRTIMER_MODE_PINNED broke this
requirement for pinned timers. There is no user space visible impact
because user space timers are not using the pinned mode, but for
consistency reasons this needs to be fixed.

Check whether the mode has the HRTIMER_MODE_REL bit set instead of
comparing with HRTIMER_MODE_ABS.


Fixes: 597d0275736d ("timers: Framework for identifying pinned timers")
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index ae7b29c4e03e..9945ea6b0e5c 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1095,7 +1095,12 @@ static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 
 	cpu_base = raw_cpu_ptr(&hrtimer_bases);
 
-	if (clock_id == CLOCK_REALTIME && mode != HRTIMER_MODE_ABS)
+	/*
+	 * Posix magic: Relative CLOCK_REALTIME timers are not affected by
+	 * clock modifications, so they needs to become CLOCK_MONOTONIC to
+	 * ensure Posix compliance.
+	 */
+	if (clock_id == CLOCK_REALTIME && mode & HRTIMER_MODE_REL)
 		clock_id = CLOCK_MONOTONIC;
 
 	base = hrtimer_clockid_to_base(clock_id);
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 07/36] hrtimer: Cleanup hrtimer_mode enum
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (5 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 06/36] hrtimer: Ensure POSIX compliance (relative CLOCK_REALTIME hrtimers) Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:43   ` [tip:timers/core] hrtimer: Clean up 'enum hrtimer_mode' tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 08/36] tracing/hrtimer: Take all clock bases and modes into account Anna-Maria Gleixner
                   ` (29 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

It's not obvious that the HRTIMER_MODE variants are bit combinations
because all modes are hard coded constants.

Change it so the bit meanings are clear and use the symbols for creating
modes which combine bits.

While at it get rid of the ugly tail comments.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 4e6a8841dcbe..28f267cf2851 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -28,13 +28,19 @@ struct hrtimer_cpu_base;
 
 /*
  * Mode arguments of xxx_hrtimer functions:
+ *
+ * HRTIMER_MODE_ABS		- Time value is absolute
+ * HRTIMER_MODE_REL		- Time value is relative to now
+ * HRTIMER_MODE_PINNED		- Timer is bound to CPU (is only considered
+ *				  when starting the timer)
  */
 enum hrtimer_mode {
-	HRTIMER_MODE_ABS = 0x0,		/* Time value is absolute */
-	HRTIMER_MODE_REL = 0x1,		/* Time value is relative to now */
-	HRTIMER_MODE_PINNED = 0x02,	/* Timer is bound to CPU */
-	HRTIMER_MODE_ABS_PINNED = 0x02,
-	HRTIMER_MODE_REL_PINNED = 0x03,
+	HRTIMER_MODE_ABS	= 0x00,
+	HRTIMER_MODE_REL	= 0x01,
+	HRTIMER_MODE_PINNED	= 0x02,
+
+	HRTIMER_MODE_ABS_PINNED = HRTIMER_MODE_ABS | HRTIMER_MODE_PINNED,
+	HRTIMER_MODE_REL_PINNED = HRTIMER_MODE_REL | HRTIMER_MODE_PINNED,
 };
 
 /*
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 08/36] tracing/hrtimer: Take all clock bases and modes into account
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (6 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 07/36] hrtimer: Cleanup hrtimer_mode enum Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:43   ` [tip:timers/core] tracing/hrtimer: Fix tracing bugs by taking " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 09/36] tracing/hrtimer: Print hrtimer mode in hrtimer_start tracepoint Anna-Maria Gleixner
                   ` (28 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

So far only CLOCK_MONOTONIC and CLOCK_REALTIME were taken into account as
well as HRTIMER_MODE_ABS/REL in hrtimer_init tracepoint. The query for
detecting timer mode ABS or REL is not valid, since the introduction of
HRTIMER_MODE_PINNED.

HRTIMER_MODE_PINNED is not evaluated in hrtimer_init() call. But for the
sake of completeness print all given modes.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/trace/events/timer.h | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h
index 16e305e69f34..c6f728037c53 100644
--- a/include/trace/events/timer.h
+++ b/include/trace/events/timer.h
@@ -136,6 +136,20 @@ DEFINE_EVENT(timer_class, timer_cancel,
 	TP_ARGS(timer)
 );
 
+#define decode_clockid(type)						\
+	__print_symbolic(type,						\
+		{ CLOCK_REALTIME,	"CLOCK_REALTIME"	},	\
+		{ CLOCK_MONOTONIC,	"CLOCK_MONOTONIC"	},	\
+		{ CLOCK_BOOTTIME,	"CLOCK_BOOTTIME"	},	\
+		{ CLOCK_TAI,		"CLOCK_TAI"		})
+
+#define decode_hrtimer_mode(mode)					\
+	__print_symbolic(mode,						\
+		{ HRTIMER_MODE_ABS,		"ABS"		},	\
+		{ HRTIMER_MODE_REL,		"REL"		},	\
+		{ HRTIMER_MODE_ABS_PINNED,	"ABS|PINNED"	},	\
+		{ HRTIMER_MODE_REL_PINNED,	"REL|PINNED"	})
+
 /**
  * hrtimer_init - called when the hrtimer is initialized
  * @hrtimer:	pointer to struct hrtimer
@@ -162,10 +176,8 @@ TRACE_EVENT(hrtimer_init,
 	),
 
 	TP_printk("hrtimer=%p clockid=%s mode=%s", __entry->hrtimer,
-		  __entry->clockid == CLOCK_REALTIME ?
-			"CLOCK_REALTIME" : "CLOCK_MONOTONIC",
-		  __entry->mode == HRTIMER_MODE_ABS ?
-			"HRTIMER_MODE_ABS" : "HRTIMER_MODE_REL")
+		  decode_clockid(__entry->clockid),
+		  decode_hrtimer_mode(__entry->mode))
 );
 
 /**
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 09/36] tracing/hrtimer: Print hrtimer mode in hrtimer_start tracepoint
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (7 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 08/36] tracing/hrtimer: Take all clock bases and modes into account Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:43   ` [tip:timers/core] tracing/hrtimer: Print the hrtimer mode in the 'hrtimer_start' tracepoint tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 10/36] hrtimer: Switch for loop to _ffs() evaluation Anna-Maria Gleixner
                   ` (27 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

The hrtimer_start tracepoint lacks the mode information. The mode is
important because consecutive starts can switch from ABS to REL or from
PINNED to non PINNED.

Add the mode information.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/trace/events/timer.h | 13 ++++++++-----
 kernel/time/hrtimer.c        | 16 +++++++++-------
 2 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h
index c6f728037c53..744b4310b24b 100644
--- a/include/trace/events/timer.h
+++ b/include/trace/events/timer.h
@@ -186,15 +186,16 @@ TRACE_EVENT(hrtimer_init,
  */
 TRACE_EVENT(hrtimer_start,
 
-	TP_PROTO(struct hrtimer *hrtimer),
+	TP_PROTO(struct hrtimer *hrtimer, enum hrtimer_mode mode),
 
-	TP_ARGS(hrtimer),
+	TP_ARGS(hrtimer, mode),
 
 	TP_STRUCT__entry(
 		__field( void *,	hrtimer		)
 		__field( void *,	function	)
 		__field( s64,		expires		)
 		__field( s64,		softexpires	)
+		__field( enum hrtimer_mode,	mode	)
 	),
 
 	TP_fast_assign(
@@ -202,12 +203,14 @@ TRACE_EVENT(hrtimer_start,
 		__entry->function	= hrtimer->function;
 		__entry->expires	= hrtimer_get_expires(hrtimer);
 		__entry->softexpires	= hrtimer_get_softexpires(hrtimer);
+		__entry->mode		= mode;
 	),
 
-	TP_printk("hrtimer=%p function=%pf expires=%llu softexpires=%llu",
-		  __entry->hrtimer, __entry->function,
+	TP_printk("hrtimer=%p function=%pf expires=%llu softexpires=%llu "
+		  "mode=%s", __entry->hrtimer, __entry->function,
 		  (unsigned long long) __entry->expires,
-		  (unsigned long long) __entry->softexpires)
+		  (unsigned long long) __entry->softexpires,
+		  decode_hrtimer_mode(__entry->mode))
 );
 
 /**
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 9945ea6b0e5c..0e000b58f0d3 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -435,10 +435,11 @@ debug_init(struct hrtimer *timer, clockid_t clockid,
 	trace_hrtimer_init(timer, clockid, mode);
 }
 
-static inline void debug_activate(struct hrtimer *timer)
+static inline void debug_activate(struct hrtimer *timer,
+				  enum hrtimer_mode mode)
 {
 	debug_hrtimer_activate(timer);
-	trace_hrtimer_start(timer);
+	trace_hrtimer_start(timer, mode);
 }
 
 static inline void debug_deactivate(struct hrtimer *timer)
@@ -828,9 +829,10 @@ EXPORT_SYMBOL_GPL(hrtimer_forward);
  * Returns 1 when the new timer is the leftmost timer in the tree.
  */
 static int enqueue_hrtimer(struct hrtimer *timer,
-			   struct hrtimer_clock_base *base)
+			   struct hrtimer_clock_base *base,
+			   enum hrtimer_mode mode)
 {
-	debug_activate(timer);
+	debug_activate(timer, mode);
 
 	base->cpu_base->active_bases |= 1 << base->index;
 
@@ -953,7 +955,7 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	/* Switch the timer base, if necessary: */
 	new_base = switch_hrtimer_base(timer, base, mode & HRTIMER_MODE_PINNED);
 
-	leftmost = enqueue_hrtimer(timer, new_base);
+	leftmost = enqueue_hrtimer(timer, new_base, mode);
 	if (!leftmost)
 		goto unlock;
 
@@ -1222,7 +1224,7 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	 */
 	if (restart != HRTIMER_NORESTART &&
 	    !(timer->state & HRTIMER_STATE_ENQUEUED))
-		enqueue_hrtimer(timer, base);
+		enqueue_hrtimer(timer, base, HRTIMER_MODE_ABS);
 
 	/*
 	 * Separate the ->running assignment from the ->state assignment.
@@ -1621,7 +1623,7 @@ static void migrate_hrtimer_list(struct hrtimer_clock_base *old_base,
 		 * sort out already expired timers and reprogram the
 		 * event device.
 		 */
-		enqueue_hrtimer(timer, new_base);
+		enqueue_hrtimer(timer, new_base, HRTIMER_MODE_ABS);
 	}
 }
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 10/36] hrtimer: Switch for loop to _ffs() evaluation
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (8 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 09/36] tracing/hrtimer: Print hrtimer mode in hrtimer_start tracepoint Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:44   ` [tip:timers/core] hrtimer: Switch 'for' " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 11/36] hrtimer: Store running timer in hrtimer_clock_base Anna-Maria Gleixner
                   ` (26 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

Looping over all clock bases to find active bits is suboptimal if not all
bases are active.

Avoid this by converting it to a __ffs() evaluation. The functionallity is
outsourced into an own function and is called via a macro as suggested by
Peter Zijlstra.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 0e000b58f0d3..43e4f6607f9d 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -448,6 +448,23 @@ static inline void debug_deactivate(struct hrtimer *timer)
 	trace_hrtimer_cancel(timer);
 }
 
+static struct hrtimer_clock_base *
+__next_base(struct hrtimer_cpu_base *cpu_base, unsigned int *active)
+{
+	unsigned int idx;
+
+	if (!*active)
+		return NULL;
+
+	idx = __ffs(*active);
+	*active &= ~(1U << idx);
+
+	return &cpu_base->clock_base[idx];
+}
+
+#define for_each_active_base(base, cpu_base, active)	\
+	while ((base = __next_base((cpu_base), &(active))))
+
 #if defined(CONFIG_NO_HZ_COMMON) || defined(CONFIG_HIGH_RES_TIMERS)
 static inline void hrtimer_update_next_timer(struct hrtimer_cpu_base *cpu_base,
 					     struct hrtimer *timer)
@@ -459,18 +476,15 @@ static inline void hrtimer_update_next_timer(struct hrtimer_cpu_base *cpu_base,
 
 static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 {
-	struct hrtimer_clock_base *base = cpu_base->clock_base;
+	struct hrtimer_clock_base *base;
 	unsigned int active = cpu_base->active_bases;
 	ktime_t expires, expires_next = KTIME_MAX;
 
 	hrtimer_update_next_timer(cpu_base, NULL);
-	for (; active; base++, active >>= 1) {
+	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *next;
 		struct hrtimer *timer;
 
-		if (!(active & 0x01))
-			continue;
-
 		next = timerqueue_getnext(&base->active);
 		timer = container_of(next, struct hrtimer, node);
 		expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
@@ -1241,16 +1255,13 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 
 static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now)
 {
-	struct hrtimer_clock_base *base = cpu_base->clock_base;
+	struct hrtimer_clock_base *base;
 	unsigned int active = cpu_base->active_bases;
 
-	for (; active; base++, active >>= 1) {
+	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *node;
 		ktime_t basenow;
 
-		if (!(active & 0x01))
-			continue;
-
 		basenow = ktime_add(now, base->offset);
 
 		while ((node = timerqueue_getnext(&base->active))) {
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 11/36] hrtimer: Store running timer in hrtimer_clock_base
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (9 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 10/36] hrtimer: Switch for loop to _ffs() evaluation Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:44   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 12/36] hrtimer: Make room in struct hrtimer_cpu_base Anna-Maria Gleixner
                   ` (25 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

The pointer to the currently running timer is stored in hrtimer_cpu_base
before the base lock is dropped and the callback is invoked.

This results in two levels of indirections and the upcoming support for
softirq based hrtimer requires splitting the "running" storage into soft
and hard irq context expiry.

Storing both in the cpu base would require conditionals in all code paths
accessing that information.

It's possible to have a per clock base sequence count and running pointer
without changing the semantics of the related mechanisms because the timer
base pointer cannot be changed while a timer is running the callback.

Unfortunately this makes cpu_clock base larger than 32 bytes on 32bit
kernels. Instead of having huge gaps due to alignment, remove the alignment
and let the compiler pack cpu base for 32bit. The resulting cache access
patterns are fortunately not really different from the current
behaviour. On 64bit kernels the 64byte alignment stays and the behaviour is
unchanged. This was determined by analyzing the resulting layout and
looking at the number of cache lines involved for the frequently used
clocks.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h | 20 +++++++++-----------
 kernel/time/hrtimer.c   | 28 +++++++++++++---------------
 2 files changed, 22 insertions(+), 26 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 28f267cf2851..1bae7b9f071d 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -118,9 +118,9 @@ struct hrtimer_sleeper {
 };
 
 #ifdef CONFIG_64BIT
-# define HRTIMER_CLOCK_BASE_ALIGN	64
+# define __hrtimer_clock_base_align	____cacheline_aligned
 #else
-# define HRTIMER_CLOCK_BASE_ALIGN	32
+# define __hrtimer_clock_base_align
 #endif
 
 /**
@@ -129,18 +129,22 @@ struct hrtimer_sleeper {
  * @index:		clock type index for per_cpu support when moving a
  *			timer to a base on another cpu.
  * @clockid:		clock id for per_cpu support
+ * @seq:		seqcount around __run_hrtimer
+ * @running:		pointer to the currently running hrtimer
  * @active:		red black tree root node for the active timers
  * @get_time:		function to retrieve the current time of the clock
  * @offset:		offset of this clock to the monotonic base
  */
 struct hrtimer_clock_base {
 	struct hrtimer_cpu_base	*cpu_base;
-	int			index;
+	unsigned int		index;
 	clockid_t		clockid;
+	seqcount_t		seq;
+	struct hrtimer		*running;
 	struct timerqueue_head	active;
 	ktime_t			(*get_time)(void);
 	ktime_t			offset;
-} __attribute__((__aligned__(HRTIMER_CLOCK_BASE_ALIGN)));
+} __hrtimer_clock_base_align;
 
 enum  hrtimer_base_type {
 	HRTIMER_BASE_MONOTONIC,
@@ -154,8 +158,6 @@ enum  hrtimer_base_type {
  * struct hrtimer_cpu_base - the per cpu clock bases
  * @lock:		lock protecting the base and associated clock bases
  *			and timers
- * @seq:		seqcount around __run_hrtimer
- * @running:		pointer to the currently running hrtimer
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
@@ -177,8 +179,6 @@ enum  hrtimer_base_type {
  */
 struct hrtimer_cpu_base {
 	raw_spinlock_t			lock;
-	seqcount_t			seq;
-	struct hrtimer			*running;
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
@@ -198,8 +198,6 @@ struct hrtimer_cpu_base {
 
 static inline void hrtimer_set_expires(struct hrtimer *timer, ktime_t time)
 {
-	BUILD_BUG_ON(sizeof(struct hrtimer_clock_base) > HRTIMER_CLOCK_BASE_ALIGN);
-
 	timer->node.expires = time;
 	timer->_softexpires = time;
 }
@@ -424,7 +422,7 @@ static inline int hrtimer_is_queued(struct hrtimer *timer)
  */
 static inline int hrtimer_callback_running(struct hrtimer *timer)
 {
-	return timer->base->cpu_base->running == timer;
+	return timer->base->running == timer;
 }
 
 /* Forward a hrtimer so it expires after now: */
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 43e4f6607f9d..80e7c15af877 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -70,7 +70,6 @@
 DEFINE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases) =
 {
 	.lock = __RAW_SPIN_LOCK_UNLOCKED(hrtimer_bases.lock),
-	.seq = SEQCNT_ZERO(hrtimer_bases.seq),
 	.clock_base =
 	{
 		{
@@ -118,7 +117,6 @@ static const int hrtimer_clock_to_base_table[MAX_CLOCKS] = {
  * timer->base->cpu_base
  */
 static struct hrtimer_cpu_base migration_cpu_base = {
-	.seq = SEQCNT_ZERO(migration_cpu_base),
 	.clock_base = { { .cpu_base = &migration_cpu_base, }, },
 };
 
@@ -1148,19 +1146,19 @@ EXPORT_SYMBOL_GPL(hrtimer_init);
  */
 bool hrtimer_active(const struct hrtimer *timer)
 {
-	struct hrtimer_cpu_base *cpu_base;
+	struct hrtimer_clock_base *base;
 	unsigned int seq;
 
 	do {
-		cpu_base = READ_ONCE(timer->base->cpu_base);
-		seq = raw_read_seqcount_begin(&cpu_base->seq);
+		base = READ_ONCE(timer->base);
+		seq = raw_read_seqcount_begin(&base->seq);
 
 		if (timer->state != HRTIMER_STATE_INACTIVE ||
-		    cpu_base->running == timer)
+		    base->running == timer)
 			return true;
 
-	} while (read_seqcount_retry(&cpu_base->seq, seq) ||
-		 cpu_base != READ_ONCE(timer->base->cpu_base));
+	} while (read_seqcount_retry(&base->seq, seq) ||
+		 base != READ_ONCE(timer->base));
 
 	return false;
 }
@@ -1194,16 +1192,16 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	lockdep_assert_held(&cpu_base->lock);
 
 	debug_deactivate(timer);
-	cpu_base->running = timer;
+	base->running = timer;
 
 	/*
 	 * Separate the ->running assignment from the ->state assignment.
 	 *
 	 * As with a regular write barrier, this ensures the read side in
-	 * hrtimer_active() cannot observe cpu_base->running == NULL &&
+	 * hrtimer_active() cannot observe base->running == NULL &&
 	 * timer->state == INACTIVE.
 	 */
-	raw_write_seqcount_barrier(&cpu_base->seq);
+	raw_write_seqcount_barrier(&base->seq);
 
 	__remove_hrtimer(timer, base, HRTIMER_STATE_INACTIVE, 0);
 	fn = timer->function;
@@ -1244,13 +1242,13 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	 * Separate the ->running assignment from the ->state assignment.
 	 *
 	 * As with a regular write barrier, this ensures the read side in
-	 * hrtimer_active() cannot observe cpu_base->running == NULL &&
+	 * hrtimer_active() cannot observe base->running.timer == NULL &&
 	 * timer->state == INACTIVE.
 	 */
-	raw_write_seqcount_barrier(&cpu_base->seq);
+	raw_write_seqcount_barrier(&base->seq);
 
-	WARN_ON_ONCE(cpu_base->running != timer);
-	cpu_base->running = NULL;
+	WARN_ON_ONCE(base->running != timer);
+	base->running = NULL;
 }
 
 static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now)
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 12/36] hrtimer: Make room in struct hrtimer_cpu_base
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (10 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 11/36] hrtimer: Store running timer in hrtimer_clock_base Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:45   ` [tip:timers/core] hrtimer: Make room in 'struct hrtimer_cpu_base' tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 13/36] hrtimer: Reduce conditional code (hres_active) Anna-Maria Gleixner
                   ` (24 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

The upcoming softirq based hrtimers support requires an additional field in
the hrtimer_cpu_base struct, which would grow the struct size beyond a
cache line.

The struct members nr_retries and nr_hangs of hrtimer_cpu_base are solely
used for diagnostic output and have no requirement to be unsigned int.

Make them unsigned short to create room for the new struct member. No
functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 1bae7b9f071d..56e56bcb6f0f 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -189,8 +189,8 @@ struct hrtimer_cpu_base {
 	ktime_t				expires_next;
 	struct hrtimer			*next_timer;
 	unsigned int			nr_events;
-	unsigned int			nr_retries;
-	unsigned int			nr_hangs;
+	unsigned short			nr_retries;
+	unsigned short			nr_hangs;
 	unsigned int			max_hang_time;
 #endif
 	struct hrtimer_clock_base	clock_base[HRTIMER_MAX_CLOCK_BASES];
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 13/36] hrtimer: Reduce conditional code (hres_active)
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (11 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 12/36] hrtimer: Make room in struct hrtimer_cpu_base Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:45   ` [tip:timers/core] hrtimer: Make the hrtimer_cpu_base::hres_active field unconditional, to simplify the code tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 14/36] hrtimer: Use accesor functions instead of direct access Anna-Maria Gleixner
                   ` (23 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

The hrtimer_cpu_base struct has the CONFIG_HIGH_RES_TIMERS conditional
struct member hres_active. All related functions to this member are
conditional as well.

There is no functional change, when the hres_active member is
unconditional with all related functions and is set to zero during
initialization.

The conditional code sections can be avoided by adding IS_ENABLED(HIGHRES)
conditionals into common functions, which ensures dead code elimination.

Suggested-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h | 20 ++++++++------------
 kernel/time/hrtimer.c   | 31 +++++++++++++++----------------
 2 files changed, 23 insertions(+), 28 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 56e56bcb6f0f..22627b3a33fe 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -161,8 +161,8 @@ enum  hrtimer_base_type {
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
- * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hres_active:	State of high resolution mode
+ * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hang_detected:	The last hrtimer interrupt detected a hang
  * @expires_next:	absolute time of the next event, is required for remote
  *			hrtimer enqueue
@@ -182,9 +182,9 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
+	unsigned int			hres_active	: 1;
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
-					hres_active	: 1,
 					hang_detected	: 1;
 	ktime_t				expires_next;
 	struct hrtimer			*next_timer;
@@ -266,16 +266,17 @@ static inline ktime_t hrtimer_cb_get_time(struct hrtimer *timer)
 	return timer->base->get_time();
 }
 
+static inline int hrtimer_is_hres_active(struct hrtimer *timer)
+{
+	return IS_ENABLED(CONFIG_HIGH_RES_TIMERS) ?
+		timer->base->cpu_base->hres_active : 0;
+}
+
 #ifdef CONFIG_HIGH_RES_TIMERS
 struct clock_event_device;
 
 extern void hrtimer_interrupt(struct clock_event_device *dev);
 
-static inline int hrtimer_is_hres_active(struct hrtimer *timer)
-{
-	return timer->base->cpu_base->hres_active;
-}
-
 /*
  * The resolution of the clocks. The resolution value is returned in
  * the clock_getres() system call to give application programmers an
@@ -298,11 +299,6 @@ extern unsigned int hrtimer_resolution;
 
 #define hrtimer_resolution	(unsigned int)LOW_RES_NSEC
 
-static inline int hrtimer_is_hres_active(struct hrtimer *timer)
-{
-	return 0;
-}
-
 static inline void clock_was_set_delayed(void) { }
 
 #endif
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 80e7c15af877..ccfe340511a0 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -512,6 +512,20 @@ static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base)
 					    offs_real, offs_boot, offs_tai);
 }
 
+/*
+ * Is the high resolution mode active ?
+ */
+static inline int __hrtimer_hres_active(struct hrtimer_cpu_base *cpu_base)
+{
+	return IS_ENABLED(CONFIG_HIGH_RES_TIMERS) ?
+		cpu_base->hres_active : 0;
+}
+
+static inline int hrtimer_hres_active(void)
+{
+	return __hrtimer_hres_active(this_cpu_ptr(&hrtimer_bases));
+}
+
 /* High resolution timer related functions */
 #ifdef CONFIG_HIGH_RES_TIMERS
 
@@ -541,19 +555,6 @@ static inline int hrtimer_is_hres_enabled(void)
 }
 
 /*
- * Is the high resolution mode active ?
- */
-static inline int __hrtimer_hres_active(struct hrtimer_cpu_base *cpu_base)
-{
-	return cpu_base->hres_active;
-}
-
-static inline int hrtimer_hres_active(void)
-{
-	return __hrtimer_hres_active(this_cpu_ptr(&hrtimer_bases));
-}
-
-/*
  * Reprogram the event source with checking both queues for the
  * next event
  * Called with interrupts disabled and base->lock held
@@ -661,7 +662,6 @@ static void hrtimer_reprogram(struct hrtimer *timer,
 static inline void hrtimer_init_hres(struct hrtimer_cpu_base *base)
 {
 	base->expires_next = KTIME_MAX;
-	base->hres_active = 0;
 }
 
 /*
@@ -720,8 +720,6 @@ void clock_was_set_delayed(void)
 
 #else
 
-static inline int __hrtimer_hres_active(struct hrtimer_cpu_base *b) { return 0; }
-static inline int hrtimer_hres_active(void) { return 0; }
 static inline int hrtimer_is_hres_enabled(void) { return 0; }
 static inline void hrtimer_switch_to_hres(void) { }
 static inline void
@@ -1600,6 +1598,7 @@ int hrtimers_prepare_cpu(unsigned int cpu)
 	}
 
 	cpu_base->cpu = cpu;
+	cpu_base->hres_active = 0;
 	hrtimer_init_hres(cpu_base);
 	return 0;
 }
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 14/36] hrtimer: Use accesor functions instead of direct access
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (12 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 13/36] hrtimer: Reduce conditional code (hres_active) Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:46   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 15/36] hrtimer: Make the remote enqueue check unconditional Anna-Maria Gleixner
                   ` (22 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

__hrtimer_hres_active() is now available unconditionally. Replace the
direct access to hrtimer_cpu_base.hres_active.

No functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index ccfe340511a0..0ed86146096a 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -564,7 +564,7 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 {
 	ktime_t expires_next;
 
-	if (!cpu_base->hres_active)
+	if (!__hrtimer_hres_active(cpu_base))
 		return;
 
 	expires_next = __hrtimer_get_next_event(cpu_base);
@@ -673,7 +673,7 @@ static void retrigger_next_event(void *arg)
 {
 	struct hrtimer_cpu_base *base = this_cpu_ptr(&hrtimer_bases);
 
-	if (!base->hres_active)
+	if (!__hrtimer_hres_active(base))
 		return;
 
 	raw_spin_lock(&base->lock);
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 15/36] hrtimer: Make the remote enqueue check unconditional
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (13 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 14/36] hrtimer: Use accesor functions instead of direct access Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:46   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 16/36] hrtimer: Make hrtimer_cpu_base.next_timer handling unconditional Anna-Maria Gleixner
                   ` (21 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

hrtimer_cpu_base.expires_next is used to cache the next event armed in the
timer hardware. The value is used to check whether an hrtimer can be
enqueued remotely. If the new hrtimer is expiring before expires_next, then
remote enqueue is not possible as the remote hrtimer hardware cannot be
accessed for reprogramming to an earlier expiry time.

The remote enqueue check is currently conditional on
CONFIG_HIGH_RES_TIMERS=y and hrtimer_cpu_base.hres_active. There is no
compelling reason to make this conditional.

Move hrtimer_cpu_base.expires_next out of the CONFIG_HIGH_RES_TIMERS=y
guarded area and remove the conditionals in hrtimer_check_target().

The check is currently a NOOP for the CONFIG_HIGH_RES_TIMERS=n and the
!hrtimer_cpu_base.hres_active case because in these cases nothing updates
hrtimer_cpu_base.expires_next yet. This will be changed with later patches
which further reduce the #ifdef zoo in this code.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h |  6 +++---
 kernel/time/hrtimer.c   | 26 ++++++--------------------
 2 files changed, 9 insertions(+), 23 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 22627b3a33fe..bb7270e8bc37 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -164,13 +164,13 @@ enum  hrtimer_base_type {
  * @hres_active:	State of high resolution mode
  * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hang_detected:	The last hrtimer interrupt detected a hang
- * @expires_next:	absolute time of the next event, is required for remote
- *			hrtimer enqueue
  * @next_timer:		Pointer to the first expiring timer
  * @nr_events:		Total number of hrtimer interrupt events
  * @nr_retries:		Total number of hrtimer interrupt retries
  * @nr_hangs:		Total number of hrtimer interrupt hangs
  * @max_hang_time:	Maximum time spent in hrtimer_interrupt
+ * @expires_next:	absolute time of the next event, is required for remote
+ *			hrtimer enqueue
  * @clock_base:		array of clock bases for this cpu
  *
  * Note: next_timer is just an optimization for __remove_hrtimer().
@@ -186,13 +186,13 @@ struct hrtimer_cpu_base {
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
 					hang_detected	: 1;
-	ktime_t				expires_next;
 	struct hrtimer			*next_timer;
 	unsigned int			nr_events;
 	unsigned short			nr_retries;
 	unsigned short			nr_hangs;
 	unsigned int			max_hang_time;
 #endif
+	ktime_t				expires_next;
 	struct hrtimer_clock_base	clock_base[HRTIMER_MAX_CLOCK_BASES];
 } ____cacheline_aligned;
 
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 0ed86146096a..b5e1aaef925c 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -154,26 +154,21 @@ struct hrtimer_clock_base *lock_hrtimer_base(const struct hrtimer *timer,
 }
 
 /*
- * With HIGHRES=y we do not migrate the timer when it is expiring
- * before the next event on the target cpu because we cannot reprogram
- * the target cpu hardware and we would cause it to fire late.
+ * We do not migrate the timer when it is expiring before the next
+ * event on the target cpu. When high resolution is enabled, we cannot
+ * reprogram the target cpu hardware and we would cause it to fire
+ * late. To keep it simple, we handle the high resolution enabled and
+ * disabled case similar.
  *
  * Called with cpu_base->lock of target cpu held.
  */
 static int
 hrtimer_check_target(struct hrtimer *timer, struct hrtimer_clock_base *new_base)
 {
-#ifdef CONFIG_HIGH_RES_TIMERS
 	ktime_t expires;
 
-	if (!new_base->cpu_base->hres_active)
-		return 0;
-
 	expires = ktime_sub(hrtimer_get_expires(timer), new_base->offset);
 	return expires <= new_base->cpu_base->expires_next;
-#else
-	return 0;
-#endif
 }
 
 static inline
@@ -657,14 +652,6 @@ static void hrtimer_reprogram(struct hrtimer *timer,
 }
 
 /*
- * Initialize the high resolution related parts of cpu_base
- */
-static inline void hrtimer_init_hres(struct hrtimer_cpu_base *base)
-{
-	base->expires_next = KTIME_MAX;
-}
-
-/*
  * Retrigger next event is called after clock was set
  *
  * Called with interrupts disabled via on_each_cpu()
@@ -729,7 +716,6 @@ static inline int hrtimer_reprogram(struct hrtimer *timer,
 {
 	return 0;
 }
-static inline void hrtimer_init_hres(struct hrtimer_cpu_base *base) { }
 static inline void retrigger_next_event(void *arg) { }
 
 #endif /* CONFIG_HIGH_RES_TIMERS */
@@ -1599,7 +1585,7 @@ int hrtimers_prepare_cpu(unsigned int cpu)
 
 	cpu_base->cpu = cpu;
 	cpu_base->hres_active = 0;
-	hrtimer_init_hres(cpu_base);
+	cpu_base->expires_next = KTIME_MAX;
 	return 0;
 }
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 16/36] hrtimer: Make hrtimer_cpu_base.next_timer handling unconditional
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (14 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 15/36] hrtimer: Make the remote enqueue check unconditional Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:46   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 17/36] hrtimer: Make hrtimer_reprogramm() unconditional Anna-Maria Gleixner
                   ` (20 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

hrtimer_cpu_base.next_timer stores the pointer to the next expiring timer
in a cpu base.

This pointer cannot be dereferenced and is solely used to check whether a
hrtimer which is removed is the hrtimer which is the first to expire in the
CPU base. If this is the case, then the timer hardware needs to be
reprogrammed to avoid an extra interrupt for nothing.

Again, this is conditional functionality, but there is no compelling reason
to make this conditional. As a preparation, hrtimer_cpu_base.next_timer
needs to be available unconditonal. Aside of that the upcoming support for
softirq based hrtimers requires access to this pointer unconditionally.

Make the update of hrtimer_cpu_base.next_timer unconditional and remove the
ifdef cruft. The impact on CONFIG_HIGH_RES_TIMERS=n && CONFIG_NOHZ=n is
marginal as it's just a store on an already dirtied cacheline.

No functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h |  4 ++--
 kernel/time/hrtimer.c   | 12 ++----------
 2 files changed, 4 insertions(+), 12 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index bb7270e8bc37..2d3e1d678a4d 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -164,13 +164,13 @@ enum  hrtimer_base_type {
  * @hres_active:	State of high resolution mode
  * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hang_detected:	The last hrtimer interrupt detected a hang
- * @next_timer:		Pointer to the first expiring timer
  * @nr_events:		Total number of hrtimer interrupt events
  * @nr_retries:		Total number of hrtimer interrupt retries
  * @nr_hangs:		Total number of hrtimer interrupt hangs
  * @max_hang_time:	Maximum time spent in hrtimer_interrupt
  * @expires_next:	absolute time of the next event, is required for remote
  *			hrtimer enqueue
+ * @next_timer:		Pointer to the first expiring timer
  * @clock_base:		array of clock bases for this cpu
  *
  * Note: next_timer is just an optimization for __remove_hrtimer().
@@ -186,13 +186,13 @@ struct hrtimer_cpu_base {
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
 					hang_detected	: 1;
-	struct hrtimer			*next_timer;
 	unsigned int			nr_events;
 	unsigned short			nr_retries;
 	unsigned short			nr_hangs;
 	unsigned int			max_hang_time;
 #endif
 	ktime_t				expires_next;
+	struct hrtimer			*next_timer;
 	struct hrtimer_clock_base	clock_base[HRTIMER_MAX_CLOCK_BASES];
 } ____cacheline_aligned;
 
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index b5e1aaef925c..cf23bfeb5235 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -459,21 +459,13 @@ __next_base(struct hrtimer_cpu_base *cpu_base, unsigned int *active)
 	while ((base = __next_base((cpu_base), &(active))))
 
 #if defined(CONFIG_NO_HZ_COMMON) || defined(CONFIG_HIGH_RES_TIMERS)
-static inline void hrtimer_update_next_timer(struct hrtimer_cpu_base *cpu_base,
-					     struct hrtimer *timer)
-{
-#ifdef CONFIG_HIGH_RES_TIMERS
-	cpu_base->next_timer = timer;
-#endif
-}
-
 static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 {
 	struct hrtimer_clock_base *base;
 	unsigned int active = cpu_base->active_bases;
 	ktime_t expires, expires_next = KTIME_MAX;
 
-	hrtimer_update_next_timer(cpu_base, NULL);
+	cpu_base->next_timer = NULL;
 	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *next;
 		struct hrtimer *timer;
@@ -483,7 +475,7 @@ static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 		expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
 		if (expires < expires_next) {
 			expires_next = expires;
-			hrtimer_update_next_timer(cpu_base, timer);
+			cpu_base->next_timer = timer;
 		}
 	}
 	/*
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 17/36] hrtimer: Make hrtimer_reprogramm() unconditional
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (15 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 16/36] hrtimer: Make hrtimer_cpu_base.next_timer handling unconditional Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:47   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 18/36] hrtimer: Make hrtimer_force_reprogramm() unconditionally available Anna-Maria Gleixner
                   ` (19 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

hrtimer_reprogram() needs to be available unconditionally for softirq based
hrtimers. Move the function and all required struct members out of the
CONFIG_HIGH_RES_TIMERS #ifdef.

There is no functional change because hrtimer_reprogram() is only invoked
when hrtimer_cpu_base.hres_active is true. Making it unconditional
increases the text size for the CONFIG_HIGH_RES_TIMERS=n case, but avoids
replication of that code for the upcoming softirq based hrtimers support.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h |   6 +--
 kernel/time/hrtimer.c   | 129 +++++++++++++++++++++++-------------------------
 2 files changed, 65 insertions(+), 70 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 2d3e1d678a4d..98ed35767ac5 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -182,10 +182,10 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
-	unsigned int			hres_active	: 1;
-#ifdef CONFIG_HIGH_RES_TIMERS
-	unsigned int			in_hrtirq	: 1,
+	unsigned int			hres_active	: 1,
+					in_hrtirq	: 1,
 					hang_detected	: 1;
+#ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			nr_events;
 	unsigned short			nr_retries;
 	unsigned short			nr_hangs;
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index cf23bfeb5235..86a1d705d052 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -582,68 +582,6 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 }
 
 /*
- * When a timer is enqueued and expires earlier than the already enqueued
- * timers, we have to check, whether it expires earlier than the timer for
- * which the clock event device was armed.
- *
- * Called with interrupts disabled and base->cpu_base.lock held
- */
-static void hrtimer_reprogram(struct hrtimer *timer,
-			      struct hrtimer_clock_base *base)
-{
-	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
-	ktime_t expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
-
-	WARN_ON_ONCE(hrtimer_get_expires_tv64(timer) < 0);
-
-	/*
-	 * If the timer is not on the current cpu, we cannot reprogram
-	 * the other cpus clock event device.
-	 */
-	if (base->cpu_base != cpu_base)
-		return;
-
-	/*
-	 * If the hrtimer interrupt is running, then it will
-	 * reevaluate the clock bases and reprogram the clock event
-	 * device. The callbacks are always executed in hard interrupt
-	 * context so we don't need an extra check for a running
-	 * callback.
-	 */
-	if (cpu_base->in_hrtirq)
-		return;
-
-	/*
-	 * CLOCK_REALTIME timer might be requested with an absolute
-	 * expiry time which is less than base->offset. Set it to 0.
-	 */
-	if (expires < 0)
-		expires = 0;
-
-	if (expires >= cpu_base->expires_next)
-		return;
-
-	/* Update the pointer to the next expiring timer */
-	cpu_base->next_timer = timer;
-
-	/*
-	 * If a hang was detected in the last timer interrupt then we
-	 * do not schedule a timer which is earlier than the expiry
-	 * which we enforced in the hang detection. We want the system
-	 * to make progress.
-	 */
-	if (cpu_base->hang_detected)
-		return;
-
-	/*
-	 * Program the timer hardware. We enforce the expiry for
-	 * events which are already in the past.
-	 */
-	cpu_base->expires_next = expires;
-	tick_program_event(expires, 1);
-}
-
-/*
  * Retrigger next event is called after clock was set
  *
  * Called with interrupts disabled via on_each_cpu()
@@ -703,16 +641,73 @@ static inline int hrtimer_is_hres_enabled(void) { return 0; }
 static inline void hrtimer_switch_to_hres(void) { }
 static inline void
 hrtimer_force_reprogram(struct hrtimer_cpu_base *base, int skip_equal) { }
-static inline int hrtimer_reprogram(struct hrtimer *timer,
-				    struct hrtimer_clock_base *base)
-{
-	return 0;
-}
 static inline void retrigger_next_event(void *arg) { }
 
 #endif /* CONFIG_HIGH_RES_TIMERS */
 
 /*
+ * When a timer is enqueued and expires earlier than the already enqueued
+ * timers, we have to check, whether it expires earlier than the timer for
+ * which the clock event device was armed.
+ *
+ * Called with interrupts disabled and base->cpu_base.lock held
+ */
+static void hrtimer_reprogram(struct hrtimer *timer,
+			      struct hrtimer_clock_base *base)
+{
+	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	ktime_t expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
+
+	WARN_ON_ONCE(hrtimer_get_expires_tv64(timer) < 0);
+
+	/*
+	 * If the timer is not on the current cpu, we cannot reprogram
+	 * the other cpus clock event device.
+	 */
+	if (base->cpu_base != cpu_base)
+		return;
+
+	/*
+	 * If the hrtimer interrupt is running, then it will
+	 * reevaluate the clock bases and reprogram the clock event
+	 * device. The callbacks are always executed in hard interrupt
+	 * context so we don't need an extra check for a running
+	 * callback.
+	 */
+	if (cpu_base->in_hrtirq)
+		return;
+
+	/*
+	 * CLOCK_REALTIME timer might be requested with an absolute
+	 * expiry time which is less than base->offset. Set it to 0.
+	 */
+	if (expires < 0)
+		expires = 0;
+
+	if (expires >= cpu_base->expires_next)
+		return;
+
+	/* Update the pointer to the next expiring timer */
+	cpu_base->next_timer = timer;
+
+	/*
+	 * If a hang was detected in the last timer interrupt then we
+	 * do not schedule a timer which is earlier than the expiry
+	 * which we enforced in the hang detection. We want the system
+	 * to make progress.
+	 */
+	if (cpu_base->hang_detected)
+		return;
+
+	/*
+	 * Program the timer hardware. We enforce the expiry for
+	 * events which are already in the past.
+	 */
+	cpu_base->expires_next = expires;
+	tick_program_event(expires, 1);
+}
+
+/*
  * Clock realtime was set
  *
  * Change the offset of the realtime clock vs. the monotonic
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 18/36] hrtimer: Make hrtimer_force_reprogramm() unconditionally available
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (16 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 17/36] hrtimer: Make hrtimer_reprogramm() unconditional Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:47   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 19/36] hrtimer: Unify handling of hrtimer remove Anna-Maria Gleixner
                   ` (18 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

hrtimer_force_reprogram() needs to be available unconditionally for softirq
based hrtimers. Move the function and all required struct members out of
the CONFIG_HIGH_RES_TIMERS #ifdef.

There is no functional change because hrtimer_force_reprogram() is only
invoked when hrtimer_cpu_base.hres_active is true and
CONFIG_HIGH_RES_TIMERS=y.

Making it unconditional increases the text size for the
CONFIG_HIGH_RES_TIMERS=n case slightly, but avoids replication of that code
for the upcoming softirq based hrtimers support. Most of the code gets
eliminated in the CONFIG_HIGH_RES_TIMERS=n case by the compiler.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 58 +++++++++++++++++++++++++--------------------------
 1 file changed, 28 insertions(+), 30 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 86a1d705d052..a54f91aa973d 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -513,34 +513,6 @@ static inline int hrtimer_hres_active(void)
 	return __hrtimer_hres_active(this_cpu_ptr(&hrtimer_bases));
 }
 
-/* High resolution timer related functions */
-#ifdef CONFIG_HIGH_RES_TIMERS
-
-/*
- * High resolution timer enabled ?
- */
-static bool hrtimer_hres_enabled __read_mostly  = true;
-unsigned int hrtimer_resolution __read_mostly = LOW_RES_NSEC;
-EXPORT_SYMBOL_GPL(hrtimer_resolution);
-
-/*
- * Enable / Disable high resolution mode
- */
-static int __init setup_hrtimer_hres(char *str)
-{
-	return (kstrtobool(str, &hrtimer_hres_enabled) == 0);
-}
-
-__setup("highres=", setup_hrtimer_hres);
-
-/*
- * hrtimer_high_res_enabled - query, if the highres mode is enabled
- */
-static inline int hrtimer_is_hres_enabled(void)
-{
-	return hrtimer_hres_enabled;
-}
-
 /*
  * Reprogram the event source with checking both queues for the
  * next event
@@ -581,6 +553,34 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 	tick_program_event(cpu_base->expires_next, 1);
 }
 
+/* High resolution timer related functions */
+#ifdef CONFIG_HIGH_RES_TIMERS
+
+/*
+ * High resolution timer enabled ?
+ */
+static bool hrtimer_hres_enabled __read_mostly  = true;
+unsigned int hrtimer_resolution __read_mostly = LOW_RES_NSEC;
+EXPORT_SYMBOL_GPL(hrtimer_resolution);
+
+/*
+ * Enable / Disable high resolution mode
+ */
+static int __init setup_hrtimer_hres(char *str)
+{
+	return (kstrtobool(str, &hrtimer_hres_enabled) == 0);
+}
+
+__setup("highres=", setup_hrtimer_hres);
+
+/*
+ * hrtimer_high_res_enabled - query, if the highres mode is enabled
+ */
+static inline int hrtimer_is_hres_enabled(void)
+{
+	return hrtimer_hres_enabled;
+}
+
 /*
  * Retrigger next event is called after clock was set
  *
@@ -639,8 +639,6 @@ void clock_was_set_delayed(void)
 
 static inline int hrtimer_is_hres_enabled(void) { return 0; }
 static inline void hrtimer_switch_to_hres(void) { }
-static inline void
-hrtimer_force_reprogram(struct hrtimer_cpu_base *base, int skip_equal) { }
 static inline void retrigger_next_event(void *arg) { }
 
 #endif /* CONFIG_HIGH_RES_TIMERS */
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 19/36] hrtimer: Unify handling of hrtimer remove
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (17 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 18/36] hrtimer: Make hrtimer_force_reprogramm() unconditionally available Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:48   ` [tip:timers/core] hrtimer: Unify hrtimer removal handling tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 20/36] hrtimer: Unify handling of remote enqueue Anna-Maria Gleixner
                   ` (17 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

When the first hrtimer on the current CPU is removed,
hrtimer_force_reprogram() is invoked but only when
CONFIG_HIGH_RES_TIMERS=y and hrtimer_cpu_base.hres_active is set.

hrtimer_force_reprogram() updates hrtimer_cpu_base.expires_next and
reprograms the clock event device. When CONFIG_HIGH_RES_TIMERS=y and
hrtimer_cpu_base.hres_active is set, a pointless hrtimer interrupt can be
prevented.

hrtimer_check_target() makes the 'can remote enqueue' decision. As soon as
hrtimer_check_target() is unconditionally available and
hrtimer_cpu_base.expires_next is updated by hrtimer_reprogram(),
hrtimer_force_reprogram() needs to be available unconditionally as well to
prevent the following scenario with CONFIG_HIGH_RES_TIMERS=n:

- the first hrtimer on this CPU is removed and hrtimer_force_reprogram() is
  not executed

- CPU goes idle (next timer is calculated and hrtimers are taken into
  account)

- a hrtimer is enqueued remote on the idle CPU: hrtimer_check_target()
  compares expiry value and hrtimer_cpu_base.expires_next. The expiry value
  is after expires_next, so the hrtimer is enqueued. This timer will fire
  late, if it expires before the effective first hrtimer on this CPU and
  the comparison was with an outdated expires_next value.

To prevent this scenario, make hrtimer_force_reprogram() unconditional
except the effective reprogramming part, which gets eliminated by the
compiler in the CONFIG_HIGH_RES_TIMERS=n case.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index a54f91aa973d..dd202c796662 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -523,9 +523,6 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 {
 	ktime_t expires_next;
 
-	if (!__hrtimer_hres_active(cpu_base))
-		return;
-
 	expires_next = __hrtimer_get_next_event(cpu_base);
 
 	if (skip_equal && expires_next == cpu_base->expires_next)
@@ -534,6 +531,9 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 	cpu_base->expires_next = expires_next;
 
 	/*
+	 * If hres is not active, hardware does not have to be
+	 * reprogrammed yet.
+	 *
 	 * If a hang was detected in the last timer interrupt then we
 	 * leave the hang delay active in the hardware. We want the
 	 * system to make progress. That also prevents the following
@@ -547,7 +547,7 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 	 * set. So we'd effectivly block all timers until the T2 event
 	 * fires.
 	 */
-	if (cpu_base->hang_detected)
+	if (!__hrtimer_hres_active(cpu_base) || cpu_base->hang_detected)
 		return;
 
 	tick_program_event(cpu_base->expires_next, 1);
@@ -846,7 +846,6 @@ static void __remove_hrtimer(struct hrtimer *timer,
 	if (!timerqueue_del(&base->active, &timer->node))
 		cpu_base->active_bases &= ~(1 << base->index);
 
-#ifdef CONFIG_HIGH_RES_TIMERS
 	/*
 	 * Note: If reprogram is false we do not update
 	 * cpu_base->next_timer. This happens when we remove the first
@@ -857,7 +856,6 @@ static void __remove_hrtimer(struct hrtimer *timer,
 	 */
 	if (reprogram && timer == cpu_base->next_timer)
 		hrtimer_force_reprogram(cpu_base, 1);
-#endif
 }
 
 /*
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 20/36] hrtimer: Unify handling of remote enqueue
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (18 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 19/36] hrtimer: Unify handling of hrtimer remove Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:48   ` [tip:timers/core] hrtimer: Unify remote enqueue handling tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 21/36] hrtimer: Make remote enqueue decision less restrictive Anna-Maria Gleixner
                   ` (16 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

hrtimer_reprogram() is conditionally invoked from hrtimer_start_range_ns()
when hrtimer_cpu_base.hres_active is true.

In the !hres_active case there is a special condition for the nohz_active
case:

  If the newly enqueued timer expires before the first expiring timer on a
  remote CPU then the remote CPU needs to be notified and woken up from a
  NOHZ idle sleep to take the new first expiring timer into account.

Previous changes have already established the prerequisites to make the
remote enqueue behaviour the same whether high resolution mode is active or
not:

  If the to be enqueued timer expires before the first expiring timer on a
  remote CPU, then it cannot be enqueued there.

This was done for the high resolution mode because there is no way to
access the remote CPU timer hardware. The same is true for NOHZ, but was
handled differently by unconditionally enqueuing the timer and waking up
the remote CPU so it can reprogram its timer. Again there is no compelling
reason for this difference.

hrtimer_check_target(), which makes the 'can remote enqueue' decision is
already unconditional, but not yet functional because nothing updates
hrtimer_cpu_base.expires_next in the !hres_active case.

To unify this the following changes are required:

 1) Make the store of the new first expiry time unconditonal in
    hrtimer_reprogram() and check __hrtimer_hres_active() before proceeding
    to the actual hardware access. This check also lets the compiler
    eliminate the rest of the function in case of CONFIG_HIGH_RES_TIMERS=n.

 2) Invoke hrtimer_reprogram() unconditionally from
    hrtimer_start_range_ns()

 3) Remove the remote wakeup special case for the !high_res && nohz_active
    case.

Confine the timers_nohz_active static key to timer.c which is the only user
now.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c       | 18 ++++++------------
 kernel/time/tick-internal.h |  6 ------
 kernel/time/timer.c         |  9 ++++++++-
 3 files changed, 14 insertions(+), 19 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index dd202c796662..c8528b932247 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -687,21 +687,24 @@ static void hrtimer_reprogram(struct hrtimer *timer,
 
 	/* Update the pointer to the next expiring timer */
 	cpu_base->next_timer = timer;
+	cpu_base->expires_next = expires;
 
 	/*
+	 * If hres is not active, hardware does not have to be
+	 * programmed yet.
+	 *
 	 * If a hang was detected in the last timer interrupt then we
 	 * do not schedule a timer which is earlier than the expiry
 	 * which we enforced in the hang detection. We want the system
 	 * to make progress.
 	 */
-	if (cpu_base->hang_detected)
+	if (!__hrtimer_hres_active(cpu_base) || cpu_base->hang_detected)
 		return;
 
 	/*
 	 * Program the timer hardware. We enforce the expiry for
 	 * events which are already in the past.
 	 */
-	cpu_base->expires_next = expires;
 	tick_program_event(expires, 1);
 }
 
@@ -938,16 +941,7 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	if (!leftmost)
 		goto unlock;
 
-	if (!hrtimer_is_hres_active(timer)) {
-		/*
-		 * Kick to reschedule the next tick to handle the new timer
-		 * on dynticks target.
-		 */
-		if (is_timers_nohz_active())
-			wake_up_nohz_cpu(new_base->cpu_base->cpu);
-	} else {
-		hrtimer_reprogram(timer, new_base);
-	}
+	hrtimer_reprogram(timer, new_base);
 unlock:
 	unlock_hrtimer_base(timer, &flags);
 }
diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index 4ac74dff59f0..e277284c2831 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -151,18 +151,12 @@ static inline void tick_nohz_init(void) { }
 #ifdef CONFIG_NO_HZ_COMMON
 extern unsigned long tick_nohz_active;
 extern void timers_update_nohz(void);
-extern struct static_key_false timers_nohz_active;
-static inline bool is_timers_nohz_active(void)
-{
-	return static_branch_unlikely(&timers_nohz_active);
-}
 # ifdef CONFIG_SMP
 extern struct static_key_false timers_migration_enabled;
 # endif
 #else /* CONFIG_NO_HZ_COMMON */
 static inline void timers_update_nohz(void) { }
 #define tick_nohz_active (0)
-static inline bool is_timers_nohz_active(void) { return false; }
 #endif
 
 DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases);
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 1e2140a23044..4b39029fbe99 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -210,7 +210,7 @@ static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]);
 
 #ifdef CONFIG_NO_HZ_COMMON
 
-DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
+static DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
 static DEFINE_MUTEX(timer_keys_mutex);
 
 static void timer_update_keys(struct work_struct *work);
@@ -260,6 +260,13 @@ int timer_migration_handler(struct ctl_table *table, int write,
 	mutex_unlock(&timer_keys_mutex);
 	return ret;
 }
+
+static inline bool is_timers_nohz_active(void)
+{
+	return static_branch_unlikely(&timers_nohz_active);
+}
+#else
+static inline bool is_timers_nohz_active(void) { return false; }
 #endif /* NO_HZ_COMMON */
 
 static unsigned long round_jiffies_common(unsigned long j, int cpu,
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 21/36] hrtimer: Make remote enqueue decision less restrictive
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (19 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 20/36] hrtimer: Unify handling of remote enqueue Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:49   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 22/36] hrtimer: Remove base argument from hrtimer_reprogram() Anna-Maria Gleixner
                   ` (15 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

The current decision whether a timer can be queued on a remote CPU checks
for timer->expiry <= remote_cpu_base.expires_next.

This is too restrictive because a timer with the same expiry time as an
existing timer will be enqueued on right-hand size of the existing timer
inside the rbtree, i.e. behind the first expiring timer.

So its safe to allow enqueuing timers with the same expiry time as the
first expiring timer on a remote CPU base.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index c8528b932247..a1c92615ac07 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -168,7 +168,7 @@ hrtimer_check_target(struct hrtimer *timer, struct hrtimer_clock_base *new_base)
 	ktime_t expires;
 
 	expires = ktime_sub(hrtimer_get_expires(timer), new_base->offset);
-	return expires <= new_base->cpu_base->expires_next;
+	return expires < new_base->cpu_base->expires_next;
 }
 
 static inline
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 22/36] hrtimer: Remove base argument from hrtimer_reprogram()
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (20 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 21/36] hrtimer: Make remote enqueue decision less restrictive Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:49   ` [tip:timers/core] hrtimer: Remove the 'base' parameter " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 23/36] hrtimer: Split hrtimer_start_range_ns() Anna-Maria Gleixner
                   ` (14 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

hrtimer_reprogram() must have access to the hrtimer_clock_base of the new
first expiring timer to access hrtimer_clock_base.offset for adjusting the
expiry time to CLOCK_MONOTONIC. This is required to evaluate whether the
new left most timer in the hrtimer_clock_base is the first expiring timer
of all clock bases in a hrtimer_cpu_base.

The only user of hrtimer_reprogram() is hrtimer_start_range_ns(), which has
a pointer to hrtimer_clock_base already and hands it in as an argument. But
hrtimer_start_range_ns() will be split for the upcoming support for softirq
based hrtimers to avoid code duplication and will lose the direct access to
the clock base pointer.

Instead of handing in timer and timer->base as an argument remove the base
argument from hrtimer_reprogram() and retrieve the clock base internally.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index a1c92615ac07..1dda56baded7 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -650,10 +650,10 @@ static inline void retrigger_next_event(void *arg) { }
  *
  * Called with interrupts disabled and base->cpu_base.lock held
  */
-static void hrtimer_reprogram(struct hrtimer *timer,
-			      struct hrtimer_clock_base *base)
+static void hrtimer_reprogram(struct hrtimer *timer)
 {
 	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	struct hrtimer_clock_base *base = timer->base;
 	ktime_t expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
 
 	WARN_ON_ONCE(hrtimer_get_expires_tv64(timer) < 0);
@@ -941,7 +941,7 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	if (!leftmost)
 		goto unlock;
 
-	hrtimer_reprogram(timer, new_base);
+	hrtimer_reprogram(timer);
 unlock:
 	unlock_hrtimer_base(timer, &flags);
 }
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 23/36] hrtimer: Split hrtimer_start_range_ns()
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (21 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 22/36] hrtimer: Remove base argument from hrtimer_reprogram() Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:49   ` [tip:timers/core] hrtimer: Factor out __hrtimer_start_range_ns() tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 24/36] hrtimer: Split __hrtimer_get_next_event() Anna-Maria Gleixner
                   ` (13 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

Preparatory patch for softirq based hrtimers to avoid code duplication. No
functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 44 ++++++++++++++++++++++++--------------------
 1 file changed, 24 insertions(+), 20 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 1dda56baded7..c7e2768ed022 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -907,22 +907,11 @@ static inline ktime_t hrtimer_update_lowres(struct hrtimer *timer, ktime_t tim,
 	return tim;
 }
 
-/**
- * hrtimer_start_range_ns - (re)start an hrtimer
- * @timer:	the timer to be added
- * @tim:	expiry time
- * @delta_ns:	"slack" range for the timer
- * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
- */
-void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
-			    u64 delta_ns, const enum hrtimer_mode mode)
+static int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
+				    u64 delta_ns, const enum hrtimer_mode mode,
+				    struct hrtimer_clock_base *base)
 {
-	struct hrtimer_clock_base *base, *new_base;
-	unsigned long flags;
-	int leftmost;
-
-	base = lock_hrtimer_base(timer, &flags);
+	struct hrtimer_clock_base *new_base;
 
 	/* Remove an active timer from the queue: */
 	remove_hrtimer(timer, base, true);
@@ -937,12 +926,27 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	/* Switch the timer base, if necessary: */
 	new_base = switch_hrtimer_base(timer, base, mode & HRTIMER_MODE_PINNED);
 
-	leftmost = enqueue_hrtimer(timer, new_base, mode);
-	if (!leftmost)
-		goto unlock;
+	return enqueue_hrtimer(timer, new_base, mode);
+}
+/**
+ * hrtimer_start_range_ns - (re)start an hrtimer
+ * @timer:	the timer to be added
+ * @tim:	expiry time
+ * @delta_ns:	"slack" range for the timer
+ * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
+ */
+void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
+			    u64 delta_ns, const enum hrtimer_mode mode)
+{
+	struct hrtimer_clock_base *base;
+	unsigned long flags;
+
+	base = lock_hrtimer_base(timer, &flags);
+
+	if (__hrtimer_start_range_ns(timer, tim, delta_ns, mode, base))
+		hrtimer_reprogram(timer);
 
-	hrtimer_reprogram(timer);
-unlock:
 	unlock_hrtimer_base(timer, &flags);
 }
 EXPORT_SYMBOL_GPL(hrtimer_start_range_ns);
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 24/36] hrtimer: Split __hrtimer_get_next_event()
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (22 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 23/36] hrtimer: Split hrtimer_start_range_ns() Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:50   ` [tip:timers/core] hrtimer: Factor out __hrtimer_next_event_base() tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 25/36] hrtimer: Use irqsave/irqrestore around __run_hrtimer() Anna-Maria Gleixner
                   ` (12 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

Preparatory patch for softirq based hrtimers to avoid code duplication. No
functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index c7e2768ed022..914e91a59a51 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -459,13 +459,13 @@ __next_base(struct hrtimer_cpu_base *cpu_base, unsigned int *active)
 	while ((base = __next_base((cpu_base), &(active))))
 
 #if defined(CONFIG_NO_HZ_COMMON) || defined(CONFIG_HIGH_RES_TIMERS)
-static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
+static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
+					 unsigned int active,
+					 ktime_t expires_next)
 {
 	struct hrtimer_clock_base *base;
-	unsigned int active = cpu_base->active_bases;
-	ktime_t expires, expires_next = KTIME_MAX;
+	ktime_t expires;
 
-	cpu_base->next_timer = NULL;
 	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *next;
 		struct hrtimer *timer;
@@ -487,6 +487,18 @@ static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 		expires_next = 0;
 	return expires_next;
 }
+
+static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
+{
+	unsigned int active = cpu_base->active_bases;
+	ktime_t expires_next = KTIME_MAX;
+
+	cpu_base->next_timer = NULL;
+
+	expires_next = __hrtimer_next_event_base(cpu_base, active, expires_next);
+
+	return expires_next;
+}
 #endif
 
 static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base)
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 25/36] hrtimer: Use irqsave/irqrestore around __run_hrtimer()
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (23 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 24/36] hrtimer: Split __hrtimer_get_next_event() Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:50   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 26/36] hrtimer: Add clock bases and hrtimer mode for soft irq context Anna-Maria Gleixner
                   ` (11 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

__run_hrtimer() is called with the hrtimer_cpu_base.lock held and
interrupts disabled. Before invoking the timer callback the base lock is
dropped, but interrupts stay disabled.

The upcoming support for softirq based hrtimers requires that interrupts
are enabled before the timer callback is invoked.

To avoid code duplication, take hrtimer_cpu_base.lock with
raw_spin_lock_irqsave(flags) at the call site and hand in the flags as
argument. So raw_spin_unlock_irqrestore() before the callback invocation
will either keep interrupts disabled in interrupt context or restore to
interrupt enabled state when called from softirq context.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 914e91a59a51..1966867f7232 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1161,7 +1161,8 @@ EXPORT_SYMBOL_GPL(hrtimer_active);
 
 static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 			  struct hrtimer_clock_base *base,
-			  struct hrtimer *timer, ktime_t *now)
+			  struct hrtimer *timer, ktime_t *now,
+			  unsigned long flags)
 {
 	enum hrtimer_restart (*fn)(struct hrtimer *);
 	int restart;
@@ -1196,11 +1197,11 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	 * protected against migration to a different CPU even if the lock
 	 * is dropped.
 	 */
-	raw_spin_unlock(&cpu_base->lock);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 	trace_hrtimer_expire_entry(timer, now);
 	restart = fn(timer);
 	trace_hrtimer_expire_exit(timer);
-	raw_spin_lock(&cpu_base->lock);
+	raw_spin_lock_irq(&cpu_base->lock);
 
 	/*
 	 * Note: We clear the running state after enqueue_hrtimer and
@@ -1228,7 +1229,8 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	base->running = NULL;
 }
 
-static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now)
+static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now,
+				 unsigned long flags)
 {
 	struct hrtimer_clock_base *base;
 	unsigned int active = cpu_base->active_bases;
@@ -1259,7 +1261,7 @@ static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now)
 			if (basenow < hrtimer_get_softexpires_tv64(timer))
 				break;
 
-			__run_hrtimer(cpu_base, base, timer, &basenow);
+			__run_hrtimer(cpu_base, base, timer, &basenow, flags);
 		}
 	}
 }
@@ -1274,13 +1276,14 @@ void hrtimer_interrupt(struct clock_event_device *dev)
 {
 	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
 	ktime_t expires_next, now, entry_time, delta;
+	unsigned long flags;
 	int retries = 0;
 
 	BUG_ON(!cpu_base->hres_active);
 	cpu_base->nr_events++;
 	dev->next_event = KTIME_MAX;
 
-	raw_spin_lock(&cpu_base->lock);
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	entry_time = now = hrtimer_update_base(cpu_base);
 retry:
 	cpu_base->in_hrtirq = 1;
@@ -1293,7 +1296,7 @@ void hrtimer_interrupt(struct clock_event_device *dev)
 	 */
 	cpu_base->expires_next = KTIME_MAX;
 
-	__hrtimer_run_queues(cpu_base, now);
+	__hrtimer_run_queues(cpu_base, now, flags);
 
 	/* Reevaluate the clock bases for the next expiry */
 	expires_next = __hrtimer_get_next_event(cpu_base);
@@ -1303,7 +1306,7 @@ void hrtimer_interrupt(struct clock_event_device *dev)
 	 */
 	cpu_base->expires_next = expires_next;
 	cpu_base->in_hrtirq = 0;
-	raw_spin_unlock(&cpu_base->lock);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 
 	/* Reprogramming necessary ? */
 	if (!tick_program_event(expires_next, 0)) {
@@ -1324,7 +1327,7 @@ void hrtimer_interrupt(struct clock_event_device *dev)
 	 * Acquire base lock for updating the offsets and retrieving
 	 * the current time.
 	 */
-	raw_spin_lock(&cpu_base->lock);
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	now = hrtimer_update_base(cpu_base);
 	cpu_base->nr_retries++;
 	if (++retries < 3)
@@ -1337,7 +1340,8 @@ void hrtimer_interrupt(struct clock_event_device *dev)
 	 */
 	cpu_base->nr_hangs++;
 	cpu_base->hang_detected = 1;
-	raw_spin_unlock(&cpu_base->lock);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
+
 	delta = ktime_sub(now, entry_time);
 	if ((unsigned int)delta > cpu_base->max_hang_time)
 		cpu_base->max_hang_time = (unsigned int) delta;
@@ -1379,6 +1383,7 @@ static inline void __hrtimer_peek_ahead_timers(void) { }
 void hrtimer_run_queues(void)
 {
 	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	unsigned long flags;
 	ktime_t now;
 
 	if (__hrtimer_hres_active(cpu_base))
@@ -1396,10 +1401,10 @@ void hrtimer_run_queues(void)
 		return;
 	}
 
-	raw_spin_lock(&cpu_base->lock);
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	now = hrtimer_update_base(cpu_base);
-	__hrtimer_run_queues(cpu_base, now);
-	raw_spin_unlock(&cpu_base->lock);
+	__hrtimer_run_queues(cpu_base, now, flags);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
 
 /*
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 26/36] hrtimer: Add clock bases and hrtimer mode for soft irq context
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (24 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 25/36] hrtimer: Use irqsave/irqrestore around __run_hrtimer() Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:51   ` [tip:timers/core] hrtimer: Add clock bases and hrtimer mode for softirq context tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 27/36] hrtimer: Prepare handling of hard and softirq based hrtimers Anna-Maria Gleixner
                   ` (10 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

hrtimer callback functions are always executed in hard interrupt
context. Users of hrtimer which need their timer function to be executed
in soft interrupt context, make use of tasklets to get the proper context.

Add additional hrtimer clock bases for timers which must expire in softirq
context, so the detour via the tasklet can be avoided. This is also
required for RT, where the majority of hrtimer is moved into softirq
hrtimer context.

The selection of the expiry mode happens via a mode bit. Introduce
HRTIMER_MODE_SOFT and the matching combinations with the ABS/REL/PINNED
bits and update the decoding of hrtimer_mode in tracepoints.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h      | 14 ++++++++++++++
 include/trace/events/timer.h |  6 +++++-
 kernel/time/hrtimer.c        | 20 ++++++++++++++++++++
 3 files changed, 39 insertions(+), 1 deletion(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 98ed35767ac5..26ae8a868ea8 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -33,14 +33,24 @@ struct hrtimer_cpu_base;
  * HRTIMER_MODE_REL		- Time value is relative to now
  * HRTIMER_MODE_PINNED		- Timer is bound to CPU (is only considered
  *				  when starting the timer)
+ * HRTIMER_MODE_SOFT		- Timer callback function will be executed in
+ *				  soft irq context
  */
 enum hrtimer_mode {
 	HRTIMER_MODE_ABS	= 0x00,
 	HRTIMER_MODE_REL	= 0x01,
 	HRTIMER_MODE_PINNED	= 0x02,
+	HRTIMER_MODE_SOFT	= 0x04,
 
 	HRTIMER_MODE_ABS_PINNED = HRTIMER_MODE_ABS | HRTIMER_MODE_PINNED,
 	HRTIMER_MODE_REL_PINNED = HRTIMER_MODE_REL | HRTIMER_MODE_PINNED,
+
+	HRTIMER_MODE_ABS_SOFT	= HRTIMER_MODE_ABS | HRTIMER_MODE_SOFT,
+	HRTIMER_MODE_REL_SOFT	= HRTIMER_MODE_REL | HRTIMER_MODE_SOFT,
+
+	HRTIMER_MODE_ABS_PINNED_SOFT = HRTIMER_MODE_ABS_PINNED | HRTIMER_MODE_SOFT,
+	HRTIMER_MODE_REL_PINNED_SOFT = HRTIMER_MODE_REL_PINNED | HRTIMER_MODE_SOFT,
+
 };
 
 /*
@@ -151,6 +161,10 @@ enum  hrtimer_base_type {
 	HRTIMER_BASE_REALTIME,
 	HRTIMER_BASE_BOOTTIME,
 	HRTIMER_BASE_TAI,
+	HRTIMER_BASE_MONOTONIC_SOFT,
+	HRTIMER_BASE_REALTIME_SOFT,
+	HRTIMER_BASE_BOOTTIME_SOFT,
+	HRTIMER_BASE_TAI_SOFT,
 	HRTIMER_MAX_CLOCK_BASES,
 };
 
diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h
index 744b4310b24b..a57e4ee989d6 100644
--- a/include/trace/events/timer.h
+++ b/include/trace/events/timer.h
@@ -148,7 +148,11 @@ DEFINE_EVENT(timer_class, timer_cancel,
 		{ HRTIMER_MODE_ABS,		"ABS"		},	\
 		{ HRTIMER_MODE_REL,		"REL"		},	\
 		{ HRTIMER_MODE_ABS_PINNED,	"ABS|PINNED"	},	\
-		{ HRTIMER_MODE_REL_PINNED,	"REL|PINNED"	})
+		{ HRTIMER_MODE_REL_PINNED,	"REL|PINNED"	},	\
+		{ HRTIMER_MODE_ABS_SOFT,	"ABS|SOFT"	},	\
+		{ HRTIMER_MODE_REL_SOFT,	"REL|SOFT"	},	\
+		{ HRTIMER_MODE_ABS_PINNED_SOFT,	"ABS|PINNED|SOFT" },	\
+		{ HRTIMER_MODE_REL_PINNED_SOFT,	"REL|PINNED|SOFT" })
 
 /**
  * hrtimer_init - called when the hrtimer is initialized
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 1966867f7232..74598e3da748 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -92,6 +92,26 @@ DEFINE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases) =
 			.clockid = CLOCK_TAI,
 			.get_time = &ktime_get_clocktai,
 		},
+		{
+			.index = HRTIMER_BASE_MONOTONIC_SOFT,
+			.clockid = CLOCK_MONOTONIC,
+			.get_time = &ktime_get,
+		},
+		{
+			.index = HRTIMER_BASE_REALTIME_SOFT,
+			.clockid = CLOCK_REALTIME,
+			.get_time = &ktime_get_real,
+		},
+		{
+			.index = HRTIMER_BASE_BOOTTIME_SOFT,
+			.clockid = CLOCK_BOOTTIME,
+			.get_time = &ktime_get_boottime,
+		},
+		{
+			.index = HRTIMER_BASE_TAI_SOFT,
+			.clockid = CLOCK_TAI,
+			.get_time = &ktime_get_clocktai,
+		},
 	}
 };
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 27/36] hrtimer: Prepare handling of hard and softirq based hrtimers
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (25 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 26/36] hrtimer: Add clock bases and hrtimer mode for soft irq context Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16  3:51   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 28/36] hrtimer: Implement support for " Anna-Maria Gleixner
                   ` (9 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

The softirq based hrtimer can utilize most of the existing hrtimers
functions, but need to operate on a different data set.

Add an active_mask argument to various functions so the hard and soft bases
can be selected. Fixup the existing callers and hand in the ACTIVE_HARD
mask.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 38 +++++++++++++++++++++++++++++---------
 1 file changed, 29 insertions(+), 9 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 74598e3da748..66af33ad2935 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -60,6 +60,15 @@
 #include "tick-internal.h"
 
 /*
+ * Masks for selecting the soft and hard context timers from
+ * cpu_base->active
+ */
+#define MASK_SHIFT		(HRTIMER_BASE_MONOTONIC_SOFT)
+#define HRTIMER_ACTIVE_HARD	((1U << MASK_SHIFT) - 1)
+#define HRTIMER_ACTIVE_SOFT	(HRTIMER_ACTIVE_HARD << MASK_SHIFT)
+#define HRTIMER_ACTIVE_ALL	(HRTIMER_ACTIVE_SOFT | HRTIMER_ACTIVE_HARD)
+
+/*
  * The timer bases:
  *
  * There are more clockids than hrtimer bases. Thus, we index
@@ -508,13 +517,24 @@ static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
 	return expires_next;
 }
 
-static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
+/*
+ * Recomputes cpu_base::*next_timer and returns the earliest expires_next but
+ * does not set cpu_base::*expires_next, that is done by hrtimer_reprogram.
+ *
+ * @active_mask must be one of:
+ *  - HRTIMER_ACTIVE,
+ *  - HRTIMER_ACTIVE_SOFT, or
+ *  - HRTIMER_ACTIVE_HARD.
+ */
+static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base,
+					unsigned int active_mask)
 {
-	unsigned int active = cpu_base->active_bases;
+	unsigned int active;
 	ktime_t expires_next = KTIME_MAX;
 
 	cpu_base->next_timer = NULL;
 
+	active = cpu_base->active_bases & active_mask;
 	expires_next = __hrtimer_next_event_base(cpu_base, active, expires_next);
 
 	return expires_next;
@@ -555,7 +575,7 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 {
 	ktime_t expires_next;
 
-	expires_next = __hrtimer_get_next_event(cpu_base);
+	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
 
 	if (skip_equal && expires_next == cpu_base->expires_next)
 		return;
@@ -1076,7 +1096,7 @@ u64 hrtimer_get_next_event(void)
 	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 
 	if (!__hrtimer_hres_active(cpu_base))
-		expires = __hrtimer_get_next_event(cpu_base);
+		expires = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
 
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 
@@ -1250,10 +1270,10 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 }
 
 static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now,
-				 unsigned long flags)
+				 unsigned long flags, unsigned int active_mask)
 {
 	struct hrtimer_clock_base *base;
-	unsigned int active = cpu_base->active_bases;
+	unsigned int active = cpu_base->active_bases & active_mask;
 
 	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *node;
@@ -1316,10 +1336,10 @@ void hrtimer_interrupt(struct clock_event_device *dev)
 	 */
 	cpu_base->expires_next = KTIME_MAX;
 
-	__hrtimer_run_queues(cpu_base, now, flags);
+	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 
 	/* Reevaluate the clock bases for the next expiry */
-	expires_next = __hrtimer_get_next_event(cpu_base);
+	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
 	/*
 	 * Store the new expiry value so the migration code can verify
 	 * against it.
@@ -1423,7 +1443,7 @@ void hrtimer_run_queues(void)
 
 	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	now = hrtimer_update_base(cpu_base);
-	__hrtimer_run_queues(cpu_base, now, flags);
+	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 28/36] hrtimer: Implement support for softirq based hrtimers
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (26 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 27/36] hrtimer: Prepare handling of hard and softirq based hrtimers Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16 10:22   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 29/36] hrtimer: Implement SOFT/HARD clock base selection Anna-Maria Gleixner
                   ` (8 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

hrtimer callbacks are always invoked in hard interrupt context. Several
users in tree require soft interrupt context for their callbacks and
achieve this by combining a hrtimer with a tasklet. The hrtimer schedules
the tasklet in hard interrupt context and the tasklet callback gets invoked
in softirq context later.

That's suboptimal and aside of that the real-time patch moves most of the
hrtimers into softirq context. So adding native support for hrtimers
expiring in softirq context is a valuable extension for both mainline and
the RT patch set.

Each valid hrtimer clock id has two associated hrtimer clock bases: one for
timers expiring in hardirq context and one for timers expiring in softirq
context.

Implement the functionality to associate a hrtimer with the hard or softirq
related clock bases and update the relevant functions to take them into
account when the next expiry time needs to be evaluated.

Add a check into the hard interrupt context handler functions to check
whether the first expiring softirq based timer has expired. If it's expired
the softirq is raised and the accounting of softirq based timers to
evaluate the next expiry time for programming the timer hardware is skipped
until the softirq processing has finished. At the end of the softirq
processing the regular processing is resumed.

Suggested-by: Thomas Gleixner <tglx@linutronix.de>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/hrtimer.h |  21 +++--
 kernel/time/hrtimer.c   | 198 +++++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 188 insertions(+), 31 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 26ae8a868ea8..c7902ca7c9f4 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -103,6 +103,7 @@ enum hrtimer_restart {
  * @base:	pointer to the timer base (per cpu and per clock)
  * @state:	state information (See bit values above)
  * @is_rel:	Set if the timer was armed relative
+ * @is_soft:	Set if hrtimer will be expired in soft interrupt context.
  *
  * The hrtimer structure must be initialized by hrtimer_init()
  */
@@ -113,6 +114,7 @@ struct hrtimer {
 	struct hrtimer_clock_base	*base;
 	u8				state;
 	u8				is_rel;
+	u8				is_soft;
 };
 
 /**
@@ -178,13 +180,18 @@ enum  hrtimer_base_type {
  * @hres_active:	State of high resolution mode
  * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hang_detected:	The last hrtimer interrupt detected a hang
+ * @softirq_activated:	displays, if the softirq is raised - update of softirq
+ *			related settings is not required then.
  * @nr_events:		Total number of hrtimer interrupt events
  * @nr_retries:		Total number of hrtimer interrupt retries
  * @nr_hangs:		Total number of hrtimer interrupt hangs
  * @max_hang_time:	Maximum time spent in hrtimer_interrupt
  * @expires_next:	absolute time of the next event, is required for remote
- *			hrtimer enqueue
+ *			hrtimer enqueue; it is the total first expiry time (hard
+ *			and soft hrtimer are taken into account)
  * @next_timer:		Pointer to the first expiring timer
+ * @softirq_expires_next: Time to check, if soft queues needs also to be expired
+ * @softirq_next_timer: Pointer to the first expiring softirq based timer
  * @clock_base:		array of clock bases for this cpu
  *
  * Note: next_timer is just an optimization for __remove_hrtimer().
@@ -196,9 +203,10 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
-	unsigned int			hres_active	: 1,
-					in_hrtirq	: 1,
-					hang_detected	: 1;
+	unsigned int			hres_active		: 1,
+					in_hrtirq		: 1,
+					hang_detected		: 1,
+					softirq_activated       : 1;
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			nr_events;
 	unsigned short			nr_retries;
@@ -207,6 +215,8 @@ struct hrtimer_cpu_base {
 #endif
 	ktime_t				expires_next;
 	struct hrtimer			*next_timer;
+	ktime_t				softirq_expires_next;
+	struct hrtimer			*softirq_next_timer;
 	struct hrtimer_clock_base	clock_base[HRTIMER_MAX_CLOCK_BASES];
 } ____cacheline_aligned;
 
@@ -379,7 +389,8 @@ extern void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
  * @timer:	the timer to be added
  * @tim:	expiry time
  * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED);
+ *		softirq based mode is considered for debug purpose only!
  */
 static inline void hrtimer_start(struct hrtimer *timer, ktime_t tim,
 				 const enum hrtimer_mode mode)
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 66af33ad2935..efacf7375159 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -411,7 +411,8 @@ static inline void debug_hrtimer_init(struct hrtimer *timer)
 	debug_object_init(timer, &hrtimer_debug_descr);
 }
 
-static inline void debug_hrtimer_activate(struct hrtimer *timer)
+static inline void debug_hrtimer_activate(struct hrtimer *timer,
+					  enum hrtimer_mode mode)
 {
 	debug_object_activate(timer, &hrtimer_debug_descr);
 }
@@ -444,8 +445,10 @@ void destroy_hrtimer_on_stack(struct hrtimer *timer)
 EXPORT_SYMBOL_GPL(destroy_hrtimer_on_stack);
 
 #else
+
 static inline void debug_hrtimer_init(struct hrtimer *timer) { }
-static inline void debug_hrtimer_activate(struct hrtimer *timer) { }
+static inline void debug_hrtimer_activate(struct hrtimer *timer,
+					  enum hrtimer_mode mode) { }
 static inline void debug_hrtimer_deactivate(struct hrtimer *timer) { }
 #endif
 
@@ -460,7 +463,7 @@ debug_init(struct hrtimer *timer, clockid_t clockid,
 static inline void debug_activate(struct hrtimer *timer,
 				  enum hrtimer_mode mode)
 {
-	debug_hrtimer_activate(timer);
+	debug_hrtimer_activate(timer, mode);
 	trace_hrtimer_start(timer, mode);
 }
 
@@ -487,7 +490,6 @@ __next_base(struct hrtimer_cpu_base *cpu_base, unsigned int *active)
 #define for_each_active_base(base, cpu_base, active)	\
 	while ((base = __next_base((cpu_base), &(active))))
 
-#if defined(CONFIG_NO_HZ_COMMON) || defined(CONFIG_HIGH_RES_TIMERS)
 static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
 					 unsigned int active,
 					 ktime_t expires_next)
@@ -504,7 +506,10 @@ static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
 		expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
 		if (expires < expires_next) {
 			expires_next = expires;
-			cpu_base->next_timer = timer;
+			if (timer->is_soft)
+				cpu_base->softirq_next_timer = timer;
+			else
+				cpu_base->next_timer = timer;
 		}
 	}
 	/*
@@ -521,25 +526,42 @@ static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
  * Recomputes cpu_base::*next_timer and returns the earliest expires_next but
  * does not set cpu_base::*expires_next, that is done by hrtimer_reprogram.
  *
+ * When a softirq is pending, we can ignore the HRTIMER_ACTIVE_SOFT bases,
+ * those timers will get run whenever the softirq gets handled, at the end of
+ * hrtimer_run_softirq(), hrtimer_update_softirq_timer() will re-add these bases.
+ *
+ * Therefore softirq values are those from the HRTIMER_ACTIVE_SOFT clock bases.
+ * The !softirq values are the minima across HRTIMER_ACTIVE_ALL, unless an actual
+ * softirq is pending, in which case they're the minima of HRTIMER_ACTIVE_HARD.
+ *
  * @active_mask must be one of:
- *  - HRTIMER_ACTIVE,
+ *  - HRTIMER_ACTIVE_ALL,
  *  - HRTIMER_ACTIVE_SOFT, or
  *  - HRTIMER_ACTIVE_HARD.
  */
-static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base,
-					unsigned int active_mask)
+static ktime_t
+__hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base, unsigned int active_mask)
 {
 	unsigned int active;
+	struct hrtimer *next_timer = NULL;
 	ktime_t expires_next = KTIME_MAX;
 
-	cpu_base->next_timer = NULL;
+	if (!cpu_base->softirq_activated && (active_mask & HRTIMER_ACTIVE_SOFT)) {
+		active = cpu_base->active_bases & HRTIMER_ACTIVE_SOFT;
+		cpu_base->softirq_next_timer = NULL;
+		expires_next = __hrtimer_next_event_base(cpu_base, active, KTIME_MAX);
+
+		next_timer = cpu_base->softirq_next_timer;
+	}
 
-	active = cpu_base->active_bases & active_mask;
-	expires_next = __hrtimer_next_event_base(cpu_base, active, expires_next);
+	if (active_mask & HRTIMER_ACTIVE_HARD) {
+		active = cpu_base->active_bases & HRTIMER_ACTIVE_HARD;
+		cpu_base->next_timer = next_timer;
+		expires_next = __hrtimer_next_event_base(cpu_base, active, expires_next);
+	}
 
 	return expires_next;
 }
-#endif
 
 static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base)
 {
@@ -547,8 +569,14 @@ static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base)
 	ktime_t *offs_boot = &base->clock_base[HRTIMER_BASE_BOOTTIME].offset;
 	ktime_t *offs_tai = &base->clock_base[HRTIMER_BASE_TAI].offset;
 
-	return ktime_get_update_offsets_now(&base->clock_was_set_seq,
+	ktime_t now = ktime_get_update_offsets_now(&base->clock_was_set_seq,
 					    offs_real, offs_boot, offs_tai);
+
+	base->clock_base[HRTIMER_BASE_REALTIME_SOFT].offset = *offs_real;
+	base->clock_base[HRTIMER_BASE_BOOTTIME_SOFT].offset = *offs_boot;
+	base->clock_base[HRTIMER_BASE_TAI_SOFT].offset = *offs_tai;
+
+	return now;
 }
 
 /*
@@ -575,7 +603,23 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 {
 	ktime_t expires_next;
 
-	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
+	/*
+	 * Find the current next expiration time.
+	 */
+	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_ALL);
+
+	if (cpu_base->next_timer && cpu_base->next_timer->is_soft) {
+		/*
+		 * When the softirq is activated, hrtimer has to be
+		 * programmed with the first hard hrtimer because soft
+		 * timer interrupt could occur too late.
+		 */
+		if (cpu_base->softirq_activated)
+			expires_next = __hrtimer_get_next_event(cpu_base,
+								HRTIMER_ACTIVE_HARD);
+		else
+			cpu_base->softirq_expires_next = expires_next;
+	}
 
 	if (skip_equal && expires_next == cpu_base->expires_next)
 		return;
@@ -702,7 +746,7 @@ static inline void retrigger_next_event(void *arg) { }
  *
  * Called with interrupts disabled and base->cpu_base.lock held
  */
-static void hrtimer_reprogram(struct hrtimer *timer)
+static void hrtimer_reprogram(struct hrtimer *timer, bool reprogram)
 {
 	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
 	struct hrtimer_clock_base *base = timer->base;
@@ -711,6 +755,37 @@ static void hrtimer_reprogram(struct hrtimer *timer)
 	WARN_ON_ONCE(hrtimer_get_expires_tv64(timer) < 0);
 
 	/*
+	 * CLOCK_REALTIME timer might be requested with an absolute
+	 * expiry time which is less than base->offset. Set it to 0.
+	 */
+	if (expires < 0)
+		expires = 0;
+
+	if (timer->is_soft) {
+		/*
+		 * soft hrtimer could be started on a remote CPU. In this
+		 * case softirq_expires_next needs to be updated on the
+		 * remote CPU. The soft hrtimer will not expire before the
+		 * first hard hrtimer on the remote CPU -
+		 * hrtimer_check_target() prevents this case.
+		 */
+		struct hrtimer_cpu_base *timer_cpu_base = base->cpu_base;
+
+		if (timer_cpu_base->softirq_activated)
+			return;
+
+		if (!ktime_before(expires, timer_cpu_base->softirq_expires_next))
+			return;
+
+		timer_cpu_base->softirq_next_timer = timer;
+		timer_cpu_base->softirq_expires_next = expires;
+
+		if (!ktime_before(expires, timer_cpu_base->expires_next) ||
+		    !reprogram)
+			return;
+	}
+
+	/*
 	 * If the timer is not on the current cpu, we cannot reprogram
 	 * the other cpus clock event device.
 	 */
@@ -727,13 +802,6 @@ static void hrtimer_reprogram(struct hrtimer *timer)
 	if (cpu_base->in_hrtirq)
 		return;
 
-	/*
-	 * CLOCK_REALTIME timer might be requested with an absolute
-	 * expiry time which is less than base->offset. Set it to 0.
-	 */
-	if (expires < 0)
-		expires = 0;
-
 	if (expires >= cpu_base->expires_next)
 		return;
 
@@ -959,6 +1027,31 @@ static inline ktime_t hrtimer_update_lowres(struct hrtimer *timer, ktime_t tim,
 	return tim;
 }
 
+static void
+hrtimer_update_softirq_timer(struct hrtimer_cpu_base *cpu_base, bool reprogram)
+{
+	ktime_t expires;
+
+	/*
+	 * Find the next SOFT expiration.
+	 */
+	expires = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_SOFT);
+
+	/*
+	 * reprogramming needs to be triggered, even if the next soft
+	 * hrtimer expires at the same time than the next hard
+	 * hrtimer. cpu_base->softirq_expires_next needs to be updated!
+	 */
+	if (expires == KTIME_MAX)
+		return;
+
+	/*
+	 * cpu_base->*next_timer is recomputed by __hrtimer_get_next_event()
+	 * cpu_base->*expires_next is only set by hrtimer_reprogram()
+	 */
+	hrtimer_reprogram(cpu_base->softirq_next_timer, reprogram);
+}
+
 static int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 				    u64 delta_ns, const enum hrtimer_mode mode,
 				    struct hrtimer_clock_base *base)
@@ -980,13 +1073,15 @@ static int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 
 	return enqueue_hrtimer(timer, new_base, mode);
 }
+
 /**
  * hrtimer_start_range_ns - (re)start an hrtimer
  * @timer:	the timer to be added
  * @tim:	expiry time
  * @delta_ns:	"slack" range for the timer
  * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED);
+ *		softirq based mode is considered for debug purpose only!
  */
 void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 			    u64 delta_ns, const enum hrtimer_mode mode)
@@ -994,10 +1089,16 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	struct hrtimer_clock_base *base;
 	unsigned long flags;
 
+	/*
+	 * Check whether the HRTIMER_MODE_SOFT bit and hrtimer.is_soft
+	 * match.
+	 */
+	WARN_ON_ONCE(!(mode & HRTIMER_MODE_SOFT) ^ !timer->is_soft);
+
 	base = lock_hrtimer_base(timer, &flags);
 
 	if (__hrtimer_start_range_ns(timer, tim, delta_ns, mode, base))
-		hrtimer_reprogram(timer);
+		hrtimer_reprogram(timer, true);
 
 	unlock_hrtimer_base(timer, &flags);
 }
@@ -1096,7 +1197,7 @@ u64 hrtimer_get_next_event(void)
 	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 
 	if (!__hrtimer_hres_active(cpu_base))
-		expires = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
+		expires = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_ALL);
 
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 
@@ -1306,6 +1407,23 @@ static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now,
 	}
 }
 
+static __latent_entropy void hrtimer_run_softirq(struct softirq_action *h)
+{
+	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	unsigned long flags;
+	ktime_t now;
+
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
+
+	now = hrtimer_update_base(cpu_base);
+	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_SOFT);
+
+	cpu_base->softirq_activated = 0;
+	hrtimer_update_softirq_timer(cpu_base, true);
+
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
+}
+
 #ifdef CONFIG_HIGH_RES_TIMERS
 
 /*
@@ -1336,10 +1454,16 @@ void hrtimer_interrupt(struct clock_event_device *dev)
 	 */
 	cpu_base->expires_next = KTIME_MAX;
 
+	if (!ktime_before(now, cpu_base->softirq_expires_next)) {
+		cpu_base->softirq_expires_next = KTIME_MAX;
+		cpu_base->softirq_activated = 1;
+		raise_softirq_irqoff(HRTIMER_SOFTIRQ);
+	}
+
 	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 
 	/* Reevaluate the clock bases for the next expiry */
-	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
+	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_ALL);
 	/*
 	 * Store the new expiry value so the migration code can verify
 	 * against it.
@@ -1443,6 +1567,13 @@ void hrtimer_run_queues(void)
 
 	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	now = hrtimer_update_base(cpu_base);
+
+	if (!ktime_before(now, cpu_base->softirq_expires_next)) {
+		cpu_base->softirq_expires_next = KTIME_MAX;
+		cpu_base->softirq_activated = 1;
+		raise_softirq_irqoff(HRTIMER_SOFTIRQ);
+	}
+
 	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
@@ -1624,6 +1755,7 @@ int hrtimers_prepare_cpu(unsigned int cpu)
 	cpu_base->cpu = cpu;
 	cpu_base->hres_active = 0;
 	cpu_base->expires_next = KTIME_MAX;
+	cpu_base->softirq_expires_next = KTIME_MAX;
 	return 0;
 }
 
@@ -1667,6 +1799,12 @@ int hrtimers_dead_cpu(unsigned int scpu)
 	BUG_ON(cpu_online(scpu));
 	tick_cancel_sched_timer(scpu);
 
+	/*
+	 * this BH disable ensures that raise_softirq_irqoff() does
+	 * not wakeup ksoftirqd (and acquire the pi-lock) while
+	 * holding the cpu_base lock
+	 */
+	local_bh_disable();
 	local_irq_disable();
 	old_base = &per_cpu(hrtimer_bases, scpu);
 	new_base = this_cpu_ptr(&hrtimer_bases);
@@ -1682,12 +1820,19 @@ int hrtimers_dead_cpu(unsigned int scpu)
 				     &new_base->clock_base[i]);
 	}
 
+	/*
+	 * The migration might have changed the first expiring softirq
+	 * timer on this CPU. Update it.
+	 */
+	hrtimer_update_softirq_timer(new_base, false);
+
 	raw_spin_unlock(&old_base->lock);
 	raw_spin_unlock(&new_base->lock);
 
 	/* Check, if we got expired work to do */
 	__hrtimer_peek_ahead_timers();
 	local_irq_enable();
+	local_bh_enable();
 	return 0;
 }
 
@@ -1696,6 +1841,7 @@ int hrtimers_dead_cpu(unsigned int scpu)
 void __init hrtimers_init(void)
 {
 	hrtimers_prepare_cpu(smp_processor_id());
+	open_softirq(HRTIMER_SOFTIRQ, hrtimer_run_softirq);
 }
 
 /**
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 29/36] hrtimer: Implement SOFT/HARD clock base selection
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (27 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 28/36] hrtimer: Implement support for " Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2018-01-16 10:22   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 30/36] can/bcm: Replace hrtimer_tasklet with softirq based hrtimer Anna-Maria Gleixner
                   ` (7 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

All prerequisites to handle hrtimers for expiry in either hard or soft
interrupt context are in place.

Add the missing bit in hrtimer_init() which associates the timer to the
hard or the soft irq clock base.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 kernel/time/hrtimer.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index efacf7375159..b40c408c8b62 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1220,8 +1220,9 @@ static inline int hrtimer_clockid_to_base(clockid_t clock_id)
 static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 			   enum hrtimer_mode mode)
 {
+	bool softtimer = !!(mode & HRTIMER_MODE_SOFT);
+	int base = softtimer ? HRTIMER_MAX_CLOCK_BASES / 2 : 0;
 	struct hrtimer_cpu_base *cpu_base;
-	int base;
 
 	memset(timer, 0, sizeof(struct hrtimer));
 
@@ -1235,7 +1236,8 @@ static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 	if (clock_id == CLOCK_REALTIME && mode & HRTIMER_MODE_REL)
 		clock_id = CLOCK_MONOTONIC;
 
-	base = hrtimer_clockid_to_base(clock_id);
+	base += hrtimer_clockid_to_base(clock_id);
+	timer->is_soft = softtimer;
 	timer->base = &cpu_base->clock_base[base];
 	timerqueue_init(&timer->node);
 }
@@ -1244,8 +1246,13 @@ static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
  * hrtimer_init - initialize a timer to the given clock
  * @timer:	the timer to be initialized
  * @clock_id:	the clock to be used
- * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL); pinned is not considered here!
+ * @mode:       The modes which are relevant for intitialization:
+ *              HRTIMER_MODE_ABS, HRTIMER_MODE_REL, HRTIMER_MODE_ABS_SOFT,
+ *              HRTIMER_MODE_REL_SOFT
+ *
+ *              The PINNED variants of the above can be handed in,
+ *              but the PINNED bit is ignored as pinning happens
+ *              when the hrtimer is started
  */
 void hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 		  enum hrtimer_mode mode)
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 30/36] can/bcm: Replace hrtimer_tasklet with softirq based hrtimer
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (28 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 29/36] hrtimer: Implement SOFT/HARD clock base selection Anna-Maria Gleixner
@ 2017-12-21 10:41 ` Anna-Maria Gleixner
  2017-12-21 10:42 ` [PATCH v4 31/36] mac80211_hwsim: Replace hrtimer tasklet with softirq hrtimer Anna-Maria Gleixner
                   ` (6 subsequent siblings)
  36 siblings, 0 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:41 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner,
	Oliver Hartkopp, linux-can, Marc Kleine-Budde

From: Thomas Gleixner <tglx@linutronix.de>

Switch the timer to HRTIMER_MODE_SOFT, which executed the timer
callback in softirq context and remove the hrtimer_tasklet.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Acked-by: Oliver Hartkopp <socketcan@hartkopp.net>
Cc: Oliver Hartkopp <socketcan@hartkopp.net>
Cc: linux-can@vger.kernel.org
Cc: Marc Kleine-Budde <mkl@pengutronix.de>
---
 net/can/bcm.c | 156 ++++++++++++++++++++--------------------------------------
 1 file changed, 52 insertions(+), 104 deletions(-)

diff --git a/net/can/bcm.c b/net/can/bcm.c
index 13690334efa3..9cc67ac257f1 100644
--- a/net/can/bcm.c
+++ b/net/can/bcm.c
@@ -102,7 +102,6 @@ struct bcm_op {
 	unsigned long frames_abs, frames_filtered;
 	struct bcm_timeval ival1, ival2;
 	struct hrtimer timer, thrtimer;
-	struct tasklet_struct tsklet, thrtsklet;
 	ktime_t rx_stamp, kt_ival1, kt_ival2, kt_lastmsg;
 	int rx_ifindex;
 	int cfsiz;
@@ -364,25 +363,34 @@ static void bcm_send_to_user(struct bcm_op *op, struct bcm_msg_head *head,
 	}
 }
 
-static void bcm_tx_start_timer(struct bcm_op *op)
+static bool bcm_tx_set_expiry(struct bcm_op *op, struct hrtimer *hrt)
 {
+	ktime_t ival;
+
 	if (op->kt_ival1 && op->count)
-		hrtimer_start(&op->timer,
-			      ktime_add(ktime_get(), op->kt_ival1),
-			      HRTIMER_MODE_ABS);
+		ival = op->kt_ival1;
 	else if (op->kt_ival2)
-		hrtimer_start(&op->timer,
-			      ktime_add(ktime_get(), op->kt_ival2),
-			      HRTIMER_MODE_ABS);
+		ival = op->kt_ival2;
+	else
+		return false;
+
+	hrtimer_set_expires(hrt, ktime_add(ktime_get(), ival));
+	return true;
 }
 
-static void bcm_tx_timeout_tsklet(unsigned long data)
+static void bcm_tx_start_timer(struct bcm_op *op)
 {
-	struct bcm_op *op = (struct bcm_op *)data;
+	if (bcm_tx_set_expiry(op, &op->timer))
+		hrtimer_start_expires(&op->timer, HRTIMER_MODE_ABS_SOFT);
+}
+
+/* bcm_tx_timeout_handler - performs cyclic CAN frame transmissions */
+static enum hrtimer_restart bcm_tx_timeout_handler(struct hrtimer *hrtimer)
+{
+	struct bcm_op *op = container_of(hrtimer, struct bcm_op, timer);
 	struct bcm_msg_head msg_head;
 
 	if (op->kt_ival1 && (op->count > 0)) {
-
 		op->count--;
 		if (!op->count && (op->flags & TX_COUNTEVT)) {
 
@@ -399,22 +407,12 @@ static void bcm_tx_timeout_tsklet(unsigned long data)
 		}
 		bcm_can_tx(op);
 
-	} else if (op->kt_ival2)
+	} else if (op->kt_ival2) {
 		bcm_can_tx(op);
+	}
 
-	bcm_tx_start_timer(op);
-}
-
-/*
- * bcm_tx_timeout_handler - performs cyclic CAN frame transmissions
- */
-static enum hrtimer_restart bcm_tx_timeout_handler(struct hrtimer *hrtimer)
-{
-	struct bcm_op *op = container_of(hrtimer, struct bcm_op, timer);
-
-	tasklet_schedule(&op->tsklet);
-
-	return HRTIMER_NORESTART;
+	return bcm_tx_set_expiry(op, &op->timer) ?
+		HRTIMER_RESTART : HRTIMER_NORESTART;
 }
 
 /*
@@ -480,7 +478,7 @@ static void bcm_rx_update_and_send(struct bcm_op *op,
 		/* do not send the saved data - only start throttle timer */
 		hrtimer_start(&op->thrtimer,
 			      ktime_add(op->kt_lastmsg, op->kt_ival2),
-			      HRTIMER_MODE_ABS);
+			      HRTIMER_MODE_ABS_SOFT);
 		return;
 	}
 
@@ -539,14 +537,21 @@ static void bcm_rx_starttimer(struct bcm_op *op)
 		return;
 
 	if (op->kt_ival1)
-		hrtimer_start(&op->timer, op->kt_ival1, HRTIMER_MODE_REL);
+		hrtimer_start(&op->timer, op->kt_ival1, HRTIMER_MODE_REL_SOFT);
 }
 
-static void bcm_rx_timeout_tsklet(unsigned long data)
+/* bcm_rx_timeout_handler - when the (cyclic) CAN frame reception timed out */
+static enum hrtimer_restart bcm_rx_timeout_handler(struct hrtimer *hrtimer)
 {
-	struct bcm_op *op = (struct bcm_op *)data;
+	struct bcm_op *op = container_of(hrtimer, struct bcm_op, timer);
 	struct bcm_msg_head msg_head;
 
+	/* if user wants to be informed, when cyclic CAN-Messages come back */
+	if ((op->flags & RX_ANNOUNCE_RESUME) && op->last_frames) {
+		/* clear received CAN frames to indicate 'nothing received' */
+		memset(op->last_frames, 0, op->nframes * op->cfsiz);
+	}
+
 	/* create notification to user */
 	msg_head.opcode  = RX_TIMEOUT;
 	msg_head.flags   = op->flags;
@@ -557,25 +562,6 @@ static void bcm_rx_timeout_tsklet(unsigned long data)
 	msg_head.nframes = 0;
 
 	bcm_send_to_user(op, &msg_head, NULL, 0);
-}
-
-/*
- * bcm_rx_timeout_handler - when the (cyclic) CAN frame reception timed out
- */
-static enum hrtimer_restart bcm_rx_timeout_handler(struct hrtimer *hrtimer)
-{
-	struct bcm_op *op = container_of(hrtimer, struct bcm_op, timer);
-
-	/* schedule before NET_RX_SOFTIRQ */
-	tasklet_hi_schedule(&op->tsklet);
-
-	/* no restart of the timer is done here! */
-
-	/* if user wants to be informed, when cyclic CAN-Messages come back */
-	if ((op->flags & RX_ANNOUNCE_RESUME) && op->last_frames) {
-		/* clear received CAN frames to indicate 'nothing received' */
-		memset(op->last_frames, 0, op->nframes * op->cfsiz);
-	}
 
 	return HRTIMER_NORESTART;
 }
@@ -583,14 +569,12 @@ static enum hrtimer_restart bcm_rx_timeout_handler(struct hrtimer *hrtimer)
 /*
  * bcm_rx_do_flush - helper for bcm_rx_thr_flush
  */
-static inline int bcm_rx_do_flush(struct bcm_op *op, int update,
-				  unsigned int index)
+static inline int bcm_rx_do_flush(struct bcm_op *op, unsigned int index)
 {
 	struct canfd_frame *lcf = op->last_frames + op->cfsiz * index;
 
 	if ((op->last_frames) && (lcf->flags & RX_THR)) {
-		if (update)
-			bcm_rx_changed(op, lcf);
+		bcm_rx_changed(op, lcf);
 		return 1;
 	}
 	return 0;
@@ -598,11 +582,8 @@ static inline int bcm_rx_do_flush(struct bcm_op *op, int update,
 
 /*
  * bcm_rx_thr_flush - Check for throttled data and send it to the userspace
- *
- * update == 0 : just check if throttled data is available  (any irq context)
- * update == 1 : check and send throttled data to userspace (soft_irq context)
  */
-static int bcm_rx_thr_flush(struct bcm_op *op, int update)
+static int bcm_rx_thr_flush(struct bcm_op *op)
 {
 	int updated = 0;
 
@@ -611,24 +592,16 @@ static int bcm_rx_thr_flush(struct bcm_op *op, int update)
 
 		/* for MUX filter we start at index 1 */
 		for (i = 1; i < op->nframes; i++)
-			updated += bcm_rx_do_flush(op, update, i);
+			updated += bcm_rx_do_flush(op, i);
 
 	} else {
 		/* for RX_FILTER_ID and simple filter */
-		updated += bcm_rx_do_flush(op, update, 0);
+		updated += bcm_rx_do_flush(op, 0);
 	}
 
 	return updated;
 }
 
-static void bcm_rx_thr_tsklet(unsigned long data)
-{
-	struct bcm_op *op = (struct bcm_op *)data;
-
-	/* push the changed data to the userspace */
-	bcm_rx_thr_flush(op, 1);
-}
-
 /*
  * bcm_rx_thr_handler - the time for blocked content updates is over now:
  *                      Check for throttled data and send it to the userspace
@@ -637,9 +610,7 @@ static enum hrtimer_restart bcm_rx_thr_handler(struct hrtimer *hrtimer)
 {
 	struct bcm_op *op = container_of(hrtimer, struct bcm_op, thrtimer);
 
-	tasklet_schedule(&op->thrtsklet);
-
-	if (bcm_rx_thr_flush(op, 0)) {
+	if (bcm_rx_thr_flush(op)) {
 		hrtimer_forward(hrtimer, ktime_get(), op->kt_ival2);
 		return HRTIMER_RESTART;
 	} else {
@@ -735,23 +706,8 @@ static struct bcm_op *bcm_find_op(struct list_head *ops,
 
 static void bcm_remove_op(struct bcm_op *op)
 {
-	if (op->tsklet.func) {
-		while (test_bit(TASKLET_STATE_SCHED, &op->tsklet.state) ||
-		       test_bit(TASKLET_STATE_RUN, &op->tsklet.state) ||
-		       hrtimer_active(&op->timer)) {
-			hrtimer_cancel(&op->timer);
-			tasklet_kill(&op->tsklet);
-		}
-	}
-
-	if (op->thrtsklet.func) {
-		while (test_bit(TASKLET_STATE_SCHED, &op->thrtsklet.state) ||
-		       test_bit(TASKLET_STATE_RUN, &op->thrtsklet.state) ||
-		       hrtimer_active(&op->thrtimer)) {
-			hrtimer_cancel(&op->thrtimer);
-			tasklet_kill(&op->thrtsklet);
-		}
-	}
+	hrtimer_cancel(&op->timer);
+	hrtimer_cancel(&op->thrtimer);
 
 	if ((op->frames) && (op->frames != &op->sframe))
 		kfree(op->frames);
@@ -979,15 +935,13 @@ static int bcm_tx_setup(struct bcm_msg_head *msg_head, struct msghdr *msg,
 		op->ifindex = ifindex;
 
 		/* initialize uninitialized (kzalloc) structure */
-		hrtimer_init(&op->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+		hrtimer_init(&op->timer, CLOCK_MONOTONIC,
+			     HRTIMER_MODE_REL_SOFT);
 		op->timer.function = bcm_tx_timeout_handler;
 
-		/* initialize tasklet for tx countevent notification */
-		tasklet_init(&op->tsklet, bcm_tx_timeout_tsklet,
-			     (unsigned long) op);
-
 		/* currently unused in tx_ops */
-		hrtimer_init(&op->thrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+		hrtimer_init(&op->thrtimer, CLOCK_MONOTONIC,
+			     HRTIMER_MODE_REL_SOFT);
 
 		/* add this bcm_op to the list of the tx_ops */
 		list_add(&op->list, &bo->tx_ops);
@@ -1150,20 +1104,14 @@ static int bcm_rx_setup(struct bcm_msg_head *msg_head, struct msghdr *msg,
 		op->rx_ifindex = ifindex;
 
 		/* initialize uninitialized (kzalloc) structure */
-		hrtimer_init(&op->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+		hrtimer_init(&op->timer, CLOCK_MONOTONIC,
+			     HRTIMER_MODE_REL_SOFT);
 		op->timer.function = bcm_rx_timeout_handler;
 
-		/* initialize tasklet for rx timeout notification */
-		tasklet_init(&op->tsklet, bcm_rx_timeout_tsklet,
-			     (unsigned long) op);
-
-		hrtimer_init(&op->thrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+		hrtimer_init(&op->thrtimer, CLOCK_MONOTONIC,
+			     HRTIMER_MODE_REL_SOFT);
 		op->thrtimer.function = bcm_rx_thr_handler;
 
-		/* initialize tasklet for rx throttle handling */
-		tasklet_init(&op->thrtsklet, bcm_rx_thr_tsklet,
-			     (unsigned long) op);
-
 		/* add this bcm_op to the list of the rx_ops */
 		list_add(&op->list, &bo->rx_ops);
 
@@ -1209,12 +1157,12 @@ static int bcm_rx_setup(struct bcm_msg_head *msg_head, struct msghdr *msg,
 			 */
 			op->kt_lastmsg = 0;
 			hrtimer_cancel(&op->thrtimer);
-			bcm_rx_thr_flush(op, 1);
+			bcm_rx_thr_flush(op);
 		}
 
 		if ((op->flags & STARTTIMER) && op->kt_ival1)
 			hrtimer_start(&op->timer, op->kt_ival1,
-				      HRTIMER_MODE_REL);
+				      HRTIMER_MODE_REL_SOFT);
 	}
 
 	/* now we can register for can_ids, if we added a new bcm_op */
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 31/36] mac80211_hwsim: Replace hrtimer tasklet with softirq hrtimer
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (29 preceding siblings ...)
  2017-12-21 10:41 ` [PATCH v4 30/36] can/bcm: Replace hrtimer_tasklet with softirq based hrtimer Anna-Maria Gleixner
@ 2017-12-21 10:42 ` Anna-Maria Gleixner
  2018-01-04 15:12   ` Johannes Berg
  2017-12-21 10:42 ` [PATCH v4 32/36] xfrm: " Anna-Maria Gleixner
                   ` (5 subsequent siblings)
  36 siblings, 1 reply; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:42 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner,
	linux-wireless, Johannes Berg, Kalle Valo

From: Thomas Gleixner <tglx@linutronix.de>

Switch the timer to HRTIMER_MODE_SOFT, which executed the timer
callback in softirq context and remove the hrtimer_tasklet.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Reviewed-by: Johannes Berg <johannes@sipsolutions.net>
Cc: linux-wireless@vger.kernel.org
Cc: Johannes Berg <johannes@sipsolutions.net>
Cc: Kalle Valo <kvalo@codeaurora.org>
---
 drivers/net/wireless/mac80211_hwsim.c | 44 ++++++++++++++++-------------------
 1 file changed, 20 insertions(+), 24 deletions(-)

diff --git a/drivers/net/wireless/mac80211_hwsim.c b/drivers/net/wireless/mac80211_hwsim.c
index 10b075a46b26..d3f1861398a7 100644
--- a/drivers/net/wireless/mac80211_hwsim.c
+++ b/drivers/net/wireless/mac80211_hwsim.c
@@ -537,7 +537,7 @@ struct mac80211_hwsim_data {
 	unsigned int rx_filter;
 	bool started, idle, scanning;
 	struct mutex mutex;
-	struct tasklet_hrtimer beacon_timer;
+	struct hrtimer beacon_timer;
 	enum ps_mode {
 		PS_DISABLED, PS_ENABLED, PS_AUTO_POLL, PS_MANUAL_POLL
 	} ps;
@@ -1418,7 +1418,7 @@ static void mac80211_hwsim_stop(struct ieee80211_hw *hw)
 {
 	struct mac80211_hwsim_data *data = hw->priv;
 	data->started = false;
-	tasklet_hrtimer_cancel(&data->beacon_timer);
+	hrtimer_cancel(&data->beacon_timer);
 	wiphy_dbg(hw->wiphy, "%s\n", __func__);
 }
 
@@ -1541,14 +1541,12 @@ static enum hrtimer_restart
 mac80211_hwsim_beacon(struct hrtimer *timer)
 {
 	struct mac80211_hwsim_data *data =
-		container_of(timer, struct mac80211_hwsim_data,
-			     beacon_timer.timer);
+		container_of(timer, struct mac80211_hwsim_data, beacon_timer);
 	struct ieee80211_hw *hw = data->hw;
 	u64 bcn_int = data->beacon_int;
-	ktime_t next_bcn;
 
 	if (!data->started)
-		goto out;
+		return HRTIMER_NORESTART;
 
 	ieee80211_iterate_active_interfaces_atomic(
 		hw, IEEE80211_IFACE_ITER_NORMAL,
@@ -1560,11 +1558,9 @@ mac80211_hwsim_beacon(struct hrtimer *timer)
 		data->bcn_delta = 0;
 	}
 
-	next_bcn = ktime_add(hrtimer_get_expires(timer),
-			     ns_to_ktime(bcn_int * 1000));
-	tasklet_hrtimer_start(&data->beacon_timer, next_bcn, HRTIMER_MODE_ABS);
-out:
-	return HRTIMER_NORESTART;
+	hrtimer_forward(&data->beacon_timer, hrtimer_get_expires(timer),
+			ns_to_ktime(bcn_int * NSEC_PER_USEC));
+	return HRTIMER_RESTART;
 }
 
 static const char * const hwsim_chanwidths[] = {
@@ -1638,15 +1634,15 @@ static int mac80211_hwsim_config(struct ieee80211_hw *hw, u32 changed)
 	mutex_unlock(&data->mutex);
 
 	if (!data->started || !data->beacon_int)
-		tasklet_hrtimer_cancel(&data->beacon_timer);
-	else if (!hrtimer_is_queued(&data->beacon_timer.timer)) {
+		hrtimer_cancel(&data->beacon_timer);
+	else if (!hrtimer_is_queued(&data->beacon_timer)) {
 		u64 tsf = mac80211_hwsim_get_tsf(hw, NULL);
 		u32 bcn_int = data->beacon_int;
 		u64 until_tbtt = bcn_int - do_div(tsf, bcn_int);
 
-		tasklet_hrtimer_start(&data->beacon_timer,
-				      ns_to_ktime(until_tbtt * 1000),
-				      HRTIMER_MODE_REL);
+		hrtimer_start(&data->beacon_timer,
+			      ns_to_ktime(until_tbtt * 1000),
+			      HRTIMER_MODE_REL_SOFT);
 	}
 
 	return 0;
@@ -1709,7 +1705,7 @@ static void mac80211_hwsim_bss_info_changed(struct ieee80211_hw *hw,
 			  info->enable_beacon, info->beacon_int);
 		vp->bcn_en = info->enable_beacon;
 		if (data->started &&
-		    !hrtimer_is_queued(&data->beacon_timer.timer) &&
+		    !hrtimer_is_queued(&data->beacon_timer) &&
 		    info->enable_beacon) {
 			u64 tsf, until_tbtt;
 			u32 bcn_int;
@@ -1717,9 +1713,9 @@ static void mac80211_hwsim_bss_info_changed(struct ieee80211_hw *hw,
 			tsf = mac80211_hwsim_get_tsf(hw, vif);
 			bcn_int = data->beacon_int;
 			until_tbtt = bcn_int - do_div(tsf, bcn_int);
-			tasklet_hrtimer_start(&data->beacon_timer,
-					      ns_to_ktime(until_tbtt * 1000),
-					      HRTIMER_MODE_REL);
+			hrtimer_start(&data->beacon_timer,
+				      ns_to_ktime(until_tbtt * 1000),
+				      HRTIMER_MODE_REL_SOFT);
 		} else if (!info->enable_beacon) {
 			unsigned int count = 0;
 			ieee80211_iterate_active_interfaces_atomic(
@@ -1728,7 +1724,7 @@ static void mac80211_hwsim_bss_info_changed(struct ieee80211_hw *hw,
 			wiphy_dbg(hw->wiphy, "  beaconing vifs remaining: %u",
 				  count);
 			if (count == 0) {
-				tasklet_hrtimer_cancel(&data->beacon_timer);
+				hrtimer_cancel(&data->beacon_timer);
 				data->beacon_int = 0;
 			}
 		}
@@ -2720,9 +2716,9 @@ static int mac80211_hwsim_new_radio(struct genl_info *info,
 				    data->debugfs,
 				    data, &hwsim_simulate_radar);
 
-	tasklet_hrtimer_init(&data->beacon_timer,
-			     mac80211_hwsim_beacon,
-			     CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
+	hrtimer_init(&data->beacon_timer, CLOCK_MONOTONIC,
+		     HRTIMER_MODE_REL_SOFT);
+	data->beacon_timer.function = mac80211_hwsim_beacon;
 
 	spin_lock_bh(&hwsim_radio_lock);
 	list_add_tail(&data->list, &hwsim_radios);
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 32/36] xfrm: Replace hrtimer tasklet with softirq hrtimer
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (30 preceding siblings ...)
  2017-12-21 10:42 ` [PATCH v4 31/36] mac80211_hwsim: Replace hrtimer tasklet with softirq hrtimer Anna-Maria Gleixner
@ 2017-12-21 10:42 ` Anna-Maria Gleixner
  2017-12-21 10:42 ` [PATCH v4 33/36] softirq: Remove tasklet_hrtimer Anna-Maria Gleixner
                   ` (4 subsequent siblings)
  36 siblings, 0 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:42 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner,
	Steffen Klassert, netdev, Herbert Xu, David S. Miller

From: Thomas Gleixner <tglx@linutronix.de>

Switch the timer to HRTIMER_MODE_SOFT, which executed the timer
callback in softirq context and remove the hrtimer_tasklet.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Cc: netdev@vger.kernel.org
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
---
 include/net/xfrm.h    |  2 +-
 net/xfrm/xfrm_state.c | 30 ++++++++++++++++++------------
 2 files changed, 19 insertions(+), 13 deletions(-)

diff --git a/include/net/xfrm.h b/include/net/xfrm.h
index dc28a98ce97c..e706ec81bd14 100644
--- a/include/net/xfrm.h
+++ b/include/net/xfrm.h
@@ -217,7 +217,7 @@ struct xfrm_state {
 	struct xfrm_stats	stats;
 
 	struct xfrm_lifetime_cur curlft;
-	struct tasklet_hrtimer	mtimer;
+	struct hrtimer		mtimer;
 
 	struct xfrm_state_offload xso;
 
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index 065d89606888..4be5fc7038af 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -426,7 +426,7 @@ static void xfrm_put_mode(struct xfrm_mode *mode)
 
 static void xfrm_state_gc_destroy(struct xfrm_state *x)
 {
-	tasklet_hrtimer_cancel(&x->mtimer);
+	hrtimer_cancel(&x->mtimer);
 	del_timer_sync(&x->rtimer);
 	kfree(x->aead);
 	kfree(x->aalg);
@@ -471,8 +471,8 @@ static void xfrm_state_gc_task(struct work_struct *work)
 
 static enum hrtimer_restart xfrm_timer_handler(struct hrtimer *me)
 {
-	struct tasklet_hrtimer *thr = container_of(me, struct tasklet_hrtimer, timer);
-	struct xfrm_state *x = container_of(thr, struct xfrm_state, mtimer);
+	struct xfrm_state *x = container_of(me, struct xfrm_state, mtimer);
+	enum hrtimer_restart ret = HRTIMER_NORESTART;
 	unsigned long now = get_seconds();
 	long next = LONG_MAX;
 	int warn = 0;
@@ -536,7 +536,8 @@ static enum hrtimer_restart xfrm_timer_handler(struct hrtimer *me)
 		km_state_expired(x, 0, 0);
 resched:
 	if (next != LONG_MAX) {
-		tasklet_hrtimer_start(&x->mtimer, ktime_set(next, 0), HRTIMER_MODE_REL);
+		hrtimer_forward_now(&x->mtimer, ktime_set(next, 0));
+		ret = HRTIMER_RESTART;
 	}
 
 	goto out;
@@ -553,7 +554,7 @@ static enum hrtimer_restart xfrm_timer_handler(struct hrtimer *me)
 
 out:
 	spin_unlock(&x->lock);
-	return HRTIMER_NORESTART;
+	return ret;
 }
 
 static void xfrm_replay_timer_handler(struct timer_list *t);
@@ -572,8 +573,8 @@ struct xfrm_state *xfrm_state_alloc(struct net *net)
 		INIT_HLIST_NODE(&x->bydst);
 		INIT_HLIST_NODE(&x->bysrc);
 		INIT_HLIST_NODE(&x->byspi);
-		tasklet_hrtimer_init(&x->mtimer, xfrm_timer_handler,
-					CLOCK_BOOTTIME, HRTIMER_MODE_ABS);
+		hrtimer_init(&x->mtimer, CLOCK_BOOTTIME, HRTIMER_MODE_ABS_SOFT);
+		x->mtimer.function = xfrm_timer_handler;
 		timer_setup(&x->rtimer, xfrm_replay_timer_handler, 0);
 		x->curlft.add_time = get_seconds();
 		x->lft.soft_byte_limit = XFRM_INF;
@@ -1029,7 +1030,9 @@ xfrm_state_find(const xfrm_address_t *daddr, const xfrm_address_t *saddr,
 				hlist_add_head_rcu(&x->byspi, net->xfrm.state_byspi + h);
 			}
 			x->lft.hard_add_expires_seconds = net->xfrm.sysctl_acq_expires;
-			tasklet_hrtimer_start(&x->mtimer, ktime_set(net->xfrm.sysctl_acq_expires, 0), HRTIMER_MODE_REL);
+			hrtimer_start(&x->mtimer,
+				      ktime_set(net->xfrm.sysctl_acq_expires, 0),
+				      HRTIMER_MODE_REL_SOFT);
 			net->xfrm.state_num++;
 			xfrm_hash_grow_check(net, x->bydst.next != NULL);
 			spin_unlock_bh(&net->xfrm.xfrm_state_lock);
@@ -1140,7 +1143,7 @@ static void __xfrm_state_insert(struct xfrm_state *x)
 		hlist_add_head_rcu(&x->byspi, net->xfrm.state_byspi + h);
 	}
 
-	tasklet_hrtimer_start(&x->mtimer, ktime_set(1, 0), HRTIMER_MODE_REL);
+	hrtimer_start(&x->mtimer, ktime_set(1, 0), HRTIMER_MODE_REL_SOFT);
 	if (x->replay_maxage)
 		mod_timer(&x->rtimer, jiffies + x->replay_maxage);
 
@@ -1244,7 +1247,9 @@ static struct xfrm_state *__find_acq_core(struct net *net,
 		x->mark.m = m->m;
 		x->lft.hard_add_expires_seconds = net->xfrm.sysctl_acq_expires;
 		xfrm_state_hold(x);
-		tasklet_hrtimer_start(&x->mtimer, ktime_set(net->xfrm.sysctl_acq_expires, 0), HRTIMER_MODE_REL);
+		hrtimer_start(&x->mtimer,
+			      ktime_set(net->xfrm.sysctl_acq_expires, 0),
+			      HRTIMER_MODE_REL_SOFT);
 		list_add(&x->km.all, &net->xfrm.state_all);
 		hlist_add_head_rcu(&x->bydst, net->xfrm.state_bydst + h);
 		h = xfrm_src_hash(net, daddr, saddr, family);
@@ -1543,7 +1548,8 @@ int xfrm_state_update(struct xfrm_state *x)
 		memcpy(&x1->lft, &x->lft, sizeof(x1->lft));
 		x1->km.dying = 0;
 
-		tasklet_hrtimer_start(&x1->mtimer, ktime_set(1, 0), HRTIMER_MODE_REL);
+		hrtimer_start(&x1->mtimer, ktime_set(1, 0),
+			      HRTIMER_MODE_REL_SOFT);
 		if (x1->curlft.use_time)
 			xfrm_state_check_expire(x1);
 
@@ -1567,7 +1573,7 @@ int xfrm_state_check_expire(struct xfrm_state *x)
 	if (x->curlft.bytes >= x->lft.hard_byte_limit ||
 	    x->curlft.packets >= x->lft.hard_packet_limit) {
 		x->km.state = XFRM_STATE_EXPIRED;
-		tasklet_hrtimer_start(&x->mtimer, 0, HRTIMER_MODE_REL);
+		hrtimer_start(&x->mtimer, 0, HRTIMER_MODE_REL_SOFT);
 		return -EINVAL;
 	}
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 33/36] softirq: Remove tasklet_hrtimer
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (31 preceding siblings ...)
  2017-12-21 10:42 ` [PATCH v4 32/36] xfrm: " Anna-Maria Gleixner
@ 2017-12-21 10:42 ` Anna-Maria Gleixner
  2017-12-21 10:42   ` Anna-Maria Gleixner
                   ` (3 subsequent siblings)
  36 siblings, 0 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:42 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner

From: Thomas Gleixner <tglx@linutronix.de>

There are no more tasklet_hrtimer users of this interface.
Remove it.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
---
 include/linux/interrupt.h | 25 -----------------------
 kernel/softirq.c          | 51 -----------------------------------------------
 2 files changed, 76 deletions(-)

diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index 69c238210325..af15bfd89598 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -618,31 +618,6 @@ extern void tasklet_kill_immediate(struct tasklet_struct *t, unsigned int cpu);
 extern void tasklet_init(struct tasklet_struct *t,
 			 void (*func)(unsigned long), unsigned long data);
 
-struct tasklet_hrtimer {
-	struct hrtimer		timer;
-	struct tasklet_struct	tasklet;
-	enum hrtimer_restart	(*function)(struct hrtimer *);
-};
-
-extern void
-tasklet_hrtimer_init(struct tasklet_hrtimer *ttimer,
-		     enum hrtimer_restart (*function)(struct hrtimer *),
-		     clockid_t which_clock, enum hrtimer_mode mode);
-
-static inline
-void tasklet_hrtimer_start(struct tasklet_hrtimer *ttimer, ktime_t time,
-			   const enum hrtimer_mode mode)
-{
-	hrtimer_start(&ttimer->timer, time, mode);
-}
-
-static inline
-void tasklet_hrtimer_cancel(struct tasklet_hrtimer *ttimer)
-{
-	hrtimer_cancel(&ttimer->timer);
-	tasklet_kill(&ttimer->tasklet);
-}
-
 /*
  * Autoprobing for irqs:
  *
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 2f5e87f1bae2..d0dd7795409f 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -584,57 +584,6 @@ void tasklet_kill(struct tasklet_struct *t)
 }
 EXPORT_SYMBOL(tasklet_kill);
 
-/*
- * tasklet_hrtimer
- */
-
-/*
- * The trampoline is called when the hrtimer expires. It schedules a tasklet
- * to run __tasklet_hrtimer_trampoline() which in turn will call the intended
- * hrtimer callback, but from softirq context.
- */
-static enum hrtimer_restart __hrtimer_tasklet_trampoline(struct hrtimer *timer)
-{
-	struct tasklet_hrtimer *ttimer =
-		container_of(timer, struct tasklet_hrtimer, timer);
-
-	tasklet_hi_schedule(&ttimer->tasklet);
-	return HRTIMER_NORESTART;
-}
-
-/*
- * Helper function which calls the hrtimer callback from
- * tasklet/softirq context
- */
-static void __tasklet_hrtimer_trampoline(unsigned long data)
-{
-	struct tasklet_hrtimer *ttimer = (void *)data;
-	enum hrtimer_restart restart;
-
-	restart = ttimer->function(&ttimer->timer);
-	if (restart != HRTIMER_NORESTART)
-		hrtimer_restart(&ttimer->timer);
-}
-
-/**
- * tasklet_hrtimer_init - Init a tasklet/hrtimer combo for softirq callbacks
- * @ttimer:	 tasklet_hrtimer which is initialized
- * @function:	 hrtimer callback function which gets called from softirq context
- * @which_clock: clock id (CLOCK_MONOTONIC/CLOCK_REALTIME)
- * @mode:	 hrtimer mode (HRTIMER_MODE_ABS/HRTIMER_MODE_REL)
- */
-void tasklet_hrtimer_init(struct tasklet_hrtimer *ttimer,
-			  enum hrtimer_restart (*function)(struct hrtimer *),
-			  clockid_t which_clock, enum hrtimer_mode mode)
-{
-	hrtimer_init(&ttimer->timer, which_clock, mode);
-	ttimer->timer.function = __hrtimer_tasklet_trampoline;
-	tasklet_init(&ttimer->tasklet, __tasklet_hrtimer_trampoline,
-		     (unsigned long)ttimer);
-	ttimer->function = function;
-}
-EXPORT_SYMBOL_GPL(tasklet_hrtimer_init);
-
 void __init softirq_init(void)
 {
 	int cpu;
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 34/36] ALSA/dummy: Replace tasklet with softirq hrtimer
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
@ 2017-12-21 10:42   ` Anna-Maria Gleixner
  2017-12-21 10:41 ` [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment Anna-Maria Gleixner
                     ` (35 subsequent siblings)
  36 siblings, 0 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:42 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner, alsa-devel,
	Takashi Iwai, Takashi Sakamoto, Jaroslav Kysela

From: Thomas Gleixner <tglx@linutronix.de>

The tasklet is used to defer the execution of snd_pcm_period_elapsed() to
the softirq context. Using the HRTIMER_MODE_SOFT mode invokes the timer
callback in softirq context as well which renders the tasklet useless.

[o-takashi: avoid stall due to a call of hrtimer_cancel() on a callback
            of hrtimer]

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Reviewed-by: Takashi Iwai <tiwai@suse.de>
Cc: alsa-devel@alsa-project.org
Cc: Takashi Iwai <tiwai@suse.com>
Cc: Takashi Sakamoto <o-takashi@sakamocchi.jp>
Cc: Jaroslav Kysela <perex@perex.cz>
Link: http://lkml.kernel.org/r/20170905161820.jtysvxtfleunbbmf@breakpoint.cc
---
 sound/drivers/dummy.c | 27 ++++++++++++---------------
 1 file changed, 12 insertions(+), 15 deletions(-)

diff --git a/sound/drivers/dummy.c b/sound/drivers/dummy.c
index 7b2b1f766b00..6ad2ff57833d 100644
--- a/sound/drivers/dummy.c
+++ b/sound/drivers/dummy.c
@@ -375,17 +375,9 @@ struct dummy_hrtimer_pcm {
 	ktime_t period_time;
 	atomic_t running;
 	struct hrtimer timer;
-	struct tasklet_struct tasklet;
 	struct snd_pcm_substream *substream;
 };
 
-static void dummy_hrtimer_pcm_elapsed(unsigned long priv)
-{
-	struct dummy_hrtimer_pcm *dpcm = (struct dummy_hrtimer_pcm *)priv;
-	if (atomic_read(&dpcm->running))
-		snd_pcm_period_elapsed(dpcm->substream);
-}
-
 static enum hrtimer_restart dummy_hrtimer_callback(struct hrtimer *timer)
 {
 	struct dummy_hrtimer_pcm *dpcm;
@@ -393,7 +385,14 @@ static enum hrtimer_restart dummy_hrtimer_callback(struct hrtimer *timer)
 	dpcm = container_of(timer, struct dummy_hrtimer_pcm, timer);
 	if (!atomic_read(&dpcm->running))
 		return HRTIMER_NORESTART;
-	tasklet_schedule(&dpcm->tasklet);
+	/*
+	 * In cases of XRUN and draining, this calls .trigger to stop PCM
+	 * substream.
+	 */
+	snd_pcm_period_elapsed(dpcm->substream);
+	if (!atomic_read(&dpcm->running))
+		return HRTIMER_NORESTART;
+
 	hrtimer_forward_now(timer, dpcm->period_time);
 	return HRTIMER_RESTART;
 }
@@ -403,7 +402,7 @@ static int dummy_hrtimer_start(struct snd_pcm_substream *substream)
 	struct dummy_hrtimer_pcm *dpcm = substream->runtime->private_data;
 
 	dpcm->base_time = hrtimer_cb_get_time(&dpcm->timer);
-	hrtimer_start(&dpcm->timer, dpcm->period_time, HRTIMER_MODE_REL);
+	hrtimer_start(&dpcm->timer, dpcm->period_time, HRTIMER_MODE_REL_SOFT);
 	atomic_set(&dpcm->running, 1);
 	return 0;
 }
@@ -413,14 +412,14 @@ static int dummy_hrtimer_stop(struct snd_pcm_substream *substream)
 	struct dummy_hrtimer_pcm *dpcm = substream->runtime->private_data;
 
 	atomic_set(&dpcm->running, 0);
-	hrtimer_cancel(&dpcm->timer);
+	if (!hrtimer_callback_running(&dpcm->timer))
+		hrtimer_cancel(&dpcm->timer);
 	return 0;
 }
 
 static inline void dummy_hrtimer_sync(struct dummy_hrtimer_pcm *dpcm)
 {
 	hrtimer_cancel(&dpcm->timer);
-	tasklet_kill(&dpcm->tasklet);
 }
 
 static snd_pcm_uframes_t
@@ -465,12 +464,10 @@ static int dummy_hrtimer_create(struct snd_pcm_substream *substream)
 	if (!dpcm)
 		return -ENOMEM;
 	substream->runtime->private_data = dpcm;
-	hrtimer_init(&dpcm->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	hrtimer_init(&dpcm->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
 	dpcm->timer.function = dummy_hrtimer_callback;
 	dpcm->substream = substream;
 	atomic_set(&dpcm->running, 0);
-	tasklet_init(&dpcm->tasklet, dummy_hrtimer_pcm_elapsed,
-		     (unsigned long)dpcm);
 	return 0;
 }
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 34/36] ALSA/dummy: Replace tasklet with softirq hrtimer
@ 2017-12-21 10:42   ` Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:42 UTC (permalink / raw)
  To: LKML
  Cc: alsa-devel, keescook, Peter Zijlstra, Takashi Iwai,
	Takashi Sakamoto, Ingo Molnar, John Stultz, Thomas Gleixner,
	Anna-Maria Gleixner, Christoph Hellwig

From: Thomas Gleixner <tglx@linutronix.de>

The tasklet is used to defer the execution of snd_pcm_period_elapsed() to
the softirq context. Using the HRTIMER_MODE_SOFT mode invokes the timer
callback in softirq context as well which renders the tasklet useless.

[o-takashi: avoid stall due to a call of hrtimer_cancel() on a callback
            of hrtimer]

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Reviewed-by: Takashi Iwai <tiwai@suse.de>
Cc: alsa-devel@alsa-project.org
Cc: Takashi Iwai <tiwai@suse.com>
Cc: Takashi Sakamoto <o-takashi@sakamocchi.jp>
Cc: Jaroslav Kysela <perex@perex.cz>
Link: http://lkml.kernel.org/r/20170905161820.jtysvxtfleunbbmf@breakpoint.cc
---
 sound/drivers/dummy.c | 27 ++++++++++++---------------
 1 file changed, 12 insertions(+), 15 deletions(-)

diff --git a/sound/drivers/dummy.c b/sound/drivers/dummy.c
index 7b2b1f766b00..6ad2ff57833d 100644
--- a/sound/drivers/dummy.c
+++ b/sound/drivers/dummy.c
@@ -375,17 +375,9 @@ struct dummy_hrtimer_pcm {
 	ktime_t period_time;
 	atomic_t running;
 	struct hrtimer timer;
-	struct tasklet_struct tasklet;
 	struct snd_pcm_substream *substream;
 };
 
-static void dummy_hrtimer_pcm_elapsed(unsigned long priv)
-{
-	struct dummy_hrtimer_pcm *dpcm = (struct dummy_hrtimer_pcm *)priv;
-	if (atomic_read(&dpcm->running))
-		snd_pcm_period_elapsed(dpcm->substream);
-}
-
 static enum hrtimer_restart dummy_hrtimer_callback(struct hrtimer *timer)
 {
 	struct dummy_hrtimer_pcm *dpcm;
@@ -393,7 +385,14 @@ static enum hrtimer_restart dummy_hrtimer_callback(struct hrtimer *timer)
 	dpcm = container_of(timer, struct dummy_hrtimer_pcm, timer);
 	if (!atomic_read(&dpcm->running))
 		return HRTIMER_NORESTART;
-	tasklet_schedule(&dpcm->tasklet);
+	/*
+	 * In cases of XRUN and draining, this calls .trigger to stop PCM
+	 * substream.
+	 */
+	snd_pcm_period_elapsed(dpcm->substream);
+	if (!atomic_read(&dpcm->running))
+		return HRTIMER_NORESTART;
+
 	hrtimer_forward_now(timer, dpcm->period_time);
 	return HRTIMER_RESTART;
 }
@@ -403,7 +402,7 @@ static int dummy_hrtimer_start(struct snd_pcm_substream *substream)
 	struct dummy_hrtimer_pcm *dpcm = substream->runtime->private_data;
 
 	dpcm->base_time = hrtimer_cb_get_time(&dpcm->timer);
-	hrtimer_start(&dpcm->timer, dpcm->period_time, HRTIMER_MODE_REL);
+	hrtimer_start(&dpcm->timer, dpcm->period_time, HRTIMER_MODE_REL_SOFT);
 	atomic_set(&dpcm->running, 1);
 	return 0;
 }
@@ -413,14 +412,14 @@ static int dummy_hrtimer_stop(struct snd_pcm_substream *substream)
 	struct dummy_hrtimer_pcm *dpcm = substream->runtime->private_data;
 
 	atomic_set(&dpcm->running, 0);
-	hrtimer_cancel(&dpcm->timer);
+	if (!hrtimer_callback_running(&dpcm->timer))
+		hrtimer_cancel(&dpcm->timer);
 	return 0;
 }
 
 static inline void dummy_hrtimer_sync(struct dummy_hrtimer_pcm *dpcm)
 {
 	hrtimer_cancel(&dpcm->timer);
-	tasklet_kill(&dpcm->tasklet);
 }
 
 static snd_pcm_uframes_t
@@ -465,12 +464,10 @@ static int dummy_hrtimer_create(struct snd_pcm_substream *substream)
 	if (!dpcm)
 		return -ENOMEM;
 	substream->runtime->private_data = dpcm;
-	hrtimer_init(&dpcm->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	hrtimer_init(&dpcm->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
 	dpcm->timer.function = dummy_hrtimer_callback;
 	dpcm->substream = substream;
 	atomic_set(&dpcm->running, 0);
-	tasklet_init(&dpcm->tasklet, dummy_hrtimer_pcm_elapsed,
-		     (unsigned long)dpcm);
 	return 0;
 }
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 35/36] usb/gadget/NCM: Replace tasklet with softirq hrtimer
@ 2017-12-21 10:42   ` Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:42 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner,
	Felipe Balbi, linux-usb

From: Thomas Gleixner <tglx@linutronix.de>

The tx_tasklet tasklet is used in invoke the hrtimer (task_timer) in
softirq context. This can be also achieved without the tasklet but
with HRTIMER_MODE_SOFT as hrtimer mode.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Acked-by: Felipe Balbi <felipe.balbi@linux.intel.com>
Cc: Felipe Balbi <balbi@kernel.org>
Cc: linux-usb@vger.kernel.org
---
 drivers/usb/gadget/function/f_ncm.c | 30 +++++++-----------------------
 1 file changed, 7 insertions(+), 23 deletions(-)

diff --git a/drivers/usb/gadget/function/f_ncm.c b/drivers/usb/gadget/function/f_ncm.c
index c5bce8e22983..5780fba620ab 100644
--- a/drivers/usb/gadget/function/f_ncm.c
+++ b/drivers/usb/gadget/function/f_ncm.c
@@ -73,9 +73,7 @@ struct f_ncm {
 	struct sk_buff			*skb_tx_ndp;
 	u16				ndp_dgram_count;
 	bool				timer_force_tx;
-	struct tasklet_struct		tx_tasklet;
 	struct hrtimer			task_timer;
-
 	bool				timer_stopping;
 };
 
@@ -1104,7 +1102,7 @@ static struct sk_buff *ncm_wrap_ntb(struct gether *port,
 
 		/* Delay the timer. */
 		hrtimer_start(&ncm->task_timer, TX_TIMEOUT_NSECS,
-			      HRTIMER_MODE_REL);
+			      HRTIMER_MODE_REL_SOFT);
 
 		/* Add the datagram position entries */
 		ntb_ndp = skb_put_zero(ncm->skb_tx_ndp, dgram_idx_len);
@@ -1148,17 +1146,15 @@ static struct sk_buff *ncm_wrap_ntb(struct gether *port,
 }
 
 /*
- * This transmits the NTB if there are frames waiting.
+ * The transmit should only be run if no skb data has been sent
+ * for a certain duration.
  */
-static void ncm_tx_tasklet(unsigned long data)
+static enum hrtimer_restart ncm_tx_timeout(struct hrtimer *data)
 {
-	struct f_ncm	*ncm = (void *)data;
-
-	if (ncm->timer_stopping)
-		return;
+	struct f_ncm *ncm = container_of(data, struct f_ncm, task_timer);
 
 	/* Only send if data is available. */
-	if (ncm->skb_tx_data) {
+	if (!ncm->timer_stopping && ncm->skb_tx_data) {
 		ncm->timer_force_tx = true;
 
 		/* XXX This allowance of a NULL skb argument to ndo_start_xmit
@@ -1171,16 +1167,6 @@ static void ncm_tx_tasklet(unsigned long data)
 
 		ncm->timer_force_tx = false;
 	}
-}
-
-/*
- * The transmit should only be run if no skb data has been sent
- * for a certain duration.
- */
-static enum hrtimer_restart ncm_tx_timeout(struct hrtimer *data)
-{
-	struct f_ncm *ncm = container_of(data, struct f_ncm, task_timer);
-	tasklet_schedule(&ncm->tx_tasklet);
 	return HRTIMER_NORESTART;
 }
 
@@ -1513,8 +1499,7 @@ static int ncm_bind(struct usb_configuration *c, struct usb_function *f)
 	ncm->port.open = ncm_open;
 	ncm->port.close = ncm_close;
 
-	tasklet_init(&ncm->tx_tasklet, ncm_tx_tasklet, (unsigned long) ncm);
-	hrtimer_init(&ncm->task_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	hrtimer_init(&ncm->task_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
 	ncm->task_timer.function = ncm_tx_timeout;
 
 	DBG(cdev, "CDC Network: %s speed IN/%s OUT/%s NOTIFY/%s\n",
@@ -1623,7 +1608,6 @@ static void ncm_unbind(struct usb_configuration *c, struct usb_function *f)
 	DBG(c->cdev, "ncm unbind\n");
 
 	hrtimer_cancel(&ncm->task_timer);
-	tasklet_kill(&ncm->tx_tasklet);
 
 	ncm_string_defs[0].id = 0;
 	usb_free_all_descriptors(f);
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [v4,35/36] usb/gadget/NCM: Replace tasklet with softirq hrtimer
@ 2017-12-21 10:42   ` Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:42 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner,
	Felipe Balbi, linux-usb

From: Thomas Gleixner <tglx@linutronix.de>

The tx_tasklet tasklet is used in invoke the hrtimer (task_timer) in
softirq context. This can be also achieved without the tasklet but
with HRTIMER_MODE_SOFT as hrtimer mode.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Acked-by: Felipe Balbi <felipe.balbi@linux.intel.com>
Cc: Felipe Balbi <balbi@kernel.org>
Cc: linux-usb@vger.kernel.org
---
 drivers/usb/gadget/function/f_ncm.c | 30 +++++++-----------------------
 1 file changed, 7 insertions(+), 23 deletions(-)

diff --git a/drivers/usb/gadget/function/f_ncm.c b/drivers/usb/gadget/function/f_ncm.c
index c5bce8e22983..5780fba620ab 100644
--- a/drivers/usb/gadget/function/f_ncm.c
+++ b/drivers/usb/gadget/function/f_ncm.c
@@ -73,9 +73,7 @@ struct f_ncm {
 	struct sk_buff			*skb_tx_ndp;
 	u16				ndp_dgram_count;
 	bool				timer_force_tx;
-	struct tasklet_struct		tx_tasklet;
 	struct hrtimer			task_timer;
-
 	bool				timer_stopping;
 };
 
@@ -1104,7 +1102,7 @@ static struct sk_buff *ncm_wrap_ntb(struct gether *port,
 
 		/* Delay the timer. */
 		hrtimer_start(&ncm->task_timer, TX_TIMEOUT_NSECS,
-			      HRTIMER_MODE_REL);
+			      HRTIMER_MODE_REL_SOFT);
 
 		/* Add the datagram position entries */
 		ntb_ndp = skb_put_zero(ncm->skb_tx_ndp, dgram_idx_len);
@@ -1148,17 +1146,15 @@ static struct sk_buff *ncm_wrap_ntb(struct gether *port,
 }
 
 /*
- * This transmits the NTB if there are frames waiting.
+ * The transmit should only be run if no skb data has been sent
+ * for a certain duration.
  */
-static void ncm_tx_tasklet(unsigned long data)
+static enum hrtimer_restart ncm_tx_timeout(struct hrtimer *data)
 {
-	struct f_ncm	*ncm = (void *)data;
-
-	if (ncm->timer_stopping)
-		return;
+	struct f_ncm *ncm = container_of(data, struct f_ncm, task_timer);
 
 	/* Only send if data is available. */
-	if (ncm->skb_tx_data) {
+	if (!ncm->timer_stopping && ncm->skb_tx_data) {
 		ncm->timer_force_tx = true;
 
 		/* XXX This allowance of a NULL skb argument to ndo_start_xmit
@@ -1171,16 +1167,6 @@ static void ncm_tx_tasklet(unsigned long data)
 
 		ncm->timer_force_tx = false;
 	}
-}
-
-/*
- * The transmit should only be run if no skb data has been sent
- * for a certain duration.
- */
-static enum hrtimer_restart ncm_tx_timeout(struct hrtimer *data)
-{
-	struct f_ncm *ncm = container_of(data, struct f_ncm, task_timer);
-	tasklet_schedule(&ncm->tx_tasklet);
 	return HRTIMER_NORESTART;
 }
 
@@ -1513,8 +1499,7 @@ static int ncm_bind(struct usb_configuration *c, struct usb_function *f)
 	ncm->port.open = ncm_open;
 	ncm->port.close = ncm_close;
 
-	tasklet_init(&ncm->tx_tasklet, ncm_tx_tasklet, (unsigned long) ncm);
-	hrtimer_init(&ncm->task_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	hrtimer_init(&ncm->task_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
 	ncm->task_timer.function = ncm_tx_timeout;
 
 	DBG(cdev, "CDC Network: %s speed IN/%s OUT/%s NOTIFY/%s\n",
@@ -1623,7 +1608,6 @@ static void ncm_unbind(struct usb_configuration *c, struct usb_function *f)
 	DBG(c->cdev, "ncm unbind\n");
 
 	hrtimer_cancel(&ncm->task_timer);
-	tasklet_kill(&ncm->tx_tasklet);
 
 	ncm_string_defs[0].id = 0;
 	usb_free_all_descriptors(f);

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v4 36/36] net/mvpp2: Replace tasklet with softirq hrtimer
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (34 preceding siblings ...)
  2017-12-21 10:42   ` [v4,35/36] " Anna-Maria Gleixner
@ 2017-12-21 10:42 ` Anna-Maria Gleixner
  2018-01-16  1:39 ` [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Ingo Molnar
  36 siblings, 0 replies; 80+ messages in thread
From: Anna-Maria Gleixner @ 2017-12-21 10:42 UTC (permalink / raw)
  To: LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, Anna-Maria Gleixner,
	Thomas Petazzoni, netdev, David S. Miller

From: Thomas Gleixner <tglx@linutronix.de>

The tx_done_tasklet tasklet is used in invoke the hrtimer
(mvpp2_hr_timer_cb) in softirq context. This can be also achieved without
the tasklet but with HRTIMER_MODE_SOFT as hrtimer mode.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: netdev@vger.kernel.org
Cc: "David S. Miller" <davem@davemloft.net>
---
 drivers/net/ethernet/marvell/mvpp2.c | 62 +++++++++++++++---------------------
 1 file changed, 25 insertions(+), 37 deletions(-)

diff --git a/drivers/net/ethernet/marvell/mvpp2.c b/drivers/net/ethernet/marvell/mvpp2.c
index 634b2f41cc9e..41f12961e4d1 100644
--- a/drivers/net/ethernet/marvell/mvpp2.c
+++ b/drivers/net/ethernet/marvell/mvpp2.c
@@ -901,9 +901,8 @@ struct mvpp2_pcpu_stats {
 /* Per-CPU port control */
 struct mvpp2_port_pcpu {
 	struct hrtimer tx_done_timer;
+	struct net_device *dev;
 	bool timer_scheduled;
-	/* Tasklet for egress finalization */
-	struct tasklet_struct tx_done_tasklet;
 };
 
 struct mvpp2_queue_vector {
@@ -6156,46 +6155,34 @@ static void mvpp2_link_event(struct net_device *dev)
 	}
 }
 
-static void mvpp2_timer_set(struct mvpp2_port_pcpu *port_pcpu)
-{
-	ktime_t interval;
-
-	if (!port_pcpu->timer_scheduled) {
-		port_pcpu->timer_scheduled = true;
-		interval = MVPP2_TXDONE_HRTIMER_PERIOD_NS;
-		hrtimer_start(&port_pcpu->tx_done_timer, interval,
-			      HRTIMER_MODE_REL_PINNED);
-	}
-}
-
-static void mvpp2_tx_proc_cb(unsigned long data)
+static enum hrtimer_restart mvpp2_hr_timer_cb(struct hrtimer *timer)
 {
-	struct net_device *dev = (struct net_device *)data;
-	struct mvpp2_port *port = netdev_priv(dev);
-	struct mvpp2_port_pcpu *port_pcpu = this_cpu_ptr(port->pcpu);
+	struct net_device *dev;
+	struct mvpp2_port *port;
+	struct mvpp2_port_pcpu *port_pcpu;
 	unsigned int tx_todo, cause;
 
+	port_pcpu = container_of(timer, struct mvpp2_port_pcpu, tx_done_timer);
+	dev = port_pcpu->dev;
+
 	if (!netif_running(dev))
-		return;
+		return HRTIMER_NORESTART;
+
 	port_pcpu->timer_scheduled = false;
+	port = netdev_priv(dev);
 
 	/* Process all the Tx queues */
 	cause = (1 << port->ntxqs) - 1;
 	tx_todo = mvpp2_tx_done(port, cause, smp_processor_id());
 
 	/* Set the timer in case not all the packets were processed */
-	if (tx_todo)
-		mvpp2_timer_set(port_pcpu);
-}
-
-static enum hrtimer_restart mvpp2_hr_timer_cb(struct hrtimer *timer)
-{
-	struct mvpp2_port_pcpu *port_pcpu = container_of(timer,
-							 struct mvpp2_port_pcpu,
-							 tx_done_timer);
-
-	tasklet_schedule(&port_pcpu->tx_done_tasklet);
+	if (tx_todo && !port_pcpu->timer_scheduled) {
+		port_pcpu->timer_scheduled = true;
+		hrtimer_forward_now(&port_pcpu->tx_done_timer,
+				    MVPP2_TXDONE_HRTIMER_PERIOD_NS);
 
+		return HRTIMER_RESTART;
+	}
 	return HRTIMER_NORESTART;
 }
 
@@ -6673,7 +6660,12 @@ static int mvpp2_tx(struct sk_buff *skb, struct net_device *dev)
 	    txq_pcpu->count > 0) {
 		struct mvpp2_port_pcpu *port_pcpu = this_cpu_ptr(port->pcpu);
 
-		mvpp2_timer_set(port_pcpu);
+		if (!port_pcpu->timer_scheduled) {
+			port_pcpu->timer_scheduled = true;
+			hrtimer_start(&port_pcpu->tx_done_timer,
+				      MVPP2_TXDONE_HRTIMER_PERIOD_NS,
+				      HRTIMER_MODE_REL_PINNED_SOFT);
+		}
 	}
 
 	return NETDEV_TX_OK;
@@ -7108,7 +7100,6 @@ static int mvpp2_stop(struct net_device *dev)
 
 			hrtimer_cancel(&port_pcpu->tx_done_timer);
 			port_pcpu->timer_scheduled = false;
-			tasklet_kill(&port_pcpu->tx_done_tasklet);
 		}
 	}
 	mvpp2_cleanup_rxqs(port);
@@ -7899,13 +7890,10 @@ static int mvpp2_port_probe(struct platform_device *pdev,
 			port_pcpu = per_cpu_ptr(port->pcpu, cpu);
 
 			hrtimer_init(&port_pcpu->tx_done_timer, CLOCK_MONOTONIC,
-				     HRTIMER_MODE_REL_PINNED);
+				     HRTIMER_MODE_REL_PINNED_SOFT);
 			port_pcpu->tx_done_timer.function = mvpp2_hr_timer_cb;
 			port_pcpu->timer_scheduled = false;
-
-			tasklet_init(&port_pcpu->tx_done_tasklet,
-				     mvpp2_tx_proc_cb,
-				     (unsigned long)dev);
+			port_pcpu->dev = dev;
 		}
 	}
 
-- 
2.11.0

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [PATCH v5 01/36] timers: Use static keys for migrate_enable/nohz_active
  2017-12-21 10:41 ` [PATCH v4 01/36] timers: Use static keys for migrate_enable/nohz_active Anna-Maria Gleixner
@ 2017-12-22 15:45   ` Sebastian Andrzej Siewior
  2018-01-14 22:13     ` Thomas Gleixner
  2018-01-11  4:25   ` [PATCH v4 " Frederic Weisbecker
  2018-01-16  3:40   ` [tip:timers/core] hrtimer: Optimize the hrtimer code by using static keys for migration_enable/nohz_active tip-bot for Thomas Gleixner
  2 siblings, 1 reply; 80+ messages in thread
From: Sebastian Andrzej Siewior @ 2017-12-22 15:45 UTC (permalink / raw)
  To: Anna-Maria Gleixner, Thomas Gleixner
  Cc: LKML, Peter Zijlstra, Ingo Molnar, keescook, Christoph Hellwig,
	John Stultz

From: Thomas Gleixner <tglx@linutronix.de>

The members migrate_enable and nohz_active in the timer/hrtimer per CPU
bases have been introduced to avoid accessing global variables for these
decisions.

Still that results in a (cache hot) load and conditional branch, which can
be avoided by using static keys.

Implement it with static keys and optimize for the most critical case of
high performance networking which tends to disable the timer migration
functionality.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
v4…v5: dropping three chunks which are non-existent since
      "timer: Use deferrable base independent of base::nohz_active"
     and I assume that it gets applied before this one.

 include/linux/hrtimer.h     |    4 --
 kernel/time/hrtimer.c       |   17 ++------
 kernel/time/tick-internal.h |   19 ++++++---
 kernel/time/tick-sched.c    |    2 -
 kernel/time/timer.c         |   85 ++++++++++++++++++++++----------------------
 5 files changed, 62 insertions(+), 65 deletions(-)

--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -153,8 +153,6 @@ enum  hrtimer_base_type {
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
- * @migration_enabled:	The migration of hrtimers to other cpus is enabled
- * @nohz_active:	The nohz functionality is enabled
  * @expires_next:	absolute time of the next event which was scheduled
  *			via clock_set_next_event()
  * @next_timer:		Pointer to the first expiring timer
@@ -178,8 +176,6 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
-	bool				migration_enabled;
-	bool				nohz_active;
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
 					hres_active	: 1,
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -178,23 +178,16 @@ hrtimer_check_target(struct hrtimer *tim
 #endif
 }
 
-#ifdef CONFIG_NO_HZ_COMMON
-static inline
-struct hrtimer_cpu_base *get_target_base(struct hrtimer_cpu_base *base,
-					 int pinned)
-{
-	if (pinned || !base->migration_enabled)
-		return base;
-	return &per_cpu(hrtimer_bases, get_nohz_timer_target());
-}
-#else
 static inline
 struct hrtimer_cpu_base *get_target_base(struct hrtimer_cpu_base *base,
 					 int pinned)
 {
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+	if (static_branch_unlikely(&timers_migration_enabled) && !pinned)
+		return &per_cpu(hrtimer_bases, get_nohz_timer_target());
+#endif
 	return base;
 }
-#endif
 
 /*
  * We switch the timer base to a power-optimized selected CPU target,
@@ -971,7 +964,7 @@ void hrtimer_start_range_ns(struct hrtim
 		 * Kick to reschedule the next tick to handle the new timer
 		 * on dynticks target.
 		 */
-		if (new_base->cpu_base->nohz_active)
+		if (is_timers_nohz_active())
 			wake_up_nohz_cpu(new_base->cpu_base->cpu);
 	} else {
 		hrtimer_reprogram(timer, new_base);
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -150,14 +150,19 @@ static inline void tick_nohz_init(void)
 
 #ifdef CONFIG_NO_HZ_COMMON
 extern unsigned long tick_nohz_active;
-#else
+extern void timers_update_nohz(void);
+extern struct static_key_false timers_nohz_active;
+static inline bool is_timers_nohz_active(void)
+{
+	return static_branch_unlikely(&timers_nohz_active);
+}
+# ifdef CONFIG_SMP
+extern struct static_key_false timers_migration_enabled;
+# endif
+#else /* CONFIG_NO_HZ_COMMON */
+static inline void timers_update_nohz(void) { }
 #define tick_nohz_active (0)
-#endif
-
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
-extern void timers_update_migration(bool update_nohz);
-#else
-static inline void timers_update_migration(bool update_nohz) { }
+static inline bool is_timers_nohz_active(void) { return false; }
 #endif
 
 DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases);
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1103,7 +1103,7 @@ static inline void tick_nohz_activate(st
 	ts->nohz_mode = mode;
 	/* One update is enough */
 	if (!test_and_set_bit(0, &tick_nohz_active))
-		timers_update_migration(true);
+		timers_update_nohz();
 }
 
 /**
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -200,8 +200,6 @@ struct timer_base {
 	unsigned long		clk;
 	unsigned long		next_expiry;
 	unsigned int		cpu;
-	bool			migration_enabled;
-	bool			nohz_active;
 	bool			is_idle;
 	bool			must_forward_clk;
 	DECLARE_BITMAP(pending_map, WHEEL_SIZE);
@@ -210,45 +208,59 @@ struct timer_base {
 
 static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]);
 
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+#ifdef CONFIG_NO_HZ_COMMON
+
+DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
+static DEFINE_MUTEX(timer_keys_mutex);
+
+static void timer_update_keys(struct work_struct *work);
+static DECLARE_WORK(timer_update_work, timer_update_keys);
+
+#ifdef CONFIG_SMP
 unsigned int sysctl_timer_migration = 1;
 
-void timers_update_migration(bool update_nohz)
+DEFINE_STATIC_KEY_FALSE(timers_migration_enabled);
+
+static void timers_update_migration(void)
 {
 	bool on = sysctl_timer_migration && tick_nohz_active;
-	unsigned int cpu;
 
-	/* Avoid the loop, if nothing to update */
-	if (this_cpu_read(timer_bases[BASE_STD].migration_enabled) == on)
-		return;
+	if (on)
+		static_branch_enable(&timers_migration_enabled);
+	else
+		static_branch_disable(&timers_migration_enabled);
+}
+#else
+static inline void timers_update_migration(void) { }
+#endif /* !CONFIG_SMP */
 
-	for_each_possible_cpu(cpu) {
-		per_cpu(timer_bases[BASE_STD].migration_enabled, cpu) = on;
-		per_cpu(timer_bases[BASE_DEF].migration_enabled, cpu) = on;
-		per_cpu(hrtimer_bases.migration_enabled, cpu) = on;
-		if (!update_nohz)
-			continue;
-		per_cpu(timer_bases[BASE_STD].nohz_active, cpu) = true;
-		per_cpu(timer_bases[BASE_DEF].nohz_active, cpu) = true;
-		per_cpu(hrtimer_bases.nohz_active, cpu) = true;
-	}
+static void timer_update_keys(struct work_struct *work)
+{
+	mutex_lock(&timer_keys_mutex);
+	timers_update_migration();
+	static_branch_enable(&timers_nohz_active);
+	mutex_unlock(&timer_keys_mutex);
+}
+
+void timers_update_nohz(void)
+{
+	schedule_work(&timer_update_work);
 }
 
 int timer_migration_handler(struct ctl_table *table, int write,
 			    void __user *buffer, size_t *lenp,
 			    loff_t *ppos)
 {
-	static DEFINE_MUTEX(mutex);
 	int ret;
 
-	mutex_lock(&mutex);
+	mutex_lock(&timer_keys_mutex);
 	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
 	if (!ret && write)
-		timers_update_migration(false);
-	mutex_unlock(&mutex);
+		timers_update_migration();
+	mutex_unlock(&timer_keys_mutex);
 	return ret;
 }
-#endif
+#endif /* NO_HZ_COMMON */
 
 static unsigned long round_jiffies_common(unsigned long j, int cpu,
 		bool force_up)
@@ -534,7 +546,7 @@ static void
 static void
 trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer)
 {
-	if (!IS_ENABLED(CONFIG_NO_HZ_COMMON) || !base->nohz_active)
+	if (!is_timers_nohz_active())
 		return;
 
 	/*
@@ -840,21 +852,20 @@ static inline struct timer_base *get_tim
 	return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK);
 }
 
-#ifdef CONFIG_NO_HZ_COMMON
 static inline struct timer_base *
 get_target_base(struct timer_base *base, unsigned tflags)
 {
-#ifdef CONFIG_SMP
-	if ((tflags & TIMER_PINNED) || !base->migration_enabled)
-		return get_timer_this_cpu_base(tflags);
-	return get_timer_cpu_base(tflags, get_nohz_timer_target());
-#else
-	return get_timer_this_cpu_base(tflags);
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+	if (static_branch_unlikely(&timers_migration_enabled) &&
+	    !(tflags & TIMER_PINNED))
+		return get_timer_cpu_base(tflags, get_nohz_timer_target());
 #endif
+	return get_timer_this_cpu_base(tflags);
 }
 
 static inline void forward_timer_base(struct timer_base *base)
 {
+#ifdef CONFIG_NO_HZ_COMMON
 	unsigned long jnow;
 
 	/*
@@ -878,16 +889,8 @@ static inline void forward_timer_base(st
 		base->clk = jnow;
 	else
 		base->clk = base->next_expiry;
-}
-#else
-static inline struct timer_base *
-get_target_base(struct timer_base *base, unsigned tflags)
-{
-	return get_timer_this_cpu_base(tflags);
-}
-
-static inline void forward_timer_base(struct timer_base *base) { }
 #endif
+}
 
 
 /*
@@ -1656,7 +1659,7 @@ void run_local_timers(void)
 	hrtimer_run_queues();
 	/* Raise the softirq only if required. */
 	if (time_before(jiffies, base->clk)) {
-		if (!IS_ENABLED(CONFIG_NO_HZ_COMMON) || !base->nohz_active)
+		if (!is_timers_nohz_active())
 			return;
 		/* CPU is awake, so check the deferrable base. */
 		base++;

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 31/36] mac80211_hwsim: Replace hrtimer tasklet with softirq hrtimer
  2017-12-21 10:42 ` [PATCH v4 31/36] mac80211_hwsim: Replace hrtimer tasklet with softirq hrtimer Anna-Maria Gleixner
@ 2018-01-04 15:12   ` Johannes Berg
  2018-01-04 15:18     ` Thomas Gleixner
  0 siblings, 1 reply; 80+ messages in thread
From: Johannes Berg @ 2018-01-04 15:12 UTC (permalink / raw)
  To: Anna-Maria Gleixner, LKML
  Cc: Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, linux-wireless, Kalle Valo

On Thu, 2017-12-21 at 11:42 +0100, Anna-Maria Gleixner wrote:
> From: Thomas Gleixner <tglx@linutronix.de>
> 
> Switch the timer to HRTIMER_MODE_SOFT, which executed the timer

I've pointed out before that this should say HRTIMER_MODE_REL_SOFT.

Anyway, since it still doesn't apply on my tree, somebody else will
have to pick it up.

johannes

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 31/36] mac80211_hwsim: Replace hrtimer tasklet with softirq hrtimer
  2018-01-04 15:12   ` Johannes Berg
@ 2018-01-04 15:18     ` Thomas Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: Thomas Gleixner @ 2018-01-04 15:18 UTC (permalink / raw)
  To: Johannes Berg
  Cc: Anna-Maria Gleixner, LKML, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz, linux-wireless, Kalle Valo

On Thu, 4 Jan 2018, Johannes Berg wrote:

> On Thu, 2017-12-21 at 11:42 +0100, Anna-Maria Gleixner wrote:
> > From: Thomas Gleixner <tglx@linutronix.de>
> > 
> > Switch the timer to HRTIMER_MODE_SOFT, which executed the timer
> 
> I've pointed out before that this should say HRTIMER_MODE_REL_SOFT.
> 
> Anyway, since it still doesn't apply on my tree, somebody else will
> have to pick it up.

Yes. I'm going to do that or decide to postpone the conversion
patches. Have not yet thought about it as I was distracted for the last 2
month by something unpleasant....

Thanks,

	tglx

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 01/36] timers: Use static keys for migrate_enable/nohz_active
  2017-12-21 10:41 ` [PATCH v4 01/36] timers: Use static keys for migrate_enable/nohz_active Anna-Maria Gleixner
  2017-12-22 15:45   ` [PATCH v5 " Sebastian Andrzej Siewior
@ 2018-01-11  4:25   ` Frederic Weisbecker
  2018-01-16  3:40   ` [tip:timers/core] hrtimer: Optimize the hrtimer code by using static keys for migration_enable/nohz_active tip-bot for Thomas Gleixner
  2 siblings, 0 replies; 80+ messages in thread
From: Frederic Weisbecker @ 2018-01-11  4:25 UTC (permalink / raw)
  To: Anna-Maria Gleixner, Sebastian Andrzej Siewior
  Cc: LKML, Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz

On Thu, Dec 21, 2017 at 11:41:30AM +0100, Anna-Maria Gleixner wrote:
> From: Thomas Gleixner <tglx@linutronix.de>
> 
> The members migrate_enable and nohz_active in the timer/hrtimer per CPU
> bases have been introduced to avoid accessing global variables for these
> decisions.
> 
> Still that results in a (cache hot) load and conditional branch, which can
> be avoided by using static keys.
> 
> Implement it with static keys and optimize for the most critical case of
> high performance networking which tends to disable the timer migration
> functionality.
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
> ---
>  include/linux/hrtimer.h     |  4 --
>  kernel/time/hrtimer.c       | 17 +++------
>  kernel/time/tick-internal.h | 19 ++++++----
>  kernel/time/tick-sched.c    |  2 +-
>  kernel/time/timer.c         | 91 +++++++++++++++++++++++----------------------
>  5 files changed, 65 insertions(+), 68 deletions(-)
> 
> diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
> index f8e1845aa464..4ac74dff59f0 100644
> --- a/kernel/time/tick-internal.h
> +++ b/kernel/time/tick-internal.h
> @@ -150,14 +150,19 @@ static inline void tick_nohz_init(void) { }
>  
>  #ifdef CONFIG_NO_HZ_COMMON
>  extern unsigned long tick_nohz_active;
> -#else
> +extern void timers_update_nohz(void);
> +extern struct static_key_false timers_nohz_active;
> +static inline bool is_timers_nohz_active(void)
> +{
> +	return static_branch_unlikely(&timers_nohz_active);

Shouldn't we expect instead that timers_nohz_active is a likely scenario?
I guess deactivating nohz is for specific workloads that can't stand the
deep state wake up latency.

Also perhaps the above symbols should be harmonized, something like:

timers_nohz_update()
timers_nohz_active_key
timers_nohz_active()

> diff --git a/kernel/time/timer.c b/kernel/time/timer.c
> index ffebcf878fba..1e2140a23044 100644
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -200,8 +200,6 @@ struct timer_base {
>  	unsigned long		clk;
>  	unsigned long		next_expiry;
>  	unsigned int		cpu;
> -	bool			migration_enabled;
> -	bool			nohz_active;
>  	bool			is_idle;
>  	bool			must_forward_clk;
>  	DECLARE_BITMAP(pending_map, WHEEL_SIZE);
> @@ -210,45 +208,59 @@ struct timer_base {
>  
>  static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]);
>  
> -#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
> +#ifdef CONFIG_NO_HZ_COMMON
> +
> +DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
> +static DEFINE_MUTEX(timer_keys_mutex);
> +
> +static void timer_update_keys(struct work_struct *work);
> +static DECLARE_WORK(timer_update_work, timer_update_keys);
> +
> +#ifdef CONFIG_SMP
>  unsigned int sysctl_timer_migration = 1;
>  
> -void timers_update_migration(bool update_nohz)
> +DEFINE_STATIC_KEY_FALSE(timers_migration_enabled);
> +
> +static void timers_update_migration(void)
>  {
>  	bool on = sysctl_timer_migration && tick_nohz_active;
> -	unsigned int cpu;
>  
> -	/* Avoid the loop, if nothing to update */
> -	if (this_cpu_read(timer_bases[BASE_STD].migration_enabled) == on)
> -		return;
> +	if (on)

You may as well put the condition directly ;)

> +		static_branch_enable(&timers_migration_enabled);
> +	else
> +		static_branch_disable(&timers_migration_enabled);
> +}

Thanks!

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment
  2017-12-21 10:41 ` [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment Anna-Maria Gleixner
@ 2018-01-11 18:28   ` Frederic Weisbecker
  2018-01-16  1:02   ` Ingo Molnar
  2018-01-16  3:40   ` [tip:timers/core] hrtimer: Correct blatantly incorrect comment tip-bot for Thomas Gleixner
  2 siblings, 0 replies; 80+ messages in thread
From: Frederic Weisbecker @ 2018-01-11 18:28 UTC (permalink / raw)
  To: Anna-Maria Gleixner
  Cc: LKML, Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz

On Thu, Dec 21, 2017 at 11:41:31AM +0100, Anna-Maria Gleixner wrote:
> From: Thomas Gleixner <tglx@linutronix.de>
> 
> The protection of a hrtimer which runs its callback against migration to a
> different CPU has nothing to do with hard interrupt context.
> 
> The protection against migration of a hrtimer running the expiry callback
> is the pointer in the cpu_base which holds a pointer to the currently
> running timer. This pointer is evaluated in the code which potentially
> switches the timer base and makes sure it's kept on the CPU on which the
> callback is running.
> 
> Reported-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>

Reviewed-by: Frederic Weisbecker <frederic@kernel.org>

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v5 01/36] timers: Use static keys for migrate_enable/nohz_active
  2017-12-22 15:45   ` [PATCH v5 " Sebastian Andrzej Siewior
@ 2018-01-14 22:13     ` Thomas Gleixner
  2018-01-14 22:30       ` [PATCH v6 " Thomas Gleixner
  0 siblings, 1 reply; 80+ messages in thread
From: Thomas Gleixner @ 2018-01-14 22:13 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior
  Cc: Anna-Maria Gleixner, LKML, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz

On Fri, 22 Dec 2017, Sebastian Andrzej Siewior wrote:
> -	if (!IS_ENABLED(CONFIG_NO_HZ_COMMON) || !base->nohz_active)
> +	if (!is_timers_nohz_active())

That's wrong. You are undoing the mainline changes. I'll send an updated one.

Thanks,

	tglx

^ permalink raw reply	[flat|nested] 80+ messages in thread

* [PATCH v6 01/36] timers: Use static keys for migrate_enable/nohz_active
  2018-01-14 22:13     ` Thomas Gleixner
@ 2018-01-14 22:30       ` Thomas Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: Thomas Gleixner @ 2018-01-14 22:30 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior
  Cc: Anna-Maria Gleixner, LKML, Peter Zijlstra, Ingo Molnar,
	Kees Cook, Christoph Hellwig, John Stultz, Frederic Weisbecker


From: Thomas Gleixner <tglx@linutronix.de>

The members migrate_enable and nohz_active in the timer/hrtimer per CPU
bases have been introduced to avoid accessing global variables for these
decisions.

Still that results in a (cache hot) load and conditional branch, which can
be avoided by using static keys.

Implement it with static keys and optimize for the most critical case of
high performance networking which tends to disable the timer migration
functionality.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Christoph Hellwig <hch@lst.de>
Link: https://lkml.kernel.org/r/20171221104205.7269-2-anna-maria@linutronix.de
---

V6: Updated to latest changes in mainline and timers/urgent. Addressed Frederics
    review comments.

---
 include/linux/hrtimer.h     |    4 --
 kernel/time/hrtimer.c       |   17 ++-------
 kernel/time/tick-internal.h |   19 ++++++----
 kernel/time/tick-sched.c    |    2 -
 kernel/time/timer.c         |   83 ++++++++++++++++++++++----------------------
 5 files changed, 60 insertions(+), 65 deletions(-)

--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -153,8 +153,6 @@ enum  hrtimer_base_type {
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
- * @migration_enabled:	The migration of hrtimers to other cpus is enabled
- * @nohz_active:	The nohz functionality is enabled
  * @expires_next:	absolute time of the next event which was scheduled
  *			via clock_set_next_event()
  * @next_timer:		Pointer to the first expiring timer
@@ -178,8 +176,6 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
-	bool				migration_enabled;
-	bool				nohz_active;
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
 					hres_active	: 1,
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -178,23 +178,16 @@ hrtimer_check_target(struct hrtimer *tim
 #endif
 }
 
-#ifdef CONFIG_NO_HZ_COMMON
-static inline
-struct hrtimer_cpu_base *get_target_base(struct hrtimer_cpu_base *base,
-					 int pinned)
-{
-	if (pinned || !base->migration_enabled)
-		return base;
-	return &per_cpu(hrtimer_bases, get_nohz_timer_target());
-}
-#else
 static inline
 struct hrtimer_cpu_base *get_target_base(struct hrtimer_cpu_base *base,
 					 int pinned)
 {
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+	if (static_branch_likely(&timers_migration_enabled) && !pinned)
+		return &per_cpu(hrtimer_bases, get_nohz_timer_target());
+#endif
 	return base;
 }
-#endif
 
 /*
  * We switch the timer base to a power-optimized selected CPU target,
@@ -969,7 +962,7 @@ void hrtimer_start_range_ns(struct hrtim
 		 * Kick to reschedule the next tick to handle the new timer
 		 * on dynticks target.
 		 */
-		if (new_base->cpu_base->nohz_active)
+		if (is_timers_nohz_active())
 			wake_up_nohz_cpu(new_base->cpu_base->cpu);
 	} else {
 		hrtimer_reprogram(timer, new_base);
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -150,14 +150,19 @@ static inline void tick_nohz_init(void)
 
 #ifdef CONFIG_NO_HZ_COMMON
 extern unsigned long tick_nohz_active;
-#else
+extern void timers_update_nohz(void);
+extern struct static_key_false timers_nohz_active;
+static inline bool is_timers_nohz_active(void)
+{
+	return static_branch_likely(&timers_nohz_active);
+}
+# ifdef CONFIG_SMP
+extern struct static_key_false timers_migration_enabled;
+# endif
+#else /* CONFIG_NO_HZ_COMMON */
+static inline void timers_update_nohz(void) { }
 #define tick_nohz_active (0)
-#endif
-
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
-extern void timers_update_migration(bool update_nohz);
-#else
-static inline void timers_update_migration(bool update_nohz) { }
+static inline bool is_timers_nohz_active(void) { return false; }
 #endif
 
 DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases);
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1107,7 +1107,7 @@ static inline void tick_nohz_activate(st
 	ts->nohz_mode = mode;
 	/* One update is enough */
 	if (!test_and_set_bit(0, &tick_nohz_active))
-		timers_update_migration(true);
+		timers_update_nohz();
 }
 
 /**
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -200,8 +200,6 @@ struct timer_base {
 	unsigned long		clk;
 	unsigned long		next_expiry;
 	unsigned int		cpu;
-	bool			migration_enabled;
-	bool			nohz_active;
 	bool			is_idle;
 	bool			must_forward_clk;
 	DECLARE_BITMAP(pending_map, WHEEL_SIZE);
@@ -210,45 +208,57 @@ struct timer_base {
 
 static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]);
 
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+#ifdef CONFIG_NO_HZ_COMMON
+
+DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
+static DEFINE_MUTEX(timer_keys_mutex);
+
+static void timer_update_keys(struct work_struct *work);
+static DECLARE_WORK(timer_update_work, timer_update_keys);
+
+#ifdef CONFIG_SMP
 unsigned int sysctl_timer_migration = 1;
 
-void timers_update_migration(bool update_nohz)
+DEFINE_STATIC_KEY_FALSE(timers_migration_enabled);
+
+static void timers_update_migration(void)
 {
-	bool on = sysctl_timer_migration && tick_nohz_active;
-	unsigned int cpu;
+	if (sysctl_timer_migration && tick_nohz_active)
+		static_branch_enable(&timers_migration_enabled);
+	else
+		static_branch_disable(&timers_migration_enabled);
+}
+#else
+static inline void timers_update_migration(void) { }
+#endif /* !CONFIG_SMP */
 
-	/* Avoid the loop, if nothing to update */
-	if (this_cpu_read(timer_bases[BASE_STD].migration_enabled) == on)
-		return;
+static void timer_update_keys(struct work_struct *work)
+{
+	mutex_lock(&timer_keys_mutex);
+	timers_update_migration();
+	static_branch_enable(&timers_nohz_active);
+	mutex_unlock(&timer_keys_mutex);
+}
 
-	for_each_possible_cpu(cpu) {
-		per_cpu(timer_bases[BASE_STD].migration_enabled, cpu) = on;
-		per_cpu(timer_bases[BASE_DEF].migration_enabled, cpu) = on;
-		per_cpu(hrtimer_bases.migration_enabled, cpu) = on;
-		if (!update_nohz)
-			continue;
-		per_cpu(timer_bases[BASE_STD].nohz_active, cpu) = true;
-		per_cpu(timer_bases[BASE_DEF].nohz_active, cpu) = true;
-		per_cpu(hrtimer_bases.nohz_active, cpu) = true;
-	}
+void timers_update_nohz(void)
+{
+	schedule_work(&timer_update_work);
 }
 
 int timer_migration_handler(struct ctl_table *table, int write,
 			    void __user *buffer, size_t *lenp,
 			    loff_t *ppos)
 {
-	static DEFINE_MUTEX(mutex);
 	int ret;
 
-	mutex_lock(&mutex);
+	mutex_lock(&timer_keys_mutex);
 	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
 	if (!ret && write)
-		timers_update_migration(false);
-	mutex_unlock(&mutex);
+		timers_update_migration();
+	mutex_unlock(&timer_keys_mutex);
 	return ret;
 }
-#endif
+#endif /* NO_HZ_COMMON */
 
 static unsigned long round_jiffies_common(unsigned long j, int cpu,
 		bool force_up)
@@ -534,7 +544,7 @@ static void
 static void
 trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer)
 {
-	if (!IS_ENABLED(CONFIG_NO_HZ_COMMON) || !base->nohz_active)
+	if (!is_timers_nohz_active())
 		return;
 
 	/*
@@ -849,21 +859,20 @@ static inline struct timer_base *get_tim
 	return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK);
 }
 
-#ifdef CONFIG_NO_HZ_COMMON
 static inline struct timer_base *
 get_target_base(struct timer_base *base, unsigned tflags)
 {
-#ifdef CONFIG_SMP
-	if ((tflags & TIMER_PINNED) || !base->migration_enabled)
-		return get_timer_this_cpu_base(tflags);
-	return get_timer_cpu_base(tflags, get_nohz_timer_target());
-#else
-	return get_timer_this_cpu_base(tflags);
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+	if (static_branch_likely(&timers_migration_enabled) &&
+	    !(tflags & TIMER_PINNED))
+		return get_timer_cpu_base(tflags, get_nohz_timer_target());
 #endif
+	return get_timer_this_cpu_base(tflags);
 }
 
 static inline void forward_timer_base(struct timer_base *base)
 {
+#ifdef CONFIG_NO_HZ_COMMON
 	unsigned long jnow;
 
 	/*
@@ -887,16 +896,8 @@ static inline void forward_timer_base(st
 		base->clk = jnow;
 	else
 		base->clk = base->next_expiry;
-}
-#else
-static inline struct timer_base *
-get_target_base(struct timer_base *base, unsigned tflags)
-{
-	return get_timer_this_cpu_base(tflags);
-}
-
-static inline void forward_timer_base(struct timer_base *base) { }
 #endif
+}
 
 
 /*

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment
  2017-12-21 10:41 ` [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment Anna-Maria Gleixner
  2018-01-11 18:28   ` Frederic Weisbecker
@ 2018-01-16  1:02   ` Ingo Molnar
  2018-01-16  3:40   ` [tip:timers/core] hrtimer: Correct blatantly incorrect comment tip-bot for Thomas Gleixner
  2 siblings, 0 replies; 80+ messages in thread
From: Ingo Molnar @ 2018-01-16  1:02 UTC (permalink / raw)
  To: Anna-Maria Gleixner
  Cc: LKML, Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz


* Anna-Maria Gleixner <anna-maria@linutronix.de> wrote:

> From: Thomas Gleixner <tglx@linutronix.de>
> 
> The protection of a hrtimer which runs its callback against migration to a
> different CPU has nothing to do with hard interrupt context.
> 
> The protection against migration of a hrtimer running the expiry callback
> is the pointer in the cpu_base which holds a pointer to the currently
> running timer. This pointer is evaluated in the code which potentially
> switches the timer base and makes sure it's kept on the CPU on which the
> callback is running.
> 
> Reported-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
> ---
>  kernel/time/hrtimer.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
> index 69d203d8b12d..aee49c0c58b9 100644
> --- a/kernel/time/hrtimer.c
> +++ b/kernel/time/hrtimer.c
> @@ -1195,9 +1195,9 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
>  		timer->is_rel = false;
>  
>  	/*
> -	 * Because we run timers from hardirq context, there is no chance
> -	 * they get migrated to another cpu, therefore its safe to unlock
> -	 * the timer base.
> +	 * The timer is marked as running in the cpu base, so it is
> +	 * protected against migration to a different CPU even if the lock
> +	 * is dropped.

Small, nit:

   s/cpu/CPU

to keep the capitalization consistent within the same sentence.

Thanks,

	Ingo

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers
  2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
                   ` (35 preceding siblings ...)
  2017-12-21 10:42 ` [PATCH v4 36/36] net/mvpp2: " Anna-Maria Gleixner
@ 2018-01-16  1:39 ` Ingo Molnar
  2018-01-16  2:03   ` Ingo Molnar
  36 siblings, 1 reply; 80+ messages in thread
From: Ingo Molnar @ 2018-01-16  1:39 UTC (permalink / raw)
  To: Anna-Maria Gleixner
  Cc: LKML, Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz


* Anna-Maria Gleixner <anna-maria@linutronix.de> wrote:

> There are quite some places in the kernel which use a combination of
> hrtimers and tasklets to make use of the precise expiry of hrtimers, which
> schedule a tasklet to bring the actual function into softirq context.
> 
> This was introduced when the previous hrtimer softirq code was
> removed. That code was implemented by expiring the timer in hard irq
> context and then deferring the execution of the callback into softirq
> context. That caused a lot of pointless shuffling between the rbtree and a
> linked list.
> 
> In recent discussions it turned out that more potential users of hrtimers
> in softirq context might come up. Aside of that the RT patches need this
> functionality as well to defer hrtimers into softirq context if their
> callbacks are not interrupt safe on RT.
> 
> This series implements a new approach by adding SOFT hrtimer mode and
> instead of doing the list shuffle, timers started with this mode are put
> into separate soft expiry hrtimer queues. These queues are evaluated only
> when the hardirq context detects that the first expiring timer in the
> softirq queues has expired. That makes the overhead in the hardirq context
> minimal.
> 
> The series reworks the code to reuse as much as possible from the existing
> facilities for the new softirq hrtimers and integrates them with all
> flavours of hrtimers (HIGH_RES=y/n - NOHZ=y/n).
> 
> To achieve this quite some of the conditionals in the existing code are
> removed for the price of adding some pointless data and state tracking to
> the HIGH_RES=n case. That's minimal, but well worth it as it increases the
> readability and maintainability of the code.
> 
> The first part of the series implements the new functionality and the
> second part converts the hrtimer/tasklet users to make use of it and
> removes struct hrtimer_tasklet and the surrounding helper functions.
> 
> This series is available from git as well:
> 
> 	git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git WIP.timers
> 
> Thanks,
> 
>         Anna-Maria

Nice work!

A general heads-up: I've applied most of this series to tip:timers/core, and while 
reviewing the changes I've tidied up a number of changelogs and titles and 
modified a few in-code comments as well - but have not changed any logic code.

No serious changes intended, but please double check the end result once I've 
pushed it out after local testing.

Thanks,

	Ingo

^ permalink raw reply	[flat|nested] 80+ messages in thread

* Re: [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers
  2018-01-16  1:39 ` [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Ingo Molnar
@ 2018-01-16  2:03   ` Ingo Molnar
  0 siblings, 0 replies; 80+ messages in thread
From: Ingo Molnar @ 2018-01-16  2:03 UTC (permalink / raw)
  To: Anna-Maria Gleixner
  Cc: LKML, Thomas Gleixner, Peter Zijlstra, Ingo Molnar, keescook,
	Christoph Hellwig, John Stultz


* Ingo Molnar <mingo@kernel.org> wrote:

> No serious changes intended, but please double check the end result once I've 
> pushed it out after local testing.

I made the fix below for the !NO_HZ_COMMON || !HIGH_RES_TIMERS case, which would 
fail with a build failure due to non-available __hrtimer_next_event_base().

These functions won't get called, so the fix I picked was to make the code 
unconditionally available.

That's the model I'd encourage going forward: we should reduce the !hrtimer and 
hrtimer differences, in fact we should eventually implement !hrtimers as a 
compatibility mode of the hrtimers code, with no separate #ifdeffery.

Thanks,

	Ingo

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 5bd0fe9b6402..ba4674e9adc2 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -487,7 +487,6 @@ __next_base(struct hrtimer_cpu_base *cpu_base, unsigned int *active)
 #define for_each_active_base(base, cpu_base, active)	\
 	while ((base = __next_base((cpu_base), &(active))))
 
-#if defined(CONFIG_NO_HZ_COMMON) || defined(CONFIG_HIGH_RES_TIMERS)
 static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
 					 unsigned int active,
 					 ktime_t expires_next)
@@ -539,7 +538,6 @@ static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base,
 
 	return expires_next;
 }
-#endif
 
 static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base)
 {

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Optimize the hrtimer code by using static keys for migration_enable/nohz_active
  2017-12-21 10:41 ` [PATCH v4 01/36] timers: Use static keys for migrate_enable/nohz_active Anna-Maria Gleixner
  2017-12-22 15:45   ` [PATCH v5 " Sebastian Andrzej Siewior
  2018-01-11  4:25   ` [PATCH v4 " Frederic Weisbecker
@ 2018-01-16  3:40   ` tip-bot for Thomas Gleixner
  2 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Thomas Gleixner @ 2018-01-16  3:40 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: fweisbec, tglx, anna-maria, john.stultz, linux-kernel, hpa,
	bigeasy, torvalds, hch, mingo, peterz

Commit-ID:  ae67badaa1643253998cb21d5782e4ea7c231a29
Gitweb:     https://git.kernel.org/tip/ae67badaa1643253998cb21d5782e4ea7c231a29
Author:     Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Sun, 14 Jan 2018 23:30:51 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:44 +0100

hrtimer: Optimize the hrtimer code by using static keys for migration_enable/nohz_active

The hrtimer_cpu_base::migration_enable and ::nohz_active fields
were originally introduced to avoid accessing global variables
for these decisions.

Still that results in a (cache hot) load and conditional branch,
which can be avoided by using static keys.

Implement it with static keys and optimize for the most critical
case of high performance networking which tends to disable the
timer migration functionality.

No change in functionality.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/alpine.DEB.2.20.1801142327490.2371@nanos
Link: https://lkml.kernel.org/r/20171221104205.7269-2-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h     |  4 ---
 kernel/time/hrtimer.c       | 17 +++-------
 kernel/time/tick-internal.h | 19 +++++++----
 kernel/time/tick-sched.c    |  2 +-
 kernel/time/timer.c         | 83 +++++++++++++++++++++++----------------------
 5 files changed, 60 insertions(+), 65 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 012c37f..79b2a8d 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -153,8 +153,6 @@ enum  hrtimer_base_type {
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
- * @migration_enabled:	The migration of hrtimers to other cpus is enabled
- * @nohz_active:	The nohz functionality is enabled
  * @expires_next:	absolute time of the next event which was scheduled
  *			via clock_set_next_event()
  * @next_timer:		Pointer to the first expiring timer
@@ -178,8 +176,6 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
-	bool				migration_enabled;
-	bool				nohz_active;
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
 					hres_active	: 1,
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index d325208..1d06d2b 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -178,23 +178,16 @@ hrtimer_check_target(struct hrtimer *timer, struct hrtimer_clock_base *new_base)
 #endif
 }
 
-#ifdef CONFIG_NO_HZ_COMMON
-static inline
-struct hrtimer_cpu_base *get_target_base(struct hrtimer_cpu_base *base,
-					 int pinned)
-{
-	if (pinned || !base->migration_enabled)
-		return base;
-	return &per_cpu(hrtimer_bases, get_nohz_timer_target());
-}
-#else
 static inline
 struct hrtimer_cpu_base *get_target_base(struct hrtimer_cpu_base *base,
 					 int pinned)
 {
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+	if (static_branch_likely(&timers_migration_enabled) && !pinned)
+		return &per_cpu(hrtimer_bases, get_nohz_timer_target());
+#endif
 	return base;
 }
-#endif
 
 /*
  * We switch the timer base to a power-optimized selected CPU target,
@@ -969,7 +962,7 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 		 * Kick to reschedule the next tick to handle the new timer
 		 * on dynticks target.
 		 */
-		if (new_base->cpu_base->nohz_active)
+		if (is_timers_nohz_active())
 			wake_up_nohz_cpu(new_base->cpu_base->cpu);
 	} else {
 		hrtimer_reprogram(timer, new_base);
diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index f8e1845..f690628 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -150,14 +150,19 @@ static inline void tick_nohz_init(void) { }
 
 #ifdef CONFIG_NO_HZ_COMMON
 extern unsigned long tick_nohz_active;
-#else
+extern void timers_update_nohz(void);
+extern struct static_key_false timers_nohz_active;
+static inline bool is_timers_nohz_active(void)
+{
+	return static_branch_likely(&timers_nohz_active);
+}
+# ifdef CONFIG_SMP
+extern struct static_key_false timers_migration_enabled;
+# endif
+#else /* CONFIG_NO_HZ_COMMON */
+static inline void timers_update_nohz(void) { }
 #define tick_nohz_active (0)
-#endif
-
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
-extern void timers_update_migration(bool update_nohz);
-#else
-static inline void timers_update_migration(bool update_nohz) { }
+static inline bool is_timers_nohz_active(void) { return false; }
 #endif
 
 DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases);
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index f7cc7ab..29a5733 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -1107,7 +1107,7 @@ static inline void tick_nohz_activate(struct tick_sched *ts, int mode)
 	ts->nohz_mode = mode;
 	/* One update is enough */
 	if (!test_and_set_bit(0, &tick_nohz_active))
-		timers_update_migration(true);
+		timers_update_nohz();
 }
 
 /**
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 0bcf00e..d530f72 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -200,8 +200,6 @@ struct timer_base {
 	unsigned long		clk;
 	unsigned long		next_expiry;
 	unsigned int		cpu;
-	bool			migration_enabled;
-	bool			nohz_active;
 	bool			is_idle;
 	bool			must_forward_clk;
 	DECLARE_BITMAP(pending_map, WHEEL_SIZE);
@@ -210,45 +208,57 @@ struct timer_base {
 
 static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]);
 
-#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+#ifdef CONFIG_NO_HZ_COMMON
+
+DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
+static DEFINE_MUTEX(timer_keys_mutex);
+
+static void timer_update_keys(struct work_struct *work);
+static DECLARE_WORK(timer_update_work, timer_update_keys);
+
+#ifdef CONFIG_SMP
 unsigned int sysctl_timer_migration = 1;
 
-void timers_update_migration(bool update_nohz)
+DEFINE_STATIC_KEY_FALSE(timers_migration_enabled);
+
+static void timers_update_migration(void)
 {
-	bool on = sysctl_timer_migration && tick_nohz_active;
-	unsigned int cpu;
+	if (sysctl_timer_migration && tick_nohz_active)
+		static_branch_enable(&timers_migration_enabled);
+	else
+		static_branch_disable(&timers_migration_enabled);
+}
+#else
+static inline void timers_update_migration(void) { }
+#endif /* !CONFIG_SMP */
 
-	/* Avoid the loop, if nothing to update */
-	if (this_cpu_read(timer_bases[BASE_STD].migration_enabled) == on)
-		return;
+static void timer_update_keys(struct work_struct *work)
+{
+	mutex_lock(&timer_keys_mutex);
+	timers_update_migration();
+	static_branch_enable(&timers_nohz_active);
+	mutex_unlock(&timer_keys_mutex);
+}
 
-	for_each_possible_cpu(cpu) {
-		per_cpu(timer_bases[BASE_STD].migration_enabled, cpu) = on;
-		per_cpu(timer_bases[BASE_DEF].migration_enabled, cpu) = on;
-		per_cpu(hrtimer_bases.migration_enabled, cpu) = on;
-		if (!update_nohz)
-			continue;
-		per_cpu(timer_bases[BASE_STD].nohz_active, cpu) = true;
-		per_cpu(timer_bases[BASE_DEF].nohz_active, cpu) = true;
-		per_cpu(hrtimer_bases.nohz_active, cpu) = true;
-	}
+void timers_update_nohz(void)
+{
+	schedule_work(&timer_update_work);
 }
 
 int timer_migration_handler(struct ctl_table *table, int write,
 			    void __user *buffer, size_t *lenp,
 			    loff_t *ppos)
 {
-	static DEFINE_MUTEX(mutex);
 	int ret;
 
-	mutex_lock(&mutex);
+	mutex_lock(&timer_keys_mutex);
 	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
 	if (!ret && write)
-		timers_update_migration(false);
-	mutex_unlock(&mutex);
+		timers_update_migration();
+	mutex_unlock(&timer_keys_mutex);
 	return ret;
 }
-#endif
+#endif /* NO_HZ_COMMON */
 
 static unsigned long round_jiffies_common(unsigned long j, int cpu,
 		bool force_up)
@@ -534,7 +544,7 @@ __internal_add_timer(struct timer_base *base, struct timer_list *timer)
 static void
 trigger_dyntick_cpu(struct timer_base *base, struct timer_list *timer)
 {
-	if (!IS_ENABLED(CONFIG_NO_HZ_COMMON) || !base->nohz_active)
+	if (!is_timers_nohz_active())
 		return;
 
 	/*
@@ -849,21 +859,20 @@ static inline struct timer_base *get_timer_base(u32 tflags)
 	return get_timer_cpu_base(tflags, tflags & TIMER_CPUMASK);
 }
 
-#ifdef CONFIG_NO_HZ_COMMON
 static inline struct timer_base *
 get_target_base(struct timer_base *base, unsigned tflags)
 {
-#ifdef CONFIG_SMP
-	if ((tflags & TIMER_PINNED) || !base->migration_enabled)
-		return get_timer_this_cpu_base(tflags);
-	return get_timer_cpu_base(tflags, get_nohz_timer_target());
-#else
-	return get_timer_this_cpu_base(tflags);
+#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
+	if (static_branch_likely(&timers_migration_enabled) &&
+	    !(tflags & TIMER_PINNED))
+		return get_timer_cpu_base(tflags, get_nohz_timer_target());
 #endif
+	return get_timer_this_cpu_base(tflags);
 }
 
 static inline void forward_timer_base(struct timer_base *base)
 {
+#ifdef CONFIG_NO_HZ_COMMON
 	unsigned long jnow;
 
 	/*
@@ -887,16 +896,8 @@ static inline void forward_timer_base(struct timer_base *base)
 		base->clk = jnow;
 	else
 		base->clk = base->next_expiry;
-}
-#else
-static inline struct timer_base *
-get_target_base(struct timer_base *base, unsigned tflags)
-{
-	return get_timer_this_cpu_base(tflags);
-}
-
-static inline void forward_timer_base(struct timer_base *base) { }
 #endif
+}
 
 
 /*

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Correct blatantly incorrect comment
  2017-12-21 10:41 ` [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment Anna-Maria Gleixner
  2018-01-11 18:28   ` Frederic Weisbecker
  2018-01-16  1:02   ` Ingo Molnar
@ 2018-01-16  3:40   ` tip-bot for Thomas Gleixner
  2 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Thomas Gleixner @ 2018-01-16  3:40 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: peterz, frederic, hpa, mingo, torvalds, anna-maria, hch, tglx,
	john.stultz, linux-kernel

Commit-ID:  d05ca13b8d3f685667b3b1748fa89285466270c5
Gitweb:     https://git.kernel.org/tip/d05ca13b8d3f685667b3b1748fa89285466270c5
Author:     Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:31 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:44 +0100

hrtimer: Correct blatantly incorrect comment

The protection of a hrtimer which runs its callback against migration to a
different CPU has nothing to do with hard interrupt context.

The protection against migration of a hrtimer running the expiry callback
is the pointer in the cpu_base which holds a pointer to the currently
running timer. This pointer is evaluated in the code which potentially
switches the timer base and makes sure it's kept on the CPU on which the
callback is running.

Reported-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-3-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 1d06d2b..7687355 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1195,9 +1195,9 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 		timer->is_rel = false;
 
 	/*
-	 * Because we run timers from hardirq context, there is no chance
-	 * they get migrated to another cpu, therefore its safe to unlock
-	 * the timer base.
+	 * The timer is marked as running in the CPU base, so it is
+	 * protected against migration to a different CPU even if the lock
+	 * is dropped.
 	 */
 	raw_spin_unlock(&cpu_base->lock);
 	trace_hrtimer_expire_entry(timer, now);

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Fix kerneldoc syntax for 'struct hrtimer_cpu_base'
  2017-12-21 10:41 ` [PATCH v4 03/36] hrtimer: Fix kerneldoc for struct hrtimer_cpu_base Anna-Maria Gleixner
@ 2018-01-16  3:41   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:41 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: john.stultz, torvalds, hch, linux-kernel, hpa, anna-maria,
	peterz, tglx, mingo

Commit-ID:  1fbc78b3c980364d4fc15db83eca4a8e7ad289da
Gitweb:     https://git.kernel.org/tip/1fbc78b3c980364d4fc15db83eca4a8e7ad289da
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:32 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:44 +0100

hrtimer: Fix kerneldoc syntax for 'struct hrtimer_cpu_base'

The '/**' sequence marks the start of a structure description. Add the
missing second asterisk. While at it adapt the ordering of the struct
members to the struct definition and document the purpose of
expires_next more precisely.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-4-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 79b2a8d..b3a382b 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -144,7 +144,7 @@ enum  hrtimer_base_type {
 	HRTIMER_MAX_CLOCK_BASES,
 };
 
-/*
+/**
  * struct hrtimer_cpu_base - the per cpu clock bases
  * @lock:		lock protecting the base and associated clock bases
  *			and timers
@@ -153,12 +153,12 @@ enum  hrtimer_base_type {
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
- * @expires_next:	absolute time of the next event which was scheduled
- *			via clock_set_next_event()
- * @next_timer:		Pointer to the first expiring timer
  * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hres_active:	State of high resolution mode
  * @hang_detected:	The last hrtimer interrupt detected a hang
+ * @expires_next:	absolute time of the next event, is required for remote
+ *			hrtimer enqueue
+ * @next_timer:		Pointer to the first expiring timer
  * @nr_events:		Total number of hrtimer interrupt events
  * @nr_retries:		Total number of hrtimer interrupt retries
  * @nr_hangs:		Total number of hrtimer interrupt hangs

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Clean up the 'int clock' parameter of schedule_hrtimeout_range_clock()
  2017-12-21 10:41 ` [PATCH v4 04/36] hrtimer: Cleanup clock argument in schedule_hrtimeout_range_clock() Anna-Maria Gleixner
@ 2018-01-16  3:41   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:41 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: peterz, john.stultz, hpa, torvalds, mingo, hch, tglx, anna-maria,
	linux-kernel

Commit-ID:  907777136f80d0cc0f714e5a389c4dfa9b4670ee
Gitweb:     https://git.kernel.org/tip/907777136f80d0cc0f714e5a389c4dfa9b4670ee
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:33 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:44 +0100

hrtimer: Clean up the 'int clock' parameter of schedule_hrtimeout_range_clock()

schedule_hrtimeout_range_clock() uses an 'int clock' parameter for the
clock ID, instead of the customary predefined "clockid_t" type.

In hrtimer coding style the canonical variable name for the clock ID is
'clock_id', therefore change the name of the parameter here as well
to make it all consistent.

While at it, clean up the description for the 'clock_id' and 'mode'
function parameters. The clock modes and the clock IDs are not
restricted as the comment suggests.

Fix the mode description as well for the callers of schedule_hrtimeout_range_clock().

No functional changes intended.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-5-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h |  2 +-
 kernel/time/hrtimer.c   | 12 ++++++------
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index b3a382b..931ce9c 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -462,7 +462,7 @@ extern int schedule_hrtimeout_range(ktime_t *expires, u64 delta,
 extern int schedule_hrtimeout_range_clock(ktime_t *expires,
 					  u64 delta,
 					  const enum hrtimer_mode mode,
-					  int clock);
+					  clockid_t clock_id);
 extern int schedule_hrtimeout(ktime_t *expires, const enum hrtimer_mode mode);
 
 /* Soft interrupt function to run the hrtimer queues: */
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 7687355..f2de328 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1662,12 +1662,12 @@ void __init hrtimers_init(void)
  * schedule_hrtimeout_range_clock - sleep until timeout
  * @expires:	timeout value (ktime_t)
  * @delta:	slack in expires timeout (ktime_t)
- * @mode:	timer mode, HRTIMER_MODE_ABS or HRTIMER_MODE_REL
- * @clock:	timer clock, CLOCK_MONOTONIC or CLOCK_REALTIME
+ * @mode:	timer mode
+ * @clock_id:	timer clock to be used
  */
 int __sched
 schedule_hrtimeout_range_clock(ktime_t *expires, u64 delta,
-			       const enum hrtimer_mode mode, int clock)
+			       const enum hrtimer_mode mode, clockid_t clock_id)
 {
 	struct hrtimer_sleeper t;
 
@@ -1688,7 +1688,7 @@ schedule_hrtimeout_range_clock(ktime_t *expires, u64 delta,
 		return -EINTR;
 	}
 
-	hrtimer_init_on_stack(&t.timer, clock, mode);
+	hrtimer_init_on_stack(&t.timer, clock_id, mode);
 	hrtimer_set_expires_range_ns(&t.timer, *expires, delta);
 
 	hrtimer_init_sleeper(&t, current);
@@ -1710,7 +1710,7 @@ schedule_hrtimeout_range_clock(ktime_t *expires, u64 delta,
  * schedule_hrtimeout_range - sleep until timeout
  * @expires:	timeout value (ktime_t)
  * @delta:	slack in expires timeout (ktime_t)
- * @mode:	timer mode, HRTIMER_MODE_ABS or HRTIMER_MODE_REL
+ * @mode:	timer mode
  *
  * Make the current task sleep until the given expiry time has
  * elapsed. The routine will return immediately unless
@@ -1749,7 +1749,7 @@ EXPORT_SYMBOL_GPL(schedule_hrtimeout_range);
 /**
  * schedule_hrtimeout - sleep until timeout
  * @expires:	timeout value (ktime_t)
- * @mode:	timer mode, HRTIMER_MODE_ABS or HRTIMER_MODE_REL
+ * @mode:	timer mode
  *
  * Make the current task sleep until the given expiry time has
  * elapsed. The routine will return immediately unless

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Fix hrtimer_start[_range_ns]() function descriptions
  2017-12-21 10:41 ` [PATCH v4 05/36] hrtimer: Fix hrtimer function description Anna-Maria Gleixner
@ 2018-01-16  3:42   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:42 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: john.stultz, torvalds, peterz, anna-maria, hch, mingo, hpa, tglx,
	linux-kernel

Commit-ID:  6de6250c759781daeadca784d0cc34ae73f3b502
Gitweb:     https://git.kernel.org/tip/6de6250c759781daeadca784d0cc34ae73f3b502
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:34 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:45 +0100

hrtimer: Fix hrtimer_start[_range_ns]() function descriptions

The hrtimer_start[_range_ns]() functions start a timer reliably on this CPU only
when HRTIMER_MODE_PINNED is set.

Furthermore the HRTIMER_MODE_PINNED mode is not considered when a hrtimer is initialized.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-6-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h | 6 +++---
 kernel/time/hrtimer.c   | 9 +++++----
 2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 931ce9c..4e6a884 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -361,11 +361,11 @@ extern void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 				   u64 range_ns, const enum hrtimer_mode mode);
 
 /**
- * hrtimer_start - (re)start an hrtimer on the current CPU
+ * hrtimer_start - (re)start an hrtimer
  * @timer:	the timer to be added
  * @tim:	expiry time
- * @mode:	expiry mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL)
+ * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
  */
 static inline void hrtimer_start(struct hrtimer *timer, ktime_t tim,
 				 const enum hrtimer_mode mode)
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index f2de328..fd08729 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -924,12 +924,12 @@ static inline ktime_t hrtimer_update_lowres(struct hrtimer *timer, ktime_t tim,
 }
 
 /**
- * hrtimer_start_range_ns - (re)start an hrtimer on the current CPU
+ * hrtimer_start_range_ns - (re)start an hrtimer
  * @timer:	the timer to be added
  * @tim:	expiry time
  * @delta_ns:	"slack" range for the timer
- * @mode:	expiry mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL)
+ * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
  */
 void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 			    u64 delta_ns, const enum hrtimer_mode mode)
@@ -1107,7 +1107,8 @@ static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
  * hrtimer_init - initialize a timer to the given clock
  * @timer:	the timer to be initialized
  * @clock_id:	the clock to be used
- * @mode:	timer mode abs/rel
+ * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
+ *		relative (HRTIMER_MODE_REL); pinned is not considered here!
  */
 void hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 		  enum hrtimer_mode mode)

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Ensure POSIX compliance (relative CLOCK_REALTIME hrtimers)
  2017-12-21 10:41 ` [PATCH v4 06/36] hrtimer: Ensure POSIX compliance (relative CLOCK_REALTIME hrtimers) Anna-Maria Gleixner
@ 2018-01-16  3:42   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:42 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hpa, anna-maria, torvalds, peterz, mingo, linux-kernel,
	john.stultz, hch, tglx

Commit-ID:  48d0c9becc7f3c66874c100c126459a9da0fdced
Gitweb:     https://git.kernel.org/tip/48d0c9becc7f3c66874c100c126459a9da0fdced
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:35 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:45 +0100

hrtimer: Ensure POSIX compliance (relative CLOCK_REALTIME hrtimers)

The POSIX specification defines that relative CLOCK_REALTIME timers are not
affected by clock modifications. Those timers have to use CLOCK_MONOTONIC
to ensure POSIX compliance.

The introduction of the additional HRTIMER_MODE_PINNED mode broke this
requirement for pinned timers.

There is no user space visible impact because user space timers are not
using pinned mode, but for consistency reasons this needs to be fixed.

Check whether the mode has the HRTIMER_MODE_REL bit set instead of
comparing with HRTIMER_MODE_ABS.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Fixes: 597d0275736d ("timers: Framework for identifying pinned timers")
Link: http://lkml.kernel.org/r/20171221104205.7269-7-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index fd08729..60faade 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1095,7 +1095,12 @@ static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 
 	cpu_base = raw_cpu_ptr(&hrtimer_bases);
 
-	if (clock_id == CLOCK_REALTIME && mode != HRTIMER_MODE_ABS)
+	/*
+	 * POSIX magic: Relative CLOCK_REALTIME timers are not affected by
+	 * clock modifications, so they needs to become CLOCK_MONOTONIC to
+	 * ensure POSIX compliance.
+	 */
+	if (clock_id == CLOCK_REALTIME && mode & HRTIMER_MODE_REL)
 		clock_id = CLOCK_MONOTONIC;
 
 	base = hrtimer_clockid_to_base(clock_id);

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Clean up 'enum hrtimer_mode'
  2017-12-21 10:41 ` [PATCH v4 07/36] hrtimer: Cleanup hrtimer_mode enum Anna-Maria Gleixner
@ 2018-01-16  3:43   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:43 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: john.stultz, linux-kernel, hch, mingo, tglx, anna-maria, peterz,
	torvalds, hpa

Commit-ID:  19b51cb5ff6ab7957bcbbec4ff812b83208f7e99
Gitweb:     https://git.kernel.org/tip/19b51cb5ff6ab7957bcbbec4ff812b83208f7e99
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:36 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:45 +0100

hrtimer: Clean up 'enum hrtimer_mode'

It's not obvious that the HRTIMER_MODE variants are bit combinations,
because all modes are hard coded constants currently.

Change it so the bit meanings are clear; and use the symbols for creating
modes which combine bits.

While at it get rid of the ugly tail comments as well.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-8-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 4e6a884..28f267c 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -28,13 +28,19 @@ struct hrtimer_cpu_base;
 
 /*
  * Mode arguments of xxx_hrtimer functions:
+ *
+ * HRTIMER_MODE_ABS		- Time value is absolute
+ * HRTIMER_MODE_REL		- Time value is relative to now
+ * HRTIMER_MODE_PINNED		- Timer is bound to CPU (is only considered
+ *				  when starting the timer)
  */
 enum hrtimer_mode {
-	HRTIMER_MODE_ABS = 0x0,		/* Time value is absolute */
-	HRTIMER_MODE_REL = 0x1,		/* Time value is relative to now */
-	HRTIMER_MODE_PINNED = 0x02,	/* Timer is bound to CPU */
-	HRTIMER_MODE_ABS_PINNED = 0x02,
-	HRTIMER_MODE_REL_PINNED = 0x03,
+	HRTIMER_MODE_ABS	= 0x00,
+	HRTIMER_MODE_REL	= 0x01,
+	HRTIMER_MODE_PINNED	= 0x02,
+
+	HRTIMER_MODE_ABS_PINNED = HRTIMER_MODE_ABS | HRTIMER_MODE_PINNED,
+	HRTIMER_MODE_REL_PINNED = HRTIMER_MODE_REL | HRTIMER_MODE_PINNED,
 };
 
 /*

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] tracing/hrtimer: Fix tracing bugs by taking all clock bases and modes into account
  2017-12-21 10:41 ` [PATCH v4 08/36] tracing/hrtimer: Take all clock bases and modes into account Anna-Maria Gleixner
@ 2018-01-16  3:43   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:43 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hpa, mingo, linux-kernel, tglx, anna-maria, torvalds, peterz,
	hch, john.stultz

Commit-ID:  91633eed73a3ac37aaece5c8c1f93a18bae616a9
Gitweb:     https://git.kernel.org/tip/91633eed73a3ac37aaece5c8c1f93a18bae616a9
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:37 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:45 +0100

tracing/hrtimer: Fix tracing bugs by taking all clock bases and modes into account

So far only CLOCK_MONOTONIC and CLOCK_REALTIME were taken into account as
well as HRTIMER_MODE_ABS/REL in the hrtimer_init tracepoint. The query for
detecting the ABS or REL timer modes is not valid anymore, it got broken
by the introduction of HRTIMER_MODE_PINNED.

HRTIMER_MODE_PINNED is not evaluated in the hrtimer_init() call, but for the
sake of completeness print all given modes.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-9-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/trace/events/timer.h | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h
index 16e305e..c6f7280 100644
--- a/include/trace/events/timer.h
+++ b/include/trace/events/timer.h
@@ -136,6 +136,20 @@ DEFINE_EVENT(timer_class, timer_cancel,
 	TP_ARGS(timer)
 );
 
+#define decode_clockid(type)						\
+	__print_symbolic(type,						\
+		{ CLOCK_REALTIME,	"CLOCK_REALTIME"	},	\
+		{ CLOCK_MONOTONIC,	"CLOCK_MONOTONIC"	},	\
+		{ CLOCK_BOOTTIME,	"CLOCK_BOOTTIME"	},	\
+		{ CLOCK_TAI,		"CLOCK_TAI"		})
+
+#define decode_hrtimer_mode(mode)					\
+	__print_symbolic(mode,						\
+		{ HRTIMER_MODE_ABS,		"ABS"		},	\
+		{ HRTIMER_MODE_REL,		"REL"		},	\
+		{ HRTIMER_MODE_ABS_PINNED,	"ABS|PINNED"	},	\
+		{ HRTIMER_MODE_REL_PINNED,	"REL|PINNED"	})
+
 /**
  * hrtimer_init - called when the hrtimer is initialized
  * @hrtimer:	pointer to struct hrtimer
@@ -162,10 +176,8 @@ TRACE_EVENT(hrtimer_init,
 	),
 
 	TP_printk("hrtimer=%p clockid=%s mode=%s", __entry->hrtimer,
-		  __entry->clockid == CLOCK_REALTIME ?
-			"CLOCK_REALTIME" : "CLOCK_MONOTONIC",
-		  __entry->mode == HRTIMER_MODE_ABS ?
-			"HRTIMER_MODE_ABS" : "HRTIMER_MODE_REL")
+		  decode_clockid(__entry->clockid),
+		  decode_hrtimer_mode(__entry->mode))
 );
 
 /**

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] tracing/hrtimer: Print the hrtimer mode in the 'hrtimer_start' tracepoint
  2017-12-21 10:41 ` [PATCH v4 09/36] tracing/hrtimer: Print hrtimer mode in hrtimer_start tracepoint Anna-Maria Gleixner
@ 2018-01-16  3:43   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:43 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: peterz, tglx, john.stultz, hch, mingo, anna-maria, torvalds, hpa,
	linux-kernel

Commit-ID:  63e2ed3659752a4850e0ef3a07f809988fcd74a4
Gitweb:     https://git.kernel.org/tip/63e2ed3659752a4850e0ef3a07f809988fcd74a4
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:38 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:46 +0100

tracing/hrtimer: Print the hrtimer mode in the 'hrtimer_start' tracepoint

The 'hrtimer_start' tracepoint lacks the mode information. The mode is
important because consecutive starts can switch from ABS to REL or from
PINNED to non PINNED.

Append the mode field.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-10-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/trace/events/timer.h | 13 ++++++++-----
 kernel/time/hrtimer.c        | 16 +++++++++-------
 2 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h
index c6f7280..744b431 100644
--- a/include/trace/events/timer.h
+++ b/include/trace/events/timer.h
@@ -186,15 +186,16 @@ TRACE_EVENT(hrtimer_init,
  */
 TRACE_EVENT(hrtimer_start,
 
-	TP_PROTO(struct hrtimer *hrtimer),
+	TP_PROTO(struct hrtimer *hrtimer, enum hrtimer_mode mode),
 
-	TP_ARGS(hrtimer),
+	TP_ARGS(hrtimer, mode),
 
 	TP_STRUCT__entry(
 		__field( void *,	hrtimer		)
 		__field( void *,	function	)
 		__field( s64,		expires		)
 		__field( s64,		softexpires	)
+		__field( enum hrtimer_mode,	mode	)
 	),
 
 	TP_fast_assign(
@@ -202,12 +203,14 @@ TRACE_EVENT(hrtimer_start,
 		__entry->function	= hrtimer->function;
 		__entry->expires	= hrtimer_get_expires(hrtimer);
 		__entry->softexpires	= hrtimer_get_softexpires(hrtimer);
+		__entry->mode		= mode;
 	),
 
-	TP_printk("hrtimer=%p function=%pf expires=%llu softexpires=%llu",
-		  __entry->hrtimer, __entry->function,
+	TP_printk("hrtimer=%p function=%pf expires=%llu softexpires=%llu "
+		  "mode=%s", __entry->hrtimer, __entry->function,
 		  (unsigned long long) __entry->expires,
-		  (unsigned long long) __entry->softexpires)
+		  (unsigned long long) __entry->softexpires,
+		  decode_hrtimer_mode(__entry->mode))
 );
 
 /**
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 60faade..f4f4658 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -435,10 +435,11 @@ debug_init(struct hrtimer *timer, clockid_t clockid,
 	trace_hrtimer_init(timer, clockid, mode);
 }
 
-static inline void debug_activate(struct hrtimer *timer)
+static inline void debug_activate(struct hrtimer *timer,
+				  enum hrtimer_mode mode)
 {
 	debug_hrtimer_activate(timer);
-	trace_hrtimer_start(timer);
+	trace_hrtimer_start(timer, mode);
 }
 
 static inline void debug_deactivate(struct hrtimer *timer)
@@ -828,9 +829,10 @@ EXPORT_SYMBOL_GPL(hrtimer_forward);
  * Returns 1 when the new timer is the leftmost timer in the tree.
  */
 static int enqueue_hrtimer(struct hrtimer *timer,
-			   struct hrtimer_clock_base *base)
+			   struct hrtimer_clock_base *base,
+			   enum hrtimer_mode mode)
 {
-	debug_activate(timer);
+	debug_activate(timer, mode);
 
 	base->cpu_base->active_bases |= 1 << base->index;
 
@@ -953,7 +955,7 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	/* Switch the timer base, if necessary: */
 	new_base = switch_hrtimer_base(timer, base, mode & HRTIMER_MODE_PINNED);
 
-	leftmost = enqueue_hrtimer(timer, new_base);
+	leftmost = enqueue_hrtimer(timer, new_base, mode);
 	if (!leftmost)
 		goto unlock;
 
@@ -1222,7 +1224,7 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	 */
 	if (restart != HRTIMER_NORESTART &&
 	    !(timer->state & HRTIMER_STATE_ENQUEUED))
-		enqueue_hrtimer(timer, base);
+		enqueue_hrtimer(timer, base, HRTIMER_MODE_ABS);
 
 	/*
 	 * Separate the ->running assignment from the ->state assignment.
@@ -1621,7 +1623,7 @@ static void migrate_hrtimer_list(struct hrtimer_clock_base *old_base,
 		 * sort out already expired timers and reprogram the
 		 * event device.
 		 */
-		enqueue_hrtimer(timer, new_base);
+		enqueue_hrtimer(timer, new_base, HRTIMER_MODE_ABS);
 	}
 }
 

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Switch 'for' loop to _ffs() evaluation
  2017-12-21 10:41 ` [PATCH v4 10/36] hrtimer: Switch for loop to _ffs() evaluation Anna-Maria Gleixner
@ 2018-01-16  3:44   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:44 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hch, tglx, anna-maria, peterz, torvalds, mingo, john.stultz,
	linux-kernel, hpa

Commit-ID:  c272ca58c3ec5631f4ab507489d9477f74efe645
Gitweb:     https://git.kernel.org/tip/c272ca58c3ec5631f4ab507489d9477f74efe645
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:39 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:46 +0100

hrtimer: Switch 'for' loop to _ffs() evaluation

Looping over all clock bases to find active bits is suboptimal if not all
bases are active.

Avoid this by converting it to a __ffs() evaluation. The functionallity is
outsourced into its own function and is called via a macro as suggested by
Peter Zijlstra.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-11-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index f4f4658..cfcf8de 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -448,6 +448,23 @@ static inline void debug_deactivate(struct hrtimer *timer)
 	trace_hrtimer_cancel(timer);
 }
 
+static struct hrtimer_clock_base *
+__next_base(struct hrtimer_cpu_base *cpu_base, unsigned int *active)
+{
+	unsigned int idx;
+
+	if (!*active)
+		return NULL;
+
+	idx = __ffs(*active);
+	*active &= ~(1U << idx);
+
+	return &cpu_base->clock_base[idx];
+}
+
+#define for_each_active_base(base, cpu_base, active)	\
+	while ((base = __next_base((cpu_base), &(active))))
+
 #if defined(CONFIG_NO_HZ_COMMON) || defined(CONFIG_HIGH_RES_TIMERS)
 static inline void hrtimer_update_next_timer(struct hrtimer_cpu_base *cpu_base,
 					     struct hrtimer *timer)
@@ -459,18 +476,15 @@ static inline void hrtimer_update_next_timer(struct hrtimer_cpu_base *cpu_base,
 
 static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 {
-	struct hrtimer_clock_base *base = cpu_base->clock_base;
+	struct hrtimer_clock_base *base;
 	unsigned int active = cpu_base->active_bases;
 	ktime_t expires, expires_next = KTIME_MAX;
 
 	hrtimer_update_next_timer(cpu_base, NULL);
-	for (; active; base++, active >>= 1) {
+	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *next;
 		struct hrtimer *timer;
 
-		if (!(active & 0x01))
-			continue;
-
 		next = timerqueue_getnext(&base->active);
 		timer = container_of(next, struct hrtimer, node);
 		expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
@@ -1241,16 +1255,13 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 
 static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now)
 {
-	struct hrtimer_clock_base *base = cpu_base->clock_base;
+	struct hrtimer_clock_base *base;
 	unsigned int active = cpu_base->active_bases;
 
-	for (; active; base++, active >>= 1) {
+	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *node;
 		ktime_t basenow;
 
-		if (!(active & 0x01))
-			continue;
-
 		basenow = ktime_add(now, base->offset);
 
 		while ((node = timerqueue_getnext(&base->active))) {

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Store running timer in hrtimer_clock_base
  2017-12-21 10:41 ` [PATCH v4 11/36] hrtimer: Store running timer in hrtimer_clock_base Anna-Maria Gleixner
@ 2018-01-16  3:44   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:44 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: mingo, linux-kernel, hch, john.stultz, torvalds, peterz, hpa,
	anna-maria, tglx

Commit-ID:  3f0b9e8eec7262648ab9c8321bf931624ee5c10a
Gitweb:     https://git.kernel.org/tip/3f0b9e8eec7262648ab9c8321bf931624ee5c10a
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:40 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:46 +0100

hrtimer: Store running timer in hrtimer_clock_base

The pointer to the currently running timer is stored in hrtimer_cpu_base
before the base lock is dropped and the callback is invoked.

This results in two levels of indirections and the upcoming support for
softirq based hrtimer requires splitting the "running" storage into soft
and hard IRQ context expiry.

Storing both in the cpu base would require conditionals in all code paths
accessing that information.

It's possible to have a per clock base sequence count and running pointer
without changing the semantics of the related mechanisms because the timer
base pointer cannot be changed while a timer is running the callback.

Unfortunately this makes cpu_clock base larger than 32 bytes on 32-bit
kernels. Instead of having huge gaps due to alignment, remove the alignment
and let the compiler pack CPU base for 32-bit kernels. The resulting cache access
patterns are fortunately not really different from the current
behaviour. On 64-bit kernels the 64-byte alignment stays and the behaviour is
unchanged. This was determined by analyzing the resulting layout and
looking at the number of cache lines involved for the frequently used
clocks.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-12-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h | 20 +++++++++-----------
 kernel/time/hrtimer.c   | 28 +++++++++++++---------------
 2 files changed, 22 insertions(+), 26 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 28f267c..1bae7b9 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -118,9 +118,9 @@ struct hrtimer_sleeper {
 };
 
 #ifdef CONFIG_64BIT
-# define HRTIMER_CLOCK_BASE_ALIGN	64
+# define __hrtimer_clock_base_align	____cacheline_aligned
 #else
-# define HRTIMER_CLOCK_BASE_ALIGN	32
+# define __hrtimer_clock_base_align
 #endif
 
 /**
@@ -129,18 +129,22 @@ struct hrtimer_sleeper {
  * @index:		clock type index for per_cpu support when moving a
  *			timer to a base on another cpu.
  * @clockid:		clock id for per_cpu support
+ * @seq:		seqcount around __run_hrtimer
+ * @running:		pointer to the currently running hrtimer
  * @active:		red black tree root node for the active timers
  * @get_time:		function to retrieve the current time of the clock
  * @offset:		offset of this clock to the monotonic base
  */
 struct hrtimer_clock_base {
 	struct hrtimer_cpu_base	*cpu_base;
-	int			index;
+	unsigned int		index;
 	clockid_t		clockid;
+	seqcount_t		seq;
+	struct hrtimer		*running;
 	struct timerqueue_head	active;
 	ktime_t			(*get_time)(void);
 	ktime_t			offset;
-} __attribute__((__aligned__(HRTIMER_CLOCK_BASE_ALIGN)));
+} __hrtimer_clock_base_align;
 
 enum  hrtimer_base_type {
 	HRTIMER_BASE_MONOTONIC,
@@ -154,8 +158,6 @@ enum  hrtimer_base_type {
  * struct hrtimer_cpu_base - the per cpu clock bases
  * @lock:		lock protecting the base and associated clock bases
  *			and timers
- * @seq:		seqcount around __run_hrtimer
- * @running:		pointer to the currently running hrtimer
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
@@ -177,8 +179,6 @@ enum  hrtimer_base_type {
  */
 struct hrtimer_cpu_base {
 	raw_spinlock_t			lock;
-	seqcount_t			seq;
-	struct hrtimer			*running;
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
@@ -198,8 +198,6 @@ struct hrtimer_cpu_base {
 
 static inline void hrtimer_set_expires(struct hrtimer *timer, ktime_t time)
 {
-	BUILD_BUG_ON(sizeof(struct hrtimer_clock_base) > HRTIMER_CLOCK_BASE_ALIGN);
-
 	timer->node.expires = time;
 	timer->_softexpires = time;
 }
@@ -424,7 +422,7 @@ static inline int hrtimer_is_queued(struct hrtimer *timer)
  */
 static inline int hrtimer_callback_running(struct hrtimer *timer)
 {
-	return timer->base->cpu_base->running == timer;
+	return timer->base->running == timer;
 }
 
 /* Forward a hrtimer so it expires after now: */
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index cfcf8de..e56805f 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -70,7 +70,6 @@
 DEFINE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases) =
 {
 	.lock = __RAW_SPIN_LOCK_UNLOCKED(hrtimer_bases.lock),
-	.seq = SEQCNT_ZERO(hrtimer_bases.seq),
 	.clock_base =
 	{
 		{
@@ -118,7 +117,6 @@ static const int hrtimer_clock_to_base_table[MAX_CLOCKS] = {
  * timer->base->cpu_base
  */
 static struct hrtimer_cpu_base migration_cpu_base = {
-	.seq = SEQCNT_ZERO(migration_cpu_base),
 	.clock_base = { { .cpu_base = &migration_cpu_base, }, },
 };
 
@@ -1148,19 +1146,19 @@ EXPORT_SYMBOL_GPL(hrtimer_init);
  */
 bool hrtimer_active(const struct hrtimer *timer)
 {
-	struct hrtimer_cpu_base *cpu_base;
+	struct hrtimer_clock_base *base;
 	unsigned int seq;
 
 	do {
-		cpu_base = READ_ONCE(timer->base->cpu_base);
-		seq = raw_read_seqcount_begin(&cpu_base->seq);
+		base = READ_ONCE(timer->base);
+		seq = raw_read_seqcount_begin(&base->seq);
 
 		if (timer->state != HRTIMER_STATE_INACTIVE ||
-		    cpu_base->running == timer)
+		    base->running == timer)
 			return true;
 
-	} while (read_seqcount_retry(&cpu_base->seq, seq) ||
-		 cpu_base != READ_ONCE(timer->base->cpu_base));
+	} while (read_seqcount_retry(&base->seq, seq) ||
+		 base != READ_ONCE(timer->base));
 
 	return false;
 }
@@ -1194,16 +1192,16 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	lockdep_assert_held(&cpu_base->lock);
 
 	debug_deactivate(timer);
-	cpu_base->running = timer;
+	base->running = timer;
 
 	/*
 	 * Separate the ->running assignment from the ->state assignment.
 	 *
 	 * As with a regular write barrier, this ensures the read side in
-	 * hrtimer_active() cannot observe cpu_base->running == NULL &&
+	 * hrtimer_active() cannot observe base->running == NULL &&
 	 * timer->state == INACTIVE.
 	 */
-	raw_write_seqcount_barrier(&cpu_base->seq);
+	raw_write_seqcount_barrier(&base->seq);
 
 	__remove_hrtimer(timer, base, HRTIMER_STATE_INACTIVE, 0);
 	fn = timer->function;
@@ -1244,13 +1242,13 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	 * Separate the ->running assignment from the ->state assignment.
 	 *
 	 * As with a regular write barrier, this ensures the read side in
-	 * hrtimer_active() cannot observe cpu_base->running == NULL &&
+	 * hrtimer_active() cannot observe base->running.timer == NULL &&
 	 * timer->state == INACTIVE.
 	 */
-	raw_write_seqcount_barrier(&cpu_base->seq);
+	raw_write_seqcount_barrier(&base->seq);
 
-	WARN_ON_ONCE(cpu_base->running != timer);
-	cpu_base->running = NULL;
+	WARN_ON_ONCE(base->running != timer);
+	base->running = NULL;
 }
 
 static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now)

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Make room in 'struct hrtimer_cpu_base'
  2017-12-21 10:41 ` [PATCH v4 12/36] hrtimer: Make room in struct hrtimer_cpu_base Anna-Maria Gleixner
@ 2018-01-16  3:45   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:45 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: linux-kernel, john.stultz, tglx, peterz, torvalds, hpa, hch,
	mingo, anna-maria

Commit-ID:  da21c5a58a7f30db69e04e06dfb6777ccbb1113c
Gitweb:     https://git.kernel.org/tip/da21c5a58a7f30db69e04e06dfb6777ccbb1113c
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:41 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:46 +0100

hrtimer: Make room in 'struct hrtimer_cpu_base'

The upcoming softirq based hrtimers support requires an additional field in
the hrtimer_cpu_base struct, which would grow the struct size beyond a
cache line.

The hrtimer_cpu_base::nr_retries and ::nr_hangs members are solely
used for diagnostic output and have no requirement to be 'unsigned int'.

Make them 'unsigned short' to create room for the new struct member.

No functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-13-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 1bae7b9..56e56bc 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -189,8 +189,8 @@ struct hrtimer_cpu_base {
 	ktime_t				expires_next;
 	struct hrtimer			*next_timer;
 	unsigned int			nr_events;
-	unsigned int			nr_retries;
-	unsigned int			nr_hangs;
+	unsigned short			nr_retries;
+	unsigned short			nr_hangs;
 	unsigned int			max_hang_time;
 #endif
 	struct hrtimer_clock_base	clock_base[HRTIMER_MAX_CLOCK_BASES];

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Make the hrtimer_cpu_base::hres_active field unconditional, to simplify the code
  2017-12-21 10:41 ` [PATCH v4 13/36] hrtimer: Reduce conditional code (hres_active) Anna-Maria Gleixner
@ 2018-01-16  3:45   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:45 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: mingo, john.stultz, anna-maria, torvalds, tglx, linux-kernel,
	peterz, hch, hpa

Commit-ID:  28bfd18bf3daa5db8bb3422ea7138c8b7d2444ac
Gitweb:     https://git.kernel.org/tip/28bfd18bf3daa5db8bb3422ea7138c8b7d2444ac
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:42 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:47 +0100

hrtimer: Make the hrtimer_cpu_base::hres_active field unconditional, to simplify the code

The hrtimer_cpu_base::hres_active_member field depends on CONFIG_HIGH_RES_TIMERS=y
currently, and all related functions to this member are conditional as well.

To simplify the code make it unconditional and set it to zero during initialization.

(This will also help with the upcoming softirq based hrtimers code.)

The conditional code sections can be avoided by adding IS_ENABLED(HIGHRES)
conditionals into common functions, which ensures dead code elimination.

There is no functional change.

Suggested-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-14-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h | 20 ++++++++------------
 kernel/time/hrtimer.c   | 31 +++++++++++++++----------------
 2 files changed, 23 insertions(+), 28 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 56e56bc..22627b3 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -161,8 +161,8 @@ enum  hrtimer_base_type {
  * @cpu:		cpu number
  * @active_bases:	Bitfield to mark bases with active timers
  * @clock_was_set_seq:	Sequence counter of clock was set events
- * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hres_active:	State of high resolution mode
+ * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hang_detected:	The last hrtimer interrupt detected a hang
  * @expires_next:	absolute time of the next event, is required for remote
  *			hrtimer enqueue
@@ -182,9 +182,9 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
+	unsigned int			hres_active	: 1;
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
-					hres_active	: 1,
 					hang_detected	: 1;
 	ktime_t				expires_next;
 	struct hrtimer			*next_timer;
@@ -266,16 +266,17 @@ static inline ktime_t hrtimer_cb_get_time(struct hrtimer *timer)
 	return timer->base->get_time();
 }
 
+static inline int hrtimer_is_hres_active(struct hrtimer *timer)
+{
+	return IS_ENABLED(CONFIG_HIGH_RES_TIMERS) ?
+		timer->base->cpu_base->hres_active : 0;
+}
+
 #ifdef CONFIG_HIGH_RES_TIMERS
 struct clock_event_device;
 
 extern void hrtimer_interrupt(struct clock_event_device *dev);
 
-static inline int hrtimer_is_hres_active(struct hrtimer *timer)
-{
-	return timer->base->cpu_base->hres_active;
-}
-
 /*
  * The resolution of the clocks. The resolution value is returned in
  * the clock_getres() system call to give application programmers an
@@ -298,11 +299,6 @@ extern unsigned int hrtimer_resolution;
 
 #define hrtimer_resolution	(unsigned int)LOW_RES_NSEC
 
-static inline int hrtimer_is_hres_active(struct hrtimer *timer)
-{
-	return 0;
-}
-
 static inline void clock_was_set_delayed(void) { }
 
 #endif
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index e56805f..b688090 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -512,6 +512,20 @@ static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base)
 					    offs_real, offs_boot, offs_tai);
 }
 
+/*
+ * Is the high resolution mode active ?
+ */
+static inline int __hrtimer_hres_active(struct hrtimer_cpu_base *cpu_base)
+{
+	return IS_ENABLED(CONFIG_HIGH_RES_TIMERS) ?
+		cpu_base->hres_active : 0;
+}
+
+static inline int hrtimer_hres_active(void)
+{
+	return __hrtimer_hres_active(this_cpu_ptr(&hrtimer_bases));
+}
+
 /* High resolution timer related functions */
 #ifdef CONFIG_HIGH_RES_TIMERS
 
@@ -541,19 +555,6 @@ static inline int hrtimer_is_hres_enabled(void)
 }
 
 /*
- * Is the high resolution mode active ?
- */
-static inline int __hrtimer_hres_active(struct hrtimer_cpu_base *cpu_base)
-{
-	return cpu_base->hres_active;
-}
-
-static inline int hrtimer_hres_active(void)
-{
-	return __hrtimer_hres_active(this_cpu_ptr(&hrtimer_bases));
-}
-
-/*
  * Reprogram the event source with checking both queues for the
  * next event
  * Called with interrupts disabled and base->lock held
@@ -661,7 +662,6 @@ static void hrtimer_reprogram(struct hrtimer *timer,
 static inline void hrtimer_init_hres(struct hrtimer_cpu_base *base)
 {
 	base->expires_next = KTIME_MAX;
-	base->hres_active = 0;
 }
 
 /*
@@ -720,8 +720,6 @@ void clock_was_set_delayed(void)
 
 #else
 
-static inline int __hrtimer_hres_active(struct hrtimer_cpu_base *b) { return 0; }
-static inline int hrtimer_hres_active(void) { return 0; }
 static inline int hrtimer_is_hres_enabled(void) { return 0; }
 static inline void hrtimer_switch_to_hres(void) { }
 static inline void
@@ -1600,6 +1598,7 @@ int hrtimers_prepare_cpu(unsigned int cpu)
 	}
 
 	cpu_base->cpu = cpu;
+	cpu_base->hres_active = 0;
 	hrtimer_init_hres(cpu_base);
 	return 0;
 }

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Use accesor functions instead of direct access
  2017-12-21 10:41 ` [PATCH v4 14/36] hrtimer: Use accesor functions instead of direct access Anna-Maria Gleixner
@ 2018-01-16  3:46   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:46 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hpa, peterz, linux-kernel, anna-maria, tglx, john.stultz,
	torvalds, hch, mingo

Commit-ID:  851cff8caf4d638d001aac6e57a3511abd94f100
Gitweb:     https://git.kernel.org/tip/851cff8caf4d638d001aac6e57a3511abd94f100
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:43 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:47 +0100

hrtimer: Use accesor functions instead of direct access

__hrtimer_hres_active() is now available unconditionally, so replace open
coded direct accesses to hrtimer_cpu_base.hres_active.

No functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-15-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index b688090..5a624f9 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -564,7 +564,7 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 {
 	ktime_t expires_next;
 
-	if (!cpu_base->hres_active)
+	if (!__hrtimer_hres_active(cpu_base))
 		return;
 
 	expires_next = __hrtimer_get_next_event(cpu_base);
@@ -673,7 +673,7 @@ static void retrigger_next_event(void *arg)
 {
 	struct hrtimer_cpu_base *base = this_cpu_ptr(&hrtimer_bases);
 
-	if (!base->hres_active)
+	if (!__hrtimer_hres_active(base))
 		return;
 
 	raw_spin_lock(&base->lock);

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Make the remote enqueue check unconditional
  2017-12-21 10:41 ` [PATCH v4 15/36] hrtimer: Make the remote enqueue check unconditional Anna-Maria Gleixner
@ 2018-01-16  3:46   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:46 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: torvalds, tglx, hch, john.stultz, hpa, peterz, anna-maria, mingo,
	linux-kernel

Commit-ID:  07a9a7eae86abb796468b225586086d7c4cb59fc
Gitweb:     https://git.kernel.org/tip/07a9a7eae86abb796468b225586086d7c4cb59fc
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:44 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:47 +0100

hrtimer: Make the remote enqueue check unconditional

hrtimer_cpu_base.expires_next is used to cache the next event armed in the
timer hardware. The value is used to check whether an hrtimer can be
enqueued remotely. If the new hrtimer is expiring before expires_next, then
remote enqueue is not possible as the remote hrtimer hardware cannot be
accessed for reprogramming to an earlier expiry time.

The remote enqueue check is currently conditional on
CONFIG_HIGH_RES_TIMERS=y and hrtimer_cpu_base.hres_active. There is no
compelling reason to make this conditional.

Move hrtimer_cpu_base.expires_next out of the CONFIG_HIGH_RES_TIMERS=y
guarded area and remove the conditionals in hrtimer_check_target().

The check is currently a NOOP for the CONFIG_HIGH_RES_TIMERS=n and the
!hrtimer_cpu_base.hres_active case because in these cases nothing updates
hrtimer_cpu_base.expires_next yet. This will be changed with later patches
which further reduce the #ifdef zoo in this code.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-16-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h |  6 +++---
 kernel/time/hrtimer.c   | 26 ++++++--------------------
 2 files changed, 9 insertions(+), 23 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 22627b3..bb7270e 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -164,13 +164,13 @@ enum  hrtimer_base_type {
  * @hres_active:	State of high resolution mode
  * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hang_detected:	The last hrtimer interrupt detected a hang
- * @expires_next:	absolute time of the next event, is required for remote
- *			hrtimer enqueue
  * @next_timer:		Pointer to the first expiring timer
  * @nr_events:		Total number of hrtimer interrupt events
  * @nr_retries:		Total number of hrtimer interrupt retries
  * @nr_hangs:		Total number of hrtimer interrupt hangs
  * @max_hang_time:	Maximum time spent in hrtimer_interrupt
+ * @expires_next:	absolute time of the next event, is required for remote
+ *			hrtimer enqueue
  * @clock_base:		array of clock bases for this cpu
  *
  * Note: next_timer is just an optimization for __remove_hrtimer().
@@ -186,13 +186,13 @@ struct hrtimer_cpu_base {
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
 					hang_detected	: 1;
-	ktime_t				expires_next;
 	struct hrtimer			*next_timer;
 	unsigned int			nr_events;
 	unsigned short			nr_retries;
 	unsigned short			nr_hangs;
 	unsigned int			max_hang_time;
 #endif
+	ktime_t				expires_next;
 	struct hrtimer_clock_base	clock_base[HRTIMER_MAX_CLOCK_BASES];
 } ____cacheline_aligned;
 
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 5a624f9..a9ab67f 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -154,26 +154,21 @@ struct hrtimer_clock_base *lock_hrtimer_base(const struct hrtimer *timer,
 }
 
 /*
- * With HIGHRES=y we do not migrate the timer when it is expiring
- * before the next event on the target cpu because we cannot reprogram
- * the target cpu hardware and we would cause it to fire late.
+ * We do not migrate the timer when it is expiring before the next
+ * event on the target cpu. When high resolution is enabled, we cannot
+ * reprogram the target cpu hardware and we would cause it to fire
+ * late. To keep it simple, we handle the high resolution enabled and
+ * disabled case similar.
  *
  * Called with cpu_base->lock of target cpu held.
  */
 static int
 hrtimer_check_target(struct hrtimer *timer, struct hrtimer_clock_base *new_base)
 {
-#ifdef CONFIG_HIGH_RES_TIMERS
 	ktime_t expires;
 
-	if (!new_base->cpu_base->hres_active)
-		return 0;
-
 	expires = ktime_sub(hrtimer_get_expires(timer), new_base->offset);
 	return expires <= new_base->cpu_base->expires_next;
-#else
-	return 0;
-#endif
 }
 
 static inline
@@ -657,14 +652,6 @@ static void hrtimer_reprogram(struct hrtimer *timer,
 }
 
 /*
- * Initialize the high resolution related parts of cpu_base
- */
-static inline void hrtimer_init_hres(struct hrtimer_cpu_base *base)
-{
-	base->expires_next = KTIME_MAX;
-}
-
-/*
  * Retrigger next event is called after clock was set
  *
  * Called with interrupts disabled via on_each_cpu()
@@ -729,7 +716,6 @@ static inline int hrtimer_reprogram(struct hrtimer *timer,
 {
 	return 0;
 }
-static inline void hrtimer_init_hres(struct hrtimer_cpu_base *base) { }
 static inline void retrigger_next_event(void *arg) { }
 
 #endif /* CONFIG_HIGH_RES_TIMERS */
@@ -1599,7 +1585,7 @@ int hrtimers_prepare_cpu(unsigned int cpu)
 
 	cpu_base->cpu = cpu;
 	cpu_base->hres_active = 0;
-	hrtimer_init_hres(cpu_base);
+	cpu_base->expires_next = KTIME_MAX;
 	return 0;
 }
 

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Make hrtimer_cpu_base.next_timer handling unconditional
  2017-12-21 10:41 ` [PATCH v4 16/36] hrtimer: Make hrtimer_cpu_base.next_timer handling unconditional Anna-Maria Gleixner
@ 2018-01-16  3:46   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:46 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: tglx, hpa, hch, linux-kernel, mingo, anna-maria, torvalds,
	john.stultz, peterz

Commit-ID:  eb27926ba05233dc4f2052cc9d4f19359ec3cd2c
Gitweb:     https://git.kernel.org/tip/eb27926ba05233dc4f2052cc9d4f19359ec3cd2c
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:45 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:47 +0100

hrtimer: Make hrtimer_cpu_base.next_timer handling unconditional

hrtimer_cpu_base.next_timer stores the pointer to the next expiring timer
in a CPU base.

This pointer cannot be dereferenced and is solely used to check whether a
hrtimer which is removed is the hrtimer which is the first to expire in the
CPU base. If this is the case, then the timer hardware needs to be
reprogrammed to avoid an extra interrupt for nothing.

Again, this is conditional functionality, but there is no compelling reason
to make this conditional. As a preparation, hrtimer_cpu_base.next_timer
needs to be available unconditonally.

Aside of that the upcoming support for softirq based hrtimers requires access
to this pointer unconditionally as well, so our motivation is not entirely
simplicity based.

Make the update of hrtimer_cpu_base.next_timer unconditional and remove the
#ifdef cruft. The impact on CONFIG_HIGH_RES_TIMERS=n && CONFIG_NOHZ=n is
marginal as it's just a store on an already dirtied cacheline.

No functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-17-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h |  4 ++--
 kernel/time/hrtimer.c   | 12 ++----------
 2 files changed, 4 insertions(+), 12 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index bb7270e..2d3e1d6 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -164,13 +164,13 @@ enum  hrtimer_base_type {
  * @hres_active:	State of high resolution mode
  * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hang_detected:	The last hrtimer interrupt detected a hang
- * @next_timer:		Pointer to the first expiring timer
  * @nr_events:		Total number of hrtimer interrupt events
  * @nr_retries:		Total number of hrtimer interrupt retries
  * @nr_hangs:		Total number of hrtimer interrupt hangs
  * @max_hang_time:	Maximum time spent in hrtimer_interrupt
  * @expires_next:	absolute time of the next event, is required for remote
  *			hrtimer enqueue
+ * @next_timer:		Pointer to the first expiring timer
  * @clock_base:		array of clock bases for this cpu
  *
  * Note: next_timer is just an optimization for __remove_hrtimer().
@@ -186,13 +186,13 @@ struct hrtimer_cpu_base {
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			in_hrtirq	: 1,
 					hang_detected	: 1;
-	struct hrtimer			*next_timer;
 	unsigned int			nr_events;
 	unsigned short			nr_retries;
 	unsigned short			nr_hangs;
 	unsigned int			max_hang_time;
 #endif
 	ktime_t				expires_next;
+	struct hrtimer			*next_timer;
 	struct hrtimer_clock_base	clock_base[HRTIMER_MAX_CLOCK_BASES];
 } ____cacheline_aligned;
 
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index a9ab67f..26abaa7 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -459,21 +459,13 @@ __next_base(struct hrtimer_cpu_base *cpu_base, unsigned int *active)
 	while ((base = __next_base((cpu_base), &(active))))
 
 #if defined(CONFIG_NO_HZ_COMMON) || defined(CONFIG_HIGH_RES_TIMERS)
-static inline void hrtimer_update_next_timer(struct hrtimer_cpu_base *cpu_base,
-					     struct hrtimer *timer)
-{
-#ifdef CONFIG_HIGH_RES_TIMERS
-	cpu_base->next_timer = timer;
-#endif
-}
-
 static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 {
 	struct hrtimer_clock_base *base;
 	unsigned int active = cpu_base->active_bases;
 	ktime_t expires, expires_next = KTIME_MAX;
 
-	hrtimer_update_next_timer(cpu_base, NULL);
+	cpu_base->next_timer = NULL;
 	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *next;
 		struct hrtimer *timer;
@@ -483,7 +475,7 @@ static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 		expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
 		if (expires < expires_next) {
 			expires_next = expires;
-			hrtimer_update_next_timer(cpu_base, timer);
+			cpu_base->next_timer = timer;
 		}
 	}
 	/*

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Make hrtimer_reprogramm() unconditional
  2017-12-21 10:41 ` [PATCH v4 17/36] hrtimer: Make hrtimer_reprogramm() unconditional Anna-Maria Gleixner
@ 2018-01-16  3:47   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:47 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: tglx, mingo, linux-kernel, peterz, torvalds, hch, john.stultz,
	hpa, anna-maria

Commit-ID:  11a9fe069e341ac53bddb8fe1a85ea986cff1a42
Gitweb:     https://git.kernel.org/tip/11a9fe069e341ac53bddb8fe1a85ea986cff1a42
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:46 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:35:47 +0100

hrtimer: Make hrtimer_reprogramm() unconditional

hrtimer_reprogram() needs to be available unconditionally for softirq based
hrtimers. Move the function and all required struct members out of the
CONFIG_HIGH_RES_TIMERS #ifdef.

There is no functional change because hrtimer_reprogram() is only invoked
when hrtimer_cpu_base.hres_active is true. Making it unconditional
increases the text size for the CONFIG_HIGH_RES_TIMERS=n case, but avoids
replication of that code for the upcoming softirq based hrtimers support.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-18-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h |   6 +--
 kernel/time/hrtimer.c   | 129 +++++++++++++++++++++++-------------------------
 2 files changed, 65 insertions(+), 70 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 2d3e1d6..98ed357 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -182,10 +182,10 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
-	unsigned int			hres_active	: 1;
-#ifdef CONFIG_HIGH_RES_TIMERS
-	unsigned int			in_hrtirq	: 1,
+	unsigned int			hres_active	: 1,
+					in_hrtirq	: 1,
 					hang_detected	: 1;
+#ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			nr_events;
 	unsigned short			nr_retries;
 	unsigned short			nr_hangs;
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 26abaa7..63d804a 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -582,68 +582,6 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 }
 
 /*
- * When a timer is enqueued and expires earlier than the already enqueued
- * timers, we have to check, whether it expires earlier than the timer for
- * which the clock event device was armed.
- *
- * Called with interrupts disabled and base->cpu_base.lock held
- */
-static void hrtimer_reprogram(struct hrtimer *timer,
-			      struct hrtimer_clock_base *base)
-{
-	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
-	ktime_t expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
-
-	WARN_ON_ONCE(hrtimer_get_expires_tv64(timer) < 0);
-
-	/*
-	 * If the timer is not on the current cpu, we cannot reprogram
-	 * the other cpus clock event device.
-	 */
-	if (base->cpu_base != cpu_base)
-		return;
-
-	/*
-	 * If the hrtimer interrupt is running, then it will
-	 * reevaluate the clock bases and reprogram the clock event
-	 * device. The callbacks are always executed in hard interrupt
-	 * context so we don't need an extra check for a running
-	 * callback.
-	 */
-	if (cpu_base->in_hrtirq)
-		return;
-
-	/*
-	 * CLOCK_REALTIME timer might be requested with an absolute
-	 * expiry time which is less than base->offset. Set it to 0.
-	 */
-	if (expires < 0)
-		expires = 0;
-
-	if (expires >= cpu_base->expires_next)
-		return;
-
-	/* Update the pointer to the next expiring timer */
-	cpu_base->next_timer = timer;
-
-	/*
-	 * If a hang was detected in the last timer interrupt then we
-	 * do not schedule a timer which is earlier than the expiry
-	 * which we enforced in the hang detection. We want the system
-	 * to make progress.
-	 */
-	if (cpu_base->hang_detected)
-		return;
-
-	/*
-	 * Program the timer hardware. We enforce the expiry for
-	 * events which are already in the past.
-	 */
-	cpu_base->expires_next = expires;
-	tick_program_event(expires, 1);
-}
-
-/*
  * Retrigger next event is called after clock was set
  *
  * Called with interrupts disabled via on_each_cpu()
@@ -703,16 +641,73 @@ static inline int hrtimer_is_hres_enabled(void) { return 0; }
 static inline void hrtimer_switch_to_hres(void) { }
 static inline void
 hrtimer_force_reprogram(struct hrtimer_cpu_base *base, int skip_equal) { }
-static inline int hrtimer_reprogram(struct hrtimer *timer,
-				    struct hrtimer_clock_base *base)
-{
-	return 0;
-}
 static inline void retrigger_next_event(void *arg) { }
 
 #endif /* CONFIG_HIGH_RES_TIMERS */
 
 /*
+ * When a timer is enqueued and expires earlier than the already enqueued
+ * timers, we have to check, whether it expires earlier than the timer for
+ * which the clock event device was armed.
+ *
+ * Called with interrupts disabled and base->cpu_base.lock held
+ */
+static void hrtimer_reprogram(struct hrtimer *timer,
+			      struct hrtimer_clock_base *base)
+{
+	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	ktime_t expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
+
+	WARN_ON_ONCE(hrtimer_get_expires_tv64(timer) < 0);
+
+	/*
+	 * If the timer is not on the current cpu, we cannot reprogram
+	 * the other cpus clock event device.
+	 */
+	if (base->cpu_base != cpu_base)
+		return;
+
+	/*
+	 * If the hrtimer interrupt is running, then it will
+	 * reevaluate the clock bases and reprogram the clock event
+	 * device. The callbacks are always executed in hard interrupt
+	 * context so we don't need an extra check for a running
+	 * callback.
+	 */
+	if (cpu_base->in_hrtirq)
+		return;
+
+	/*
+	 * CLOCK_REALTIME timer might be requested with an absolute
+	 * expiry time which is less than base->offset. Set it to 0.
+	 */
+	if (expires < 0)
+		expires = 0;
+
+	if (expires >= cpu_base->expires_next)
+		return;
+
+	/* Update the pointer to the next expiring timer */
+	cpu_base->next_timer = timer;
+
+	/*
+	 * If a hang was detected in the last timer interrupt then we
+	 * do not schedule a timer which is earlier than the expiry
+	 * which we enforced in the hang detection. We want the system
+	 * to make progress.
+	 */
+	if (cpu_base->hang_detected)
+		return;
+
+	/*
+	 * Program the timer hardware. We enforce the expiry for
+	 * events which are already in the past.
+	 */
+	cpu_base->expires_next = expires;
+	tick_program_event(expires, 1);
+}
+
+/*
  * Clock realtime was set
  *
  * Change the offset of the realtime clock vs. the monotonic

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Make hrtimer_force_reprogramm() unconditionally available
  2017-12-21 10:41 ` [PATCH v4 18/36] hrtimer: Make hrtimer_force_reprogramm() unconditionally available Anna-Maria Gleixner
@ 2018-01-16  3:47   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:47 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hch, anna-maria, mingo, john.stultz, peterz, torvalds, tglx, hpa,
	linux-kernel

Commit-ID:  ebba2c723f38a766546b2eaf828c522576c791d4
Gitweb:     https://git.kernel.org/tip/ebba2c723f38a766546b2eaf828c522576c791d4
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:47 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:53:28 +0100

hrtimer: Make hrtimer_force_reprogramm() unconditionally available

hrtimer_force_reprogram() needs to be available unconditionally for softirq
based hrtimers. Move the function and all required struct members out of
the CONFIG_HIGH_RES_TIMERS #ifdef.

There is no functional change because hrtimer_force_reprogram() is only
invoked when hrtimer_cpu_base.hres_active is true and
CONFIG_HIGH_RES_TIMERS=y.

Making it unconditional increases the text size for the
CONFIG_HIGH_RES_TIMERS=n case slightly, but avoids replication of that code
for the upcoming softirq based hrtimers support. Most of the code gets
eliminated in the CONFIG_HIGH_RES_TIMERS=n case by the compiler.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-19-anna-maria@linutronix.de
[ Made it build on !CONFIG_HIGH_RES_TIMERS ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 60 ++++++++++++++++++++++++---------------------------
 1 file changed, 28 insertions(+), 32 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 63d804a..2b3222e 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -458,7 +458,6 @@ __next_base(struct hrtimer_cpu_base *cpu_base, unsigned int *active)
 #define for_each_active_base(base, cpu_base, active)	\
 	while ((base = __next_base((cpu_base), &(active))))
 
-#if defined(CONFIG_NO_HZ_COMMON) || defined(CONFIG_HIGH_RES_TIMERS)
 static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 {
 	struct hrtimer_clock_base *base;
@@ -487,7 +486,6 @@ static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 		expires_next = 0;
 	return expires_next;
 }
-#endif
 
 static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base)
 {
@@ -513,34 +511,6 @@ static inline int hrtimer_hres_active(void)
 	return __hrtimer_hres_active(this_cpu_ptr(&hrtimer_bases));
 }
 
-/* High resolution timer related functions */
-#ifdef CONFIG_HIGH_RES_TIMERS
-
-/*
- * High resolution timer enabled ?
- */
-static bool hrtimer_hres_enabled __read_mostly  = true;
-unsigned int hrtimer_resolution __read_mostly = LOW_RES_NSEC;
-EXPORT_SYMBOL_GPL(hrtimer_resolution);
-
-/*
- * Enable / Disable high resolution mode
- */
-static int __init setup_hrtimer_hres(char *str)
-{
-	return (kstrtobool(str, &hrtimer_hres_enabled) == 0);
-}
-
-__setup("highres=", setup_hrtimer_hres);
-
-/*
- * hrtimer_high_res_enabled - query, if the highres mode is enabled
- */
-static inline int hrtimer_is_hres_enabled(void)
-{
-	return hrtimer_hres_enabled;
-}
-
 /*
  * Reprogram the event source with checking both queues for the
  * next event
@@ -581,6 +551,34 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 	tick_program_event(cpu_base->expires_next, 1);
 }
 
+/* High resolution timer related functions */
+#ifdef CONFIG_HIGH_RES_TIMERS
+
+/*
+ * High resolution timer enabled ?
+ */
+static bool hrtimer_hres_enabled __read_mostly  = true;
+unsigned int hrtimer_resolution __read_mostly = LOW_RES_NSEC;
+EXPORT_SYMBOL_GPL(hrtimer_resolution);
+
+/*
+ * Enable / Disable high resolution mode
+ */
+static int __init setup_hrtimer_hres(char *str)
+{
+	return (kstrtobool(str, &hrtimer_hres_enabled) == 0);
+}
+
+__setup("highres=", setup_hrtimer_hres);
+
+/*
+ * hrtimer_high_res_enabled - query, if the highres mode is enabled
+ */
+static inline int hrtimer_is_hres_enabled(void)
+{
+	return hrtimer_hres_enabled;
+}
+
 /*
  * Retrigger next event is called after clock was set
  *
@@ -639,8 +637,6 @@ void clock_was_set_delayed(void)
 
 static inline int hrtimer_is_hres_enabled(void) { return 0; }
 static inline void hrtimer_switch_to_hres(void) { }
-static inline void
-hrtimer_force_reprogram(struct hrtimer_cpu_base *base, int skip_equal) { }
 static inline void retrigger_next_event(void *arg) { }
 
 #endif /* CONFIG_HIGH_RES_TIMERS */

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Unify hrtimer removal handling
  2017-12-21 10:41 ` [PATCH v4 19/36] hrtimer: Unify handling of hrtimer remove Anna-Maria Gleixner
@ 2018-01-16  3:48   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:48 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: tglx, hch, torvalds, peterz, linux-kernel, mingo, anna-maria,
	john.stultz, hpa

Commit-ID:  61bb4bcb79c7afcd0bf0d20aef4704977172fd60
Gitweb:     https://git.kernel.org/tip/61bb4bcb79c7afcd0bf0d20aef4704977172fd60
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:48 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:53:58 +0100

hrtimer: Unify hrtimer removal handling

When the first hrtimer on the current CPU is removed,
hrtimer_force_reprogram() is invoked but only when
CONFIG_HIGH_RES_TIMERS=y and hrtimer_cpu_base.hres_active is set.

hrtimer_force_reprogram() updates hrtimer_cpu_base.expires_next and
reprograms the clock event device. When CONFIG_HIGH_RES_TIMERS=y and
hrtimer_cpu_base.hres_active is set, a pointless hrtimer interrupt can be
prevented.

hrtimer_check_target() makes the 'can remote enqueue' decision. As soon as
hrtimer_check_target() is unconditionally available and
hrtimer_cpu_base.expires_next is updated by hrtimer_reprogram(),
hrtimer_force_reprogram() needs to be available unconditionally as well to
prevent the following scenario with CONFIG_HIGH_RES_TIMERS=n:

- the first hrtimer on this CPU is removed and hrtimer_force_reprogram() is
  not executed

- CPU goes idle (next timer is calculated and hrtimers are taken into
  account)

- a hrtimer is enqueued remote on the idle CPU: hrtimer_check_target()
  compares expiry value and hrtimer_cpu_base.expires_next. The expiry value
  is after expires_next, so the hrtimer is enqueued. This timer will fire
  late, if it expires before the effective first hrtimer on this CPU and
  the comparison was with an outdated expires_next value.

To prevent this scenario, make hrtimer_force_reprogram() unconditional
except the effective reprogramming part, which gets eliminated by the
compiler in the CONFIG_HIGH_RES_TIMERS=n case.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-20-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 2b3222e..e6a78ae 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -521,9 +521,6 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 {
 	ktime_t expires_next;
 
-	if (!__hrtimer_hres_active(cpu_base))
-		return;
-
 	expires_next = __hrtimer_get_next_event(cpu_base);
 
 	if (skip_equal && expires_next == cpu_base->expires_next)
@@ -532,6 +529,9 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 	cpu_base->expires_next = expires_next;
 
 	/*
+	 * If hres is not active, hardware does not have to be
+	 * reprogrammed yet.
+	 *
 	 * If a hang was detected in the last timer interrupt then we
 	 * leave the hang delay active in the hardware. We want the
 	 * system to make progress. That also prevents the following
@@ -545,7 +545,7 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 	 * set. So we'd effectivly block all timers until the T2 event
 	 * fires.
 	 */
-	if (cpu_base->hang_detected)
+	if (!__hrtimer_hres_active(cpu_base) || cpu_base->hang_detected)
 		return;
 
 	tick_program_event(cpu_base->expires_next, 1);
@@ -844,7 +844,6 @@ static void __remove_hrtimer(struct hrtimer *timer,
 	if (!timerqueue_del(&base->active, &timer->node))
 		cpu_base->active_bases &= ~(1 << base->index);
 
-#ifdef CONFIG_HIGH_RES_TIMERS
 	/*
 	 * Note: If reprogram is false we do not update
 	 * cpu_base->next_timer. This happens when we remove the first
@@ -855,7 +854,6 @@ static void __remove_hrtimer(struct hrtimer *timer,
 	 */
 	if (reprogram && timer == cpu_base->next_timer)
 		hrtimer_force_reprogram(cpu_base, 1);
-#endif
 }
 
 /*

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Unify remote enqueue handling
  2017-12-21 10:41 ` [PATCH v4 20/36] hrtimer: Unify handling of remote enqueue Anna-Maria Gleixner
@ 2018-01-16  3:48   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:48 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hch, peterz, torvalds, linux-kernel, john.stultz, mingo, tglx,
	anna-maria, hpa

Commit-ID:  14c803419de6acba08e143d51813ac5e0f3443b8
Gitweb:     https://git.kernel.org/tip/14c803419de6acba08e143d51813ac5e0f3443b8
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:49 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:53:58 +0100

hrtimer: Unify remote enqueue handling

hrtimer_reprogram() is conditionally invoked from hrtimer_start_range_ns()
when hrtimer_cpu_base.hres_active is true.

In the !hres_active case there is a special condition for the nohz_active
case:

  If the newly enqueued timer expires before the first expiring timer on a
  remote CPU then the remote CPU needs to be notified and woken up from a
  NOHZ idle sleep to take the new first expiring timer into account.

Previous changes have already established the prerequisites to make the
remote enqueue behaviour the same whether high resolution mode is active or
not:

  If the to be enqueued timer expires before the first expiring timer on a
  remote CPU, then it cannot be enqueued there.

This was done for the high resolution mode because there is no way to
access the remote CPU timer hardware. The same is true for NOHZ, but was
handled differently by unconditionally enqueuing the timer and waking up
the remote CPU so it can reprogram its timer. Again there is no compelling
reason for this difference.

hrtimer_check_target(), which makes the 'can remote enqueue' decision is
already unconditional, but not yet functional because nothing updates
hrtimer_cpu_base.expires_next in the !hres_active case.

To unify this the following changes are required:

 1) Make the store of the new first expiry time unconditonal in
    hrtimer_reprogram() and check __hrtimer_hres_active() before proceeding
    to the actual hardware access. This check also lets the compiler
    eliminate the rest of the function in case of CONFIG_HIGH_RES_TIMERS=n.

 2) Invoke hrtimer_reprogram() unconditionally from
    hrtimer_start_range_ns()

 3) Remove the remote wakeup special case for the !high_res && nohz_active
    case.

Confine the timers_nohz_active static key to timer.c which is the only user
now.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-21-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c       | 18 ++++++------------
 kernel/time/tick-internal.h |  6 ------
 kernel/time/timer.c         |  9 ++++++++-
 3 files changed, 14 insertions(+), 19 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index e6a78ae..1c68bf2 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -685,21 +685,24 @@ static void hrtimer_reprogram(struct hrtimer *timer,
 
 	/* Update the pointer to the next expiring timer */
 	cpu_base->next_timer = timer;
+	cpu_base->expires_next = expires;
 
 	/*
+	 * If hres is not active, hardware does not have to be
+	 * programmed yet.
+	 *
 	 * If a hang was detected in the last timer interrupt then we
 	 * do not schedule a timer which is earlier than the expiry
 	 * which we enforced in the hang detection. We want the system
 	 * to make progress.
 	 */
-	if (cpu_base->hang_detected)
+	if (!__hrtimer_hres_active(cpu_base) || cpu_base->hang_detected)
 		return;
 
 	/*
 	 * Program the timer hardware. We enforce the expiry for
 	 * events which are already in the past.
 	 */
-	cpu_base->expires_next = expires;
 	tick_program_event(expires, 1);
 }
 
@@ -936,16 +939,7 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	if (!leftmost)
 		goto unlock;
 
-	if (!hrtimer_is_hres_active(timer)) {
-		/*
-		 * Kick to reschedule the next tick to handle the new timer
-		 * on dynticks target.
-		 */
-		if (is_timers_nohz_active())
-			wake_up_nohz_cpu(new_base->cpu_base->cpu);
-	} else {
-		hrtimer_reprogram(timer, new_base);
-	}
+	hrtimer_reprogram(timer, new_base);
 unlock:
 	unlock_hrtimer_base(timer, &flags);
 }
diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index f690628..e277284 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -151,18 +151,12 @@ static inline void tick_nohz_init(void) { }
 #ifdef CONFIG_NO_HZ_COMMON
 extern unsigned long tick_nohz_active;
 extern void timers_update_nohz(void);
-extern struct static_key_false timers_nohz_active;
-static inline bool is_timers_nohz_active(void)
-{
-	return static_branch_likely(&timers_nohz_active);
-}
 # ifdef CONFIG_SMP
 extern struct static_key_false timers_migration_enabled;
 # endif
 #else /* CONFIG_NO_HZ_COMMON */
 static inline void timers_update_nohz(void) { }
 #define tick_nohz_active (0)
-static inline bool is_timers_nohz_active(void) { return false; }
 #endif
 
 DECLARE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases);
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index d530f72..48150ab 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -210,7 +210,7 @@ static DEFINE_PER_CPU(struct timer_base, timer_bases[NR_BASES]);
 
 #ifdef CONFIG_NO_HZ_COMMON
 
-DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
+static DEFINE_STATIC_KEY_FALSE(timers_nohz_active);
 static DEFINE_MUTEX(timer_keys_mutex);
 
 static void timer_update_keys(struct work_struct *work);
@@ -258,6 +258,13 @@ int timer_migration_handler(struct ctl_table *table, int write,
 	mutex_unlock(&timer_keys_mutex);
 	return ret;
 }
+
+static inline bool is_timers_nohz_active(void)
+{
+	return static_branch_unlikely(&timers_nohz_active);
+}
+#else
+static inline bool is_timers_nohz_active(void) { return false; }
 #endif /* NO_HZ_COMMON */
 
 static unsigned long round_jiffies_common(unsigned long j, int cpu,

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Make remote enqueue decision less restrictive
  2017-12-21 10:41 ` [PATCH v4 21/36] hrtimer: Make remote enqueue decision less restrictive Anna-Maria Gleixner
@ 2018-01-16  3:49   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:49 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: torvalds, hch, tglx, mingo, linux-kernel, hpa, peterz,
	john.stultz, anna-maria

Commit-ID:  2ac2dccce9d16a7b1a8fddf69a955d249375bce4
Gitweb:     https://git.kernel.org/tip/2ac2dccce9d16a7b1a8fddf69a955d249375bce4
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:50 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:53:58 +0100

hrtimer: Make remote enqueue decision less restrictive

The current decision whether a timer can be queued on a remote CPU checks
for timer->expiry <= remote_cpu_base.expires_next.

This is too restrictive because a timer with the same expiry time as an
existing timer will be enqueued on right-hand size of the existing timer
inside the rbtree, i.e. behind the first expiring timer.

So its safe to allow enqueuing timers with the same expiry time as the
first expiring timer on a remote CPU base.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-22-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 1c68bf2..f4a56fb 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -168,7 +168,7 @@ hrtimer_check_target(struct hrtimer *timer, struct hrtimer_clock_base *new_base)
 	ktime_t expires;
 
 	expires = ktime_sub(hrtimer_get_expires(timer), new_base->offset);
-	return expires <= new_base->cpu_base->expires_next;
+	return expires < new_base->cpu_base->expires_next;
 }
 
 static inline

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Remove the 'base' parameter from hrtimer_reprogram()
  2017-12-21 10:41 ` [PATCH v4 22/36] hrtimer: Remove base argument from hrtimer_reprogram() Anna-Maria Gleixner
@ 2018-01-16  3:49   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:49 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: mingo, peterz, tglx, hch, anna-maria, torvalds, john.stultz,
	linux-kernel, hpa

Commit-ID:  3ec7a3ee9f15f6dcac1591902d85b94c2a4b520d
Gitweb:     https://git.kernel.org/tip/3ec7a3ee9f15f6dcac1591902d85b94c2a4b520d
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:51 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:53:59 +0100

hrtimer: Remove the 'base' parameter from hrtimer_reprogram()

hrtimer_reprogram() must have access to the hrtimer_clock_base of the new
first expiring timer to access hrtimer_clock_base.offset for adjusting the
expiry time to CLOCK_MONOTONIC. This is required to evaluate whether the
new left most timer in the hrtimer_clock_base is the first expiring timer
of all clock bases in a hrtimer_cpu_base.

The only user of hrtimer_reprogram() is hrtimer_start_range_ns(), which has
a pointer to hrtimer_clock_base() already and hands it in as a parameter. But
hrtimer_start_range_ns() will be split for the upcoming support for softirq
based hrtimers to avoid code duplication and will lose the direct access to
the clock base pointer.

Instead of handing in timer and timer->base as a parameter remove the base
parameter from hrtimer_reprogram() instead and retrieve the clock base internally.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-23-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index f4a56fb..33a6c99 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -648,10 +648,10 @@ static inline void retrigger_next_event(void *arg) { }
  *
  * Called with interrupts disabled and base->cpu_base.lock held
  */
-static void hrtimer_reprogram(struct hrtimer *timer,
-			      struct hrtimer_clock_base *base)
+static void hrtimer_reprogram(struct hrtimer *timer)
 {
 	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	struct hrtimer_clock_base *base = timer->base;
 	ktime_t expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
 
 	WARN_ON_ONCE(hrtimer_get_expires_tv64(timer) < 0);
@@ -939,7 +939,7 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	if (!leftmost)
 		goto unlock;
 
-	hrtimer_reprogram(timer, new_base);
+	hrtimer_reprogram(timer);
 unlock:
 	unlock_hrtimer_base(timer, &flags);
 }

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Factor out __hrtimer_start_range_ns()
  2017-12-21 10:41 ` [PATCH v4 23/36] hrtimer: Split hrtimer_start_range_ns() Anna-Maria Gleixner
@ 2018-01-16  3:49   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:49 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: peterz, mingo, anna-maria, hch, torvalds, tglx, john.stultz,
	linux-kernel, hpa

Commit-ID:  138a6b7ae4dedde5513678f57b275eee19c41b6a
Gitweb:     https://git.kernel.org/tip/138a6b7ae4dedde5513678f57b275eee19c41b6a
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:52 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 02:53:59 +0100

hrtimer: Factor out __hrtimer_start_range_ns()

Preparatory patch for softirq based hrtimers to avoid code duplication,
factor out the __hrtimer_start_range_ns() function from hrtimer_start_range_ns().

No functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-24-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 44 ++++++++++++++++++++++++--------------------
 1 file changed, 24 insertions(+), 20 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 33a6c99..4142e6f 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -905,22 +905,11 @@ static inline ktime_t hrtimer_update_lowres(struct hrtimer *timer, ktime_t tim,
 	return tim;
 }
 
-/**
- * hrtimer_start_range_ns - (re)start an hrtimer
- * @timer:	the timer to be added
- * @tim:	expiry time
- * @delta_ns:	"slack" range for the timer
- * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
- */
-void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
-			    u64 delta_ns, const enum hrtimer_mode mode)
+static int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
+				    u64 delta_ns, const enum hrtimer_mode mode,
+				    struct hrtimer_clock_base *base)
 {
-	struct hrtimer_clock_base *base, *new_base;
-	unsigned long flags;
-	int leftmost;
-
-	base = lock_hrtimer_base(timer, &flags);
+	struct hrtimer_clock_base *new_base;
 
 	/* Remove an active timer from the queue: */
 	remove_hrtimer(timer, base, true);
@@ -935,12 +924,27 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	/* Switch the timer base, if necessary: */
 	new_base = switch_hrtimer_base(timer, base, mode & HRTIMER_MODE_PINNED);
 
-	leftmost = enqueue_hrtimer(timer, new_base, mode);
-	if (!leftmost)
-		goto unlock;
+	return enqueue_hrtimer(timer, new_base, mode);
+}
+/**
+ * hrtimer_start_range_ns - (re)start an hrtimer
+ * @timer:	the timer to be added
+ * @tim:	expiry time
+ * @delta_ns:	"slack" range for the timer
+ * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
+ */
+void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
+			    u64 delta_ns, const enum hrtimer_mode mode)
+{
+	struct hrtimer_clock_base *base;
+	unsigned long flags;
+
+	base = lock_hrtimer_base(timer, &flags);
+
+	if (__hrtimer_start_range_ns(timer, tim, delta_ns, mode, base))
+		hrtimer_reprogram(timer);
 
-	hrtimer_reprogram(timer);
-unlock:
 	unlock_hrtimer_base(timer, &flags);
 }
 EXPORT_SYMBOL_GPL(hrtimer_start_range_ns);

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Factor out __hrtimer_next_event_base()
  2017-12-21 10:41 ` [PATCH v4 24/36] hrtimer: Split __hrtimer_get_next_event() Anna-Maria Gleixner
@ 2018-01-16  3:50   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:50 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: mingo, john.stultz, torvalds, linux-kernel, hpa, peterz, hch,
	tglx, anna-maria

Commit-ID:  ad38f596d8e4babc19be8b21a7a49debffb4a7f5
Gitweb:     https://git.kernel.org/tip/ad38f596d8e4babc19be8b21a7a49debffb4a7f5
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:53 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 03:00:43 +0100

hrtimer: Factor out __hrtimer_next_event_base()

Preparatory patch for softirq based hrtimers to avoid code duplication.

No functional change.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-25-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 4142e6f..5d9b81d 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -458,13 +458,13 @@ __next_base(struct hrtimer_cpu_base *cpu_base, unsigned int *active)
 #define for_each_active_base(base, cpu_base, active)	\
 	while ((base = __next_base((cpu_base), &(active))))
 
-static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
+static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
+					 unsigned int active,
+					 ktime_t expires_next)
 {
 	struct hrtimer_clock_base *base;
-	unsigned int active = cpu_base->active_bases;
-	ktime_t expires, expires_next = KTIME_MAX;
+	ktime_t expires;
 
-	cpu_base->next_timer = NULL;
 	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *next;
 		struct hrtimer *timer;
@@ -487,6 +487,18 @@ static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
 	return expires_next;
 }
 
+static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
+{
+	unsigned int active = cpu_base->active_bases;
+	ktime_t expires_next = KTIME_MAX;
+
+	cpu_base->next_timer = NULL;
+
+	expires_next = __hrtimer_next_event_base(cpu_base, active, expires_next);
+
+	return expires_next;
+}
+
 static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base)
 {
 	ktime_t *offs_real = &base->clock_base[HRTIMER_BASE_REALTIME].offset;

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Use irqsave/irqrestore around __run_hrtimer()
  2017-12-21 10:41 ` [PATCH v4 25/36] hrtimer: Use irqsave/irqrestore around __run_hrtimer() Anna-Maria Gleixner
@ 2018-01-16  3:50   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:50 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: torvalds, hpa, tglx, anna-maria, mingo, hch, peterz, john.stultz,
	linux-kernel

Commit-ID:  dd934aa8ad1fbaab3d916125c7fe42fff75aa7ff
Gitweb:     https://git.kernel.org/tip/dd934aa8ad1fbaab3d916125c7fe42fff75aa7ff
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:54 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 03:00:47 +0100

hrtimer: Use irqsave/irqrestore around __run_hrtimer()

__run_hrtimer() is called with the hrtimer_cpu_base.lock held and
interrupts disabled. Before invoking the timer callback the base lock is
dropped, but interrupts stay disabled.

The upcoming support for softirq based hrtimers requires that interrupts
are enabled before the timer callback is invoked.

To avoid code duplication, take hrtimer_cpu_base.lock with
raw_spin_lock_irqsave(flags) at the call site and hand in the flags as
a parameter. So raw_spin_unlock_irqrestore() before the callback invocation
will either keep interrupts disabled in interrupt context or restore to
interrupt enabled state when called from softirq context.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-26-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 5d9b81d..31ccd86 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1159,7 +1159,8 @@ EXPORT_SYMBOL_GPL(hrtimer_active);
 
 static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 			  struct hrtimer_clock_base *base,
-			  struct hrtimer *timer, ktime_t *now)
+			  struct hrtimer *timer, ktime_t *now,
+			  unsigned long flags)
 {
 	enum hrtimer_restart (*fn)(struct hrtimer *);
 	int restart;
@@ -1194,11 +1195,11 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	 * protected against migration to a different CPU even if the lock
 	 * is dropped.
 	 */
-	raw_spin_unlock(&cpu_base->lock);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 	trace_hrtimer_expire_entry(timer, now);
 	restart = fn(timer);
 	trace_hrtimer_expire_exit(timer);
-	raw_spin_lock(&cpu_base->lock);
+	raw_spin_lock_irq(&cpu_base->lock);
 
 	/*
 	 * Note: We clear the running state after enqueue_hrtimer and
@@ -1226,7 +1227,8 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 	base->running = NULL;
 }
 
-static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now)
+static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now,
+				 unsigned long flags)
 {
 	struct hrtimer_clock_base *base;
 	unsigned int active = cpu_base->active_bases;
@@ -1257,7 +1259,7 @@ static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now)
 			if (basenow < hrtimer_get_softexpires_tv64(timer))
 				break;
 
-			__run_hrtimer(cpu_base, base, timer, &basenow);
+			__run_hrtimer(cpu_base, base, timer, &basenow, flags);
 		}
 	}
 }
@@ -1272,13 +1274,14 @@ void hrtimer_interrupt(struct clock_event_device *dev)
 {
 	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
 	ktime_t expires_next, now, entry_time, delta;
+	unsigned long flags;
 	int retries = 0;
 
 	BUG_ON(!cpu_base->hres_active);
 	cpu_base->nr_events++;
 	dev->next_event = KTIME_MAX;
 
-	raw_spin_lock(&cpu_base->lock);
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	entry_time = now = hrtimer_update_base(cpu_base);
 retry:
 	cpu_base->in_hrtirq = 1;
@@ -1291,7 +1294,7 @@ retry:
 	 */
 	cpu_base->expires_next = KTIME_MAX;
 
-	__hrtimer_run_queues(cpu_base, now);
+	__hrtimer_run_queues(cpu_base, now, flags);
 
 	/* Reevaluate the clock bases for the next expiry */
 	expires_next = __hrtimer_get_next_event(cpu_base);
@@ -1301,7 +1304,7 @@ retry:
 	 */
 	cpu_base->expires_next = expires_next;
 	cpu_base->in_hrtirq = 0;
-	raw_spin_unlock(&cpu_base->lock);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 
 	/* Reprogramming necessary ? */
 	if (!tick_program_event(expires_next, 0)) {
@@ -1322,7 +1325,7 @@ retry:
 	 * Acquire base lock for updating the offsets and retrieving
 	 * the current time.
 	 */
-	raw_spin_lock(&cpu_base->lock);
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	now = hrtimer_update_base(cpu_base);
 	cpu_base->nr_retries++;
 	if (++retries < 3)
@@ -1335,7 +1338,8 @@ retry:
 	 */
 	cpu_base->nr_hangs++;
 	cpu_base->hang_detected = 1;
-	raw_spin_unlock(&cpu_base->lock);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
+
 	delta = ktime_sub(now, entry_time);
 	if ((unsigned int)delta > cpu_base->max_hang_time)
 		cpu_base->max_hang_time = (unsigned int) delta;
@@ -1377,6 +1381,7 @@ static inline void __hrtimer_peek_ahead_timers(void) { }
 void hrtimer_run_queues(void)
 {
 	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	unsigned long flags;
 	ktime_t now;
 
 	if (__hrtimer_hres_active(cpu_base))
@@ -1394,10 +1399,10 @@ void hrtimer_run_queues(void)
 		return;
 	}
 
-	raw_spin_lock(&cpu_base->lock);
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	now = hrtimer_update_base(cpu_base);
-	__hrtimer_run_queues(cpu_base, now);
-	raw_spin_unlock(&cpu_base->lock);
+	__hrtimer_run_queues(cpu_base, now, flags);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
 
 /*

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Add clock bases and hrtimer mode for softirq context
  2017-12-21 10:41 ` [PATCH v4 26/36] hrtimer: Add clock bases and hrtimer mode for soft irq context Anna-Maria Gleixner
@ 2018-01-16  3:51   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:51 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hch, linux-kernel, anna-maria, peterz, hpa, tglx, torvalds,
	mingo, john.stultz

Commit-ID:  98ecadd4305d8677ba77162152485798d47dcc85
Gitweb:     https://git.kernel.org/tip/98ecadd4305d8677ba77162152485798d47dcc85
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:55 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 03:00:50 +0100

hrtimer: Add clock bases and hrtimer mode for softirq context

Currently hrtimer callback functions are always executed in hard interrupt
context. Users of hrtimers, which need their timer function to be executed
in soft interrupt context, make use of tasklets to get the proper context.

Add additional hrtimer clock bases for timers which must expire in softirq
context, so the detour via the tasklet can be avoided. This is also
required for RT, where the majority of hrtimer is moved into softirq
hrtimer context.

The selection of the expiry mode happens via a mode bit. Introduce
HRTIMER_MODE_SOFT and the matching combinations with the ABS/REL/PINNED
bits and update the decoding of hrtimer_mode in tracepoints.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-27-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h      | 14 ++++++++++++++
 include/trace/events/timer.h |  6 +++++-
 kernel/time/hrtimer.c        | 20 ++++++++++++++++++++
 3 files changed, 39 insertions(+), 1 deletion(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 98ed357..26ae8a8 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -33,14 +33,24 @@ struct hrtimer_cpu_base;
  * HRTIMER_MODE_REL		- Time value is relative to now
  * HRTIMER_MODE_PINNED		- Timer is bound to CPU (is only considered
  *				  when starting the timer)
+ * HRTIMER_MODE_SOFT		- Timer callback function will be executed in
+ *				  soft irq context
  */
 enum hrtimer_mode {
 	HRTIMER_MODE_ABS	= 0x00,
 	HRTIMER_MODE_REL	= 0x01,
 	HRTIMER_MODE_PINNED	= 0x02,
+	HRTIMER_MODE_SOFT	= 0x04,
 
 	HRTIMER_MODE_ABS_PINNED = HRTIMER_MODE_ABS | HRTIMER_MODE_PINNED,
 	HRTIMER_MODE_REL_PINNED = HRTIMER_MODE_REL | HRTIMER_MODE_PINNED,
+
+	HRTIMER_MODE_ABS_SOFT	= HRTIMER_MODE_ABS | HRTIMER_MODE_SOFT,
+	HRTIMER_MODE_REL_SOFT	= HRTIMER_MODE_REL | HRTIMER_MODE_SOFT,
+
+	HRTIMER_MODE_ABS_PINNED_SOFT = HRTIMER_MODE_ABS_PINNED | HRTIMER_MODE_SOFT,
+	HRTIMER_MODE_REL_PINNED_SOFT = HRTIMER_MODE_REL_PINNED | HRTIMER_MODE_SOFT,
+
 };
 
 /*
@@ -151,6 +161,10 @@ enum  hrtimer_base_type {
 	HRTIMER_BASE_REALTIME,
 	HRTIMER_BASE_BOOTTIME,
 	HRTIMER_BASE_TAI,
+	HRTIMER_BASE_MONOTONIC_SOFT,
+	HRTIMER_BASE_REALTIME_SOFT,
+	HRTIMER_BASE_BOOTTIME_SOFT,
+	HRTIMER_BASE_TAI_SOFT,
 	HRTIMER_MAX_CLOCK_BASES,
 };
 
diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h
index 744b431..a57e4ee 100644
--- a/include/trace/events/timer.h
+++ b/include/trace/events/timer.h
@@ -148,7 +148,11 @@ DEFINE_EVENT(timer_class, timer_cancel,
 		{ HRTIMER_MODE_ABS,		"ABS"		},	\
 		{ HRTIMER_MODE_REL,		"REL"		},	\
 		{ HRTIMER_MODE_ABS_PINNED,	"ABS|PINNED"	},	\
-		{ HRTIMER_MODE_REL_PINNED,	"REL|PINNED"	})
+		{ HRTIMER_MODE_REL_PINNED,	"REL|PINNED"	},	\
+		{ HRTIMER_MODE_ABS_SOFT,	"ABS|SOFT"	},	\
+		{ HRTIMER_MODE_REL_SOFT,	"REL|SOFT"	},	\
+		{ HRTIMER_MODE_ABS_PINNED_SOFT,	"ABS|PINNED|SOFT" },	\
+		{ HRTIMER_MODE_REL_PINNED_SOFT,	"REL|PINNED|SOFT" })
 
 /**
  * hrtimer_init - called when the hrtimer is initialized
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 31ccd86..e2353f5 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -92,6 +92,26 @@ DEFINE_PER_CPU(struct hrtimer_cpu_base, hrtimer_bases) =
 			.clockid = CLOCK_TAI,
 			.get_time = &ktime_get_clocktai,
 		},
+		{
+			.index = HRTIMER_BASE_MONOTONIC_SOFT,
+			.clockid = CLOCK_MONOTONIC,
+			.get_time = &ktime_get,
+		},
+		{
+			.index = HRTIMER_BASE_REALTIME_SOFT,
+			.clockid = CLOCK_REALTIME,
+			.get_time = &ktime_get_real,
+		},
+		{
+			.index = HRTIMER_BASE_BOOTTIME_SOFT,
+			.clockid = CLOCK_BOOTTIME,
+			.get_time = &ktime_get_boottime,
+		},
+		{
+			.index = HRTIMER_BASE_TAI_SOFT,
+			.clockid = CLOCK_TAI,
+			.get_time = &ktime_get_clocktai,
+		},
 	}
 };
 

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Prepare handling of hard and softirq based hrtimers
  2017-12-21 10:41 ` [PATCH v4 27/36] hrtimer: Prepare handling of hard and softirq based hrtimers Anna-Maria Gleixner
@ 2018-01-16  3:51   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16  3:51 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: mingo, hpa, hch, peterz, linux-kernel, john.stultz, tglx,
	torvalds, anna-maria

Commit-ID:  c458b1d102036eaa2c70e03000c959bd491c2037
Gitweb:     https://git.kernel.org/tip/c458b1d102036eaa2c70e03000c959bd491c2037
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:56 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 03:01:20 +0100

hrtimer: Prepare handling of hard and softirq based hrtimers

The softirq based hrtimer can utilize most of the existing hrtimers
functions, but need to operate on a different data set.

Add an 'active_mask' parameter to various functions so the hard and soft bases
can be selected. Fixup the existing callers and hand in the ACTIVE_HARD
mask.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-28-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 38 +++++++++++++++++++++++++++++---------
 1 file changed, 29 insertions(+), 9 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index e2353f5..ba4674e 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -60,6 +60,15 @@
 #include "tick-internal.h"
 
 /*
+ * Masks for selecting the soft and hard context timers from
+ * cpu_base->active
+ */
+#define MASK_SHIFT		(HRTIMER_BASE_MONOTONIC_SOFT)
+#define HRTIMER_ACTIVE_HARD	((1U << MASK_SHIFT) - 1)
+#define HRTIMER_ACTIVE_SOFT	(HRTIMER_ACTIVE_HARD << MASK_SHIFT)
+#define HRTIMER_ACTIVE_ALL	(HRTIMER_ACTIVE_SOFT | HRTIMER_ACTIVE_HARD)
+
+/*
  * The timer bases:
  *
  * There are more clockids than hrtimer bases. Thus, we index
@@ -507,13 +516,24 @@ static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
 	return expires_next;
 }
 
-static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base)
+/*
+ * Recomputes cpu_base::*next_timer and returns the earliest expires_next but
+ * does not set cpu_base::*expires_next, that is done by hrtimer_reprogram.
+ *
+ * @active_mask must be one of:
+ *  - HRTIMER_ACTIVE,
+ *  - HRTIMER_ACTIVE_SOFT, or
+ *  - HRTIMER_ACTIVE_HARD.
+ */
+static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base,
+					unsigned int active_mask)
 {
-	unsigned int active = cpu_base->active_bases;
+	unsigned int active;
 	ktime_t expires_next = KTIME_MAX;
 
 	cpu_base->next_timer = NULL;
 
+	active = cpu_base->active_bases & active_mask;
 	expires_next = __hrtimer_next_event_base(cpu_base, active, expires_next);
 
 	return expires_next;
@@ -553,7 +573,7 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 {
 	ktime_t expires_next;
 
-	expires_next = __hrtimer_get_next_event(cpu_base);
+	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
 
 	if (skip_equal && expires_next == cpu_base->expires_next)
 		return;
@@ -1074,7 +1094,7 @@ u64 hrtimer_get_next_event(void)
 	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 
 	if (!__hrtimer_hres_active(cpu_base))
-		expires = __hrtimer_get_next_event(cpu_base);
+		expires = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
 
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 
@@ -1248,10 +1268,10 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base,
 }
 
 static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now,
-				 unsigned long flags)
+				 unsigned long flags, unsigned int active_mask)
 {
 	struct hrtimer_clock_base *base;
-	unsigned int active = cpu_base->active_bases;
+	unsigned int active = cpu_base->active_bases & active_mask;
 
 	for_each_active_base(base, cpu_base, active) {
 		struct timerqueue_node *node;
@@ -1314,10 +1334,10 @@ retry:
 	 */
 	cpu_base->expires_next = KTIME_MAX;
 
-	__hrtimer_run_queues(cpu_base, now, flags);
+	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 
 	/* Reevaluate the clock bases for the next expiry */
-	expires_next = __hrtimer_get_next_event(cpu_base);
+	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
 	/*
 	 * Store the new expiry value so the migration code can verify
 	 * against it.
@@ -1421,7 +1441,7 @@ void hrtimer_run_queues(void)
 
 	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	now = hrtimer_update_base(cpu_base);
-	__hrtimer_run_queues(cpu_base, now, flags);
+	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
 

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Implement support for softirq based hrtimers
  2017-12-21 10:41 ` [PATCH v4 28/36] hrtimer: Implement support for " Anna-Maria Gleixner
@ 2018-01-16 10:22   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16 10:22 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hch, anna-maria, torvalds, tglx, mingo, peterz, linux-kernel,
	hpa, john.stultz

Commit-ID:  5da70160462e80b0ab8a6960cdd0cdd476907523
Gitweb:     https://git.kernel.org/tip/5da70160462e80b0ab8a6960cdd0cdd476907523
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:57 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 09:51:22 +0100

hrtimer: Implement support for softirq based hrtimers

hrtimer callbacks are always invoked in hard interrupt context. Several
users in tree require soft interrupt context for their callbacks and
achieve this by combining a hrtimer with a tasklet. The hrtimer schedules
the tasklet in hard interrupt context and the tasklet callback gets invoked
in softirq context later.

That's suboptimal and aside of that the real-time patch moves most of the
hrtimers into softirq context. So adding native support for hrtimers
expiring in softirq context is a valuable extension for both mainline and
the RT patch set.

Each valid hrtimer clock id has two associated hrtimer clock bases: one for
timers expiring in hardirq context and one for timers expiring in softirq
context.

Implement the functionality to associate a hrtimer with the hard or softirq
related clock bases and update the relevant functions to take them into
account when the next expiry time needs to be evaluated.

Add a check into the hard interrupt context handler functions to check
whether the first expiring softirq based timer has expired. If it's expired
the softirq is raised and the accounting of softirq based timers to
evaluate the next expiry time for programming the timer hardware is skipped
until the softirq processing has finished. At the end of the softirq
processing the regular processing is resumed.

Suggested-by: Thomas Gleixner <tglx@linutronix.de>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-29-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 include/linux/hrtimer.h |  21 ++++--
 kernel/time/hrtimer.c   | 196 ++++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 188 insertions(+), 29 deletions(-)

diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 26ae8a8..c7902ca 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -103,6 +103,7 @@ enum hrtimer_restart {
  * @base:	pointer to the timer base (per cpu and per clock)
  * @state:	state information (See bit values above)
  * @is_rel:	Set if the timer was armed relative
+ * @is_soft:	Set if hrtimer will be expired in soft interrupt context.
  *
  * The hrtimer structure must be initialized by hrtimer_init()
  */
@@ -113,6 +114,7 @@ struct hrtimer {
 	struct hrtimer_clock_base	*base;
 	u8				state;
 	u8				is_rel;
+	u8				is_soft;
 };
 
 /**
@@ -178,13 +180,18 @@ enum  hrtimer_base_type {
  * @hres_active:	State of high resolution mode
  * @in_hrtirq:		hrtimer_interrupt() is currently executing
  * @hang_detected:	The last hrtimer interrupt detected a hang
+ * @softirq_activated:	displays, if the softirq is raised - update of softirq
+ *			related settings is not required then.
  * @nr_events:		Total number of hrtimer interrupt events
  * @nr_retries:		Total number of hrtimer interrupt retries
  * @nr_hangs:		Total number of hrtimer interrupt hangs
  * @max_hang_time:	Maximum time spent in hrtimer_interrupt
  * @expires_next:	absolute time of the next event, is required for remote
- *			hrtimer enqueue
+ *			hrtimer enqueue; it is the total first expiry time (hard
+ *			and soft hrtimer are taken into account)
  * @next_timer:		Pointer to the first expiring timer
+ * @softirq_expires_next: Time to check, if soft queues needs also to be expired
+ * @softirq_next_timer: Pointer to the first expiring softirq based timer
  * @clock_base:		array of clock bases for this cpu
  *
  * Note: next_timer is just an optimization for __remove_hrtimer().
@@ -196,9 +203,10 @@ struct hrtimer_cpu_base {
 	unsigned int			cpu;
 	unsigned int			active_bases;
 	unsigned int			clock_was_set_seq;
-	unsigned int			hres_active	: 1,
-					in_hrtirq	: 1,
-					hang_detected	: 1;
+	unsigned int			hres_active		: 1,
+					in_hrtirq		: 1,
+					hang_detected		: 1,
+					softirq_activated       : 1;
 #ifdef CONFIG_HIGH_RES_TIMERS
 	unsigned int			nr_events;
 	unsigned short			nr_retries;
@@ -207,6 +215,8 @@ struct hrtimer_cpu_base {
 #endif
 	ktime_t				expires_next;
 	struct hrtimer			*next_timer;
+	ktime_t				softirq_expires_next;
+	struct hrtimer			*softirq_next_timer;
 	struct hrtimer_clock_base	clock_base[HRTIMER_MAX_CLOCK_BASES];
 } ____cacheline_aligned;
 
@@ -379,7 +389,8 @@ extern void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
  * @timer:	the timer to be added
  * @tim:	expiry time
  * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED);
+ *		softirq based mode is considered for debug purpose only!
  */
 static inline void hrtimer_start(struct hrtimer *timer, ktime_t tim,
 				 const enum hrtimer_mode mode)
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index ba4674e..d93e3e7 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -411,7 +411,8 @@ static inline void debug_hrtimer_init(struct hrtimer *timer)
 	debug_object_init(timer, &hrtimer_debug_descr);
 }
 
-static inline void debug_hrtimer_activate(struct hrtimer *timer)
+static inline void debug_hrtimer_activate(struct hrtimer *timer,
+					  enum hrtimer_mode mode)
 {
 	debug_object_activate(timer, &hrtimer_debug_descr);
 }
@@ -444,8 +445,10 @@ void destroy_hrtimer_on_stack(struct hrtimer *timer)
 EXPORT_SYMBOL_GPL(destroy_hrtimer_on_stack);
 
 #else
+
 static inline void debug_hrtimer_init(struct hrtimer *timer) { }
-static inline void debug_hrtimer_activate(struct hrtimer *timer) { }
+static inline void debug_hrtimer_activate(struct hrtimer *timer,
+					  enum hrtimer_mode mode) { }
 static inline void debug_hrtimer_deactivate(struct hrtimer *timer) { }
 #endif
 
@@ -460,7 +463,7 @@ debug_init(struct hrtimer *timer, clockid_t clockid,
 static inline void debug_activate(struct hrtimer *timer,
 				  enum hrtimer_mode mode)
 {
-	debug_hrtimer_activate(timer);
+	debug_hrtimer_activate(timer, mode);
 	trace_hrtimer_start(timer, mode);
 }
 
@@ -503,7 +506,10 @@ static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
 		expires = ktime_sub(hrtimer_get_expires(timer), base->offset);
 		if (expires < expires_next) {
 			expires_next = expires;
-			cpu_base->next_timer = timer;
+			if (timer->is_soft)
+				cpu_base->softirq_next_timer = timer;
+			else
+				cpu_base->next_timer = timer;
 		}
 	}
 	/*
@@ -520,21 +526,39 @@ static ktime_t __hrtimer_next_event_base(struct hrtimer_cpu_base *cpu_base,
  * Recomputes cpu_base::*next_timer and returns the earliest expires_next but
  * does not set cpu_base::*expires_next, that is done by hrtimer_reprogram.
  *
+ * When a softirq is pending, we can ignore the HRTIMER_ACTIVE_SOFT bases,
+ * those timers will get run whenever the softirq gets handled, at the end of
+ * hrtimer_run_softirq(), hrtimer_update_softirq_timer() will re-add these bases.
+ *
+ * Therefore softirq values are those from the HRTIMER_ACTIVE_SOFT clock bases.
+ * The !softirq values are the minima across HRTIMER_ACTIVE_ALL, unless an actual
+ * softirq is pending, in which case they're the minima of HRTIMER_ACTIVE_HARD.
+ *
  * @active_mask must be one of:
- *  - HRTIMER_ACTIVE,
+ *  - HRTIMER_ACTIVE_ALL,
  *  - HRTIMER_ACTIVE_SOFT, or
  *  - HRTIMER_ACTIVE_HARD.
  */
-static ktime_t __hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base,
-					unsigned int active_mask)
+static ktime_t
+__hrtimer_get_next_event(struct hrtimer_cpu_base *cpu_base, unsigned int active_mask)
 {
 	unsigned int active;
+	struct hrtimer *next_timer = NULL;
 	ktime_t expires_next = KTIME_MAX;
 
-	cpu_base->next_timer = NULL;
+	if (!cpu_base->softirq_activated && (active_mask & HRTIMER_ACTIVE_SOFT)) {
+		active = cpu_base->active_bases & HRTIMER_ACTIVE_SOFT;
+		cpu_base->softirq_next_timer = NULL;
+		expires_next = __hrtimer_next_event_base(cpu_base, active, KTIME_MAX);
+
+		next_timer = cpu_base->softirq_next_timer;
+	}
 
-	active = cpu_base->active_bases & active_mask;
-	expires_next = __hrtimer_next_event_base(cpu_base, active, expires_next);
+	if (active_mask & HRTIMER_ACTIVE_HARD) {
+		active = cpu_base->active_bases & HRTIMER_ACTIVE_HARD;
+		cpu_base->next_timer = next_timer;
+		expires_next = __hrtimer_next_event_base(cpu_base, active, expires_next);
+	}
 
 	return expires_next;
 }
@@ -545,8 +569,14 @@ static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base)
 	ktime_t *offs_boot = &base->clock_base[HRTIMER_BASE_BOOTTIME].offset;
 	ktime_t *offs_tai = &base->clock_base[HRTIMER_BASE_TAI].offset;
 
-	return ktime_get_update_offsets_now(&base->clock_was_set_seq,
+	ktime_t now = ktime_get_update_offsets_now(&base->clock_was_set_seq,
 					    offs_real, offs_boot, offs_tai);
+
+	base->clock_base[HRTIMER_BASE_REALTIME_SOFT].offset = *offs_real;
+	base->clock_base[HRTIMER_BASE_BOOTTIME_SOFT].offset = *offs_boot;
+	base->clock_base[HRTIMER_BASE_TAI_SOFT].offset = *offs_tai;
+
+	return now;
 }
 
 /*
@@ -573,7 +603,23 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)
 {
 	ktime_t expires_next;
 
-	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
+	/*
+	 * Find the current next expiration time.
+	 */
+	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_ALL);
+
+	if (cpu_base->next_timer && cpu_base->next_timer->is_soft) {
+		/*
+		 * When the softirq is activated, hrtimer has to be
+		 * programmed with the first hard hrtimer because soft
+		 * timer interrupt could occur too late.
+		 */
+		if (cpu_base->softirq_activated)
+			expires_next = __hrtimer_get_next_event(cpu_base,
+								HRTIMER_ACTIVE_HARD);
+		else
+			cpu_base->softirq_expires_next = expires_next;
+	}
 
 	if (skip_equal && expires_next == cpu_base->expires_next)
 		return;
@@ -700,7 +746,7 @@ static inline void retrigger_next_event(void *arg) { }
  *
  * Called with interrupts disabled and base->cpu_base.lock held
  */
-static void hrtimer_reprogram(struct hrtimer *timer)
+static void hrtimer_reprogram(struct hrtimer *timer, bool reprogram)
 {
 	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
 	struct hrtimer_clock_base *base = timer->base;
@@ -709,6 +755,37 @@ static void hrtimer_reprogram(struct hrtimer *timer)
 	WARN_ON_ONCE(hrtimer_get_expires_tv64(timer) < 0);
 
 	/*
+	 * CLOCK_REALTIME timer might be requested with an absolute
+	 * expiry time which is less than base->offset. Set it to 0.
+	 */
+	if (expires < 0)
+		expires = 0;
+
+	if (timer->is_soft) {
+		/*
+		 * soft hrtimer could be started on a remote CPU. In this
+		 * case softirq_expires_next needs to be updated on the
+		 * remote CPU. The soft hrtimer will not expire before the
+		 * first hard hrtimer on the remote CPU -
+		 * hrtimer_check_target() prevents this case.
+		 */
+		struct hrtimer_cpu_base *timer_cpu_base = base->cpu_base;
+
+		if (timer_cpu_base->softirq_activated)
+			return;
+
+		if (!ktime_before(expires, timer_cpu_base->softirq_expires_next))
+			return;
+
+		timer_cpu_base->softirq_next_timer = timer;
+		timer_cpu_base->softirq_expires_next = expires;
+
+		if (!ktime_before(expires, timer_cpu_base->expires_next) ||
+		    !reprogram)
+			return;
+	}
+
+	/*
 	 * If the timer is not on the current cpu, we cannot reprogram
 	 * the other cpus clock event device.
 	 */
@@ -725,13 +802,6 @@ static void hrtimer_reprogram(struct hrtimer *timer)
 	if (cpu_base->in_hrtirq)
 		return;
 
-	/*
-	 * CLOCK_REALTIME timer might be requested with an absolute
-	 * expiry time which is less than base->offset. Set it to 0.
-	 */
-	if (expires < 0)
-		expires = 0;
-
 	if (expires >= cpu_base->expires_next)
 		return;
 
@@ -957,6 +1027,31 @@ static inline ktime_t hrtimer_update_lowres(struct hrtimer *timer, ktime_t tim,
 	return tim;
 }
 
+static void
+hrtimer_update_softirq_timer(struct hrtimer_cpu_base *cpu_base, bool reprogram)
+{
+	ktime_t expires;
+
+	/*
+	 * Find the next SOFT expiration.
+	 */
+	expires = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_SOFT);
+
+	/*
+	 * reprogramming needs to be triggered, even if the next soft
+	 * hrtimer expires at the same time than the next hard
+	 * hrtimer. cpu_base->softirq_expires_next needs to be updated!
+	 */
+	if (expires == KTIME_MAX)
+		return;
+
+	/*
+	 * cpu_base->*next_timer is recomputed by __hrtimer_get_next_event()
+	 * cpu_base->*expires_next is only set by hrtimer_reprogram()
+	 */
+	hrtimer_reprogram(cpu_base->softirq_next_timer, reprogram);
+}
+
 static int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 				    u64 delta_ns, const enum hrtimer_mode mode,
 				    struct hrtimer_clock_base *base)
@@ -978,13 +1073,15 @@ static int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 
 	return enqueue_hrtimer(timer, new_base, mode);
 }
+
 /**
  * hrtimer_start_range_ns - (re)start an hrtimer
  * @timer:	the timer to be added
  * @tim:	expiry time
  * @delta_ns:	"slack" range for the timer
  * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED)
+ *		relative (HRTIMER_MODE_REL), and pinned (HRTIMER_MODE_PINNED);
+ *		softirq based mode is considered for debug purpose only!
  */
 void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 			    u64 delta_ns, const enum hrtimer_mode mode)
@@ -992,10 +1089,16 @@ void hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,
 	struct hrtimer_clock_base *base;
 	unsigned long flags;
 
+	/*
+	 * Check whether the HRTIMER_MODE_SOFT bit and hrtimer.is_soft
+	 * match.
+	 */
+	WARN_ON_ONCE(!(mode & HRTIMER_MODE_SOFT) ^ !timer->is_soft);
+
 	base = lock_hrtimer_base(timer, &flags);
 
 	if (__hrtimer_start_range_ns(timer, tim, delta_ns, mode, base))
-		hrtimer_reprogram(timer);
+		hrtimer_reprogram(timer, true);
 
 	unlock_hrtimer_base(timer, &flags);
 }
@@ -1094,7 +1197,7 @@ u64 hrtimer_get_next_event(void)
 	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 
 	if (!__hrtimer_hres_active(cpu_base))
-		expires = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
+		expires = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_ALL);
 
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 
@@ -1304,6 +1407,23 @@ static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now,
 	}
 }
 
+static __latent_entropy void hrtimer_run_softirq(struct softirq_action *h)
+{
+	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	unsigned long flags;
+	ktime_t now;
+
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
+
+	now = hrtimer_update_base(cpu_base);
+	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_SOFT);
+
+	cpu_base->softirq_activated = 0;
+	hrtimer_update_softirq_timer(cpu_base, true);
+
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
+}
+
 #ifdef CONFIG_HIGH_RES_TIMERS
 
 /*
@@ -1334,10 +1454,16 @@ retry:
 	 */
 	cpu_base->expires_next = KTIME_MAX;
 
+	if (!ktime_before(now, cpu_base->softirq_expires_next)) {
+		cpu_base->softirq_expires_next = KTIME_MAX;
+		cpu_base->softirq_activated = 1;
+		raise_softirq_irqoff(HRTIMER_SOFTIRQ);
+	}
+
 	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 
 	/* Reevaluate the clock bases for the next expiry */
-	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_HARD);
+	expires_next = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_ALL);
 	/*
 	 * Store the new expiry value so the migration code can verify
 	 * against it.
@@ -1441,6 +1567,13 @@ void hrtimer_run_queues(void)
 
 	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 	now = hrtimer_update_base(cpu_base);
+
+	if (!ktime_before(now, cpu_base->softirq_expires_next)) {
+		cpu_base->softirq_expires_next = KTIME_MAX;
+		cpu_base->softirq_activated = 1;
+		raise_softirq_irqoff(HRTIMER_SOFTIRQ);
+	}
+
 	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
@@ -1622,6 +1755,7 @@ int hrtimers_prepare_cpu(unsigned int cpu)
 	cpu_base->cpu = cpu;
 	cpu_base->hres_active = 0;
 	cpu_base->expires_next = KTIME_MAX;
+	cpu_base->softirq_expires_next = KTIME_MAX;
 	return 0;
 }
 
@@ -1665,6 +1799,12 @@ int hrtimers_dead_cpu(unsigned int scpu)
 	BUG_ON(cpu_online(scpu));
 	tick_cancel_sched_timer(scpu);
 
+	/*
+	 * this BH disable ensures that raise_softirq_irqoff() does
+	 * not wakeup ksoftirqd (and acquire the pi-lock) while
+	 * holding the cpu_base lock
+	 */
+	local_bh_disable();
 	local_irq_disable();
 	old_base = &per_cpu(hrtimer_bases, scpu);
 	new_base = this_cpu_ptr(&hrtimer_bases);
@@ -1680,12 +1820,19 @@ int hrtimers_dead_cpu(unsigned int scpu)
 				     &new_base->clock_base[i]);
 	}
 
+	/*
+	 * The migration might have changed the first expiring softirq
+	 * timer on this CPU. Update it.
+	 */
+	hrtimer_update_softirq_timer(new_base, false);
+
 	raw_spin_unlock(&old_base->lock);
 	raw_spin_unlock(&new_base->lock);
 
 	/* Check, if we got expired work to do */
 	__hrtimer_peek_ahead_timers();
 	local_irq_enable();
+	local_bh_enable();
 	return 0;
 }
 
@@ -1694,6 +1841,7 @@ int hrtimers_dead_cpu(unsigned int scpu)
 void __init hrtimers_init(void)
 {
 	hrtimers_prepare_cpu(smp_processor_id());
+	open_softirq(HRTIMER_SOFTIRQ, hrtimer_run_softirq);
 }
 
 /**

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] hrtimer: Implement SOFT/HARD clock base selection
  2017-12-21 10:41 ` [PATCH v4 29/36] hrtimer: Implement SOFT/HARD clock base selection Anna-Maria Gleixner
@ 2018-01-16 10:22   ` tip-bot for Anna-Maria Gleixner
  0 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Anna-Maria Gleixner @ 2018-01-16 10:22 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: torvalds, peterz, hpa, linux-kernel, hch, tglx, mingo,
	john.stultz, anna-maria

Commit-ID:  42f42da41b54c191ae6a775e84a86c100d66c5e8
Gitweb:     https://git.kernel.org/tip/42f42da41b54c191ae6a775e84a86c100d66c5e8
Author:     Anna-Maria Gleixner <anna-maria@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:41:58 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 09:51:22 +0100

hrtimer: Implement SOFT/HARD clock base selection

All prerequisites to handle hrtimers for expiry in either hard or soft
interrupt context are in place.

Add the missing bit in hrtimer_init() which associates the timer to the
hard or the softirq clock base.

Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-30-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/time/hrtimer.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index d93e3e7..3d20158 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1220,8 +1220,9 @@ static inline int hrtimer_clockid_to_base(clockid_t clock_id)
 static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 			   enum hrtimer_mode mode)
 {
+	bool softtimer = !!(mode & HRTIMER_MODE_SOFT);
+	int base = softtimer ? HRTIMER_MAX_CLOCK_BASES / 2 : 0;
 	struct hrtimer_cpu_base *cpu_base;
-	int base;
 
 	memset(timer, 0, sizeof(struct hrtimer));
 
@@ -1235,7 +1236,8 @@ static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 	if (clock_id == CLOCK_REALTIME && mode & HRTIMER_MODE_REL)
 		clock_id = CLOCK_MONOTONIC;
 
-	base = hrtimer_clockid_to_base(clock_id);
+	base += hrtimer_clockid_to_base(clock_id);
+	timer->is_soft = softtimer;
 	timer->base = &cpu_base->clock_base[base];
 	timerqueue_init(&timer->node);
 }
@@ -1244,8 +1246,13 @@ static void __hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
  * hrtimer_init - initialize a timer to the given clock
  * @timer:	the timer to be initialized
  * @clock_id:	the clock to be used
- * @mode:	timer mode: absolute (HRTIMER_MODE_ABS) or
- *		relative (HRTIMER_MODE_REL); pinned is not considered here!
+ * @mode:       The modes which are relevant for intitialization:
+ *              HRTIMER_MODE_ABS, HRTIMER_MODE_REL, HRTIMER_MODE_ABS_SOFT,
+ *              HRTIMER_MODE_REL_SOFT
+ *
+ *              The PINNED variants of the above can be handed in,
+ *              but the PINNED bit is ignored as pinning happens
+ *              when the hrtimer is started
  */
 void hrtimer_init(struct hrtimer *timer, clockid_t clock_id,
 		  enum hrtimer_mode mode)

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] ALSA/dummy: Replace tasklet with softirq hrtimer
  2017-12-21 10:42   ` Anna-Maria Gleixner
  (?)
@ 2018-01-16 10:23   ` tip-bot for Thomas Gleixner
  -1 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Thomas Gleixner @ 2018-01-16 10:23 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: anna-maria, tiwai, tglx, hch, o-takashi, tiwai, torvalds,
	john.stultz, linux-kernel, hpa, peterz, mingo, perex

Commit-ID:  b03bbbe08ff04d80136b6aac152954ef308a4909
Gitweb:     https://git.kernel.org/tip/b03bbbe08ff04d80136b6aac152954ef308a4909
Author:     Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:42:03 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 09:51:22 +0100

ALSA/dummy: Replace tasklet with softirq hrtimer

The tasklet is used to defer the execution of snd_pcm_period_elapsed() to
the softirq context. Using the HRTIMER_MODE_SOFT mode invokes the timer
callback in softirq context as well which renders the tasklet useless.

[o-takashi: avoid stall due to a call of hrtimer_cancel() on a callback of hrtimer]

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Reviewed-by: Takashi Iwai <tiwai@suse.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Jaroslav Kysela <perex@perex.cz>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Takashi Iwai <tiwai@suse.com>
Cc: Takashi Sakamoto <o-takashi@sakamocchi.jp>
Cc: alsa-devel@alsa-project.org
Cc: keescook@chromium.org
Link: http://lkml.kernel.org/r/20171221104205.7269-35-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 sound/drivers/dummy.c | 27 ++++++++++++---------------
 1 file changed, 12 insertions(+), 15 deletions(-)

diff --git a/sound/drivers/dummy.c b/sound/drivers/dummy.c
index 7b2b1f7..6ad2ff5 100644
--- a/sound/drivers/dummy.c
+++ b/sound/drivers/dummy.c
@@ -375,17 +375,9 @@ struct dummy_hrtimer_pcm {
 	ktime_t period_time;
 	atomic_t running;
 	struct hrtimer timer;
-	struct tasklet_struct tasklet;
 	struct snd_pcm_substream *substream;
 };
 
-static void dummy_hrtimer_pcm_elapsed(unsigned long priv)
-{
-	struct dummy_hrtimer_pcm *dpcm = (struct dummy_hrtimer_pcm *)priv;
-	if (atomic_read(&dpcm->running))
-		snd_pcm_period_elapsed(dpcm->substream);
-}
-
 static enum hrtimer_restart dummy_hrtimer_callback(struct hrtimer *timer)
 {
 	struct dummy_hrtimer_pcm *dpcm;
@@ -393,7 +385,14 @@ static enum hrtimer_restart dummy_hrtimer_callback(struct hrtimer *timer)
 	dpcm = container_of(timer, struct dummy_hrtimer_pcm, timer);
 	if (!atomic_read(&dpcm->running))
 		return HRTIMER_NORESTART;
-	tasklet_schedule(&dpcm->tasklet);
+	/*
+	 * In cases of XRUN and draining, this calls .trigger to stop PCM
+	 * substream.
+	 */
+	snd_pcm_period_elapsed(dpcm->substream);
+	if (!atomic_read(&dpcm->running))
+		return HRTIMER_NORESTART;
+
 	hrtimer_forward_now(timer, dpcm->period_time);
 	return HRTIMER_RESTART;
 }
@@ -403,7 +402,7 @@ static int dummy_hrtimer_start(struct snd_pcm_substream *substream)
 	struct dummy_hrtimer_pcm *dpcm = substream->runtime->private_data;
 
 	dpcm->base_time = hrtimer_cb_get_time(&dpcm->timer);
-	hrtimer_start(&dpcm->timer, dpcm->period_time, HRTIMER_MODE_REL);
+	hrtimer_start(&dpcm->timer, dpcm->period_time, HRTIMER_MODE_REL_SOFT);
 	atomic_set(&dpcm->running, 1);
 	return 0;
 }
@@ -413,14 +412,14 @@ static int dummy_hrtimer_stop(struct snd_pcm_substream *substream)
 	struct dummy_hrtimer_pcm *dpcm = substream->runtime->private_data;
 
 	atomic_set(&dpcm->running, 0);
-	hrtimer_cancel(&dpcm->timer);
+	if (!hrtimer_callback_running(&dpcm->timer))
+		hrtimer_cancel(&dpcm->timer);
 	return 0;
 }
 
 static inline void dummy_hrtimer_sync(struct dummy_hrtimer_pcm *dpcm)
 {
 	hrtimer_cancel(&dpcm->timer);
-	tasklet_kill(&dpcm->tasklet);
 }
 
 static snd_pcm_uframes_t
@@ -465,12 +464,10 @@ static int dummy_hrtimer_create(struct snd_pcm_substream *substream)
 	if (!dpcm)
 		return -ENOMEM;
 	substream->runtime->private_data = dpcm;
-	hrtimer_init(&dpcm->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	hrtimer_init(&dpcm->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
 	dpcm->timer.function = dummy_hrtimer_callback;
 	dpcm->substream = substream;
 	atomic_set(&dpcm->running, 0);
-	tasklet_init(&dpcm->tasklet, dummy_hrtimer_pcm_elapsed,
-		     (unsigned long)dpcm);
 	return 0;
 }
 

^ permalink raw reply related	[flat|nested] 80+ messages in thread

* [tip:timers/core] usb/gadget/NCM: Replace tasklet with softirq hrtimer
  2017-12-21 10:42   ` [v4,35/36] " Anna-Maria Gleixner
  (?)
@ 2018-01-16 10:23   ` tip-bot for Thomas Gleixner
  -1 siblings, 0 replies; 80+ messages in thread
From: tip-bot for Thomas Gleixner @ 2018-01-16 10:23 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hch, balbi, torvalds, john.stultz, hpa, tglx, linux-kernel,
	felipe.balbi, anna-maria, mingo, peterz

Commit-ID:  b1a31a5f5f27ff8aba42b545a1c721941f735107
Gitweb:     https://git.kernel.org/tip/b1a31a5f5f27ff8aba42b545a1c721941f735107
Author:     Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Thu, 21 Dec 2017 11:42:04 +0100
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 16 Jan 2018 09:51:23 +0100

usb/gadget/NCM: Replace tasklet with softirq hrtimer

The tx_tasklet tasklet is used in invoke the hrtimer (task_timer) in
softirq context. This can be also achieved without the tasklet but
with HRTIMER_MODE_SOFT as hrtimer mode.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Anna-Maria Gleixner <anna-maria@linutronix.de>
Acked-by: Felipe Balbi <felipe.balbi@linux.intel.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Felipe Balbi <balbi@kernel.org>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: keescook@chromium.org
Cc: linux-usb@vger.kernel.org
Link: http://lkml.kernel.org/r/20171221104205.7269-36-anna-maria@linutronix.de
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 drivers/usb/gadget/function/f_ncm.c | 30 +++++++-----------------------
 1 file changed, 7 insertions(+), 23 deletions(-)

diff --git a/drivers/usb/gadget/function/f_ncm.c b/drivers/usb/gadget/function/f_ncm.c
index c5bce8e..5780fba 100644
--- a/drivers/usb/gadget/function/f_ncm.c
+++ b/drivers/usb/gadget/function/f_ncm.c
@@ -73,9 +73,7 @@ struct f_ncm {
 	struct sk_buff			*skb_tx_ndp;
 	u16				ndp_dgram_count;
 	bool				timer_force_tx;
-	struct tasklet_struct		tx_tasklet;
 	struct hrtimer			task_timer;
-
 	bool				timer_stopping;
 };
 
@@ -1104,7 +1102,7 @@ static struct sk_buff *ncm_wrap_ntb(struct gether *port,
 
 		/* Delay the timer. */
 		hrtimer_start(&ncm->task_timer, TX_TIMEOUT_NSECS,
-			      HRTIMER_MODE_REL);
+			      HRTIMER_MODE_REL_SOFT);
 
 		/* Add the datagram position entries */
 		ntb_ndp = skb_put_zero(ncm->skb_tx_ndp, dgram_idx_len);
@@ -1148,17 +1146,15 @@ err:
 }
 
 /*
- * This transmits the NTB if there are frames waiting.
+ * The transmit should only be run if no skb data has been sent
+ * for a certain duration.
  */
-static void ncm_tx_tasklet(unsigned long data)
+static enum hrtimer_restart ncm_tx_timeout(struct hrtimer *data)
 {
-	struct f_ncm	*ncm = (void *)data;
-
-	if (ncm->timer_stopping)
-		return;
+	struct f_ncm *ncm = container_of(data, struct f_ncm, task_timer);
 
 	/* Only send if data is available. */
-	if (ncm->skb_tx_data) {
+	if (!ncm->timer_stopping && ncm->skb_tx_data) {
 		ncm->timer_force_tx = true;
 
 		/* XXX This allowance of a NULL skb argument to ndo_start_xmit
@@ -1171,16 +1167,6 @@ static void ncm_tx_tasklet(unsigned long data)
 
 		ncm->timer_force_tx = false;
 	}
-}
-
-/*
- * The transmit should only be run if no skb data has been sent
- * for a certain duration.
- */
-static enum hrtimer_restart ncm_tx_timeout(struct hrtimer *data)
-{
-	struct f_ncm *ncm = container_of(data, struct f_ncm, task_timer);
-	tasklet_schedule(&ncm->tx_tasklet);
 	return HRTIMER_NORESTART;
 }
 
@@ -1513,8 +1499,7 @@ static int ncm_bind(struct usb_configuration *c, struct usb_function *f)
 	ncm->port.open = ncm_open;
 	ncm->port.close = ncm_close;
 
-	tasklet_init(&ncm->tx_tasklet, ncm_tx_tasklet, (unsigned long) ncm);
-	hrtimer_init(&ncm->task_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	hrtimer_init(&ncm->task_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
 	ncm->task_timer.function = ncm_tx_timeout;
 
 	DBG(cdev, "CDC Network: %s speed IN/%s OUT/%s NOTIFY/%s\n",
@@ -1623,7 +1608,6 @@ static void ncm_unbind(struct usb_configuration *c, struct usb_function *f)
 	DBG(c->cdev, "ncm unbind\n");
 
 	hrtimer_cancel(&ncm->task_timer);
-	tasklet_kill(&ncm->tx_tasklet);
 
 	ncm_string_defs[0].id = 0;
 	usb_free_all_descriptors(f);

^ permalink raw reply related	[flat|nested] 80+ messages in thread

end of thread, other threads:[~2018-01-16 10:28 UTC | newest]

Thread overview: 80+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-12-21 10:41 [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 01/36] timers: Use static keys for migrate_enable/nohz_active Anna-Maria Gleixner
2017-12-22 15:45   ` [PATCH v5 " Sebastian Andrzej Siewior
2018-01-14 22:13     ` Thomas Gleixner
2018-01-14 22:30       ` [PATCH v6 " Thomas Gleixner
2018-01-11  4:25   ` [PATCH v4 " Frederic Weisbecker
2018-01-16  3:40   ` [tip:timers/core] hrtimer: Optimize the hrtimer code by using static keys for migration_enable/nohz_active tip-bot for Thomas Gleixner
2017-12-21 10:41 ` [PATCH v4 02/36] hrtimer: Correct blantanly wrong comment Anna-Maria Gleixner
2018-01-11 18:28   ` Frederic Weisbecker
2018-01-16  1:02   ` Ingo Molnar
2018-01-16  3:40   ` [tip:timers/core] hrtimer: Correct blatantly incorrect comment tip-bot for Thomas Gleixner
2017-12-21 10:41 ` [PATCH v4 03/36] hrtimer: Fix kerneldoc for struct hrtimer_cpu_base Anna-Maria Gleixner
2018-01-16  3:41   ` [tip:timers/core] hrtimer: Fix kerneldoc syntax for 'struct hrtimer_cpu_base' tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 04/36] hrtimer: Cleanup clock argument in schedule_hrtimeout_range_clock() Anna-Maria Gleixner
2018-01-16  3:41   ` [tip:timers/core] hrtimer: Clean up the 'int clock' parameter of schedule_hrtimeout_range_clock() tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 05/36] hrtimer: Fix hrtimer function description Anna-Maria Gleixner
2018-01-16  3:42   ` [tip:timers/core] hrtimer: Fix hrtimer_start[_range_ns]() function descriptions tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 06/36] hrtimer: Ensure POSIX compliance (relative CLOCK_REALTIME hrtimers) Anna-Maria Gleixner
2018-01-16  3:42   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 07/36] hrtimer: Cleanup hrtimer_mode enum Anna-Maria Gleixner
2018-01-16  3:43   ` [tip:timers/core] hrtimer: Clean up 'enum hrtimer_mode' tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 08/36] tracing/hrtimer: Take all clock bases and modes into account Anna-Maria Gleixner
2018-01-16  3:43   ` [tip:timers/core] tracing/hrtimer: Fix tracing bugs by taking " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 09/36] tracing/hrtimer: Print hrtimer mode in hrtimer_start tracepoint Anna-Maria Gleixner
2018-01-16  3:43   ` [tip:timers/core] tracing/hrtimer: Print the hrtimer mode in the 'hrtimer_start' tracepoint tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 10/36] hrtimer: Switch for loop to _ffs() evaluation Anna-Maria Gleixner
2018-01-16  3:44   ` [tip:timers/core] hrtimer: Switch 'for' " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 11/36] hrtimer: Store running timer in hrtimer_clock_base Anna-Maria Gleixner
2018-01-16  3:44   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 12/36] hrtimer: Make room in struct hrtimer_cpu_base Anna-Maria Gleixner
2018-01-16  3:45   ` [tip:timers/core] hrtimer: Make room in 'struct hrtimer_cpu_base' tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 13/36] hrtimer: Reduce conditional code (hres_active) Anna-Maria Gleixner
2018-01-16  3:45   ` [tip:timers/core] hrtimer: Make the hrtimer_cpu_base::hres_active field unconditional, to simplify the code tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 14/36] hrtimer: Use accesor functions instead of direct access Anna-Maria Gleixner
2018-01-16  3:46   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 15/36] hrtimer: Make the remote enqueue check unconditional Anna-Maria Gleixner
2018-01-16  3:46   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 16/36] hrtimer: Make hrtimer_cpu_base.next_timer handling unconditional Anna-Maria Gleixner
2018-01-16  3:46   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 17/36] hrtimer: Make hrtimer_reprogramm() unconditional Anna-Maria Gleixner
2018-01-16  3:47   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 18/36] hrtimer: Make hrtimer_force_reprogramm() unconditionally available Anna-Maria Gleixner
2018-01-16  3:47   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 19/36] hrtimer: Unify handling of hrtimer remove Anna-Maria Gleixner
2018-01-16  3:48   ` [tip:timers/core] hrtimer: Unify hrtimer removal handling tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 20/36] hrtimer: Unify handling of remote enqueue Anna-Maria Gleixner
2018-01-16  3:48   ` [tip:timers/core] hrtimer: Unify remote enqueue handling tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 21/36] hrtimer: Make remote enqueue decision less restrictive Anna-Maria Gleixner
2018-01-16  3:49   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 22/36] hrtimer: Remove base argument from hrtimer_reprogram() Anna-Maria Gleixner
2018-01-16  3:49   ` [tip:timers/core] hrtimer: Remove the 'base' parameter " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 23/36] hrtimer: Split hrtimer_start_range_ns() Anna-Maria Gleixner
2018-01-16  3:49   ` [tip:timers/core] hrtimer: Factor out __hrtimer_start_range_ns() tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 24/36] hrtimer: Split __hrtimer_get_next_event() Anna-Maria Gleixner
2018-01-16  3:50   ` [tip:timers/core] hrtimer: Factor out __hrtimer_next_event_base() tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 25/36] hrtimer: Use irqsave/irqrestore around __run_hrtimer() Anna-Maria Gleixner
2018-01-16  3:50   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 26/36] hrtimer: Add clock bases and hrtimer mode for soft irq context Anna-Maria Gleixner
2018-01-16  3:51   ` [tip:timers/core] hrtimer: Add clock bases and hrtimer mode for softirq context tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 27/36] hrtimer: Prepare handling of hard and softirq based hrtimers Anna-Maria Gleixner
2018-01-16  3:51   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 28/36] hrtimer: Implement support for " Anna-Maria Gleixner
2018-01-16 10:22   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 29/36] hrtimer: Implement SOFT/HARD clock base selection Anna-Maria Gleixner
2018-01-16 10:22   ` [tip:timers/core] " tip-bot for Anna-Maria Gleixner
2017-12-21 10:41 ` [PATCH v4 30/36] can/bcm: Replace hrtimer_tasklet with softirq based hrtimer Anna-Maria Gleixner
2017-12-21 10:42 ` [PATCH v4 31/36] mac80211_hwsim: Replace hrtimer tasklet with softirq hrtimer Anna-Maria Gleixner
2018-01-04 15:12   ` Johannes Berg
2018-01-04 15:18     ` Thomas Gleixner
2017-12-21 10:42 ` [PATCH v4 32/36] xfrm: " Anna-Maria Gleixner
2017-12-21 10:42 ` [PATCH v4 33/36] softirq: Remove tasklet_hrtimer Anna-Maria Gleixner
2017-12-21 10:42 ` [PATCH v4 34/36] ALSA/dummy: Replace tasklet with softirq hrtimer Anna-Maria Gleixner
2017-12-21 10:42   ` Anna-Maria Gleixner
2018-01-16 10:23   ` [tip:timers/core] " tip-bot for Thomas Gleixner
2017-12-21 10:42 ` [PATCH v4 35/36] usb/gadget/NCM: " Anna-Maria Gleixner
2017-12-21 10:42   ` [v4,35/36] " Anna-Maria Gleixner
2018-01-16 10:23   ` [tip:timers/core] " tip-bot for Thomas Gleixner
2017-12-21 10:42 ` [PATCH v4 36/36] net/mvpp2: " Anna-Maria Gleixner
2018-01-16  1:39 ` [PATCH v4 00/36] hrtimer: Provide softirq context hrtimers Ingo Molnar
2018-01-16  2:03   ` Ingo Molnar

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.