[PATCH v2] timer: document TIMER_PINNED

[PATCH v2] timer: document TIMER_PINNED

[Peter Xu]

The flag hints the user that the pinned timers will always be run on a
static CPU (because that should be what "pinned" means...) but that's
not the truth, at least with current implementation.

For example, currently if we setup a pinned timer but later on we call
mod_timer() upon the pinned timer, the mod_timer() will still try to
run the timer on the current processor and migrate the timer if
necessary.  In other words, the suggested way to arm a pinned timer
should be add_timer_on() always.  mod_timer() can be used in this case
only if current processor is the one that we want to pin the timer on.

Document it a bit with the definition of TIMER_PINNED so that all
future users will use it correctly.

CC: Thomas Gleixner <tglx@linutronix.de>
CC: linux-kernel@vger.kernel.org
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/linux/timer.h | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/include/linux/timer.h b/include/linux/timer.h
index 7b066fd38248..1b96cc623a12 100644
--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -36,19 +36,31 @@ struct timer_list {
 #define __TIMER_LOCKDEP_MAP_INITIALIZER(_kn)
 #endif
 
-/*
- * A deferrable timer will work normally when the system is busy, but
- * will not cause a CPU to come out of idle just to service it; instead,
- * the timer will be serviced when the CPU eventually wakes up with a
- * subsequent non-deferrable timer.
+/**
+ * @TIMER_DEFERRABLE: A deferrable timer will work normally when the
+ * system is busy, but will not cause a CPU to come out of idle just
+ * to service it; instead, the timer will be serviced when the CPU
+ * eventually wakes up with a subsequent non-deferrable timer.
  *
- * An irqsafe timer is executed with IRQ disabled and it's safe to wait for
- * the completion of the running instance from IRQ handlers, for example,
- * by calling del_timer_sync().
+ * @TIMER_IRQSAFE: An irqsafe timer is executed with IRQ disabled and
+ * it's safe to wait for the completion of the running instance from
+ * IRQ handlers, for example, by calling del_timer_sync().
  *
  * Note: The irq disabled callback execution is a special case for
  * workqueue locking issues. It's not meant for executing random crap
  * with interrupts disabled. Abuse is monitored!
+ *
+ * @TIMER_PINNED: A pinned timer will not be affected by any timer
+ * placement heuristics (like, NOHZ) and will always be run on the CPU
+ * when the timer was enqueued.
+ *
+ * Note: Because enqueuing of timers can actually migrate the timer
+ * from one CPU to another, pinned timers are not guaranteed to stay
+ * on the initialy selected CPU.  They move to the CPU on which the
+ * enqueue function is invoked via mod_timer() or add_timer().  If the
+ * timer should be placed on a particular CPU, then add_timer_on() has
+ * to be used.  It is also suggested that the user should always use
+ * add_timer_on() explicitly for pinned timers.
  */
 #define TIMER_CPUMASK		0x0003FFFF
 #define TIMER_MIGRATING		0x00040000
-- 
2.21.0

Re: [PATCH v2] timer: document TIMER_PINNED

[Thomas Gleixner]

On Thu, 27 Jun 2019, Peter Xu wrote:
> + * @TIMER_PINNED: A pinned timer will not be affected by any timer
> + * placement heuristics (like, NOHZ) and will always be run on the CPU
> + * when the timer was enqueued.

s/when/on which/

> + *
> + * Note: Because enqueuing of timers can actually migrate the timer
> + * from one CPU to another, pinned timers are not guaranteed to stay
> + * on the initialy selected CPU.  They move to the CPU on which the
> + * enqueue function is invoked via mod_timer() or add_timer().  If the
> + * timer should be placed on a particular CPU, then add_timer_on() has
> + * to be used.  It is also suggested that the user should always use
> + * add_timer_on() explicitly for pinned timers.

That last sentence is not correct. add_timer_on() has limitations over
mod_timer(). As pinned prevents the timer from being queued on a remote CPU
mod timer is perfectly fine for many cases.

add_timer_on() is really about queueing a timer on a dedicated CPU, which
is often enough a remote CPU.

Thanks,

	tglx

Re: [PATCH v2] timer: document TIMER_PINNED

[Peter Xu]

On Thu, Jun 27, 2019 at 11:10:08PM +0200, Thomas Gleixner wrote:
> On Thu, 27 Jun 2019, Peter Xu wrote:
> > + * @TIMER_PINNED: A pinned timer will not be affected by any timer
> > + * placement heuristics (like, NOHZ) and will always be run on the CPU
> > + * when the timer was enqueued.
> 
> s/when/on which/

Fixed.

> 
> > + *
> > + * Note: Because enqueuing of timers can actually migrate the timer
> > + * from one CPU to another, pinned timers are not guaranteed to stay
> > + * on the initialy selected CPU.  They move to the CPU on which the
> > + * enqueue function is invoked via mod_timer() or add_timer().  If the
> > + * timer should be placed on a particular CPU, then add_timer_on() has
> > + * to be used.  It is also suggested that the user should always use
> > + * add_timer_on() explicitly for pinned timers.
> 
> That last sentence is not correct. add_timer_on() has limitations over
> mod_timer(). As pinned prevents the timer from being queued on a remote CPU
> mod timer is perfectly fine for many cases.
> 
> add_timer_on() is really about queueing a timer on a dedicated CPU, which
> is often enough a remote CPU.

Frankly speaking I still think add_timer_on() is preferred here
because mod_timer() users will really need to be careful to make sure
they'll pin the timers correctly all the time, and I assume that's why
we've tried to find all the TIMER_PINNED users and tried to make sure
there's nothing wrong on using them during previous discussion (and
more than half of them do use add_timer_on() which seems to be good).
In all cases, I'll take your suggestion to drop the last sentence.

Thanks for reviewing this document patch.  I'll repost.

-- 
Peter Xu