From: Thomas Gleixner <tglx@linutronix.de>
To: LKML <linux-kernel@vger.kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@kernel.org>,
Preeti U Murthy <preeti@linux.vnet.ibm.com>,
Viresh Kumar <viresh.kumar@linaro.org>,
Marcelo Tosatti <mtosatt@redhat.com>,
Frederic Weisbecker <fweisbec@gmail.com>
Subject: [patch 00/39] hrtimer/tick: Optimizations, cleanups and solutions for various issues
Date: Tue, 14 Apr 2015 21:08:23 -0000 [thread overview]
Message-ID: <20150414203303.702062272@linutronix.de> (raw)
When I returned from my break I got offended by a pile of patches
which kill the patient with the cure.
The issues at hand:
- NOHZ: Get rid of the softirq invocation
- hrtimer: Use the active_bases field in order to avoid evaluating
inactive bases
- hrtimer: Cache footprint issues
Aside of that Peter and I were discussing for a long time to get rid
of the hrtimer softirq.
After staring at all of it for quite some time it occured to me that
all issues are related in one way or the other. So I sat down and
reworked the code in various ways:
- Reduce the data size, so the hrtimer clock bases can be made
cache line aligned.
- Consolidate everything on the high resolution timer
implementation and get rid of dubious optimizations for the non
highres case which bloat code and data.
- Implement the active_bases mechanism proper and avoid touching
inactive hrtimer clock bases. This includes a conditional update
mechanism for the hrtimer clock offsets to update them only when
they changed, which they do seldom enough instead of polluting 4
cache lines in every tick/hrtimer interrupt.
- Get rid of the softirq deferment and simply enforce a hrtimer
interrupt when the timer was already expired. This allows to
remove the ugly __hrtimer_start_range_ns() interface and to
cleanup the usage sites (sched/perf). As a consequence this also
gets rid of the forward loops in the tick nohz code.
- Analogous to the hrtimer enforcement, force a tick interrupt for
NOHZ non highres systems when the forwarding code tries to fire
an expired tick. This allows to get rid of the softirq invocation
in the NOHZ code.
- A cleanup of the code which evaluates the next timer event: Use
nsec based calculations instead of the jiffy magic. That makes it
actually readable by some definition of readable.
- While doing the above I had to audit quite some usage sites of
various hrtimer interfaces, which revealed some entertaining
bugs. The fixes have been posted in a seperate series
already. Some other bogosities have been removed as part of this
series.
The total change size of this overhaul is:
37 files changed, 515 insertions(+), 794 deletions(-)
The resulting text size of hrtimers.o shrinks in the range of 8-10%
depending on the architecture.
The cache foot print of the hrtimer per cpu data shrinks as well.
x8684 i386 ARM ARM64 power64
Before: 328 248 280 328 328 Bytes
6 4 5 6 6 cache lines (64byte)
After: 320 192 192 320 320 Bytes
5 3 3 5 5 cache lines (64byte)
Note, that the new code avoids to touch the inactive clock bases which
are now cache line aligned and therefor reduces the cache foot print
in normal usage scenarios significantly.
I did some perf measurements on an isolated core running
- hrtimer centric workloads
- idle scenarios with periodic wakeups of various length
The patches reduce the number of instructions executed during the test
runs between 2.5 and 6% depending on the scenario and the cache misses
between 3 and 8%.
For your convenience this series is also available at:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/wip
Note: The branch is temporary and not meant to base other work on it.
Thanks,
tglx
---
arch/x86/kernel/cpu/perf_event_intel_rapl.c | 5
arch/x86/kernel/cpu/perf_event_intel_uncore.c | 5
drivers/power/reset/ltc2952-poweroff.c | 18
drivers/staging/ozwpan/ozpd.c | 8
include/linux/alarmtimer.h | 4
include/linux/hrtimer.h | 101 ++---
include/linux/interrupt.h | 7
include/linux/rcupdate.h | 6
include/linux/rcutree.h | 2
include/linux/timekeeper_internal.h | 2
include/linux/timer.h | 7
include/linux/timerqueue.h | 8
include/trace/events/irq.h | 1
kernel/events/core.c | 9
kernel/futex.c | 5
kernel/locking/rtmutex.c | 5
kernel/rcu/tree_plugin.h | 14
kernel/sched/core.c | 28 -
kernel/sched/deadline.c | 12
kernel/sched/fair.c | 2
kernel/softirq.c | 2
kernel/time/alarmtimer.c | 17
kernel/time/hrtimer.c | 525 +++++++++-----------------
kernel/time/posix-timers.c | 17
kernel/time/tick-broadcast-hrtimer.c | 8
kernel/time/tick-internal.h | 2
kernel/time/tick-sched.c | 288 +++++---------
kernel/time/tick-sched.h | 2
kernel/time/timekeeping.c | 55 --
kernel/time/timekeeping.h | 10
kernel/time/timer.c | 79 +--
kernel/time/timer_list.c | 14
lib/timerqueue.c | 10
net/core/pktgen.c | 2
net/sched/sch_api.c | 5
sound/core/hrtimer.c | 9
sound/drivers/pcsp/pcsp.c | 15
37 files changed, 515 insertions(+), 794 deletions(-)
next reply other threads:[~2015-04-14 21:08 UTC|newest]
Thread overview: 123+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-04-14 21:08 Thomas Gleixner [this message]
2015-04-14 21:08 ` [patch 01/39] hrtimer: Update active_bases before calling hrtimer_force_reprogram() Thomas Gleixner
2015-04-14 21:08 ` [patch 02/39] hrtimer: Get rid of the resolution field in hrtimer_clock_base Thomas Gleixner
2015-04-15 6:29 ` Frans Klaver
2015-04-15 6:32 ` Frans Klaver
2015-04-20 8:34 ` Preeti U Murthy
2015-04-22 19:05 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 03/39] net: sched: Use hrtimer_resolution instead of hrtimer_get_res() Thomas Gleixner
2015-04-16 16:04 ` David Miller
2015-04-22 19:05 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 04/39] sound: " Thomas Gleixner
2015-04-16 8:07 ` Takashi Iwai
2015-04-16 9:08 ` Thomas Gleixner
2015-04-22 19:05 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 05/39] hrtimer: Get rid " Thomas Gleixner
2015-04-22 19:05 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 06/39] hrtimer: Make the statistics fields smaller Thomas Gleixner
2015-04-22 19:06 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 07/39] hrtimer: Get rid of softirq time Thomas Gleixner
2015-04-22 19:06 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 08/39] hrtimer: Make offset update smarter Thomas Gleixner
2015-04-20 9:30 ` Preeti U Murthy
2015-04-22 19:06 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 09/39] hrtimer: Use a bits for various boolean indicators Thomas Gleixner
2015-04-22 19:07 ` [tip:timers/core] hrtimer: Use " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 10/39] hrtimer: Use cpu_base->active_base for hotpath iterators Thomas Gleixner
2015-04-20 11:16 ` Preeti U Murthy
2015-04-21 11:53 ` Thomas Gleixner
2015-04-22 3:13 ` Preeti U Murthy
2015-04-22 19:07 ` [tip:timers/core] hrtimer: Use cpu_base-> active_base " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 11/39] hrtimer: Cache line align the hrtimer cpu base Thomas Gleixner
2015-04-22 19:07 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 12/39] hrtimer: Align the hrtimer clock bases as well Thomas Gleixner
2015-04-22 19:07 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 13/39] timerqueue: Let timerqueue_add/del return information Thomas Gleixner
2015-04-22 19:08 ` [tip:timers/core] timerqueue: Let timerqueue_add/ del " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 14/39] hrtimer: Make use of timerqueue_add/del return values Thomas Gleixner
2015-04-22 19:08 ` [tip:timers/core] hrtimer: Make use of timerqueue_add/ del " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 15/39] hrtimer: Keep pointer to first timer and simplify __remove_hrtimer() Thomas Gleixner
2015-04-22 19:08 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 16/39] hrtimer: Get rid of hrtimer softirq Thomas Gleixner
2015-04-22 19:09 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 17/39] tick: sched: Remove hrtimer_active() checks Thomas Gleixner
2015-04-16 13:37 ` Frederic Weisbecker
2015-04-22 19:09 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 18/39] tick: sched: Force tick interrupt and get rid of softirq magic Thomas Gleixner
2015-04-22 14:22 ` Frederic Weisbecker
2015-04-22 14:32 ` Thomas Gleixner
2015-04-23 11:47 ` Frederic Weisbecker
2015-04-23 13:07 ` Thomas Gleixner
2015-04-23 16:14 ` Frederic Weisbecker
2015-04-22 19:09 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 19/39] tick: sched: Restructure code Thomas Gleixner
2015-04-22 19:09 ` [tip:timers/core] tick: Sched: " tip-bot for Thomas Gleixner
2015-04-14 21:08 ` [patch 20/39] tick: nohz: Rework next timer evaluation Thomas Gleixner
2015-04-16 16:42 ` Paul E. McKenney
2015-04-21 12:04 ` Thomas Gleixner
2015-04-22 19:10 ` [tip:timers/core] tick: Nohz: " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 21/39] x86: perf: Use hrtimer_start() Thomas Gleixner
2015-04-22 19:10 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 22/39] x86: perf: uncore: " Thomas Gleixner
2015-04-22 19:10 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 23/39] perf: core: " Thomas Gleixner
2015-04-22 19:11 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 24/39] sched: core: Use hrtimer_start[_expires]() Thomas Gleixner
2015-04-22 19:11 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 25/39] sched: deadline: Use hrtimer_start() Thomas Gleixner
2015-04-22 19:11 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 26/39] hrtimer: Get rid of __hrtimer_start_range_ns() Thomas Gleixner
2015-04-22 19:11 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 27/39] hrtimer: Make hrtimer_start() a inline wrapper Thomas Gleixner
2015-04-22 19:12 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 28/39] hrtimer: Remove bogus hrtimer_active() check Thomas Gleixner
2015-04-22 19:12 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 29/39] hrtimer: Rmove " Thomas Gleixner
2015-04-22 19:12 ` [tip:timers/core] futex: Remove " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 30/39] rtmutex: " Thomas Gleixner
2015-04-22 19:13 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 31/39] net: core: pktgen: " Thomas Gleixner
2015-04-16 16:04 ` David Miller
2015-04-22 19:13 ` [tip:timers/core] net: core: pktgen: Remove bogus hrtimer_active( ) check tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 32/39] alarmtimer: Get rid of unused return value Thomas Gleixner
2015-04-22 19:13 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 34/39] tick: broadcast-hrtimer: Remove overly clever return value abuse Thomas Gleixner
2015-04-17 10:33 ` Preeti U Murthy
2015-04-22 19:13 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 35/39] hrtimer: Remove hrtimer_start() return value Thomas Gleixner
2015-04-22 19:14 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 36/39] hrtimer: Avoid locking in hrtimer_cancel() if timer not active Thomas Gleixner
2015-04-22 19:14 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 37/39] staging: ozwpan: Remove hrtimer_active() check Thomas Gleixner
2015-04-14 21:09 ` [patch 38/39] timer: Remove pointless return value of do_usleep_range() Thomas Gleixner
2015-04-22 19:14 ` [tip:timers/core] " tip-bot for Thomas Gleixner
2015-04-14 21:09 ` [patch 39/39] timer: Put usleep_range into the __sched section Thomas Gleixner
2015-04-22 19:14 ` [tip:timers/core] " tip-bot for Thomas Gleixner
[not found] ` <20150414203503.322172417@linutronix.de>
2015-04-14 21:38 ` [patch 33/39] power: reset: ltc2952: Remove bogus hrtimer_start() return value checks Frans Klaver
2015-04-30 15:49 ` Sebastian Reichel
-- strict thread matches above, loose matches on Subject: below --
2015-04-07 2:10 [PATCH V2 0/2] hrtimer: Iterate only over active clock-bases Viresh Kumar
2015-04-07 2:10 ` [PATCH V2 1/2] hrtimer: update '->active_bases' before calling hrtimer_force_reprogram() Viresh Kumar
2015-04-22 19:04 ` [tip:timers/core] hrtimer: Update active_bases " tip-bot for Viresh Kumar
2015-04-07 2:10 ` [PATCH V2 2/2] hrtimer: Iterate only over active clock-bases Viresh Kumar
2015-04-08 12:10 ` Peter Zijlstra
2015-04-08 20:11 ` Thomas Gleixner
2015-04-09 2:42 ` Viresh Kumar
2015-04-09 6:28 ` [PATCH] hrtimer: Replace cpu_base->active_bases with a direct check of the active list Ingo Molnar
2015-04-09 6:38 ` Ingo Molnar
2015-04-09 6:39 ` [PATCH] hrtimer: Only iterate over active bases in migrate_hrtimers() Ingo Molnar
2015-04-09 6:53 ` [PATCH] hrtimer: Replace timerqueue_getnext() uses with direct access to 'active.next' Ingo Molnar
2015-04-09 7:10 ` [PATCH] hrtimers: Use consistent variable names for timerqueue_node iterations Ingo Molnar
2015-04-09 6:57 ` [PATCH] hrtimer: Replace cpu_base->active_bases with a direct check of the active list Peter Zijlstra
2015-04-09 7:09 ` Ingo Molnar
2015-04-09 7:20 ` Ingo Molnar
2015-04-09 8:58 ` Thomas Gleixner
2015-04-09 8:58 ` Peter Zijlstra
2015-04-09 9:18 ` Thomas Gleixner
2015-04-09 9:31 ` Peter Zijlstra
2015-04-09 9:56 ` Thomas Gleixner
2015-04-13 5:53 ` Preeti U Murthy
2015-04-13 7:53 ` Thomas Gleixner
2015-04-09 8:03 ` Peter Zijlstra
2015-04-09 8:10 ` Ingo Molnar
2015-04-09 8:53 ` Thomas Gleixner
2015-04-09 9:18 ` Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150414203303.702062272@linutronix.de \
--to=tglx@linutronix.de \
--cc=fweisbec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=mtosatt@redhat.com \
--cc=peterz@infradead.org \
--cc=preeti@linux.vnet.ibm.com \
--cc=viresh.kumar@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).