linux-mips.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Thomas Gleixner <tglx@linutronix.de>
To: Vincenzo Frascino <vincenzo.frascino@arm.com>
Cc: linux-arch@vger.kernel.org,
	LAK <linux-arm-kernel@lists.infradead.org>,
	LKML <linux-kernel@vger.kernel.org>,
	linux-mips@vger.kernel.org, linux-kselftest@vger.kernel.org,
	catalin.marinas@arm.com, Will Deacon <will.deacon@arm.com>,
	Arnd Bergmann <arnd@arndb.de>,
	linux@armlinux.org.uk, Ralf Baechle <ralf@linux-mips.org>,
	paul.burton@mips.com, Daniel Lezcano <daniel.lezcano@linaro.org>,
	salyzyn@android.com, pcc@google.com, shuah@kernel.org,
	0x7f454c46@gmail.com, linux@rasmusvillemoes.dk,
	huw@codeweavers.com, sthotton@marvell.com,
	andre.przywara@arm.com, Andy Lutomirski <luto@kernel.org>
Subject: lib/vdso: Make delta calculation work correctly
Date: Wed, 26 Jun 2019 12:02:00 +0200 (CEST)	[thread overview]
Message-ID: <alpine.DEB.2.21.1906261159230.32342@nanos.tec.linutronix.de> (raw)
In-Reply-To: <4ff43de4-fe51-eed4-a155-31a05edf2f11@arm.com>

The x86 vdso implementation on which the generic vdso library is based on
has subtle (unfortunately undocumented) twists:

 1) The code assumes that the clocksource mask is U64_MAX which means that
    no bits are masked. Which is true for any valid x86 VDSO clocksource.
    Stupidly it still did the mask operation for no reason and at the wrong
    place right after reading the clocksource.

 2) It contains a sanity check to catch the case where slightly
    unsynchronized TSC values can be overserved which would cause the delta
    calculation to make a huge jump. It therefore checks whether the
    current TSC value is larger than the value on which the current
    conversion is based on. If it's not larger the base value is used to
    prevent time jumps.

#1 Is not only stupid for the X86 case because it does the masking for no
reason it is also completely wrong for clocksources with a smaller mask
which can legitimately wrap around during a conversion period. The core
timekeeping code does it correct by applying the mask after the delta
calculation:

	(now - base) & mask

#2 is equally broken for clocksources which have smaller masks and can wrap
around during a conversion period because there the now > base check is
just wrong and causes stale time stamps and time going backwards issues.

Unbreak it by:

  1) Removing the mask operation from the clocksource read which makes the
     fallback detection work for all clocksources

  2) Replacing the conditional delta calculation with a overrideable inline
     function.

#2 could reuse clocksource_delta() from the timekeeping code but that
results in a significant performance hit for the x86 VSDO. The timekeeping
core code must have the non optimized version as it has to operate
correctly with clocksources which have smaller masks as well to handle the
case where TSC is discarded as timekeeper clocksource and replaced by HPET
or pmtimer. For the VDSO there is no replacement clocksource. If TSC is
unusable the syscall is enforced which does the right thing.

To accomodate to the needs of various architectures provide an overrideable
inline function which defaults to the regular delta calculation with
masking:

	(now - base) & mask

Override it for x86 with the non-masking and checking version.

This unbreaks the ARM64 syscall fallback operation, allows to use
clocksources with arbitrary width and preserves the performance
optimization for x86.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/vdso/gettimeofday.h |   27 +++++++++++++++++++++++++++
 lib/vdso/gettimeofday.c                  |   19 +++++++++++++++----
 2 files changed, 42 insertions(+), 4 deletions(-)

--- a/arch/x86/include/asm/vdso/gettimeofday.h
+++ b/arch/x86/include/asm/vdso/gettimeofday.h
@@ -229,6 +229,33 @@ static __always_inline const struct vdso
 	return __vdso_data;
 }
 
+/*
+ * x86 specific delta calculation.
+ *
+ * The regular implementation assumes that clocksource reads are globally
+ * monotonic. The TSC can be slightly off across sockets which can cause
+ * the regular delta calculation (@cycles - @last) to return a huge time
+ * jump.
+ *
+ * Therefore it needs to be verified that @cycles are greater than
+ * @last. If not then use @last, which is the base time of the current
+ * conversion period.
+ *
+ * This variant also removes the masking of the subtraction because the
+ * clocksource mask of all VDSO capable clocksources on x86 is U64_MAX
+ * which would result in a pointless operation. The compiler cannot
+ * optimize it away as the mask comes from the vdso data and is not compile
+ * time constant.
+ */
+static __always_inline
+u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult)
+{
+	if (cycles > last)
+		return (cycles - last) * mult;
+	return 0;
+}
+#define vdso_calc_delta vdso_calc_delta
+
 #endif /* !__ASSEMBLY__ */
 
 #endif /* __ASM_VDSO_GETTIMEOFDAY_H */
--- a/lib/vdso/gettimeofday.c
+++ b/lib/vdso/gettimeofday.c
@@ -26,6 +26,18 @@
 #include <asm/vdso/gettimeofday.h>
 #endif /* ENABLE_COMPAT_VDSO */
 
+#ifndef vdso_calc_delta
+/*
+ * Default implementation which works for all sane clocksources. That
+ * obviously excludes x86/TSC.
+ */
+static __always_inline
+u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult)
+{
+	return ((cyles - last) & mask) * mult;
+}
+#endif
+
 static int do_hres(const struct vdso_data *vd, clockid_t clk,
 		   struct __kernel_timespec *ts)
 {
@@ -35,14 +47,13 @@ static int do_hres(const struct vdso_dat
 
 	do {
 		seq = vdso_read_begin(vd);
-		cycles = __arch_get_hw_counter(vd->clock_mode) &
-			vd->mask;
+		cycles = __arch_get_hw_counter(vd->clock_mode);
 		ns = vdso_ts->nsec;
 		last = vd->cycle_last;
 		if (unlikely((s64)cycles < 0))
 			return clock_gettime_fallback(clk, ts);
-		if (cycles > last)
-			ns += (cycles - last) * vd->mult;
+
+		ns += vdso_calc_delta(cycles, last, vd->mask, vd->mult);
 		ns >>= vd->shift;
 		sec = vdso_ts->sec;
 	} while (unlikely(vdso_read_retry(vd, seq)));

  reply	other threads:[~2019-06-26 10:02 UTC|newest]

Thread overview: 108+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-21  9:52 [PATCH v7 00/25] Unify vDSOs across more architectures Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 01/25] kernel: Standardize vdso_datapage Vincenzo Frascino
2019-06-24 13:56   ` Catalin Marinas
2019-06-21  9:52 ` [PATCH v7 02/25] kernel: Define gettimeofday vdso common code Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 03/25] kernel: Unify update_vsyscall implementation Vincenzo Frascino
2019-06-21 10:49   ` Huw Davies
2019-06-21  9:52 ` [PATCH v7 04/25] arm64: Substitute gettimeofday with C implementation Vincenzo Frascino
2019-06-24 13:36   ` Will Deacon
2019-06-24 13:59     ` Vincenzo Frascino
2019-06-25 16:18     ` [PATCH 1/3] lib/vdso: Delay mask application in do_hres() Vincenzo Frascino
2019-06-25 16:18       ` [PATCH 2/3] arm64: Fix __arch_get_hw_counter() implementation Vincenzo Frascino
2019-06-25 16:18       ` [PATCH 3/3] arm64: compat: " Vincenzo Frascino
2019-06-25 17:02       ` [PATCH 1/3] lib/vdso: Delay mask application in do_hres() Thomas Gleixner
2019-06-25 18:27         ` Thomas Gleixner
2019-06-25 20:15           ` Andy Lutomirski
2019-06-25 22:24             ` Thomas Gleixner
2019-06-26  6:38         ` Thomas Gleixner
2019-06-26  9:25           ` Vincenzo Frascino
2019-06-26 10:02             ` Thomas Gleixner [this message]
2019-06-26 11:08               ` lib/vdso: Make delta calculation work correctly Vincenzo Frascino
2019-06-24 13:58   ` [PATCH v7 04/25] arm64: Substitute gettimeofday with C implementation Catalin Marinas
2019-06-25 15:33   ` Dave Martin
2019-06-26 13:27     ` Vincenzo Frascino
2019-06-26 16:14       ` Dave Martin
2019-06-26 19:01         ` Vincenzo Frascino
2019-06-27 10:01           ` Dave Martin
2019-06-27 10:57             ` Vincenzo Frascino
2019-06-27 11:27               ` Dave Martin
2019-06-27 11:59                 ` Vincenzo Frascino
2019-06-27 14:38                   ` Dave Martin
2019-06-27 15:34                     ` Vincenzo Frascino
2019-06-25 17:43   ` [PATCH] arm64: vdso: Fix compilation with clang < 8 Vincenzo Frascino
2019-06-26 11:36   ` [PATCH v2] arm64: vdso: Fix compilation with clang older then 8 Vincenzo Frascino
     [not found]   ` <CGME20190628130921eucas1p239935b0771032c331911eacc1a69dd2e@eucas1p2.samsung.com>
2019-06-28 13:09     ` [PATCH v7 04/25] arm64: Substitute gettimeofday with C implementation Marek Szyprowski
2019-06-28 14:32       ` Vincenzo Frascino
2019-06-28 16:50         ` Sylwester Nawrocki
2019-06-29  6:58           ` Vincenzo Frascino
2019-07-08 12:57             ` Sylwester Nawrocki
2019-07-08 13:09               ` Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 05/25] arm64: Build vDSO with -ffixed-x18 Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 06/25] arm64: compat: Add missing syscall numbers Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 07/25] arm64: compat: Expose signal related structures Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 08/25] arm64: compat: Generate asm offsets for signals Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 09/25] lib: vdso: Add compat support Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 10/25] arm64: compat: Add vDSO Vincenzo Frascino
2019-06-24 14:00   ` Catalin Marinas
2019-07-10  4:02   ` John Stultz
2019-07-10  6:12     ` Thomas Gleixner
2019-07-10  9:48       ` Vincenzo Frascino
2019-07-10  8:27     ` Will Deacon
2019-07-10  8:58       ` Thomas Gleixner
2019-07-10  9:12         ` Will Deacon
2019-07-10  9:47     ` Vincenzo Frascino
2019-07-10 13:41       ` Vincenzo Frascino
2019-07-10 13:04   ` [PATCH] arm64: vdso: Fix ABI regression in compat vdso Vincenzo Frascino
2019-07-10 13:25     ` Will Deacon
2019-07-10 13:42       ` Vincenzo Frascino
2019-07-10 14:01   ` [PATCH v2] " Vincenzo Frascino
2019-07-10 15:44     ` John Stultz
2019-07-10 15:53       ` Vincenzo Frascino
2019-07-11  9:45     ` Will Deacon
2019-07-11 10:34       ` Thomas Gleixner
2019-07-11 11:32         ` Will Deacon
2019-06-21  9:52 ` [PATCH v7 11/25] arm64: Refactor vDSO code Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 12/25] arm64: compat: vDSO setup for compat layer Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 13/25] arm64: elf: vDSO code page discovery Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 14/25] arm64: compat: Get sigreturn trampolines from vDSO Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 15/25] arm64: Add vDSO compat support Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 16/25] arm: Add support for generic vDSO Vincenzo Frascino
2019-12-04 13:51   ` [PATCH v7 16/25] arm: Add support for generic vDSO (causing crash) Guenter Roeck
2019-12-04 13:58     ` Vincenzo Frascino
2019-12-04 16:16       ` Guenter Roeck
2019-12-04 17:15         ` Vincenzo Frascino
2019-12-04 19:39           ` Guenter Roeck
2019-12-05  9:42           ` Philippe Mathieu-Daudé
2019-12-05 10:00             ` Vincenzo Frascino
2019-12-05 11:02               ` Arnd Bergmann
2019-12-05 14:56                 ` Philippe Mathieu-Daudé
2019-06-21  9:52 ` [PATCH v7 17/25] arm: Add clock_getres entry point Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 18/25] arm: Add clock_gettime64 " Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 19/25] mips: Add support for generic vDSO Vincenzo Frascino
2019-07-26  5:15   ` Paul Burton
2019-07-26 16:29     ` [PATCH 0/2] mips: vdso: Fix Makefile Vincenzo Frascino
2019-07-26 16:29       ` [PATCH 1/2] mips: vdso: Fix source path Vincenzo Frascino
2019-07-26 16:29       ` [PATCH 2/2] mips: vdso: Fix flip/flop vdso building bug Vincenzo Frascino
2019-07-28 22:20       ` [PATCH 0/2] mips: vdso: Fix Makefile Paul Burton
2019-06-21  9:52 ` [PATCH v7 20/25] mips: Add clock_getres entry point Vincenzo Frascino
2019-07-26  5:15   ` Paul Burton
2019-06-21  9:52 ` [PATCH v7 21/25] mips: Add clock_gettime64 " Vincenzo Frascino
2019-07-26  5:15   ` Paul Burton
2019-06-21  9:52 ` [PATCH v7 22/25] x86: Add support for generic vDSO Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 23/25] x86: Add clock_getres entry point Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 24/25] x86: Add clock_gettime64 " Vincenzo Frascino
2019-06-21  9:52 ` [PATCH v7 25/25] kselftest: Extend vDSO selftest Vincenzo Frascino
2019-06-24  0:34 ` [PATCH v7 00/25] Unify vDSOs across more architectures Thomas Gleixner
2019-06-24  1:15   ` Andy Lutomirski
2019-06-24  7:42     ` Thomas Gleixner
2019-06-24 13:21   ` Vincenzo Frascino
2019-06-24 14:18   ` Thomas Gleixner
2019-06-24 14:23     ` Russell King - ARM Linux admin
2019-06-24 14:49       ` Catalin Marinas
2019-06-24 16:20         ` Vincenzo Frascino
2019-10-25 11:42         ` Geert Uytterhoeven
2019-06-24 18:41   ` Paul Burton
2019-06-24 23:16     ` Vincenzo Frascino
2019-06-25 17:11       ` Paul Burton
2019-06-25 17:17         ` Vincenzo Frascino
2019-06-24 12:50 ` Andre Przywara

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=alpine.DEB.2.21.1906261159230.32342@nanos.tec.linutronix.de \
    --to=tglx@linutronix.de \
    --cc=0x7f454c46@gmail.com \
    --cc=andre.przywara@arm.com \
    --cc=arnd@arndb.de \
    --cc=catalin.marinas@arm.com \
    --cc=daniel.lezcano@linaro.org \
    --cc=huw@codeweavers.com \
    --cc=linux-arch@vger.kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-mips@vger.kernel.org \
    --cc=linux@armlinux.org.uk \
    --cc=linux@rasmusvillemoes.dk \
    --cc=luto@kernel.org \
    --cc=paul.burton@mips.com \
    --cc=pcc@google.com \
    --cc=ralf@linux-mips.org \
    --cc=salyzyn@android.com \
    --cc=shuah@kernel.org \
    --cc=sthotton@marvell.com \
    --cc=vincenzo.frascino@arm.com \
    --cc=will.deacon@arm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).