[PATCH] Revert "clocksource: Make clocksource validation work for all clocksources"

[PATCH] Revert "clocksource: Make clocksource validation work for all clocksources"

[Michael Zhivich]

Revert commit 1f45f1f33c8c ("clocksource: Make clocksource validation work
for all clocksources") to restore correct clocksource_delta() computation
for clocksources that wrap frequently, while retaining the check for tsc
drifting.

Truncating result of clocksource_delta() to 0 causes incorrect behavior for
clocksources that wrap frequently (e.g. acpi_pm which is only 24-bit wide).
In particular, large time deltas (e.g. last = 0x000000, now = 0x800000)
will be incorrectly computed as 0.

If acpi_pm is used as the clocksource watchdog, and machine is under heavy
load, the time period for the watchdog check may be significantly longer
than the requested 0.5 seconds.  If the watchdog check is delayed by 2
seconds (observed behavior), then acpi_pm time delta will be 

	2.5 sec * 3579545 ticks/sec = 8948863 = 0x888c3f

which will be treated as negative and truncated to 0.  This behavior will
cause tsc to be incorrectly declared unstable in clocksource_watchdog(), as
it no longer agrees with acpi_pm.

Signed-off-by: Michael Zhivich <mzhivich@akamai.com>
---
 kernel/time/timekeeping_internal.h | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/kernel/time/timekeeping_internal.h b/kernel/time/timekeeping_internal.h
index bcbb52db2256..21463e20d008 100644
--- a/kernel/time/timekeeping_internal.h
+++ b/kernel/time/timekeeping_internal.h
@@ -18,11 +18,7 @@ static inline u64 clocksource_delta(u64 now, u64 last, u64 mask)
 {
 	u64 ret = (now - last) & mask;
 
-	/*
-	 * Prevent time going backwards by checking the MSB of mask in
-	 * the result. If set, return 0.
-	 */
-	return ret & ~(mask >> 1) ? 0 : ret;
+	return (s64) ret > 0 ? ret : 0;
 }
 #else
 static inline u64 clocksource_delta(u64 now, u64 last, u64 mask)
-- 
2.19.1

Re: [PATCH] Revert "clocksource: Make clocksource validation work for all clocksources"

[John Stultz]

On Mon, Nov 12, 2018 at 10:56 AM, Michael Zhivich <mzhivich@akamai.com> wrote:
> Revert commit 1f45f1f33c8c ("clocksource: Make clocksource validation work
> for all clocksources") to restore correct clocksource_delta() computation
> for clocksources that wrap frequently, while retaining the check for tsc
> drifting.
>
> Truncating result of clocksource_delta() to 0 causes incorrect behavior for
> clocksources that wrap frequently (e.g. acpi_pm which is only 24-bit wide).
> In particular, large time deltas (e.g. last = 0x000000, now = 0x800000)
> will be incorrectly computed as 0.
>
> If acpi_pm is used as the clocksource watchdog, and machine is under heavy
> load, the time period for the watchdog check may be significantly longer
> than the requested 0.5 seconds.  If the watchdog check is delayed by 2
> seconds (observed behavior), then acpi_pm time delta will be
>
>         2.5 sec * 3579545 ticks/sec = 8948863 = 0x888c3f
>
> which will be treated as negative and truncated to 0.  This behavior will
> cause tsc to be incorrectly declared unstable in clocksource_watchdog(), as
> it no longer agrees with acpi_pm.

Thanks for raising this issue and submitting the patch!

Yea, this is a concern particularly with quick wrapping clocksources.
Though I worry that if you're already blocking the watchdog from
running for 2.5 seconds, you're likely to also block the watchdog for
more then 5 seconds, which if I'm remembering would result in the same
problem?  In other words, does this really solve the problem, or does
it just push the bar a little further out?

So, I'm wondering to really fix this, do we need to find some way to
raise the priority of the clocksource watchdog, so it isn't deferred
for quite so long?

thanks
-john

Re: [PATCH] Revert "clocksource: Make clocksource validation work for all clocksources"

[Zhivich, Michael]

On 11/12/18, 5:23 PM, "John Stultz" <john.stultz@linaro.org> wrote:

    On Mon, Nov 12, 2018 at 10:56 AM, Michael Zhivich <mzhivich@akamai.com> wrote:
    > Revert commit 1f45f1f33c8c ("clocksource: Make clocksource validation work
    > for all clocksources") to restore correct clocksource_delta() computation
    > for clocksources that wrap frequently, while retaining the check for tsc
    > drifting.
    >
    > Truncating result of clocksource_delta() to 0 causes incorrect behavior for
    > clocksources that wrap frequently (e.g. acpi_pm which is only 24-bit wide).
    > In particular, large time deltas (e.g. last = 0x000000, now = 0x800000)
    > will be incorrectly computed as 0.
    >
    > If acpi_pm is used as the clocksource watchdog, and machine is under heavy
    > load, the time period for the watchdog check may be significantly longer
    > than the requested 0.5 seconds.  If the watchdog check is delayed by 2
    > seconds (observed behavior), then acpi_pm time delta will be
    >
    >         2.5 sec * 3579545 ticks/sec = 8948863 = 0x888c3f
    >
    > which will be treated as negative and truncated to 0.  This behavior will
    > cause tsc to be incorrectly declared unstable in clocksource_watchdog(), as
    > it no longer agrees with acpi_pm.
    
    Thanks for raising this issue and submitting the patch!
    
    Yea, this is a concern particularly with quick wrapping clocksources.
    Though I worry that if you're already blocking the watchdog from
    running for 2.5 seconds, you're likely to also block the watchdog for
    more then 5 seconds, which if I'm remembering would result in the same
    problem?  In other words, does this really solve the problem, or does
    it just push the bar a little further out?
    
    So, I'm wondering to really fix this, do we need to find some way to
    raise the priority of the clocksource watchdog, so it isn't deferred
    for quite so long?
    
    thanks
    -john
    

Thanks for the quick response.  

In principle, I agree that a proper solution would have to ensure that watchdog timer is not blocked for too long.  My understanding is that watchdog work is triggered via TIMER softirq and will get pushed to ksoftirqd when the system is busy.  In particular, it appears that do_softirq() and invoke_softirq() both check ksoftirqd_running() before deciding to actually do work.

One interesting bit is that ksoftirq_running() will disregard an active ksoftirqd thread and return "false" when HI_SOFTIRQ or TASKLET_SOFTIRQ is set (resulting in at least a single pass over pending softirqs).  I think it would make sense to add TIMER_SOFTIRQ to this exception list as well.

In fact, looking back at the commit log, I'm finding similar thoughts on the subject:

    commit 3c53776e29f81719efcf8f7a6e30cdf753bee94d
    Author: Linus Torvalds <torvalds@linux-foundation.org> 
    Date:   Mon Jan 8 11:51:04 2018 -0800

    Mark HI and TASKLET softirq synchronous
    ...
    We should probably also consider the timer softirqs to be synchronous
    and not be delayed to ksoftirqd (since they were the issue with the
    earlier watchdog problems), but that should be done as a separate patch.
    This does only the tasklet cases.

If that makes sense, I'm happy to write up a patch.

Thanks,
~ Michael