From: Feng Tang <feng.tang@intel.com>
To: "Paul E. McKenney" <paulmck@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>,
<linux-kernel@vger.kernel.org>, <john.stultz@linaro.org>,
<sboyd@kernel.org>, <corbet@lwn.net>, <Mark.Rutland@arm.com>,
<maz@kernel.org>, <kernel-team@meta.com>,
<neeraju@codeaurora.org>, <ak@linux.intel.com>,
<zhengjun.xing@intel.com>, Chris Mason <clm@meta.com>,
John Stultz <jstultz@google.com>,
Waiman Long <longman@redhat.com>
Subject: Re: [PATCH clocksource 1/3] clocksource: Reject bogus watchdog clocksource measurements
Date: Wed, 30 Nov 2022 09:38:00 +0800 [thread overview]
Message-ID: <Y4az+FT5YjpAWjZc@feng-clx> (raw)
In-Reply-To: <20221129192915.GD4001@paulmck-ThinkPad-P17-Gen-1>
On Tue, Nov 29, 2022 at 11:29:15AM -0800, Paul E. McKenney wrote:
[...]
> > > > IIUC, this will make TSC to watchdog HPET every 500 ms. We have got
> > > > report that the 500ms watchdog timer had big impact on some parallel
> > > > workload on big servers, that was another factor for us to seek
> > > > stopping the timer.
> > >
> > > Another approach would be to slow it down. Given the tighter bounds
> > > on skew, it could be done every (say) 10 seconds while allowing
> > > 2 milliseconds skew instead of the current 100 microseconds.
> >
> > Yes, this can reduce the OS noise much. One problem is if we make it
> > a general interface, there is some clocksource whose warp time is
> > less than 10 seconds, like ACPI PM_TIMER (3-4 seconds), and I don't
> > know if other ARCHs have similar cases.
>
> Maybe a simpler approach is for systems with such high sensitivity to
> OS noise to simply disable the clocksource watchdog. ;-)
That's what the reported did, test with and without "tsc=reliable"
parameter :)
And AFAIK, many customers with big server farms hate to add more
cmdline parameters when we suggested so.
> > > > Is this about the concern of possible TSC frequency calibration
> > > > issue, as the 40 ms per second drift between HPET and TSC? With
> > > > b50db7095fe0 backported, we also have another patch to force TSC
> > > > calibration for those platforms which get the TSC freq directly
> > > > from CPUID or MSR and don't have such info in dmesg:
> > > > "tsc: Refined TSC clocksource calibration: 2693.509 MHz"
> > > >
> > > > https://lore.kernel.org/lkml/20220509144110.9242-1-feng.tang@intel.com/
> > > >
> > > > We did met tsc calibration issue due to some firmware issue, and
> > > > this can help to catch it. You can try it if you think it's relevant.
> > >
> > > I am giving this a go, thank you!
> >
> > Thanks for spending time testing it!
>
> And here are the results from setting tsc_force_recalibrate to 1:
>
> $ dmesg | grep -E 'calibrat|clocksource'
> [ 5.272939] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
> [ 16.830644] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 76450417870 ns
> [ 17.938020] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x36a8d32ce31, max_idle_ns: 881590731004 ns
> [ 24.548583] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
> [ 49.762432] clocksource: Switched to clocksource tsc-early
> [ 50.076769] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
> [ 55.615946] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x36a8d32ce31, max_idle_ns: 881590731004 ns
> [ 55.640270] clocksource: Switched to clocksource tsc
> [ 56.694371] tsc: Warning: TSC freq calibrated by CPUID/MSR differs from what is calibrated by HW timer, please check with vendor!!
> [ 56.724550] tsc: Previous calibrated TSC freq: 1896.000 MHz
> [ 56.737646] tsc: TSC freq recalibrated by [HPET]: 1975.000 MHz
Looks like there is really something wrong here. I assume the first
number '1896 MHz' is got from CPUID(0x15)'s math calculation.
I thinks 2 more things could be try:
* add "nohpet" to the cmdline, so the tsc_force_recalibrate should use
ACPI PM_TIMER to do the calibration, say a third-party check.
* If the system don't have auto-adjusted time setting like NTP, I
guess the system time will have obvious drift comparing to a normal
clock or a mobile phone time, as the deviation is about 4%, which
is 2.4 minutes per hour.
> Apologies for the delay, but reconfigurations put the system off the
> net for some time.
No problem at all, it's your holiday time! Thanks for trying this!
Thanks,
Feng
> Thanx, Paul
next prev parent reply other threads:[~2022-11-30 1:41 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-14 23:28 [PATCH clocksource 0/3] Reject bogus watchdog clocksource measurements Paul E. McKenney
2022-11-14 23:28 ` [PATCH clocksource 1/3] clocksource: " Paul E. McKenney
2022-11-17 21:57 ` Thomas Gleixner
2022-11-17 23:09 ` Paul E. McKenney
2022-11-21 0:55 ` Feng Tang
2022-11-21 15:21 ` Paul E. McKenney
2022-11-21 18:14 ` Paul E. McKenney
2022-11-22 15:55 ` Feng Tang
2022-11-22 22:07 ` Paul E. McKenney
2022-11-23 2:36 ` Feng Tang
2022-11-23 21:23 ` Paul E. McKenney
2022-11-28 2:15 ` Feng Tang
2022-11-29 19:29 ` Paul E. McKenney
2022-11-30 1:38 ` Feng Tang [this message]
2022-11-30 4:12 ` Paul E. McKenney
2022-11-30 4:49 ` Feng Tang
2022-11-30 5:16 ` Paul E. McKenney
2022-11-30 5:35 ` Feng Tang
2022-11-30 5:50 ` Paul E. McKenney
2022-11-30 6:00 ` Feng Tang
2022-12-01 17:24 ` Paul E. McKenney
2022-12-02 1:10 ` Feng Tang
2022-12-02 1:44 ` Paul E. McKenney
2022-12-02 2:02 ` Feng Tang
2022-12-02 22:24 ` Paul E. McKenney
2022-12-03 2:51 ` Feng Tang
2022-11-14 23:28 ` [PATCH clocksource 2/3] clocksource: Add comments to classify bogus measurements Paul E. McKenney
2022-11-14 23:28 ` [PATCH clocksource 3/3] clocksource: Exponential backoff for load-induced bogus watchdog reads Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y4az+FT5YjpAWjZc@feng-clx \
--to=feng.tang@intel.com \
--cc=Mark.Rutland@arm.com \
--cc=ak@linux.intel.com \
--cc=clm@meta.com \
--cc=corbet@lwn.net \
--cc=john.stultz@linaro.org \
--cc=jstultz@google.com \
--cc=kernel-team@meta.com \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=maz@kernel.org \
--cc=neeraju@codeaurora.org \
--cc=paulmck@kernel.org \
--cc=sboyd@kernel.org \
--cc=tglx@linutronix.de \
--cc=zhengjun.xing@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).