From: Zhenzhong Duan <zhenzhong.duan@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: Zhenzhong Duan <zhenzhong.duan@oracle.com>,
John Stultz <john.stultz@linaro.org>,
Thomas Gleixner <tglx@linutronix.de>,
Stephen Boyd <sboyd@kernel.org>, Waiman Long <longman@redhat.com>,
Srinivas Eeda <srinivas.eeda@oracle.com>
Subject: [PATCH 1/2] acpi_pm: Fix bootup softlockup due to PMTMR counter read contention
Date: Thu, 14 Mar 2019 16:42:11 +0800 [thread overview]
Message-ID: <1552552932-21821-1-git-send-email-zhenzhong.duan@oracle.com> (raw)
During bootup stage of a large system with many CPUs, with nohpet, PMTMR
is temporarily selected as the clock source which can lead to a softlockup
because of the following reasons:
1) There is a single PMTMR counter shared by all the CPUs.
2) PMTMR counter reading is a very slow operation.
At bootup stage tick device is firstly initialized in periodic mode and
then switch to one-shot mode when a high resolution clocksource is
initialized. Between clocksoure initialization and switching to one-shot
mode, there is small window where timer interrupt triggers.
Due to PMTMR read contention, the 1ms(HZ=1000) interval isn't enough
for all the CPUs to process timer interrupt in periodic mode.
Then CPUs are busy processing interrupt one by one without a break,
tick_clock_notify() have no chance to be called and we never switch
to one-shot mode. Finally the system may crash because of a NMI
watchdog soft lockup, logs:
[ 20.181521] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff,
max_idle_ns: 2085701024 ns
[ 44.273786] BUG: soft lockup - CPU#48 stuck for 23s! [swapper/48:0]
[ 44.279992] BUG: soft lockup - CPU#49 stuck for 23s! [migration/49:307]
[ 44.285169] BUG: soft lockup - CPU#50 stuck for 23s! [migration/50:313]
In one-shot mode, the contention is still there but next event is always
set with a future value. We may missed some ticks, but the timer code is
smart enough to pick up those missed ticks.
By moving tick_clock_notify() into stop_machine, kernel changes to one-shot
mode early before the contention accumulate and lockup system.
This patch also address the same issue of commit f99fd22e4d4b ("x86/hpet:
Reduce HPET counter read contention") in a simple way, so that commit could
be reverted.
Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
Tested-by: Kin Cho <kin.cho@oracle.com>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Stephen Boyd <sboyd@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Srinivas Eeda <srinivas.eeda@oracle.com>
---
kernel/time/timekeeping.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index f986e19..815c92d 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -1378,6 +1378,7 @@ static int change_clocksource(void *data)
write_seqcount_end(&tk_core.seq);
raw_spin_unlock_irqrestore(&timekeeper_lock, flags);
+ tick_clock_notify();
return 0;
}
@@ -1396,7 +1397,6 @@ int timekeeping_notify(struct clocksource *clock)
if (tk->tkr_mono.clock == clock)
return 0;
stop_machine(change_clocksource, clock, NULL);
- tick_clock_notify();
return tk->tkr_mono.clock == clock ? 0 : -1;
}
--
1.8.3.1
next reply other threads:[~2019-03-15 8:41 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-03-14 8:42 Zhenzhong Duan [this message]
2019-03-14 8:42 ` [PATCH 2/2] Revert "x86/hpet: Reduce HPET counter read contention" Zhenzhong Duan
2019-03-15 9:25 ` Peter Zijlstra
2019-03-15 9:29 ` Peter Zijlstra
2019-03-15 14:17 ` Waiman Long
2019-03-18 8:44 ` Zhenzhong Duan
2019-03-18 15:35 ` Waiman Long
2019-03-20 10:24 ` Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1552552932-21821-1-git-send-email-zhenzhong.duan@oracle.com \
--to=zhenzhong.duan@oracle.com \
--cc=john.stultz@linaro.org \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=sboyd@kernel.org \
--cc=srinivas.eeda@oracle.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).