linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: kan.liang@intel.com
To: linux-kernel@vger.kernel.org
Cc: dzickus@redhat.com, mingo@kernel.org, akpm@linux-foundation.org,
	babu.moger@oracle.com, atomlin@redhat.com, prarit@redhat.com,
	torvalds@linux-foundation.org, peterz@infradead.org,
	tglx@linutronix.de, eranian@google.com, acme@redhat.com,
	ak@linux.intel.com, Kan Liang <Kan.liang@intel.com>,
	stable@vger.kernel.org
Subject: [PATCH V2] kernel/watchdog: fix spurious hard lockups
Date: Wed, 21 Jun 2017 07:41:18 -0700	[thread overview]
Message-ID: <20170621144118.5939-1-kan.liang@intel.com> (raw)

From: Kan Liang <Kan.liang@intel.com>

Some users reported spurious NMI watchdog timeouts.

We now have more and more systems where the Turbo range is wide enough
that the NMI watchdog expires faster than the soft watchdog timer that
updates the interrupt tick the NMI watchdog relies on.

This problem was originally added by commit 58687acba592
("lockup_detector: Combine nmi_watchdog and softlockup detector").
Previously the NMI watchdog would always check jiffies, which were
ticking fast enough. But now the backing is quite slow so the expire
time becomes more sensitive.

For mainline the right fix is to switch the NMI watchdog to reference
cycles, which tick always at the same rate independent of turbo mode.
But this is requires some complicated changes in perf, which are too
difficult to backport. Since we need a stable fix to just increase the
NMI watchdog rate here to avoid the spurious timeouts. This is not an
ideal fix because a 3x as large Turbo range could still fail, but for
now that's not likely.

Signed-off-by: Kan Liang <Kan.liang@intel.com>
Cc: stable@vger.kernel.org
Fixes: 58687acba592 ("lockup_detector: Combine nmi_watchdog and
softlockup detector")
---

The right fix for mainline can be found here.
perf/x86/intel: enable CPU ref_cycles for GP counter
perf/x86/intel, watchdog: Switch NMI watchdog to ref cycles on x86
https://patchwork.kernel.org/patch/9779087/
https://patchwork.kernel.org/patch/9779089/

Change since V1:
 - Restrict the period in hw_nmi.c for Intel platform. (Don Zickus)

 arch/x86/kernel/apic/hw_nmi.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c
index c73c9fb281e1..716d44e986f9 100644
--- a/arch/x86/kernel/apic/hw_nmi.c
+++ b/arch/x86/kernel/apic/hw_nmi.c
@@ -20,9 +20,15 @@
 #include <linux/delay.h>
 
 #ifdef CONFIG_HARDLOCKUP_DETECTOR
+/*
+ * The NMI watchdog relies on PERF_COUNT_HW_CPU_CYCLES event, which
+ * can tick faster than the measured CPU Frequency due to Turbo mode.
+ * That can lead to spurious timeouts.
+ * To workaround the issue, extending the period by 3 times.
+ */
 u64 hw_nmi_get_sample_period(int watchdog_thresh)
 {
-	return (u64)(cpu_khz) * 1000 * watchdog_thresh;
+	return (u64)(cpu_khz) * 1000 * watchdog_thresh * 3;
 }
 #endif
 
-- 
2.11.0

             reply	other threads:[~2017-06-21 14:42 UTC|newest]

Thread overview: 37+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-06-21 14:41 kan.liang [this message]
2017-06-21 15:12 ` [PATCH V2] kernel/watchdog: fix spurious hard lockups Thomas Gleixner
2017-06-21 15:47   ` Liang, Kan
2017-06-21 17:40     ` Prarit Bhargava
2017-06-21 17:07   ` Andi Kleen
2017-06-21 19:59     ` Thomas Gleixner
2017-06-21 21:53 ` Thomas Gleixner
2017-06-22 15:33   ` Thomas Gleixner
2017-06-22 15:44   ` Don Zickus
2017-06-22 15:48     ` Liang, Kan
2017-06-23  8:01     ` Thomas Gleixner
2017-06-23 16:29       ` Don Zickus
2017-06-23 21:50         ` Thomas Gleixner
2017-06-26 20:19           ` Don Zickus
2017-06-26 20:30             ` Thomas Gleixner
2017-06-27 20:12             ` Don Zickus
2017-06-27 20:49               ` Liang, Kan
2017-06-27 21:09                 ` Don Zickus
2017-06-27 23:48                 ` Andi Kleen
2017-06-28 19:00                   ` Don Zickus
2017-06-28 20:14                     ` Andi Kleen
2017-06-29 15:44                       ` Don Zickus
2017-06-29 16:12                         ` Andi Kleen
2017-06-29 16:26                           ` Don Zickus
2017-06-29 16:36                             ` Andi Kleen
2017-07-17  1:24               ` Liang, Kan
2017-07-17  7:14                 ` Thomas Gleixner
2017-07-17 12:18                   ` Liang, Kan
2017-07-17 13:13                     ` Thomas Gleixner
2017-07-17 14:46                       ` Liang, Kan
2017-07-17 15:00                         ` Thomas Gleixner
2017-07-17 14:46                 ` Don Zickus
2017-08-15  1:16                   ` Liang, Kan
2017-08-15  1:28                     ` Linus Torvalds
2017-08-15  7:50                     ` Thomas Gleixner
2017-08-17 15:45                       ` Liang, Kan
2017-08-18 10:39                       ` [tip:core/urgent] kernel/watchdog: Prevent false positives with turbo modes tip-bot for Thomas Gleixner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170621144118.5939-1-kan.liang@intel.com \
    --to=kan.liang@intel.com \
    --cc=acme@redhat.com \
    --cc=ak@linux.intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=atomlin@redhat.com \
    --cc=babu.moger@oracle.com \
    --cc=dzickus@redhat.com \
    --cc=eranian@google.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=prarit@redhat.com \
    --cc=stable@vger.kernel.org \
    --cc=tglx@linutronix.de \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).