All of lore.kernel.org
 help / color / mirror / Atom feed
From: Andi Kleen <ak@suse.de>
To: Ingo Molnar <mingo@elte.hu>, Andi Kleen <ak@suse.de>,
	Len Brown <lenb@kernel.org>,
	patches@x86-64.org, linux-kernel@vger.kernel.org
Subject: [PATCH x86 for review II] [38/39] x86: fix laptop bootup hang in init_acpi()
Date: Mon, 12 Feb 2007 08:38:26 +0100 (CET)	[thread overview]
Message-ID: <20070212073826.CC4AE13DD5@wotan.suse.de> (raw)
In-Reply-To: <20070212837.963446000@suse.de>


From: Ingo Molnar <mingo@elte.hu>

During kernel bootup, a new T60 laptop (CoreDuo, 32-bit) hangs about
10%-20% of the time in acpi_init():

 Calling initcall 0xc055ce1a: topology_init+0x0/0x2f()
 Calling initcall 0xc055d75e: mtrr_init_finialize+0x0/0x2c()
 Calling initcall 0xc05664f3: param_sysfs_init+0x0/0x175()
 Calling initcall 0xc014cb65: pm_sysrq_init+0x0/0x17()
 Calling initcall 0xc0569f99: init_bio+0x0/0xf4()
 Calling initcall 0xc056b865: genhd_device_init+0x0/0x50()
 Calling initcall 0xc056c4bd: fbmem_init+0x0/0x87()
 Calling initcall 0xc056dd74: acpi_init+0x0/0x1ee()

It's a hard hang that not even an NMI could punch through!  Frustratingly,
adding printks or function tracing to the ACPI code made the hangs go away
...

After some time an additional detail emerged: disabling the NMI watchdog
made these occasional hangs go away.

So i spent the better part of today trying to debug this and trying out
various theories when i finally found the likely reason for the hang: if
acpi_ns_initialize_devices() executes an _INI AML method and an NMI
happens to hit that AML execution in the wrong moment, the machine would
hang.  (my theory is that this must be some sort of chipset setup method
doing stores to chipset mmio registers?)

Unfortunately given the characteristics of the hang it was sheer
impossible to figure out which of the numerous AML methods is impacted
by this problem.

As a workaround i wrote an interface to disable chipset-based NMIs while
executing _INI sections - and indeed this fixed the hang.  I did a
boot-loop of 100 separate reboots and none hung - while without the patch
it would hang every 5-10 attempts.  Out of caution i did not touch the
nmi_watchdog=2 case (it's not related to the chipset anyway and didnt
hang).

I implemented this for both x86_64 and i686, tested the i686 laptop both
with nmi_watchdog=1 [which triggered the hangs] and nmi_watchdog=2, and
tested an Athlon64 box with the 64-bit kernel as well. Everything builds
and works with the patch applied.

Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andi Kleen <ak@suse.de>
Cc: Andi Kleen <ak@suse.de>
Cc: Len Brown <lenb@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 arch/i386/kernel/nmi.c          |   28 ++++++++++++++++++++++++++++
 arch/x86_64/kernel/nmi.c        |   27 +++++++++++++++++++++++++++
 drivers/acpi/namespace/nsinit.c |    9 +++++++++
 include/linux/nmi.h             |    9 ++++++++-
 4 files changed, 72 insertions(+), 1 deletion(-)

Index: linux/arch/i386/kernel/nmi.c
===================================================================
--- linux.orig/arch/i386/kernel/nmi.c
+++ linux/arch/i386/kernel/nmi.c
@@ -383,6 +383,34 @@ void enable_timer_nmi_watchdog(void)
 	}
 }
 
+static void __acpi_nmi_disable(void *__unused)
+{
+	apic_write_around(APIC_LVT0, APIC_DM_NMI | APIC_LVT_MASKED);
+}
+
+/*
+ * Disable timer based NMIs on all CPUs:
+ */
+void acpi_nmi_disable(void)
+{
+	if (atomic_read(&nmi_active) && nmi_watchdog == NMI_IO_APIC)
+		on_each_cpu(__acpi_nmi_disable, NULL, 0, 1);
+}
+
+static void __acpi_nmi_enable(void *__unused)
+{
+	apic_write_around(APIC_LVT0, APIC_DM_NMI);
+}
+
+/*
+ * Enable timer based NMIs on all CPUs:
+ */
+void acpi_nmi_enable(void)
+{
+	if (atomic_read(&nmi_active) && nmi_watchdog == NMI_IO_APIC)
+		on_each_cpu(__acpi_nmi_enable, NULL, 0, 1);
+}
+
 #ifdef CONFIG_PM
 
 static int nmi_pm_active; /* nmi_active before suspend */
Index: linux/arch/x86_64/kernel/nmi.c
===================================================================
--- linux.orig/arch/x86_64/kernel/nmi.c
+++ linux/arch/x86_64/kernel/nmi.c
@@ -368,6 +368,33 @@ void enable_timer_nmi_watchdog(void)
 	}
 }
 
+static void __acpi_nmi_disable(void *__unused)
+{
+	apic_write(APIC_LVT0, APIC_DM_NMI | APIC_LVT_MASKED);
+}
+
+/*
+ * Disable timer based NMIs on all CPUs:
+ */
+void acpi_nmi_disable(void)
+{
+	if (atomic_read(&nmi_active) && nmi_watchdog == NMI_IO_APIC)
+		on_each_cpu(__acpi_nmi_disable, NULL, 0, 1);
+}
+
+static void __acpi_nmi_enable(void *__unused)
+{
+	apic_write(APIC_LVT0, APIC_DM_NMI);
+}
+
+/*
+ * Enable timer based NMIs on all CPUs:
+ */
+void acpi_nmi_enable(void)
+{
+	if (atomic_read(&nmi_active) && nmi_watchdog == NMI_IO_APIC)
+		on_each_cpu(__acpi_nmi_enable, NULL, 0, 1);
+}
 #ifdef CONFIG_PM
 
 static int nmi_pm_active; /* nmi_active before suspend */
Index: linux/drivers/acpi/namespace/nsinit.c
===================================================================
--- linux.orig/drivers/acpi/namespace/nsinit.c
+++ linux/drivers/acpi/namespace/nsinit.c
@@ -45,6 +45,7 @@
 #include <acpi/acnamesp.h>
 #include <acpi/acdispat.h>
 #include <acpi/acinterp.h>
+#include <linux/nmi.h>
 
 #define _COMPONENT          ACPI_NAMESPACE
 ACPI_MODULE_NAME("nsinit")
@@ -534,7 +535,15 @@ acpi_ns_init_one_device(acpi_handle obj_
 	info->parameter_type = ACPI_PARAM_ARGS;
 	info->flags = ACPI_IGNORE_RETURN_VALUE;
 
+	/*
+	 * Some hardware relies on this being executed as atomically
+	 * as possible (without an NMI being received in the middle of
+	 * this) - so disable NMIs and initialize the device:
+	 */
+	acpi_nmi_disable();
 	status = acpi_ns_evaluate(info);
+	acpi_nmi_enable();
+
 	if (ACPI_SUCCESS(status)) {
 		walk_info->num_INI++;
 
Index: linux/include/linux/nmi.h
===================================================================
--- linux.orig/include/linux/nmi.h
+++ linux/include/linux/nmi.h
@@ -17,8 +17,15 @@
 #ifdef ARCH_HAS_NMI_WATCHDOG
 #include <asm/nmi.h>
 extern void touch_nmi_watchdog(void);
+extern void acpi_nmi_disable(void);
+extern void acpi_nmi_enable(void);
 #else
-# define touch_nmi_watchdog() touch_softlockup_watchdog()
+static inline void touch_nmi_watchdog(void)
+{
+	touch_softlockup_watchdog();
+}
+static inline void acpi_nmi_disable(void) { }
+static inline void acpi_nmi_enable(void) { }
 #endif
 
 #ifndef trigger_all_cpu_backtrace

  parent reply	other threads:[~2007-02-12  7:39 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2007-02-12  7:37 [PATCH x86 for review II] [1/39] i386: move startup_32() in text.head section Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [2/39] x86_64: Break init() in two parts to avoid MODPOST warnings Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [3/39] i386: arch/i386/kernel/cpu/mcheck/mce.c should #include <asm/mce.h> Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [4/39] i386: add idle notifier Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [5/39] i386: improve sched_clock() on i686 Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [6/39] i386: romsignature/checksum cleanup Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [7/39] x86_64: Fix fake numa for x86_64 machines with big IO hole Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [8/39] x86_64: Remove fastcall references in x86_64 code Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [9/39] x86_64: Use constant instead of raw number in x86_64 ioperm.c Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [10/39] x86_64: Handle 32 bit PerfMon Counter writes cleanly in x86_64 nmi_watchdog Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [11/39] i386: Handle 32 bit PerfMon Counter writes cleanly in i386 nmi_watchdog Andi Kleen
2007-02-12  7:37 ` [PATCH x86 for review II] [12/39] i386: Handle 32 bit PerfMon Counter writes cleanly in oprofile Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [13/39] i386: CONFIG_PHYSICAL_ALIGN limited to 4M? Andi Kleen
2007-02-13  6:36   ` Rene Herman
2007-02-12  7:38 ` [PATCH x86 for review II] [14/39] x86_64: cleanup Doc/x86_64/ files Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [15/39] x86_64: list x86_64 quilt tree Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [16/39] x86: simplify notify_page_fault() Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [17/39] x86_64: Tighten mce_amd driver MSR reads Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [18/39] x86_64: Allow to run a program when a machine check event is detected Andi Kleen
2007-02-12  7:54   ` Oliver Neukum
2007-02-12  8:04     ` Andi Kleen
2007-02-12  8:11       ` Bauke Jan Douma
2007-02-12 15:05       ` [patches] " Pavel Machek
2007-02-12  7:38 ` [PATCH x86 for review II] [19/39] x86_64: remove get_pmd() Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [20/39] i386: Small cleanup to TLB flush code Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [21/39] i386: rdmsr_on_cpu, wrmsr_on_cpu Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [22/39] x86_64: Kconfig typos Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [23/39] i386: use smp_call_function_single() Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [24/39] " Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [25/39] x86_64: Fix preprocessor condition Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [26/39] i386: fix 32-bit ioctls on x64_32 Andi Kleen
2007-02-12 13:24   ` Giuliano Procida
2007-02-12 22:28     ` Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [27/39] i386: APM on i386 Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [28/39] i386: fix size_or_mask and size_and_mask Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [29/39] x86_64: - Ignore long SMI interrupts in clock calibration code - update 1 Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [30/39] x86_64: Check return value of putreg in PTRACE_SETREGS Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [31/39] x86_64: Unexport __supported_pte_mask Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [32/39] x86_64: x86_64 - Fix FS/GS registers for VT execution Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [33/39] x86_64: Fix off by one error in IOMMU boundary checking Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [34/39] i386: Use stack arguments for calling into EFI Andi Kleen
2007-02-12 19:45   ` Frédéric RISS
2007-02-12  7:38 ` [PATCH x86 for review II] [35/39] x86_64: Don't reserve ROMs Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [36/39] x86_64: define dma noncoherent API functions Andi Kleen
2007-02-12  7:38 ` [PATCH x86 for review II] [37/39] x86_64: robustify bad_dma_address handling Andi Kleen
2007-02-12  7:38 ` Andi Kleen [this message]
2007-02-12  7:38 ` [PATCH x86 for review II] [39/39] i386: All Transmeta CPUs have constant TSCs Andi Kleen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20070212073826.CC4AE13DD5@wotan.suse.de \
    --to=ak@suse.de \
    --cc=lenb@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=patches@x86-64.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.