Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

* Re: [tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path
       [not found] <tip-d9bc9be89629445758670220787683e37c93f6c1@git.kernel.org>
@ 2012-02-12  1:04 ` Yinghai Lu
  2012-02-12  3:13   ` Eric W. Biederman
                     ` (2 more replies)
  0 siblings, 3 replies; 63+ messages in thread
From: Yinghai Lu @ 2012-02-12  1:04 UTC (permalink / raw)
  To: linux-kernel, mingo, hpa, yinghai, torvalds, kexec, vgoyal,
	ebiederm, akpm, tglx, dzickus, mingo
  Cc: linux-tip-commits

On Sat, Feb 11, 2012 at 3:09 PM, tip-bot for Don Zickus
<dzickus@redhat.com> wrote:
> Commit-ID:  d9bc9be89629445758670220787683e37c93f6c1
> Gitweb:     http://git.kernel.org/tip/d9bc9be89629445758670220787683e37c93f6c1
> Author:     Don Zickus <dzickus@redhat.com>
> AuthorDate: Thu, 9 Feb 2012 16:53:41 -0500
> Committer:  Ingo Molnar <mingo@elte.hu>
> CommitDate: Sat, 11 Feb 2012 15:38:53 +0100
>
> x86/kdump: No need to disable ioapic/lapic in crash path
>
> A customer of ours noticed when their machine crashed, kdump did
> not work but hung instead.  Using their firmware dumping
> solution they grabbed a vmcore and decoded the stacks on the
> cpus.  What they noticed seemed to be a rare deadlock with the
> ioapic_lock.
>
>  CPU4:
>  machine_crash_shutdown
>  -> machine_ops.crash_shutdown
>    -> native_machine_crash_shutdown
>       -> kdump_nmi_shootdown_cpus ------> Send NMI to other CPUs
>       -> disable_IO_APIC
>          -> clear_IO_APIC
>             -> clear_IO_APIC_pin
>                -> ioapic_read_entry
>                   -> spin_lock_irqsave(&ioapic_lock, flags)
>                   ---Infinite loop here---
>
>  CPU0:
>  do_IRQ
>  -> handle_irq
>    -> handle_edge_irq
>        -> ack_apic_edge
>           -> move_native_irq
>               -> mask_IO_APIC_irq
>                  -> mask_IO_APIC_irq_desc
>                     -> spin_lock_irqsave(&ioapic_lock, flags)
>                     ---Receive NMI here after getting spinlock---
>                        -> nmi
>                           -> do_nmi
>                              -> crash_nmi_callback
>                              ---Infinite loop here---
>
> The problem is that although kdump tries to shutdown minimal
> hardware, it still needs to disable the IO APIC.  This requires
> spinlocks which may be held by another cpu.  This other cpu is
> being held infinitely in an NMI context by kdump in order to
> serialize the crashing path.  Instant deadlock.
>
> Eric brought up a point that because the boot code was
> restructured we may not need to disable the io apic any more in
> the crash path.  The original concern that led to the
> development of disable_IO_APIC, was that the jiffies calibration
> on boot up relied on the PIT timer for reference.  Access to the
> PIT required 8259 interrupts to be working.  This wouldn't work
> if the ioapic needed to be configured.  So on panic path, the
> ioapic was reconfigured to use virtual wire mode to allow the 8259 to passthrough.
>
> Those concerns don't hold true now, thanks to the jiffies
> calibration code not needing the PIT.  As a result, we can
> remove this call and simplify the locking needed in the panic
> path.
>
> The same work allowed us to remove the need to disable the local
> apic on shutdown too.  This should allow us to jump to the
> second a little faster.
>
> I tested kdump on an Ivy Bridge platform, a Pentium4 and an old
> athlon that did not have an ioapic.  All three were successful.
>
> I also tested using lkdtm that would use jprobes to panic the
> system when entering do_IRQ.  The idea was to see how the system
> reacted with an interrupt pending in the second kernel.  My
> core2 quad successfully kdump'd 3 times in a row with no issues.
>
> v2: removed the disable lapic code too

with this commit, kdump is not working anymore on my setups with
Nehalem, Westmere, sandbridge.
these setup all have VT-d enabled.

After reverting this commit, kdump is working again.

So assume you need to drop this patch.

Thanks

Yinghai Lu

^ permalink raw reply	[flat|nested] 63+ messages in thread