From: Wu Bo <wubo40@huawei.com>
To: <linux-edac@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
<openipmi-developer@lists.sourceforge.net>
Cc: <tony.luck@intel.com>, <minyard@acm.org>, <bp@alien8.de>,
<tglx@linutronix.de>, <andi@firstfloor.org>, <mingo@redhat.com>,
<x86@kernel.org>, <hpa@zytor.com>, <arnd@arndb.de>,
<gregkh@linuxfoundation.org>, <hidehiro.kawai.ez@hitachi.com>,
<linfeilong@huawei.com>, <liuzhiqiang26@huawei.com>
Subject: Re: [RFC PATCH] mce: don't not enable IRQ in wait_for_panic()
Date: Mon, 21 Sep 2020 19:35:50 +0800 [thread overview]
Message-ID: <2262c2ee-3272-987a-0bdb-a0ce55a1d43c@huawei.com> (raw)
In-Reply-To: <1600339070-570840-1-git-send-email-wubo40@huawei.com>
On 2020/9/17 18:37, Wu Bo wrote:
> In my virtual machine (have 4 cpus), Use mce_inject to inject errors
> into the system. After mce-inject injects an uncorrectable error,
> there is a probability that the virtual machine is not reset immediately,
> but hangs for more than 3000 seconds, and appeared unable to
> handle kernel paging request.
>
> The analysis reasons are as follows:
> 1) MCE appears on all CPUs, Currently all CPUs are in the NMI interrupt
> context. cpu0 is the first to seize the opportunity to run panic
> routines, and panic event should stop the other processors before
> do ipmi flush_messages(). but cpu1, cpu2 and cpu3 has already
> in NMI interrupt context, So the Second NMI interrupt(IPI)
> will not be processed again by cpu1, cpu2 and cpu3.
> At this time, cpu1,cpu2 and cpu3 did not stopped.
>
> 2) cpu1, cpu2 and cpu3 are waitting for cpu0 to finish the panic process.
> if a timeout waiting for other CPUs happened, do wait_for_panic(),
> the irq is enabled in the wait_for_panic() function.
>
> 3) ipmi IRQ occurs on the cpu3, and the cpu0 is doing the panic,
> they have the opportunity to call the smi_event_handler()
> function concurrently. the ipmi IRQ affects the panic process of cpu0.
>
> CPU0 CPU3
>
> |-nmi_handle do mce_panic |-nmi_handle do_machine_check
> | |
> |-panic() |-wait_for_panic()
> | |
> |-stop other cpus ---- NMI ------> (Ignore, already in nmi interrupt)
> | |
> |-notifier call(ipmi panic_event) |<-ipmi IRQ occurs
> | |
> \|/ \|/
> do smi_event_handler() do smi_event_handler()
>
> My understanding is that panic() runs with only one operational CPU
> in the system, other CPUs should be stopped, if other CPUs does not stop,
> at least IRQ interrupts should be disabled. The x86 architecture,
> disable IRQ interrupt will not affect IPI when do mce panic,
> because IPI is notified through NMI interrupt. If my analysis
> is not right, please correct me, thanks.
>
> Steps to reproduce (Have a certain probability):
> 1. # vim /tmp/uncorrected
> CPU 1 BANK 4
> STATUS uncorrected 0xc0
> MCGSTATUS EIPV MCIP
> ADDR 0x1234
> RIP 0xdeadbabe
> RAISINGCPU 0
> MCGCAP SER CMCI TES 0x6
>
> 2. # modprobe mce_inject
> 3. # cd /tmp
> 4. # mce-inject uncorrected
>
Hi,
I tested the 5.9-rc5 version and found that the problem still exists. Is
there a good solution ?
Best regards,
Wu Bo
prev parent reply other threads:[~2020-09-21 11:36 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-17 10:37 [RFC PATCH] mce: don't not enable IRQ in wait_for_panic() Wu Bo
2020-09-17 15:10 ` Corey Minyard
2020-09-21 11:35 ` Wu Bo [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2262c2ee-3272-987a-0bdb-a0ce55a1d43c@huawei.com \
--to=wubo40@huawei.com \
--cc=andi@firstfloor.org \
--cc=arnd@arndb.de \
--cc=bp@alien8.de \
--cc=gregkh@linuxfoundation.org \
--cc=hidehiro.kawai.ez@hitachi.com \
--cc=hpa@zytor.com \
--cc=linfeilong@huawei.com \
--cc=linux-edac@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=liuzhiqiang26@huawei.com \
--cc=mingo@redhat.com \
--cc=minyard@acm.org \
--cc=openipmi-developer@lists.sourceforge.net \
--cc=tglx@linutronix.de \
--cc=tony.luck@intel.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).