From: Andy Lutomirski <luto@amacapital.net>
To: "Luck, Tony" <tony.luck@intel.com>
Cc: Oleg Nesterov <oleg@redhat.com>, Borislav Petkov <bp@alien8.de>,
X86 ML <x86@kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Andi Kleen <andi@firstfloor.org>
Subject: Re: [RFC PATCH] x86, entry: Switch stacks on a paranoid entry from userspace
Date: Thu, 13 Nov 2014 14:23:24 -0800 [thread overview]
Message-ID: <CALCETrVx7hGZodWfAoU1oee1SW7XXSn3Lr8Vs9JTOUMNtLXY4A@mail.gmail.com> (raw)
In-Reply-To: <3908561D78D1C84285E8C5FCA982C28F3292CB9A@ORSMSX114.amr.corp.intel.com>
On Thu, Nov 13, 2014 at 10:43 AM, Luck, Tony <tony.luck@intel.com> wrote:
>> printk seems to work just fine in do_machine_check. Any chance you
>> can instrument, for each cpu, all entries to do_machine_check, all
>> calls to do_machine_check, all returns, and everything that tries to
>> do memory_failure?
>
> I first added a printk() just for the cpu that calls do_machine_check()
>
> printk("MCE: regs = %p\n", regs);
>
> to see if something went wonky when jumping to the kernel stack.
> But that printed the same value every time (same process is consuming
> and recovering from errors). Maybe this took longer to hit the problem
> case - I ran to 1500ish errors instead of just 400 in the previous two tests.
> But not sure if that is a significant change.
>
> Then I added printk() for every entry/return on all cpus. This just locked
> up on the third injection. Serial console looked to have stopped printing
> after the first - so I put in bigger delays into my test program between injection
> and consumption, and before looping around for the next cycle to give
> time for all the messages (4-socket HSW-EX ... there are a lot of cpus
> printing messages). But now it is taking a lot longer to get through
> injection/consumption iterations. At 226 now and counting.
>
>> Also, shouldn't there be a local_irq_enable before memory_failure and
>> a local_irq_disable after it? It wouldn't surprise me if you've
>> deadlocked somewhere. Lockdep could also have something interesting
>> to say.
> Added enable/disable.
>
>> should still be deliverable. Is it possible that we really need an
>> IRET to unmask NMIs? This seems unlikely.)
>
> If that were the problem, wouldn't we fail on iteration 2, instead of
> 400+ ?
>
> -Tony
There could be a timer interrupt or something. But I agree, it seems
implausible.
Are you sure that this works in an unmodified kernel? The timeout
code seems highly questionable to me. For example, there's this:
if ((c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xe)) &&
cfg->monarch_timeout < 0)
cfg->monarch_timeout = USEC_PER_SEC;
which presumably determines monarch_timeout on your system and sets it
to 1000000. But then there's this:
#define SPINUNIT 100 /* 100ns */
which smells like unit error to me. On top of that, it seems likely
to me that the cpu could execute a loop iteration in much less than
100ns, since the only thing that should be anything other than an L1
hit or a correctly predicted branch is the rmb(), which is lfence,
which is probably just a few ns. So you have 10k iterations at, say,
10ns each, allowing about 100µs to synchronize, and if an SMI hits at
an inopportune time, boom.
Also, rmb, seriously? I would understand smp_rmb() or cpu_relax() or
even barrier(), but rmb() seems completely bogus if harmless.
--Andy
--
Andy Lutomirski
AMA Capital Management, LLC
next prev parent reply other threads:[~2014-11-13 22:23 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-11-11 20:56 [RFC PATCH] x86, entry: Switch stacks on a paranoid entry from userspace Andy Lutomirski
2014-11-11 21:36 ` Borislav Petkov
2014-11-11 22:00 ` Luck, Tony
2014-11-11 22:15 ` Andy Lutomirski
2014-11-11 22:12 ` Andy Lutomirski
2014-11-11 22:33 ` Borislav Petkov
2014-11-11 22:40 ` Andy Lutomirski
2014-11-11 23:09 ` Borislav Petkov
2014-11-11 23:21 ` Andy Lutomirski
2014-11-12 0:22 ` Luck, Tony
2014-11-12 0:40 ` Andy Lutomirski
2014-11-12 1:06 ` Luck, Tony
2014-11-12 2:01 ` Andy Lutomirski
2014-11-12 2:06 ` Tony Luck
2014-11-12 10:30 ` Borislav Petkov
2014-11-12 15:48 ` Andy Lutomirski
2014-11-12 16:22 ` Borislav Petkov
2014-11-12 17:17 ` Luck, Tony
2014-11-12 17:30 ` Borislav Petkov
2014-11-13 18:04 ` Borislav Petkov
2014-11-14 21:56 ` Luck, Tony
2014-11-14 22:07 ` Andy Lutomirski
2014-11-17 18:50 ` Borislav Petkov
2014-11-17 19:57 ` Andy Lutomirski
2014-11-17 20:03 ` Borislav Petkov
2014-11-17 20:05 ` Andy Lutomirski
2014-11-17 21:55 ` Luck, Tony
2014-11-17 22:26 ` Andy Lutomirski
2014-11-17 23:16 ` Luck, Tony
2014-11-18 0:05 ` Andy Lutomirski
2014-11-18 0:22 ` Luck, Tony
2014-11-18 0:55 ` Andy Lutomirski
2014-11-18 18:30 ` Luck, Tony
2014-11-18 23:04 ` Andy Lutomirski
2014-11-18 23:26 ` Luck, Tony
2014-11-18 16:12 ` Borislav Petkov
2014-11-12 22:00 ` Oleg Nesterov
2014-11-12 23:17 ` Andy Lutomirski
2014-11-12 23:41 ` Luck, Tony
2014-11-13 0:02 ` Andy Lutomirski
2014-11-13 0:31 ` Luck, Tony
2014-11-13 1:34 ` Andy Lutomirski
2014-11-13 3:03 ` Andy Lutomirski
2014-11-13 18:43 ` Luck, Tony
2014-11-13 22:23 ` Andy Lutomirski [this message]
2014-11-13 22:25 ` Andy Lutomirski
2014-11-13 22:33 ` Luck, Tony
2014-11-13 22:47 ` Andy Lutomirski
2014-11-13 23:13 ` Andy Lutomirski
2014-11-14 0:50 ` Andy Lutomirski
2014-11-14 1:20 ` Luck, Tony
2014-11-14 1:36 ` Andy Lutomirski
2014-11-14 17:49 ` Luck, Tony
2014-11-14 19:10 ` Andy Lutomirski
2014-11-14 19:37 ` Luck, Tony
2014-11-14 18:27 ` Luck, Tony
2014-11-14 10:34 ` Borislav Petkov
2014-11-14 17:18 ` Andy Lutomirski
2014-11-14 17:24 ` Borislav Petkov
2014-11-14 17:26 ` Andy Lutomirski
2014-11-14 18:53 ` Borislav Petkov
2014-11-13 10:59 ` Borislav Petkov
2014-11-13 21:23 ` Borislav Petkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CALCETrVx7hGZodWfAoU1oee1SW7XXSn3Lr8Vs9JTOUMNtLXY4A@mail.gmail.com \
--to=luto@amacapital.net \
--cc=andi@firstfloor.org \
--cc=bp@alien8.de \
--cc=linux-kernel@vger.kernel.org \
--cc=oleg@redhat.com \
--cc=peterz@infradead.org \
--cc=tony.luck@intel.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).