All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Paul E. McKenney" <paulmck@kernel.org>
To: "Luck, Tony" <tony.luck@intel.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"linux-edac@vger.kernel.org" <linux-edac@vger.kernel.org>,
	"bp@alien8.de" <bp@alien8.de>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"hpa@zytor.com" <hpa@zytor.com>,
	"kernel-team@fb.com" <kernel-team@fb.com>
Subject: Re: [PATCH RFC x86/mce] Make mce_timed_out() identify holdout CPUs
Date: Wed, 6 Jan 2021 11:17:08 -0800	[thread overview]
Message-ID: <20210106191708.GB2743@paulmck-ThinkPad-P72> (raw)
In-Reply-To: <3513b04e2bb543d2871ca8c152dcf5ae@intel.com>

On Wed, Jan 06, 2021 at 06:39:30PM +0000, Luck, Tony wrote:
> > The "Timeout: Not all CPUs entered broadcast exception handler" message
> > will appear from time to time given enough systems, but this message does
> > not identify which CPUs failed to enter the broadcast exception handler.
> > This information would be valuable if available, for example, in order to
> > correlated with other hardware-oriented error messages.  This commit
> > therefore maintains a cpumask_t of CPUs that have entered this handler,
> > and prints out which ones failed to enter in the event of a timeout.
> 
> I tried doing this a while back, but found that in my test case where I forced
> an error that would cause both threads from one core to be "missing", the
> output was highly unpredictable. Some random number of extra CPUs were
> reported as missing. After I added some extra breadcrumbs it became clear
> that pretty much all the CPUs (except the missing pair) entered do_machine_check(),
> but some got hung up at various points beyond the entry point. My only theory
> was that they were trying to snoop caches from the dead core (or access some
> other resource held by the dead core) and so they hung too.
> 
> Your code is much neater than mine ... and perhaps works in other cases, but
> maybe the message needs to allow for the fact that some of the cores that
> are reported missing may just be collateral damage from the initial problem.

Understood.  The system is probably not in the best shape if this code
is ever executed, after all.  ;-)

So how about like this?

	pr_info("%s: MCE holdout CPUs (may include false positives): %*pbl\n",

Easy enough if so!

> If I get time in the next day or two, I'll run my old test against your code to
> see what happens.

Thank you very much in advance!

For my own testing, is this still the right thing to use?

	https://github.com/andikleen/mce-inject

							Thanx, Paul

  reply	other threads:[~2021-01-06 19:18 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-06 17:41 [PATCH RFC x86/mce] Make mce_timed_out() identify holdout CPUs Paul E. McKenney
2021-01-06 18:32 ` Borislav Petkov
2021-01-06 19:13   ` Paul E. McKenney
2021-01-07  7:07     ` Borislav Petkov
2021-01-07 17:08       ` Paul E. McKenney
2021-01-08 12:31         ` Borislav Petkov
2021-01-08 14:55           ` Paul E. McKenney
2021-01-08 16:57             ` Borislav Petkov
2021-01-06 18:39 ` Luck, Tony
2021-01-06 19:17   ` Paul E. McKenney [this message]
2021-01-06 22:49     ` Luck, Tony
2021-01-06 23:23       ` Paul E. McKenney
2021-01-07  0:26         ` Luck, Tony
2021-01-07  0:41           ` Paul E. McKenney
2021-01-08 17:09 ` [tip: ras/core] x86/mce: " tip-bot2 for Paul E. McKenney

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210106191708.GB2743@paulmck-ThinkPad-P72 \
    --to=paulmck@kernel.org \
    --cc=bp@alien8.de \
    --cc=hpa@zytor.com \
    --cc=kernel-team@fb.com \
    --cc=linux-edac@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=tglx@linutronix.de \
    --cc=tony.luck@intel.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.