From: "Michal Suchánek" <msuchanek@suse.de>
To: Nicholas Piggin <npiggin@gmail.com>
Cc: ro@suse.de, linuxppc-dev@lists.ozlabs.org
Subject: Re: KVM on POWER8 host lock up since 10d91611f426 ("powerpc/64s: Reimplement book3s idle code in C")
Date: Mon, 31 Aug 2020 11:15:23 +0200 [thread overview]
Message-ID: <20200831091523.GC29521@kitsune.suse.cz> (raw)
In-Reply-To: <1598835313.5688ngko4f.astroid@bobo.none>
On Mon, Aug 31, 2020 at 11:14:18AM +1000, Nicholas Piggin wrote:
> Excerpts from Michal Suchánek's message of August 31, 2020 6:11 am:
> > Hello,
> >
> > on POWER8 KVM hosts lock up since commit 10d91611f426 ("powerpc/64s:
> > Reimplement book3s idle code in C").
> >
> > The symptom is host locking up completely after some hours of KVM
> > workload with messages like
> >
> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 71
> > 2020-08-30T10:51:31+00:00 obs-power8-01 kernel: KVM: couldn't grab cpu 47
> >
> > printed before the host locks up.
> >
> > The machines run sandboxed builds which is a mixed workload resulting in
> > IO/single core/mutiple core load over time and there are periods of no
> > activity and no VMS runnig as well. The VMs are shortlived so VM
> > setup/terdown is somewhat excercised as well.
> >
> > POWER9 with the new guest entry fast path does not seem to be affected.
> >
> > Reverted the patch and the followup idle fixes on top of 5.2.14 and
> > re-applied commit a3f3072db6ca ("powerpc/powernv/idle: Restore IAMR
> > after idle") which gives same idle code as 5.1.16 and the kernel seems
> > stable.
> >
> > Config is attached.
> >
> > I cannot easily revert this commit, especially if I want to use the same
> > kernel on POWER8 and POWER9 - many of the POWER9 fixes are applicable
> > only to the new idle code.
> >
> > Any idea what can be the problem?
>
> So hwthread_state is never getting back to to HWTHREAD_IN_IDLE on
> those threads. I wonder what they are doing. POWER8 doesn't have a good
> NMI IPI and I don't know if it supports pdbg dumping registers from the
> BMC unfortunately.
It may be possible to set up fadump with a later kernel version that
supports it on powernv and dump the whole kernel.
Thanks
Michal
next prev parent reply other threads:[~2020-08-31 9:17 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-30 20:11 KVM on POWER8 host lock up since 10d91611f426 ("powerpc/64s: Reimplement book3s idle code in C") Michal Suchánek
2020-08-31 1:14 ` Nicholas Piggin
2020-08-31 8:48 ` Ruediger Oertel
2020-08-31 12:58 ` Michael Ellerman
2020-08-31 13:13 ` Ruediger Oertel
2020-08-31 9:03 ` Ruediger Oertel
2020-08-31 9:07 ` Michal Suchánek
2020-08-31 9:15 ` Michal Suchánek [this message]
2020-08-31 10:50 ` Michael Ellerman
2020-09-07 13:13 ` Nicholas Piggin
2020-09-07 13:25 ` Michal Suchánek
2020-10-16 20:14 ` Michal Suchánek
2020-10-19 1:00 ` Nicholas Piggin
2020-10-19 4:50 ` Nicholas Piggin
2020-10-21 17:00 ` Michal Suchánek
2021-01-14 12:40 ` Michal Suchánek
2021-01-14 13:08 ` Nicholas Piggin
2021-11-02 15:48 ` Michal Suchánek
2021-11-05 1:47 ` Nicholas Piggin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200831091523.GC29521@kitsune.suse.cz \
--to=msuchanek@suse.de \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=npiggin@gmail.com \
--cc=ro@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).