On Mon, Aug 30, 2021 at 05:44:00PM +0200, BALATON Zoltan wrote:
> On Mon, 30 Aug 2021, David Gibson wrote:
> > On Sun, Aug 29, 2021 at 04:09:54AM +0000, Joseph wrote:
> > > Hi Mark, Cédric, Greg at the openbsd-ppc ML,
> > > 
> > > It is great to talk to you. Thank you for taking on the conversation.
> > > 
> > > Right, OpenBSD implements powernv meaning it runs on bare metal on
> > > Power9, that is great.
> > > 
> > > What I wanted to ask about with this thread is:
> > > 
> > > To have the same utility of Power9 as of AMD64, it would be great to
> > > have a high speed virtualizer, like what OpenBSD's VMM or Linux' KVM-
> > > QEMU accomplish on AMD64.
> > > 
> > > Linux KVM-QEMU supports Power9 already so that's a great start: One
> > > runs Linux powernv bare metal on Power9 hardware. Each VM is a
> > > KVM-QEMU instance.
> > > 
> > > Is there any way today to make Linux-KVM-QEMU as VM host run
> > > OpenBSD as a high speed (say >90% of bare metal performance, here
> > > presume KVM uses PCIe passthrough) VM guest -
> > 
> > I'm afraid this is more or less impossible, without adding
> > PAPR/pseries support to OpenBSD.  The fundamental problem is that the
> > virtualization facilities on the POWER chip don't really allow
> > efficient full hardware virtualization, only para-virtualization and
> > PAPR is that para-virtualized environment.
> > 
> > That's why the "powernv" machine type doesn't utilize KVM and is fully
> > emulated and therefore slow.  It might be possible to use the
> > "powernv" machine type with the "PR" implementation of KVM - that's a
> > KVM implementation which works by running the entire guest in
> > userspace and emulating all privileged instructions.  But:
> > 
> > * KVM PR doesn't currently work properly on POWER9, and getting it
> >   working would be a significant amount of work
> > * The way KVM PR works means it's very fiddly to get right, so it's
> >   unlikely to ever be suitable for production work
> > * Depending on host and guest cpu models there might be a few corner
> >   cases it can never get exactly right
> 
> Out of curiosity what are the problems with KVM-PR on POWER9 currently and

I don't know entirely.  My point is that I've never had the time
and/or interest to investigate (and to my knowledge no-one else has
either).  The biggest is likely to be that PR also needs to emulate to
some extent the guest CPU's MMU.  That means adding support for the
POWER9 radix-MMU, which would be avery large job.  There are probably
other problems as well: I vaguely recall that if you attempt to boot a
kernel, the first problem you hit is a new-in-POWER9 privileged SPR
which PR doesn't emulate.

> what are the corner cases that it can never get right?

Well, I'm not certain they exist (at least in a way that can't be
worked around) - but I'm not certain they don't either.  In particular
behavioural differences in userspace (i.e. MSR[PR] == 1) between host
and guest CPUs couldn't be handled by PR (since it wouldn't get the
necessary trap to emulate).  Those are rare, of course, since the
architecture is pretty strict about user mode behaviour, but I can't
swear that none exist.  There are certainly new non-privileged
instructions that have been added, but it might be possible to work
around those (trap the illegal instruction and emulate if guest has it
and host doesn't, hope the guest doesn't rely on the 0x700 trap if
host has it and guest doesn't).

Or of course you could have a big matrix of host/guest CPU userspace
compatibility, but that in itself is not a trivial job.

> This info may be
> useful for those interested in fixing and using it and having it listed here
> may save time debugging some known problems.
> 
> Regards,
> BALATON Zoltan


-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson