Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[Cédric Le Goater]

[ ... ] 

> OpenBSD probably uses XIVE in a different way than Linux though.
If it is running under the skiboot firmware (Like on the Talos 
system), it is necessarily using an OPAL interface, either the 
P8 legacy interface (on top of XIVE HW) or the XIVE native 
interface.     

  https://github.com/open-power/skiboot/blob/master/doc/opal-api/index.rst
  https://github.com/open-power/skiboot/blob/master/doc/xive.rst


[ ... ]

> qemu-system-ppc64 -M powernv9 -cpu power9 -m 2G -smp 1 \
>                   -serial mon:stdio \
>                   -device ich9-ahci,id=ahci0,bus=pcie.0 \
>                   -device qemu-xhci,id=usb0,bus=pcie.2 \
>                   -bios /usr/local/share/qemu/skiboot.lid \
>                   -kernel ./pnor.BOOTKERNEL \
>                   -drive id=disk,file=miniroot70.img,if=none \
>                   -device ide-hd,bus=ahci0.0,drive=disk \
>                   -drive id=disk1,file=power.img,if=none \
>                   -device ide-hd,bus=ahci0.1,drive=disk1 \
>                   -device e1000e,bus=pcie.1
> 
> miniroot70.img is the OpenBSD installer image that can be found at:
> 
>   http://cdn.openbsd.org/pub/OpenBSD/snapshots/powerpc64/miniroot70.img
> 
> power.img is the disk image to install on; you can probably leave that out.
> 
> Not sure where I got pnor.BOOTKERNEL from anymore.

That's the file we would be interested in. I suppose this is the same 
image used to boot the Talos ? 

> This command boots into the installer, but hangs at the "Welcome to
> the OpenBSD installer" message.  It is supposed to print a prompt
> after that, but that never happens.

It shouldn't be too hard to debug with all the FW images.  

[ ... ]

>> I thought the BSD folks were working on POWER9 baremetal support,
>> PowerNV platform, on top of OPAL/skiboot. Is that completed ?
> 
> Yup. OpenBSD boots fine on the Raptor Talos/Blackbird machines that we have.

Great ! Where can we get the kernel/rootfs images loaded by skiboot ?  
That would make a good acceptance test for the QEMU PowerNV machines.

Thanks,

C.
 

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[Joseph]

Hi Mark, Cédric, Greg at the openbsd-ppc ML,

It is great to talk to you. Thank you for taking on the conversation.

Right, OpenBSD implements powernv meaning it runs on bare metal on
Power9, that is great.

What I wanted to ask about with this thread is:

To have the same utility of Power9 as of AMD64, it would be great to
have a high speed virtualizer, like what OpenBSD's VMM or Linux' KVM-
QEMU accomplish on AMD64.

Linux KVM-QEMU supports Power9 already so that's a great start: One
runs Linux powernv bare metal on Power9 hardware. Each VM is a
KVM-QEMU instance.

Is there any way today to make Linux-KVM-QEMU as VM host run
OpenBSD as a high speed (say >90% of bare metal performance, here
presume KVM uses PCIe passthrough) VM guest -

From the discussion I gather it's not possible without OpenBSD being
extended with pseries mode support, is it so?

(Great to see that Power hardware is on par with or faster than AMD64,
so the hardware part of this equation is clearly there: Power10 is
advertised to be released late this year, lithography will be down
from 14nm to 7, they say performance will 3x and electricity 1/3.)

Thanks!
Joseph

On Saturday, August 28th, 2021 at 7:06 PM, Cédric Le Goater <clg@kaod.org> wrote:
> [ ... ]
>
> > OpenBSD probably uses XIVE in a different way than Linux though.
>
> If it is running under the skiboot firmware (Like on the Talos
>
> system), it is necessarily using an OPAL interface, either the
>
> P8 legacy interface (on top of XIVE HW) or the XIVE native
>
> interface.
>
> https://github.com/open-power/skiboot/blob/master/doc/opal-api/index.rst
>
> https://github.com/open-power/skiboot/blob/master/doc/xive.rst
>
> [ ... ]
>
> > qemu-system-ppc64 -M powernv9 -cpu power9 -m 2G -smp 1 \
> >
> > -serial mon:stdio \
> >
> > -device ich9-ahci,id=ahci0,bus=pcie.0 \
> >
> > -device qemu-xhci,id=usb0,bus=pcie.2 \
> >
> > -bios /usr/local/share/qemu/skiboot.lid \
> >
> > -kernel ./pnor.BOOTKERNEL \
> >
> > -drive id=disk,file=miniroot70.img,if=none \
> >
> > -device ide-hd,bus=ahci0.0,drive=disk \
> >
> > -drive id=disk1,file=power.img,if=none \
> >
> > -device ide-hd,bus=ahci0.1,drive=disk1 \
> >
> > -device e1000e,bus=pcie.1
> >
> > miniroot70.img is the OpenBSD installer image that can be found at:
> >
> > http://cdn.openbsd.org/pub/OpenBSD/snapshots/powerpc64/miniroot70.img
> >
> > power.img is the disk image to install on; you can probably leave that out.
> >
> > Not sure where I got pnor.BOOTKERNEL from anymore.
>
> That's the file we would be interested in. I suppose this is the same
>
> image used to boot the Talos ?
>
> > This command boots into the installer, but hangs at the "Welcome to
> >
> > the OpenBSD installer" message. It is supposed to print a prompt
> >
> > after that, but that never happens.
>
> It shouldn't be too hard to debug with all the FW images.
>
> [ ... ]
>
> > > I thought the BSD folks were working on POWER9 baremetal support,
> > >
> > > PowerNV platform, on top of OPAL/skiboot. Is that completed ?
> >
> > Yup. OpenBSD boots fine on the Raptor Talos/Blackbird machines that we have.
>
> Great ! Where can we get the kernel/rootfs images loaded by skiboot ?
>
> That would make a good acceptance test for the QEMU PowerNV machines.
>
> Thanks,
>
> C.

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[Joseph]

> runs Linux powernv bare metal on Power9 hardware. Each VM is a
> KVM-QEMU instance.

> > > If you want to also run OpenBSD inside a VM, then OpenBSD must
> > > implement proper support to be able to run in the paravirtualized
> > > PAPR environment provided by KVM-QEMU on POWER. The OpenBSD statement
> > > seem to indicate this is missing. Nothing special "should" be needed
> > > on the KVM-QEMU side.

Terminology, what is PAPR an abbreviation of, is pseries (hardware
interface) equivalent with PAPR?

Also all discussed above is supported by Raptor Talos/Blackbird
hardware right - the Raptor machine exports PowerNV to the host system
that runs on bare metal, and the host system (= Linux QEMU-KVM for
now) will export a virtualized pseries/papr Power9 machine to the VM
guest, which runs at close to full bare metal speed?

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[Cédric Le Goater]

On 8/29/21 6:20 AM, Joseph wrote:
>> runs Linux powernv bare metal on Power9 hardware. Each VM is a
>> KVM-QEMU instance.
> 
>>>> If you want to also run OpenBSD inside a VM, then OpenBSD must
>>>> implement proper support to be able to run in the paravirtualized
>>>> PAPR environment provided by KVM-QEMU on POWER. The OpenBSD statement
>>>> seem to indicate this is missing. Nothing special "should" be needed
>>>> on the KVM-QEMU side.
> 
> Terminology, what is PAPR an abbreviation of

Power Architecture Platform Reference

https://openpowerfoundation.org/wp-content/uploads/2020/07/LoPAR-20200611.pdf

> is pseries (hardware interface) equivalent with PAPR?

pseries was the name of IBM powerpc servers. It is the name of the Linux 
platform supporting PAPR and also the name of the QEMU machine implementing 
PAPR.

> Also all discussed above is supported by Raptor Talos/Blackbird
> hardware right - the Raptor machine exports PowerNV to the host system
> that runs on bare metal, and the host system (= Linux QEMU-KVM for
> now) will export a virtualized pseries/papr Power9 machine to the VM
> guest, which runs at close to full bare metal speed?

yes. There is a slight overhead depending on how much the hypervisor needs
to be involved.  
 
C.

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3435 bytes --]

On Sun, Aug 29, 2021 at 04:09:54AM +0000, Joseph wrote:
> Hi Mark, Cédric, Greg at the openbsd-ppc ML,
> 
> It is great to talk to you. Thank you for taking on the conversation.
> 
> Right, OpenBSD implements powernv meaning it runs on bare metal on
> Power9, that is great.
> 
> What I wanted to ask about with this thread is:
> 
> To have the same utility of Power9 as of AMD64, it would be great to
> have a high speed virtualizer, like what OpenBSD's VMM or Linux' KVM-
> QEMU accomplish on AMD64.
> 
> Linux KVM-QEMU supports Power9 already so that's a great start: One
> runs Linux powernv bare metal on Power9 hardware. Each VM is a
> KVM-QEMU instance.
> 
> Is there any way today to make Linux-KVM-QEMU as VM host run
> OpenBSD as a high speed (say >90% of bare metal performance, here
> presume KVM uses PCIe passthrough) VM guest -

I'm afraid this is more or less impossible, without adding
PAPR/pseries support to OpenBSD.  The fundamental problem is that the
virtualization facilities on the POWER chip don't really allow
efficient full hardware virtualization, only para-virtualization and
PAPR is that para-virtualized environment.

That's why the "powernv" machine type doesn't utilize KVM and is fully
emulated and therefore slow.  It might be possible to use the
"powernv" machine type with the "PR" implementation of KVM - that's a
KVM implementation which works by running the entire guest in
userspace and emulating all privileged instructions.  But:

 * KVM PR doesn't currently work properly on POWER9, and getting it
   working would be a significant amount of work
 * The way KVM PR works means it's very fiddly to get right, so it's
   unlikely to ever be suitable for production work
 * Depending on host and guest cpu models there might be a few corner
   cases it can never get exactly right
 * KVM PR will never be all that fast, although it will be
   significantly faster than full emulation

> From the discussion I gather it's not possible without OpenBSD being
> extended with pseries mode support, is it so?

Essentially yes.  The situation is different from x86.  On x86 the
bare metal PC platform was the original "normal" one, and then that
platform was made possible as a virtual guest.  There were some
alternative paravirtualized guest platforms (notably XenPV) but they
never really caught on.

On POWER, the paravirtualized PAPR platform was the original "normal"
one, as provided by PowerVM, which was included in the POWER
firmware.  Then bare metal machines became available and the powernv
platform was created.  KVM kept the same paravirtualized PAPR platform
for the guest as PowerVM used for compatibility, rather than inventing
a new one.

> (Great to see that Power hardware is on par with or faster than AMD64,
> so the hardware part of this equation is clearly there: Power10 is
> advertised to be released late this year, lithography will be down
> from 14nm to 7, they say performance will 3x and electricity 1/3.)

So.. if you want to run OpenBSD on POWER10 you will definitely need
PAPR support, because POWER10 won't support bare metal OS access at
all (it will be back to PowerVM always, like POWER5..POWER7).

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[BALATON Zoltan]

[-- Attachment #1: Type: text/plain, Size: 2205 bytes --]

On Mon, 30 Aug 2021, David Gibson wrote:
> On Sun, Aug 29, 2021 at 04:09:54AM +0000, Joseph wrote:
>> Hi Mark, Cédric, Greg at the openbsd-ppc ML,
>>
>> It is great to talk to you. Thank you for taking on the conversation.
>>
>> Right, OpenBSD implements powernv meaning it runs on bare metal on
>> Power9, that is great.
>>
>> What I wanted to ask about with this thread is:
>>
>> To have the same utility of Power9 as of AMD64, it would be great to
>> have a high speed virtualizer, like what OpenBSD's VMM or Linux' KVM-
>> QEMU accomplish on AMD64.
>>
>> Linux KVM-QEMU supports Power9 already so that's a great start: One
>> runs Linux powernv bare metal on Power9 hardware. Each VM is a
>> KVM-QEMU instance.
>>
>> Is there any way today to make Linux-KVM-QEMU as VM host run
>> OpenBSD as a high speed (say >90% of bare metal performance, here
>> presume KVM uses PCIe passthrough) VM guest -
>
> I'm afraid this is more or less impossible, without adding
> PAPR/pseries support to OpenBSD.  The fundamental problem is that the
> virtualization facilities on the POWER chip don't really allow
> efficient full hardware virtualization, only para-virtualization and
> PAPR is that para-virtualized environment.
>
> That's why the "powernv" machine type doesn't utilize KVM and is fully
> emulated and therefore slow.  It might be possible to use the
> "powernv" machine type with the "PR" implementation of KVM - that's a
> KVM implementation which works by running the entire guest in
> userspace and emulating all privileged instructions.  But:
>
> * KVM PR doesn't currently work properly on POWER9, and getting it
>   working would be a significant amount of work
> * The way KVM PR works means it's very fiddly to get right, so it's
>   unlikely to ever be suitable for production work
> * Depending on host and guest cpu models there might be a few corner
>   cases it can never get exactly right

Out of curiosity what are the problems with KVM-PR on POWER9 currently and 
what are the corner cases that it can never get right? This info may be 
useful for those interested in fixing and using it and having it listed 
here may save time debugging some known problems.

Regards,
BALATON Zoltan

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[Michal Suchánek]

On Mon, Aug 30, 2021 at 04:57:21PM +1000, David Gibson wrote:
> On Sun, Aug 29, 2021 at 04:09:54AM +0000, Joseph wrote:
> > Hi Mark, Cédric, Greg at the openbsd-ppc ML,

> 
> So.. if you want to run OpenBSD on POWER10 you will definitely need
> PAPR support, because POWER10 won't support bare metal OS access at
> all (it will be back to PowerVM always, like POWER5..POWER7).

Hello,

what is stopping powernv support other than lack of firmware support?

Doesn't PowerVM use teh very same vrtualization features that KVM does?

Thanks

Michal

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4019 bytes --]

On Mon, Aug 30, 2021 at 05:44:00PM +0200, BALATON Zoltan wrote:
> On Mon, 30 Aug 2021, David Gibson wrote:
> > On Sun, Aug 29, 2021 at 04:09:54AM +0000, Joseph wrote:
> > > Hi Mark, Cédric, Greg at the openbsd-ppc ML,
> > > 
> > > It is great to talk to you. Thank you for taking on the conversation.
> > > 
> > > Right, OpenBSD implements powernv meaning it runs on bare metal on
> > > Power9, that is great.
> > > 
> > > What I wanted to ask about with this thread is:
> > > 
> > > To have the same utility of Power9 as of AMD64, it would be great to
> > > have a high speed virtualizer, like what OpenBSD's VMM or Linux' KVM-
> > > QEMU accomplish on AMD64.
> > > 
> > > Linux KVM-QEMU supports Power9 already so that's a great start: One
> > > runs Linux powernv bare metal on Power9 hardware. Each VM is a
> > > KVM-QEMU instance.
> > > 
> > > Is there any way today to make Linux-KVM-QEMU as VM host run
> > > OpenBSD as a high speed (say >90% of bare metal performance, here
> > > presume KVM uses PCIe passthrough) VM guest -
> > 
> > I'm afraid this is more or less impossible, without adding
> > PAPR/pseries support to OpenBSD.  The fundamental problem is that the
> > virtualization facilities on the POWER chip don't really allow
> > efficient full hardware virtualization, only para-virtualization and
> > PAPR is that para-virtualized environment.
> > 
> > That's why the "powernv" machine type doesn't utilize KVM and is fully
> > emulated and therefore slow.  It might be possible to use the
> > "powernv" machine type with the "PR" implementation of KVM - that's a
> > KVM implementation which works by running the entire guest in
> > userspace and emulating all privileged instructions.  But:
> > 
> > * KVM PR doesn't currently work properly on POWER9, and getting it
> >   working would be a significant amount of work
> > * The way KVM PR works means it's very fiddly to get right, so it's
> >   unlikely to ever be suitable for production work
> > * Depending on host and guest cpu models there might be a few corner
> >   cases it can never get exactly right
> 
> Out of curiosity what are the problems with KVM-PR on POWER9 currently and

I don't know entirely.  My point is that I've never had the time
and/or interest to investigate (and to my knowledge no-one else has
either).  The biggest is likely to be that PR also needs to emulate to
some extent the guest CPU's MMU.  That means adding support for the
POWER9 radix-MMU, which would be avery large job.  There are probably
other problems as well: I vaguely recall that if you attempt to boot a
kernel, the first problem you hit is a new-in-POWER9 privileged SPR
which PR doesn't emulate.

> what are the corner cases that it can never get right?

Well, I'm not certain they exist (at least in a way that can't be
worked around) - but I'm not certain they don't either.  In particular
behavioural differences in userspace (i.e. MSR[PR] == 1) between host
and guest CPUs couldn't be handled by PR (since it wouldn't get the
necessary trap to emulate).  Those are rare, of course, since the
architecture is pretty strict about user mode behaviour, but I can't
swear that none exist.  There are certainly new non-privileged
instructions that have been added, but it might be possible to work
around those (trap the illegal instruction and emulate if guest has it
and host doesn't, hope the guest doesn't rely on the 0x700 trap if
host has it and guest doesn't).

Or of course you could have a big matrix of host/guest CPU userspace
compatibility, but that in itself is not a trivial job.

> This info may be
> useful for those interested in fixing and using it and having it listed here
> may save time debugging some known problems.
> 
> Regards,
> BALATON Zoltan


-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1936 bytes --]

On Mon, Aug 30, 2021 at 06:04:31PM +0200, Michal Suchánek wrote:
> On Mon, Aug 30, 2021 at 04:57:21PM +1000, David Gibson wrote:
> > On Sun, Aug 29, 2021 at 04:09:54AM +0000, Joseph wrote:
> > > Hi Mark, Cédric, Greg at the openbsd-ppc ML,
> 
> > 
> > So.. if you want to run OpenBSD on POWER10 you will definitely need
> > PAPR support, because POWER10 won't support bare metal OS access at
> > all (it will be back to PowerVM always, like POWER5..POWER7).
> 
> Hello,
> 
> what is stopping powernv support other than lack of firmware
> support?

The virtualization features of the CPU simply aren't designed to do
full virtualization, only para-virtualization (which is what PAPR
is).  A guest in the POWER model is *aware* that it doesn't have
hypervisor privilege, but a powernv OS expects to have hypervisor
privilege.

I don't know that it's strictly impossible to provide a virtualized
powernv environment.  Specifically, it might be possible to take an
approach similar to KVM PR at a higher level: run the guest in
supervisor mode, but trap and emulate all hypervisor privileged
instructions.  However, I'm not certain that all the things that need
to be trapped do in fact trap, it would require a bunch of careful
research.  Even if it's possible, it would be a big job: essentially a
completely new KVM implementation.

> Doesn't PowerVM use teh very same vrtualization features that KVM
> does?

Yes, that's exactly the point.  The (para-)virtualized guest
environment provided by both PowerVM and KVM is PAPR (the "pseries"
machine type in qemu terms).  "powernv" is the bare metal,
hypervisor-privileged, host environment (that's even where the name
comes from "POWER Non Virtualized").

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[Cédric Le Goater]

On 8/30/21 6:04 PM, Michal Suchánek wrote:
> On Mon, Aug 30, 2021 at 04:57:21PM +1000, David Gibson wrote:
>> On Sun, Aug 29, 2021 at 04:09:54AM +0000, Joseph wrote:
>>> Hi Mark, Cédric, Greg at the openbsd-ppc ML,
> 
>>
>> So.. if you want to run OpenBSD on POWER10 you will definitely need
>> PAPR support, because POWER10 won't support bare metal OS access at
>> all (it will be back to PowerVM always, like POWER5..POWER7).
> 
> Hello,
> 
> what is stopping powernv support other than lack of firmware support?

Yes. that's the main issue. OpenPOWER firmware should be available
for people to use on P10 but it won't have any support AFAIK. We just 
published skiboot as part of it. 
 
> Doesn't PowerVM use teh very same vrtualization features that KVM does?

The hypervisor on the host (pHyp) replaces the software stack : 

    skiboot + Linux(PowerNV) + KVM + QEMU + SLOF

and it comes with its own system management driven from the FSP/BMC.

C. 

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

On Tue, Aug 31, 2021 at 09:42:05AM +0200, Cédric Le Goater wrote:
> On 8/30/21 6:04 PM, Michal Suchánek wrote:
> > On Mon, Aug 30, 2021 at 04:57:21PM +1000, David Gibson wrote:
> >> On Sun, Aug 29, 2021 at 04:09:54AM +0000, Joseph wrote:
> >>> Hi Mark, Cédric, Greg at the openbsd-ppc ML,
> > 
> >>
> >> So.. if you want to run OpenBSD on POWER10 you will definitely need
> >> PAPR support, because POWER10 won't support bare metal OS access at
> >> all (it will be back to PowerVM always, like POWER5..POWER7).
> > 
> > Hello,
> > 
> > what is stopping powernv support other than lack of firmware support?
> 
> Yes. that's the main issue. OpenPOWER firmware should be available
> for people to use on P10 but it won't have any support AFAIK. We just 
> published skiboot as part of it.

Thanks for your reply.  I realized that my reply misunderstood the
question, sorry about that.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[Fabiano Rosas]

David Gibson <david@gibson.dropbear.id.au> writes:

>> > * KVM PR doesn't currently work properly on POWER9, and getting it
>> >   working would be a significant amount of work
>> > * The way KVM PR works means it's very fiddly to get right, so it's
>> >   unlikely to ever be suitable for production work
>> > * Depending on host and guest cpu models there might be a few corner
>> >   cases it can never get exactly right
>> 
>> Out of curiosity what are the problems with KVM-PR on POWER9 currently and
>
> I don't know entirely.  My point is that I've never had the time
> and/or interest to investigate (and to my knowledge no-one else has
> either).  The biggest is likely to be that PR also needs to emulate to
> some extent the guest CPU's MMU.  That means adding support for the
> POWER9 radix-MMU, which would be avery large job.  There are probably
> other problems as well: I vaguely recall that if you attempt to boot a
> kernel, the first problem you hit is a new-in-POWER9 privileged SPR
> which PR doesn't emulate.

Just as a data point, I recently ran KVM-PR in POWER9/HPT. Works both in
bare-metal and nested. For a very simple guest, at least.

qemu-system-ppc64 -accel kvm -display none -vga none -machine \
pseries,cap-ccf-assist=off,cap-large-decr=off,cap-fwnmi=off,kvm-type=PR,kernel-irqchip=off \
-serial mon:stdio -smp 2 -m 1024 -kernel ~/vmlinux-guest -initrd \
~/initramfs.img -append "console=hvc0 debug mitigations=off"

I needed kernel-irqchip=off to avoid an issue with XICS and
mitigations=off because it seems we don't emulate the mtspr TRIG2
instruction which is used for L1 cache flush in some mitigations.

About running PR in radix, the code has this comment:

static int kvmppc_core_check_processor_compat_pr(void)
{
	/*
	 * PR KVM can work on POWER9 inside a guest partition
	 * running in HPT mode.  It can't work if we are using
	 * radix translation (because radix provides no way for
	 * a process to have unique translations in quadrant 3).
	 */
	if (cpu_has_feature(CPU_FTR_ARCH_300) && radix_enabled())
		return -EIO;
	return 0;
}

which might indicate it would not be at all possible to support PR in
Radix? I don't know enough about PR to understand why.

>> what are the corner cases that it can never get right?
>
> Well, I'm not certain they exist (at least in a way that can't be
> worked around) - but I'm not certain they don't either.  In particular
> behavioural differences in userspace (i.e. MSR[PR] == 1) between host
> and guest CPUs couldn't be handled by PR (since it wouldn't get the
> necessary trap to emulate).  Those are rare, of course, since the
> architecture is pretty strict about user mode behaviour, but I can't
> swear that none exist.  There are certainly new non-privileged
> instructions that have been added, but it might be possible to work
> around those (trap the illegal instruction and emulate if guest has it
> and host doesn't, hope the guest doesn't rely on the 0x700 trap if
> host has it and guest doesn't).
>
> Or of course you could have a big matrix of host/guest CPU userspace
> compatibility, but that in itself is not a trivial job.
>
>> This info may be
>> useful for those interested in fixing and using it and having it listed here
>> may save time debugging some known problems.
>> 
>> Regards,
>> BALATON Zoltan

QEMU-KVM offers OPAL firmware interface? OpenBSD guest support?

[Joseph]

Hi KVM-PPC emailing list,

I just emailed this to QEMU-PPC, and then realized it should probably
go here too. Any cues much appreciated.

 --

Hi QEMU PPC emailing list!

(And maintainers https://wiki.qemu.org/Documentation/Platforms/POWER )

https://www.openbsd.org/powerpc64.html says "OpenBSD/powerpc64 does
not run under a hypervisor such as PowerVM or PowerKVM".

QEMU-KVM is not listed here but I have heard no success report
anywhere, so I presume it doesn't work.

From talking to people, I gather that the limit to running OpenBSD
as VM guest on POWER, is that it operates based on the OPAL firmware
interface, and for some reason previous VM:s did not export it. But
also, I may have gotten this detail wrong.
(https://www.talospace.com/2020/07/when-will-openpower-openbsd-be-now-now.html)

Watching the POWER QEMU Wiki page https://wiki.qemu.org/Documentation/Platforms/POWER
I don't see any mentioning of any of this .

Can you who work with QEMU POWER support help clarify, can OpenBSD
run as a QEMU-KVM guest - if so what are the steps to get it going -
and or are any updates to QEMU coming that will enable it?

Any cues much appreciated.

I brought up this same question on the OpenBSD PPC emailing list,
https://marc.info/?l=openbsd-ppc&m=163006851125546&w=2 , waiting for
response.

Thanks!
Joseph